Image sending / receiving system, device, method and storage medium

Abstract

Provided are an image transmission / reception system, an image transmission device, an image reception device, an image transmission / reception method, and a program, which achieve a reduction in the amount of image data communication while reducing the risk of displaying an erroneous moving image that is different from the image to be originally displayed. An amplification unit (44) generates a first image to be displayed based on first image data. A substitute image generation unit (48) generates a substitute image of a second image to be displayed based on second image data, the substitute image being an estimation result based on the first image data. A display control unit (50) displays an image generated based on the second image data or an image generated based on the substitute image after displaying the image generated based on the first image data, depending on whether the second image data is received successfully / failed. A transmission control unit (30) controls whether to send the second image data to a terminal (12) based on the possibility that the substitute image generation unit (48) successfully estimates the second image to be displayed based on the first image data.

Description

Technical Field

[0001] The present invention relates to an image transmitting / receiving system, an image transmitting device, an image receiving device, an image transmitting / receiving method and a program. Background Art

[0002] In the cloud gaming service technology that has received much attention in recent years, when a user playing a video game operates a terminal that can communicate with a cloud server configured to run a video game program, an operation signal based on the operation is sent from the terminal to the cloud server. The cloud server then performs video game processing including processing based on the operation signal, thereby generating a play image depicting the situation in the video game. The cloud server then sends image data based on the play image to the terminal, and the terminal displays an image generated based on the image data. This series of processing stages is repeatedly performed to display a motion image depicting the situation in the video game on the terminal. Summary of the invention

[0003] Technical issues

[0004] As the frequency of image data transmission from the cloud server to the terminal increases, a higher frame rate and smoother moving images can be displayed on the terminal, but the communication volume between the cloud server and the terminal increases.

[0005] Here, in order to reduce the communication volume while preventing the smoothness of the displayed moving images from decreasing, it is conceivable to reduce the frequency of sending image data, and for the unsent image data, generate an image by estimating the image to be generated based on the image data in question in the terminal.

[0006] However, for example, in the case where there is a sudden scene switch between the game images, or in the case where the generated game images include objects that have not appeared in the past, it is difficult for the terminal to estimate the image to be generated based on the image data that has not been sent. In this case, there is a risk of displaying a false moving image that is different from the image that should be displayed. Please note that this applies not only to cloud gaming services, but also to general image-based data transmission.

[0007] The present invention is made in view of the above situation, and its purpose is to provide an image sending / receiving system, an image sending device, an image receiving device, an image sending / receiving method and a program, which can reduce the amount of image data communication while reducing the risk of displaying erroneous motion images that are different from the images that should be displayed.

[0008] Solution to the problem

[0009] In order to solve the above-mentioned problems, according to the present invention, there is provided an image transmission / reception system including an image transmission device and an image receiving device. The image transmission device includes an image generation unit configured to generate a first image and generate a second image after generating the first image; an image data transmission unit configured to transmit first image data generated based on the first image; and a transmission control unit configured to control whether to transmit second image data generated based on the second image. The image receiving device includes an image data receiving unit configured to receive first image data; a to-be-displayed image generation unit configured to generate a first to-be-displayed image based on the first image data; an alternative image generation unit configured to generate an alternative image by estimating a second to-be-displayed image generated by the to-be-displayed image generation unit based on the second image data based on the first image data; and a display control unit configured to display any one of the second to-be-displayed images generated based on the second image data and the alternative image after displaying the first to-be-displayed image, depending on whether reception of the second image data is successful. The transmission control unit controls whether to transmit the second image data based on the possibility that the alternative image generation unit successfully estimates the second to-be-displayed image based on the first image data.

[0010] In one aspect of the present invention, the sending control unit controls whether to send the second image data based on the possibility that the alternative image generating unit successfully estimates the second image to be displayed based on the first image data, the possibility being determined based on the feature amount of the first image and the feature amount of the second image.

[0011] In this aspect, the sending control unit can control whether to send the second image data based on the possibility that the alternative image generation unit successfully estimates the second image to be displayed based on the first image data, and the possibility is determined based on the time sequence of feature quantities including the feature quantity of the first image and the feature quantity of the second image.

[0012] In addition, in one aspect of the present invention, the sending control unit controls whether to send second image data indicating a reduced image based on the possibility that the alternative image generation unit successfully estimates the second image to be displayed based on the first image data, and the possibility is determined based on a feature quantity including parameters for encoding the reduced image obtained by reducing the second image.

[0013] Alternatively, the image data sending unit sends first image data indicating a first reduced image obtained by reducing the first image, and the sending control unit controls whether to send second image data indicating the second reduced image based on the possibility that the alternative image generating unit successfully estimates the second image to be displayed based on the first image data, the possibility being determined based on a feature quantity including parameters for encoding the first reduced image and a feature quantity including parameters for encoding the second reduced image obtained by reducing the second image.

[0014] Here, the feature amount may indicate at least any one of multiple information indicating feature points of an image, edge strength of an image, depth of pixels included in an image, texture type of an image, optical flow in an image, and moving direction and speed of a rectangular area in an image.

[0015] Furthermore, the transmission control unit may perform control of transmitting the feature amount data indicating the feature amount to the image receiving device in priority to transmitting the image data indicating the image.

[0016] Furthermore, in one aspect of the present invention, the transmission control unit controls whether to transmit the second image data based on whether the second image includes a key frame.

[0017] Furthermore, in one aspect of the present invention, the transmission control unit controls whether to transmit the second image data based on whether a scene in the second image is switched from a scene in the first image.

[0018] Furthermore, in one aspect of the present invention, the transmission control unit controls whether to transmit the second image data based on whether the second image includes an image of an object that does not appear in the first image.

[0019] Furthermore, in one aspect of the present invention, the substitute image generation unit starts generating the substitute image based on the first image data before checking whether reception of the second image data is successful.

[0020] Furthermore, in one aspect of the present invention, when the transmission speed of the image transmitting device is higher than a predetermined speed, the transmission control unit performs control of transmitting the second image data regardless of the possibility that the alternative image generating unit successfully estimates the second image to be displayed based on the first image data.

[0021] In addition, according to the present invention, there is provided an image sending device, comprising: an image generating unit, configured to generate a first image and generate a second image after generating the first image; an image data sending unit, configured to send first image data generated based on the first image to an image receiving device, the image receiving device comprising an image data receiving unit, configured to receive the first image data; an image-to-be-displayed generating unit, configured to generate a first image-to-be-displayed based on the first image data; an alternative image generating unit, configured to generate an alternative image by estimating, based on the first image data, a second image-to-be-displayed generated by the image-to-be-displayed generating unit based on the second image, to generate an alternative image; and a display control unit, configured to, after displaying the first image-to-be-displayed, display any one of the second images-to-be-displayed generated based on the second image data and the alternative image, depending on whether reception of the second image data is successful; and a sending control unit, configured to control whether to send the second image data generated based on the second image to the image receiving device, based on the possibility that the alternative image generating unit successfully estimates the second image-to-be-displayed based on the first image data.

[0022] In addition, according to the present invention, there is provided an image receiving device, comprising an image data receiving unit, configured to receive first image data from an image sending device; a to-be-displayed image generating unit, configured to generate a first to-be-displayed image based on the first image data; a substitute image generating unit, configured to generate a substitute image by estimating, based on the first image data, a second to-be-displayed image generated by the to-be-displayed image generating unit based on the second image data; and a display control unit, configured to, after displaying the first to-be-displayed image, display any one of the second to-be-displayed images generated based on the second image data and the substitute image, depending on whether the reception of the second image data is successful and whether the second image data controlled by the image sending device is sent based on the possibility that the substitute image generating unit successfully estimates the second image to be displayed based on the first image data.

[0023] Furthermore, according to the present invention, there is provided an image transmission / reception method, comprising: generating a first image through a first image generation step of an image transmission device; generating a second image after generating the first image through a second image generation step of the image transmission device; transmitting the first image data generated based on the first image through an image data transmission step of the image transmission device; controlling whether to transmit the second image data generated based on the second image through a transmission control step of the image transmission device; receiving the first image data through an image data reception step of an image reception device; generating a first image to be displayed based on the first image data through a to-be-displayed image generation step of the image reception device; generating a substitute image by estimating the second image to be displayed generated based on the second image data in the to-be-displayed image generation step based on the first image data through a substitute image generation step of the image reception device; and displaying any one of the second image to be displayed generated based on the second image data and the substitute image after displaying the first image to be displayed, depending on whether reception of the second image data is successful, through a display control step of the image reception device. In the transmission control step, the image transmission device controls whether to transmit the second image data based on the possibility that the second image to be displayed is successfully estimated based on the first image data in the substitute image generation step.

[0024] In addition, according to the present invention, there is provided a program for causing a computer to execute: a first image generating process for generating a first image; a second image generating process for generating a second image after generating the first image; an image data sending process for sending the first image data generated based on the first image to an image receiving device, the image receiving device including an image data receiving unit configured to receive the first image data; a to-be-displayed image generating unit configured to generate the first to-be-displayed image based on the first image data; an alternative image generating unit configured to generate an alternative image by estimating, based on the first image data, a second to-be-displayed image generated by the to-be-displayed image generating unit based on the second image, to generate an alternative image; and a display control unit configured to display, after displaying the first to-be-displayed image, any one of the second to-be-displayed images generated based on the second image data and the alternative image, depending on whether reception of the second image data is successful; and a sending control process for controlling whether the second image data generated based on the second image is sent to the image receiving device based on the possibility that the alternative image generating unit successfully estimates the second to-be-displayed image based on the first image data.

[0025] In addition, according to the present invention, another program is provided for causing a computer to execute: an image data receiving process, receiving first image data from an image sending device; a to-be-displayed image generating process, generating a first to-be-displayed image based on the first image data; an alternative image generating process, generating an alternative image by estimating, based on the first image data, a second to-be-displayed image generated by an to-be-displayed image generating unit based on the second image data to generate an alternative image; and a display control process, after displaying the first to-be-displayed image, displaying any one of the second to-be-displayed images generated based on the second image data and the alternative image, depending on whether the reception of the second image data is successful and whether the second image data controlled by the image sending device is sent based on the possibility of successfully estimating the second image to be displayed based on the first image data in the alternative image generating process. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a diagram showing an exemplary overall configuration of a cloud gaming system according to an embodiment of the present invention.

[0027] Figure 2 is a functional block diagram illustrating exemplary functions implemented in a cloud gaming system according to an embodiment of the present invention.

[0028] Figure 3 is a flowchart showing an exemplary process flow executed in a cloud server according to an embodiment of the present invention.

[0029] Figure 4 is a diagram schematically illustrating an exemplary success estimation likelihood determination model.

[0030] Figure 5 It is schematically shown Figure 4 A diagram of an exemplary learning of a success estimation likelihood determination model is shown.

[0031] Figure 6 is a flowchart showing an exemplary process flow executed in a terminal according to an embodiment of the present invention.

[0032] Figure 7 is a diagram schematically showing an exemplary enlarged model.

[0033] Figure 8 It is schematically shown Figure 7 Figure 2 shows an exemplary learning diagram of the zoomed-in model.

[0034] Fig. 9 is a diagram schematically illustrating an exemplary alternative image generation model.

[0035] Fig.10 It is schematically shown Fig. 9 Figure 2. Example learning diagram of an alternative image generation model shown.

[0036] Fig.11 is a diagram schematically illustrating an exemplary alternative image update model.

[0037] Fig.12 It is schematically shown Fig.11 Figure 2. Example learning diagram of the surrogate image update model shown.

[0038] Fig.13 FIG. 4 is a flowchart showing another exemplary process flow executed in a terminal according to an embodiment of the present invention.

[0039] Fig.14 is a diagram schematically showing another exemplary enlarged model.

[0040] Fig.15 It is schematically shown Fig.14 Figure 2 shows an exemplary learning diagram of the zoomed-in model.

[0041] Fig.16 is a diagram schematically illustrating another exemplary alternative image generation model.

[0042] Fig.17 It is schematically shown Fig.16 Figure 2. Example learning diagram of an alternative image generation model shown.

[0043] Fig.18 is a diagram schematically showing an exemplary feature amount data update model.

[0044] Fig.19 It is schematically shown Fig.18 A diagram showing an exemplary learning of a feature quantity data update model is shown.

[0045] Fig. 20 is a diagram schematically illustrating another exemplary alternative image update model.

[0046] Fig.21 It is schematically shown Fig. 20 Figure 2. Example learning diagram of the surrogate image update model shown. DETAILED DESCRIPTION

[0047] Figure 1 is a diagram showing an exemplary overall configuration of a cloud gaming system 1 according to an embodiment of the present invention. Figure 1 As shown, the cloud gaming system 1 according to the present embodiment includes a cloud server 10 and a terminal 12, each of which mainly includes a computer. The cloud server 10 and the terminal 12 are connected to a computer network 14 such as the Internet so that the cloud server 10 and the terminal 12 can communicate with each other.

[0048] The cloud server 10 according to the present embodiment is, for example, a server computer configured to execute a video game program based on a cloud game service. The cloud server 10 transmits a moving image depicting a situation in a video game to a terminal 12 used by a user playing the video game.

[0049] like Figure 1 As shown, the cloud server 10 includes, for example, a processor 10a, a storage unit 10b, a communication unit 10c, and an encoding / decoding unit 10d.

[0050] The processor 10a is, for example, a program control device such as a central processing unit (CPU), and performs various types of information processing according to a program stored in the storage unit 10b. The processor 10a according to the present embodiment also includes a graphics processing unit (GPU) configured to draw an image in a frame buffer based on a graphics command or data provided from the CPU.

[0051] The storage unit 10b is, for example, a storage element such as a read-only memory (ROM) or a random access memory (RAM) or a hard disk drive. The storage unit 10b stores, for example, a program executed by the processor 10a. In addition, the storage unit 10b according to the present embodiment has a frame buffer in which the GPU included in the processor 10a draws an image.

[0052] The communication unit 10 c is, for example, a communication interface for transmitting and receiving data to and from a computer such as the terminal 12 via the computer network 14 .

[0053] The encoding / decoding unit 10 d includes, for example, an encoder configured to encode an image to generate image data and a decoder configured to decode the image data to generate an image.

[0054] The terminal 12 according to the present embodiment is, for example, a computer used by a user who uses a cloud gaming service, such as a video game console, a personal computer, a tablet device, or a smartphone.

[0055] like Figure 1 As shown, the terminal 12 includes, for example, a processor 12a, a storage unit 12b, a communication unit 12c, an encoding / decoding unit 12d, an operation unit 12e, and a display unit 12f.

[0056] The processor 12a is, for example, a program control device such as a CPU, and performs various types of information processing according to a program stored in the storage unit 12b. The processor 12a according to the present embodiment also includes a GPU configured to draw an image in a frame buffer based on a graphic command or data provided from the CPU.

[0057] The storage unit 12b is, for example, a storage element such as a ROM or a RAM or a hard disk drive. The storage unit 12b stores, for example, a program executed by the processor 12a. In addition, the storage unit 12b according to the present embodiment has a frame buffer in which the GPU included in the processor 12a draws an image.

[0058] The communication unit 12 c is, for example, a communication interface for transmitting / receiving data to / from a computer such as the cloud server 10 via the computer network 14 .

[0059] The encoding / decoding unit 12d includes, for example, an encoder and a decoder. The encoder encodes an input image to generate image data indicating the image. Furthermore, the decoder decodes the input image data to output an image indicated by the image data.

[0060] The operation unit 12e is, for example, an operation member for performing operation input to the processor 12a.

[0061] The display unit 12f is, for example, a display device such as a liquid crystal display or an organic electroluminescence (EL) display.

[0062] When an operation related to a video game is performed through the operating unit 12e, the terminal 12 according to the present embodiment sends an operation signal indicating the operation to the cloud server 10. Then, the cloud server 10 performs video game processing according to the operation signal. Then, a play image is generated that describes the situation in the video game affected by the operation signal. In the present embodiment, the video game processing and play image generation are performed at a predetermined frame rate (e.g., 60fps). Note that in the present embodiment, the video game processing and play image generation can be performed at a variable frame rate.

[0063] Here, in the present embodiment, instead of image data indicating a play image, image data indicating an image obtained by reducing the play image is sent from the cloud server 10 to the terminal 12. Hereinafter, the play image before reduction is referred to as an "original image", and the image obtained by reducing the original image is referred to as a "reduced image". Here, the original image may be a 4K image, and the reduced image may be a 2K image.

[0064] Then, the terminal 12 generates an image with a higher resolution than the above-mentioned reduced image based on the image data received from the cloud server 10. Hereinafter, the image generated in this way with a higher resolution than the reduced image is referred to as an "enlarged image". Then, the enlarged image is displayed on the display unit 12f of the terminal 12. This series of processing stages is repeatedly performed so that a moving image including the series of enlarged images is displayed on the terminal 12.

[0065] As the frequency of image data transmission from the cloud server 10 to the terminal 12 increases, a higher frame rate and smoother moving images can be displayed on the terminal 12, but the communication volume between the cloud server 10 and the terminal 12 increases.

[0066] Here, in order to reduce the amount of communication while preventing the smoothness of the displayed moving images from decreasing, it is conceivable to reduce the frequency of sending image data, and for image data that is not sent, generate an image by estimating the image to be generated (hereinafter referred to as the "alternative image") in terminal 12 based on the image data in question.

[0067] However, for example, in the case where there is a sudden scene switch between the play images, or in the case where the play images generated include objects that have not appeared in the past, it is difficult for the terminal 12 to estimate the image to be generated based on the unsent image data. In this case, there is a risk of displaying a false motion image that is different from the image that should be displayed.

[0068] Therefore, in this embodiment, as described below, the transmission of image data is controlled based on the possibility that the terminal 12 successfully estimates the above-mentioned image to be generated, thereby reducing the image data communication volume while reducing the risk of displaying a false moving image different from the image that should be displayed.

[0069] Now, the functions of the cloud gaming system 1 according to the present embodiment and the processing performed in the cloud gaming system 1 are further described mainly in terms of image data transmission control processing.

[0070] Figure 2 1 is a functional block diagram showing exemplary functions implemented in the cloud gaming system 1 according to the present embodiment. Note that in the cloud gaming system 1 according to the present embodiment, it is not necessary to implement Figure 2 All functions shown in the Figure 2 Functions other than those shown in (for example, video game processing functions based on operation signals, etc.)

[0071] like Figure 2 As shown, the cloud server 10 according to this embodiment functionally includes, for example, an original image generation unit 20, a reduction unit 22, an encoding processing unit 24, a feature data generation unit 26, a feature data storage unit 28, a sending control unit 30 and a data sending unit 32.

[0072] The original image generation unit 20, the reduction unit 22 and the transmission control unit 30 are mainly implemented by the processor 10a. The encoding processing unit 24 is mainly implemented by the encoding / decoding unit 10d. The feature data generation unit 26 is mainly implemented by the processor 10a and the storage unit 10b. The feature data storage unit 28 is mainly implemented by the storage unit 10b. The data transmission unit 32 is mainly implemented by the communication unit 10c. The cloud server 10 of this embodiment plays the role of an image transmission device configured to transmit image data.

[0073] The above functions are implemented by the processor 10a executing a program including instructions corresponding to the above functions, which has been installed on the cloud server 10 as a computer. The program is provided to the cloud server 10 through a computer-readable information storage medium (such as an optical disk, a magnetic disk, a magnetic tape, a magneto-optical disk, or a flash memory) or via, for example, the Internet.

[0074] In addition, if Figure 2 As shown in , the terminal 12 according to the present embodiment functionally includes, for example, a data receiving unit 40, a decoding processing unit 42, an amplification unit 44, an image storage unit 46, an alternative image generation unit 48, and a display control unit 50. The data receiving unit 40 is mainly implemented by the communication unit 12c. The decoding processing unit 42 is mainly implemented by the encoding / decoding unit 12d. The amplification unit 44 and the alternative image generation unit 48 are mainly implemented by the processor 12a and the storage unit 12b. The image storage unit 46 is mainly implemented by the storage unit 12b. The display control unit 50 is mainly implemented by the processor 12a and the display unit 12f. The terminal 12 of the present embodiment functions as an image receiving device configured to receive image data.

[0075] The above functions are implemented by the processor 12a executing a program including instructions corresponding to the above functions, which has been installed on the terminal 12 as a computer. The program is provided to the terminal 12 through a computer-readable information storage medium (such as an optical disk, magnetic disk, magnetic tape, magneto-optical disk or flash memory) or via, for example, the Internet.

[0076] The original image generation unit 20 of the present embodiment generates, for example, the above-mentioned original image. As described above, the original image is, for example, a play image that describes a situation in a video game.

[0077] The reduction unit 22 of the present embodiment generates a reduced image by, for example, reducing the original image generated by the original image generation unit 20 .

[0078] The encoding processing unit 24 of the present embodiment generates, for example, image data indicating a reduced image obtained by reducing the original image. For example, the encoding processing unit 24 encodes the reduced image generated by the reduction unit 22 to generate image data indicating the reduced image. Here, in the encoding of the reduced image, the reduced image may be compressed so that image data smaller than the reduced image may be generated. Examples of encoding methods for encoding reduced images include MPEG-4 Advanced Video Codec (AVC) / H.264 / and High Efficiency Video Codec (HEVC) / H265.

[0079] The feature amount data generating unit 26 of the present embodiment generates, for example, feature amount data indicating at least one of a feature amount of the original image generated by the original image generating unit 20 and a feature amount corresponding to a parameter for reduced image encoding. The feature amount data is data that cannot be identified only from the image data indicating the reduced image.

[0080] Then, the feature amount data generating unit 26 of the present embodiment stores the generated feature amount data in, for example, the feature amount data storage unit 28. Here, for example, the feature amount data may be stored in the feature amount data storage unit 28 in association with a timestamp indicating the timing at which the feature amount data has been generated, information indicating the position of the feature amount data in the generation order, and the like. In this way, in the present embodiment, for each of the plurality of feature amount data stored in the feature amount data storage unit 28, the timing at which the feature amount data has been generated and the position of the feature amount data in the generation order are identifiable.

[0081] Here, the above-mentioned feature quantity may indicate at least one of the feature points or edge strength indicating the original image, the depth of each pixel included in the original image, the texture type or optical flow of the original image, and the moving direction and speed of the rectangular area in the image. For example, the feature quantity data generating unit 26 can extract the feature quantity of the original image to generate feature quantity data indicating the extracted feature quantity. The feature quantity data may include, for example, data indicating the texture type of the original image (for example, data indicating the position of the edge area, flat area, high density area, detail area, and crowd area). In addition, the feature quantity data may include data indicating the position of the Harris corner feature point or the edge strength of the original image. In addition, the feature quantity data may include data indicating the optical flow in the original image. In addition, the feature quantity data may include depth data indicating the depth of each pixel included in the original image.

[0082] Furthermore, the feature amount data generating unit 26 may perform an object recognition process on the original image. Furthermore, the feature amount data may include data indicating a feature amount of the original image, including data indicating an object in the original image recognized by the object recognition process.

[0083] In addition, the feature amount data generating unit 26 may generate feature amount data indicating feature amounts corresponding to the parameters used by the encoding processing unit 24 for encoding. The feature amount data may include, for example, motion estimation (ME) information data indicating the moving direction and speed of a rectangular area in an image. In addition, the feature amount data may include data about CU unit allocation information in units of CU unit size, which indicates the position to which CU units of the CU unit size have been allocated. In addition, the feature amount data may include data indicating a region of interest (ROI), which is an area to which a high bit rate has been allocated to achieve higher image quality. In addition, the feature amount data may include data indicating a quantization parameter value. In addition, the feature amount data may include data indicating whether there is a scene switch or data indicating whether a frame is a key frame (I frame).

[0084] Furthermore, for example, the feature amount data generating unit 26 may generate feature amount data indicating the feature amount of the original image and the feature amount corresponding to the parameter used by the encoding processing unit 24 for encoding. Furthermore, for example, the feature amount data generating unit 26 may generate feature amount data including data generated based on the data indicating the feature amount of the original image and the data indicating the feature amount corresponding to the parameter used by the encoding processing unit 24 for encoding. For example, the feature amount data generating unit 26 may decide whether to perform super-resolution processing based on the data indicating the feature amount of the original image and the data indicating the feature amount corresponding to the parameter used by the encoding processing unit 24 for encoding. Then, the feature amount data generating unit 26 may generate feature amount data including data indicating whether to perform super-resolution processing.

[0085] The feature amount data storage unit 28 of the present embodiment stores, for example, the feature amount data generated by the feature amount data generation unit 26 .

[0086] The transmission control unit 30 of the present embodiment controls, for example, whether to transmit the image data generated by the encoding processing unit 24 to the terminal 12. Here, for example, the transmission control unit 30 controls the data transmission unit 32 to control whether to transmit the image data to the terminal 12.

[0087] The data transmission unit 32 of this embodiment transmits the image data generated by the encoding processing unit 24 to the terminal 12, for example. In this embodiment, the data transmission unit 32 transmits or does not transmit the image data to the terminal 12 according to the control of the transmission control unit 30. Here, the data transmission unit 32 may discard the image data not transmitted to the terminal 12.

[0088] The data receiving unit 40 of the present embodiment receives the above-mentioned image data transmitted from the cloud server 10 , for example.

[0089] In the present embodiment, the transmission of image data by the data transmission unit 32 and the reception of image data by the data reception unit 40 are repeatedly performed.

[0090] The decoding processing unit 42 of the present embodiment decodes the image data received by the data receiving unit 40, for example, to generate an image. Hereinafter, the image generated in this manner is referred to as a "decoded image". In the present embodiment, the decoded image is an image having the same resolution as the reduced image (e.g., 2K image). In the case where the reduced image is encoded in an irreversible manner, the decoded image is generally different from the reduced image.

[0091] The enlargement unit 44 of the present embodiment generates an image to be displayed on the display unit 12f based on the image data received by the data receiving unit 40, for example. Here, for example, as the image to be displayed, an enlarged image having a higher resolution than the reduced image generated by the reduction unit 22 is generated. The enlarged image may be an image having the same resolution as the original image (for example, a 4K image). In the present embodiment, an image to be displayed that is the same as or similar to the original image is generated based on the image data based on the original image. The details of the enlarged image generation process will be described later.

[0092] The image storage unit 46 of the present embodiment stores, for example, images generated in the terminal 12. Here, the image storage unit 46 may store an enlarged image generated by the enlarging unit 44. Furthermore, the image storage unit 46 may store a substitute image generated by the substitute image generating unit 48, as described below.

[0093] The substitute image generation unit 48 of the present embodiment generates, for example, a substitute image corresponding to the second image data received at a second timing which is predetermined to be received after the first timing, based on the first image data received at the first timing. For example, it is assumed that the original image generation unit 20 generates a first original image, and generates a second original image after generating the first original image. Then, the first image data is generated based on the first original image, and the second image data is generated based on the second original image. In this case, the substitute image generation unit 48 can generate a substitute image based on the first image data by estimating the enlarged image to be generated by the enlargement unit 44 based on the second image data. In addition, the substitute image generation unit 48 can generate a substitute image by estimating the second original image based on the first image data.

[0094] Here, for example, the substitute image generation unit 48 of the present embodiment may start generating a substitute image corresponding to the second image data based on the first image data before checking whether the second image data has been successfully received. Here, for example, based on the received first image data, a substitute image corresponding to the second image data that is scheduled to be received after the first image data may be generated. The substitute image may be an image having a higher resolution than the reduced image (for example, an image having the same resolution as the enlarged image (for example, a 4K image)). The details of the substitute image generation process are described later.

[0095] The display control unit 50 of the present embodiment displays, for example, a first image to be displayed generated based on the first image data. Afterwards, the display control unit 50 of the present embodiment displays, for example, a second image to be displayed generated based on the second image data and any one of a replacement image based on the first image data, depending on whether the second image data has been successfully received. Here, for example, the display control unit 50 can display, according to whether the second image data has been successfully received, any one of a second image to be displayed generated based on the second image data and a replacement image generated based on the above-mentioned first image data. Here, for example, the display control unit 50 causes the display unit 12f of the terminal 12 to display any one of an enlarged image and a replacement image.

[0096] Here, reference Figure 3 The flowchart of FIG. 1 describes an exemplary flow of image data generation and transmission processing performed in the cloud server 10 according to the present embodiment. The image data generation and transmission processing is repeatedly performed at a predetermined frame rate (for example, 60 fps). Figure 3 Note that the processing in S101 to S108 may be repeatedly performed at a variable frame rate.

[0097] First, the original image generation unit 20 generates an original image in a frame (S101). Here, as described above, an original image in a frame depicting a situation in a video game affected by an operation signal received from the terminal 12 can be generated.

[0098] Then, the reduction unit 22 performs a reduction process on the original image generated in the process of S101 to generate a reduced image ( S102 ).

[0099] Then, the encoding processing unit 24 performs encoding processing on the reduced image generated in the process of S102 to generate image data ( S103 ).

[0100] Then, the feature amount data generating unit 26 generates feature amount data (S104). For example, in the process of S104, the feature amount data generating unit 26 may generate feature amount data indicating the feature amount of the original image generated in the process of S101. Here, for example, feature amount data indicating the feature amount extracted from the original image generated in the process of S101 may be generated. Furthermore, for example, the feature amount data generating unit 26 may generate feature amount data indicating the feature amount corresponding to the parameter used for encoding in S103. Furthermore, for example, the feature amount data generating unit 26 may generate feature amount data indicating both the feature amount extracted from the original image generated in the process of S101 and the feature amount corresponding to the parameter used for encoding in S103.

[0101] Then, the feature amount data generating unit 26 stores the feature amount data generated in the process of S104 in the feature amount data storing unit 28 ( S105 ).

[0102] Then, the transmission control unit 30 determines the possibility that the substitute image generation unit 48 successfully estimates the enlarged image in the frame based on the image data transmitted in the immediately previous frame (S106). Here, an evaluation value indicating the possibility may be identified. For example, an evaluation value "1" may be identified in the case where the possibility of success is high, and an evaluation value "0" may be identified in the case where the possibility of success is low.

[0103] In the case where it is determined that there is a sudden scene switch between images or the generated image includes an image of an object that has not appeared in the past, the sending control unit 30 can identify an evaluation value of "0", otherwise an evaluation value of "1" can be identified. Here, for example, the sending control unit 30 can determine the above possibility based on a feature quantity corresponding to a parameter used for reduced image encoding in a frame, which is indicated by feature quantity data generated in the process of S104. For example, in the case where the feature quantity data in the frame includes data indicating whether there is a scene switch or data indicating that the frame is a key frame (I frame), an evaluation value of "0" can be identified, otherwise an evaluation value of "1" can be identified. In addition, for example, in the case where the size of the image data generated in the encoding process is larger than a predetermined size, an evaluation value of "0" can be identified, otherwise an evaluation value of "1" can be identified.

[0104] In addition, the transmission control unit 30 may evaluate the above-mentioned possibility based on, for example, first feature amount data which is feature amount data in the immediately preceding frame and second feature amount data which is feature amount data in the frame in question, the first feature amount data and the second feature amount data being stored in the feature amount data storage unit 28. Here, for example, the first feature amount data may be data indicating the feature amount of the original image in the immediately preceding frame, and the second feature amount data may be data indicating the feature amount of the original image in the frame in question. In addition, the first feature amount data may be data indicating the feature amount corresponding to the parameter used for reduced image encoding in the immediately preceding frame. The second feature amount data may be data indicating the feature amount corresponding to the parameter used for reduced image encoding in the frame in question.

[0105] For example, when the difference between the histogram corresponding to the second feature amount data and the histogram corresponding to the first feature amount data is greater than a predetermined difference, an evaluation value of "0" may be identified, otherwise an evaluation value of "1" may be identified. In addition, for example, when the object indicated by the second feature amount data includes an object not indicated by the first feature amount data, an evaluation value of "0" may be identified, otherwise an evaluation value of "1" may be identified.

[0106] Then, the data transmission unit 32 checks whether it has been determined in the process of S106 that the possibility that the substitute image generation unit 48 has successfully estimated a substitute image based on the image data transmitted in the immediately previous frame is low ( S107 ).

[0107] In the case where it has been determined that the probability of the substitute image generation unit 48 succeeding in the estimation is high (for example, in the case where the identification evaluation value is "1") (S107: N), the process returns to the process of S101. In this case, the data transmission unit 32 does not transmit the image data generated in the process of S103 to the terminal 12.

[0108] When it has been determined that the alternative image generation unit 48 has a low probability of success in the estimation (for example, when the identification evaluation value is "0") (S107: Y), the data sending unit 32 sends the image data generated in the processing of S103 to the terminal 12 (S108), and the processing returns to the processing of S101.

[0109] Here, in the case where the transmission speed of the cloud server 10 is higher than the predetermined speed, the transmission control unit 30 may perform control to transmit the second image data regardless of the possibility of successfully estimating the second image to be displayed based on the first image data. In this case, the data transmission unit 32 may transmit the second image data regardless of the possibility of successfully estimating the second image to be displayed based on the first image data.

[0110] For example, the transmission control unit 30 may always measure the transmission speed of the cloud server 10 based on the result of data transmission from the cloud server 10 to the terminal 12. Then, in the case where the current transmission speed is higher than the predetermined speed, the transmission control unit 30 may perform control to transmit the second image data regardless of the result of the determination in the process of S106. In this case, the process of S108 is performed regardless of the result of the determination in the process of S106.

[0111] As described above, the transmission control unit 30 can control whether to transmit the second image data to the terminal 12 based on the possibility that the substitute image generation unit 48 successfully estimates the second image to be displayed based on the first image data.

[0112] The possibility that the terminal 12 successfully estimates the second original image based on the first image data can be determined based on the feature amount of the first original image and the feature amount of the second original image. Alternatively, the possibility can be determined based on the feature amount corresponding to the parameter for encoding the reduced image obtained by reducing the original image. For example, the possibility can be determined based on the above-mentioned feature amount of the reduced image obtained by reducing the first original image and the above-mentioned feature amount of the reduced image obtained by reducing the second original image.

[0113] In addition, in the case where the second image is a key frame, the possibility that the substitute image generation unit 48 successfully estimates the second image to be displayed based on the first image data is low. Based on this, the transmission control unit 30 can control whether to transmit the second image data based on whether the second image is a key frame. In this case, it can be determined based on the above-mentioned second feature amount data whether the second image is a key frame.

[0114] In addition, in the case where the scene in the second image is switched from the scene in the first image, the possibility that the alternative image generation unit 48 successfully estimates the second image to be displayed based on the first image data is low. Based on this, the transmission control unit 30 can control whether to send the second image data based on whether the scene in the second image is switched from the scene in the first image. In this case, it can be determined based on the first image and the second image whether the scene in the second image is switched from the scene in the first image. Alternatively, it can be determined based on at least one of the above-mentioned first feature amount data and the second feature amount data whether the scene in the second image is switched from the scene in the first image.

[0115] In addition, in the case where the second image includes an image of an object that does not appear in the first image, the possibility of successfully estimating the second image to be displayed based on the first image data by the alternative image generation unit 48 is low. Based on this, the transmission control unit 30 can control whether to send the second image data based on whether the second image includes an image of an object that does not appear in the first image. In this case, it can be determined based on the first image and the second image whether the second image includes an image of an object that does not appear in the first image. Alternatively, it can be determined based on at least one of the above-mentioned first feature amount data and the second feature amount data whether the second image includes an image of an object that does not appear in the first image.

[0116] In addition, in the process of S106 in the above exemplary process, it is possible to use Figure 4 The above determination is performed using the successful estimation possibility evaluation model 60 of the training machine learning model illustrated in the example. Here, the successful estimation possibility evaluation model 60 may adopt deep learning. The successful estimation possibility evaluation model 60 may include coupling topology information and coupling strength parameter information about the nodes included in the neural network.

[0117] In this case, the first feature quantity data as the feature quantity data in the immediately previous frame and the second feature quantity data as the feature quantity data in the frame in question stored in the feature quantity data storage unit 28 are input to the success estimation possibility evaluation model 60. Then, the success estimation possibility evaluation model 60 outputs the success estimation possibility data based on the input. Here, for example, in the case where the possibility of successful estimation is high, the success estimation possibility data having a value of 1 can be output, and in the case where the possibility of successful estimation is low, the success estimation possibility data having a value of 0 can be output. Here, the success estimation possibility evaluation model 60 is included in the sending control unit 30, for example.

[0118] Figure 5 It is schematically shown Figure 4 A diagram of exemplary learning of the success estimation possibility evaluation model 60 is shown. For example, learning using a plurality of learning data is performed by the success estimation possibility evaluation model 60. The learning data includes, for example, learning input data including first learning input feature quantity data and second learning input feature quantity data and success estimation possibility training data as training data to be compared with the success estimation possibility data output from the success estimation possibility evaluation model 60.

[0119] By processing similar to the above processing, first learning input feature quantity data can be generated, which is feature quantity data corresponding to a high-resolution image (for example, a 4K image) used as a first frame image included in a given motion image. Then, by processing similar to the above processing, second learning input feature quantity data can be generated, which is feature quantity data corresponding to a second frame image, which is a frame image in a frame next to the first frame image in the above given motion image. Then, learning input data including the first learning input feature quantity data and the second learning input feature quantity data can be generated.

[0120] Then, the user can decide the value of the success estimation possibility training data corresponding to the learning input data. For example, the user can check the comparison between the first frame image and the second frame image with his / her eyes, or check the comparison between the substitute images generated by the substitute image generation unit 48 based on the first frame image and the second frame image, thereby determining the value of the success estimation possibility training data. In addition, for example, the user can decide the value of the success estimation possibility training data based on an empirical rule. Then, learning data including the success estimation possibility training data having the value determined in this way and the learning input data corresponding to the success estimation possibility training data can be generated.

[0121] By using supervised learning of a plurality of pieces of learning data generated in this manner based on a plurality of motion images and a plurality of frame images, it is possible to perform learning of the success estimation possibility evaluation model 60. For example, the success estimation possibility data output when the learning input data included in the learning data is input to the success estimation possibility evaluation model 60 and the success estimation possibility training data included in the learning data in question can be compared with each other. Then, based on the result of the comparison, the parameters of the success estimation possibility evaluation model 60 can be updated by back propagation so that the learning of the success estimation possibility evaluation model 60 can be performed.

[0122] Note that the learning of the success estimation possibility evaluation model 60 is not limited to the above method. For example, the learning of the success estimation possibility evaluation model 60 can be performed by unsupervised learning or reinforcement learning.

[0123] In addition, the success estimation possibility assessment model 60 can receive as input at least one of the original image in the immediately previous frame as the basis of the first feature quantity data and the original image in the frame in question as the basis of the second feature quantity data. In this case, learning data including at least one of the first frame image and the second frame image can be used to perform learning of the success estimation possibility assessment model 60, and the learning data corresponds to the input to the success estimation possibility assessment model 60.

[0124] Next, refer to Figure 6 The flowchart of FIG. 1 describes an exemplary flow of the enlarged image or substitute image display control process executed in the terminal 12 according to the present embodiment. The process is repeatedly executed at a predetermined frame rate (for example, 60 fps). Figure 6 Note that the processing in S201 to S212 may be repeatedly performed at a variable frame rate.

[0125] First, the data receiving unit 40 receives the image data transmitted from the cloud server 10 in the process of S108 ( S201 ).

[0126] Then, the decoding processing unit 42 performs a decoding process on the image data received in the process of S201 to generate a decoded image ( S202 ).

[0127] Then, the amplification unit 44 checks whether the reception of all the image data in the frame is successful (S203). In the case where the terminal 12 has not received the image data, or in the case where the terminal 12 has received only the damaged image data, the reception failure of all the image data in the frame is confirmed in the process of S203. In addition, in the case where the decoded image has not been generated in the process of S202, the reception failure of all the image data in the frame can be confirmed in the process of S203. In addition, here, in the case where the reception of all the image data has failed, the data receiving unit 40 can notify the amplification unit 44 of the reception failure of all the image data. Then, when the amplification unit 44 receives the notification, the reception failure of all the image data in the frame can be confirmed in the process of S203.

[0128] It is assumed that it is confirmed in the process of S203 that the reception of all the image data in the frame has been successful (S203: Y). In this case, the enlargement unit 44 performs an enlargement process on the decoded image generated in the process of S202 to generate an enlarged image (S204). Here, the enlargement process using the enlargement model 62 is performed, and the enlargement model 62 is Figure 7 Here, the amplified model 62 may adopt deep learning. The amplified model 62 may include coupling topology information and coupling strength parameter information about the nodes included in the neural network.

[0129] In the process of S204, for example, the decoded image in the frame generated in the process of S202 is input to the enlargement model 62. Then, the enlargement model 62 outputs the enlarged image based on the input. Here, for example, the enlargement model 62 is included in the enlargement unit 44.

[0130] Figure 8 It is schematically shown Figure 71 is a diagram of exemplary learning of the enlarged model 62 shown. For example, learning using a plurality of pieces of learning data is performed by the enlarged model 62. The learning data includes, for example, learning input data including a learning input image and an enlarged training image as training data to be compared with the enlarged image output from the enlarged model 62.

[0131] For example, by performing the above-mentioned reduction processing, the above-mentioned encoding processing, and the above-mentioned decoding processing on a high-resolution image (for example, a 4K image), a learning input image which is a low-resolution image (for example, a 2K image) can be generated.

[0132] Then, learning data including the above-mentioned learning input image and the enlarged training image as the above-mentioned high-resolution image can be generated. Then, by supervised learning using a plurality of pieces of learning data generated in this manner based on a plurality of high-resolution images, learning of the enlarged model 62 can be performed. For example, an enlarged image output when the learning input data included in the learning data is input to the enlarged model 62 and an enlarged training image included in the learning data in question can be compared with each other. Then, based on the result of the comparison, the parameters of the enlarged model 62 can be updated by back propagation so that learning of the enlarged model 62 can be performed.

[0133] Note that the learning of the amplification model 62 is not limited to the above method. For example, the learning of the amplification model 62 can be performed by unsupervised learning or reinforcement learning.

[0134] When the process of S204 ends, the substitute image generation unit 48 starts to generate a substitute image of the next frame based on the image generated according to the image data received in the process of S201 as a basic image for substitute image generation (S205). Here, for example, the enlargement unit 44 may store the enlarged image generated in the process of S204 as a basic image in the image storage unit 46. In addition, for example, the enlargement unit 44 may store the decoded image generated in the process of S202 as a basic image in the image storage unit 46. Then, the substitute image generation unit 48 may start substitute image generation based on the basic image stored in the image storage unit 46.

[0135] Here, the substitute image generation process is started using the substitute image generation model 64. The substitute image generation model 64 is Fig. 9 Here, the substitute image generation model 64 may adopt deep learning. The substitute image generation model 64 may include coupling topology information and coupling strength parameter information about the nodes included in the neural network.

[0136] In the process of S205, for example, the basic image stored in the image storage unit 46 in the frame is input to the substitute image generation model 64. Then, the substitute image generation model 64 outputs the substitute image based on the input. Here, the substitute image generation model 64 is included in the substitute image generation unit 48, for example.

[0137] Fig.10 It is schematically shown Fig. 9 1 is a diagram of exemplary learning of the substitute image generation model 64. For example, learning using a plurality of pieces of learning data is performed by the substitute image generation model 64. The learning data includes, for example, learning input data including a learning input image and a substitute training image as training data to be compared with the substitute image output from the substitute image generation model 64.

[0138] For example, it is assumed that the basic image is a decoded image. In this case, the above-mentioned reduction processing, the above-mentioned encoding processing, and the above-mentioned decoding processing can be performed on a high-resolution image (for example, a 4K image) as a frame image included in a given motion image. Thus, a learning input image as a low-resolution image (for example, a 2K image) can be generated. In addition, for example, it is assumed that the basic image is an enlarged image. In this case, the above-mentioned reduction processing, the above-mentioned encoding processing, the above-mentioned decoding processing, and the above-mentioned enlargement processing can be performed on a high-resolution image (for example, a 4K image) as a frame image included in a motion image. Thus, a learning input image as a high-resolution image can be generated.

[0139] Then, learning data can be generated, which includes learning input data containing the above-mentioned learning input image, and includes a substitute training image of a frame image in a frame next to a frame image corresponding to the learning input image in the above-mentioned given motion image. Then, by using supervised learning of multiple learning data generated in this way based on multiple motion images and multiple frame images, learning of the substitute image generation model 64 can be performed. For example, a substitute image output when the learning input data included in the learning data is input to the substitute image generation model 64 and a substitute training image included in the learning data in question can be compared with each other. Then, based on the result of the comparison, the parameters of the substitute image generation model 64 can be updated by back propagation so that learning of the substitute image generation model 64 can be performed.

[0140] Note that the learning of the substitute image generation model 64 is not limited to the above method. For example, the learning of the substitute image generation model 64 can be performed by unsupervised learning or reinforcement learning.

[0141] Then, the substitute image generation unit 48 stores the substitute image of the frame next to it generated in the process started in S205 in the image storage unit 46 ( S206 ).

[0142] Then, the display control unit 50 causes the display unit 12f to display the enlarged image generated in the process of S204 (S207), and the process returns to the process of S201.

[0143] Assume that it is confirmed in the process of S203 that the reception of all the image data in the frame has failed (S203: Y). In this case, the substitute image generation unit 48 checks whether the reception of part of the image data in the frame is successful (S208). In the case where part of the image data received by the terminal 12 is damaged, the reception failure of part of the image data in the frame is confirmed in the process of S208. In addition, in the case where the decoded image generated in the process of S202 has a missing portion, the reception failure of part of the image data in the frame can be confirmed in the process of S208. In addition, here, in the case of the reception failure of part of the image data, the data receiving unit 40 can notify the substitute image generation unit 48 of the reception failure of the part of the image data. Then, when the substitute image generation unit 48 receives the notification, the reception failure of part of the image data in the frame can be confirmed in the process of S208.

[0144] Assume that it is confirmed in the process of S208 that the reception of the partial image data in the frame has been successful (S208: Y). In this case, the substitute image generation unit 48 updates the substitute image of the frame based on the partial image data received in the frame (S209). In the process of S209, the partial image data received in the frame is reflected in the substitute image of the frame. Here, the substitute image update process using the substitute image update model 66 is performed, and the substitute image update model 66 is Fig.11 Here, the alternative image update model 66 may adopt deep learning. The alternative image update model 66 may include coupling topology information and coupling strength parameter information about the nodes included in the neural network.

[0145] In the process of S209, for example, the first substitute image stored in the process of S206 in the immediately preceding frame and the partial image data which is a part of the image data received in the frame in question are input to the substitute image update model 66. Then, the substitute image update model 66 outputs the second substitute image based on the input. In this way, the first substitute image stored in the process of S206 in the immediately preceding frame is updated to the second substitute image. Here, the substitute image update model 66 is included in the substitute image generation unit 48, for example.

[0146] Fig.12 It is schematically shown Fig.11FIG. 6 is a diagram of exemplary learning of the substitute image update model 66. For example, learning using a plurality of pieces of learning data is performed by the substitute image update model 66. The learning data includes, for example, learning input data including a learning input image and learning input partial image data, and a substitute training image as training data to be compared with the substitute image output from the substitute image update model 66.

[0147] For example, it is assumed that the basic image is a decoded image. In this case, the above-mentioned reduction processing, the above-mentioned encoding processing, and the above-mentioned decoding processing can be performed on a high-resolution image (for example, a 4K image) as a frame image included in a given motion image. Thus, a learning input image as a low-resolution image (for example, a 2K image) can be generated. In addition, for example, it is assumed that the basic image is an enlarged image. In this case, the above-mentioned reduction processing, the above-mentioned encoding processing, the above-mentioned decoding processing, and the above-mentioned enlargement processing can be performed on a high-resolution image (for example, a 4K image) as a frame image included in a motion image. Thus, a learning input image as a high-resolution image can be generated.

[0148] Then, learning input partial image data can be generated as a part of image data generated by performing the above-mentioned reduction processing and the above-mentioned encoding processing on a frame image in a frame next to a frame image corresponding to the learning input image in the above-mentioned given moving image.

[0149] Then, learning data can be generated, which includes the learning input data including the above-mentioned learning input image and the above-mentioned learning input partial image data, and a substitute training image, which is a frame image in a frame next to the frame image corresponding to the learning input image in the above-mentioned given motion image. Then, by using supervised learning of multiple learning data generated in this way based on multiple motion images and multiple frame images, learning of the substitute image update model 66 can be performed. For example, the substitute image output when the learning input data included in the learning data is input to the substitute image update model 66 and the substitute training image included in the learning data in question can be compared with each other. Then, based on the result of the comparison, the parameters of the substitute image update model 66 can be updated by back propagation so that learning of the substitute image update model 66 can be performed.

[0150] Note that the learning of the substitute image update model 66 is not limited to the above method. For example, the learning of the substitute image update model 66 can be performed by unsupervised learning or reinforcement learning.

[0151] When the processing of S209 ends or it is confirmed in the processing of S208 that the reception of the partial image data in the frame has not been successful (S208: N), the substitute image generation unit 48 starts to generate the substitute image of the next frame (S210). Here, when the processing of S209 ends, the substitute image generation unit 48 may store the substitute image updated in the processing of S209 as a basic image in the image storage unit 46. Then, the substitute image generation unit 48 may start to generate the substitute image of the next frame based on the basic image. In addition, in the processing of S208, when the reception of the partial image data in the frame has not been successful, the substitute image stored in the processing of S206 in the immediately previous frame may be stored in the image storage unit 46 as a basic image. Then, the substitute image generation unit 48 may start to generate the substitute image of the next frame based on the basic image.

[0152] Here, a substitute image generation process using a substitute image generation model 64 is performed. The substitute image generation model 64 is Fig. 9 Here, the alternative image generation model 64 may employ deep learning.

[0153] Then, the substitute image generation unit 48 stores the substitute image of the frame next to the one generated in the process started in S210 in the image storage unit 46 ( S211 ).

[0154] Then, the display control unit 50 causes the display unit 12f to display a substitute image of the frame (S212), and the process returns to the process of S201.

[0155] In this processing example, it is assumed that reception of all image data in the nth frame is confirmed to have succeeded in the nth frame (S203: Y). In this case, an enlarged image generated based on the image data received in the nth frame is displayed (S207). Furthermore, in this case, in the nth frame, a substitute image of the (n+1)th frame is generated based on the enlarged image (S205 and S206).

[0156] Assume that it is confirmed in the nth frame that the reception of all the image data in the nth frame has failed, but the reception of part of the image data in the nth frame has succeeded (S203: N and S208: Y). In this case, in the nth frame, the replacement image of the nth frame generated in the (n-1)th frame is updated (S209). Then, the updated replacement image of the nth frame is displayed (S212). In addition, in this case, in the nth frame, the replacement image of the (n+1)th frame is generated based on the updated replacement image of the nth frame (S210 and S211).

[0157] Assume that it is confirmed in the nth frame that the reception of all the image data in the nth frame has failed, and the reception of part of the image data in the nth frame has not yet succeeded (S203: N and S208: Y). In this case, in the nth frame, the replacement image of the nth frame generated in the (n-1)th frame is displayed (S212). In addition, in this case, in the nth frame, the replacement image of the (n+1)th frame is generated based on the replacement image of the nth frame generated in the (n-1)th frame (S210 and S211).

[0158] Furthermore, in this processing example, before starting to generate a substitute image for the next frame and ending to store the substitute image for the next frame, display of an enlarged image or display of a substitute image for the frame in question may be performed.

[0159] In addition, the substitute image update model 66 may output a determination value indicating whether an erroneous image different from the image that should be displayed is output as the second substitute image. For example, in the case where an erroneous image different from the image that should be displayed is output as the second substitute image, a determination value of "1" may be output, and in the case where the image that should be displayed is output as the second substitute image, a determination value of "0" may be output. In this case, the learning of the substitute image update model 66 may be performed using training data including a given learning determination value based on learning input data, which is compared with a determination value output from the substitute image update model 66 when the learning input data is received. In this case, in the process of S209, the substitute image update model 66 may also output a determination value.

[0160] Then, based on the determination value, in the process of S212, the display control unit 50 can control whether to make the display unit 12f display the substitute image generated in the frame. For example, in the case where the substitute image update model 66 outputs the determination value "1", the display control unit 50 can perform control to prevent the display of the substitute image. In addition, in the case where the substitute image update model 66 outputs the determination value "0", the display control unit 50 can perform control to display the substitute image.

[0161] In addition, in a similar manner, the substitute image generation model 64 may output a determination value indicating whether an erroneous image different from the image that should be displayed is output as a substitute image. For example, in the case where an erroneous image different from the image that should be displayed is output as a substitute image, a determination value "1" may be output, and in the case where the image that should be displayed is output as a substitute image, a determination value "0" may be output. In this case, the learning of the substitute image generation model 64 may be performed using training data including a given learning determination value based on learning input data, which is compared with a determination value output from the substitute image generation model 64 when the learning input data is received. In this case, in the processing of S205 or S210, the substitute image generation model 64 may also output a determination value.

[0162] Then, based on the determination value, in the process of S212 of the next frame, the display control unit 50 can control whether to make the display unit 12f display the substitute image generated in the frame in question. For example, in the case where the substitute image generation model 64 outputs the determination value "1", the display control unit 50 can perform control to prevent the display of the substitute image. In addition, in the case where the substitute image generation model 64 outputs the determination value "0", the display control unit 50 can perform control to display the substitute image.

[0163] Here, for example, in the case where any one of the substitute image generation model 64 and the substitute image update model 66 outputs a determination value of “1”, the display control unit 50 may perform control to prevent display of the substitute image.

[0164] In addition, Figure 3 In the process of S108 of FIG. 1 , the data transmission unit 32 may transmit the image data generated in the process of S103 and the feature quantity data generated in the process of S104 and associated with the image data to the terminal 12. In this case, the transmission control unit 30 may perform control to prioritize the transmission of the feature quantity data to the terminal 12 over the transmission of the image data. In this case, the data transmission unit 32 may prioritize the transmission of the feature quantity data to the terminal 12 over the transmission of the image data. For example, the transmission of the feature quantity data to the terminal 12 may be prioritized over the transmission of the image data by using the quality of service (QoS) or priority control technology.

[0165] Here, reference Fig.13 The flowchart of FIG. 1 describes an example flow of an enlarged image or substitute image display control process executed in the terminal 12 when the image data and the feature amount data are transmitted to the terminal 12. The process is repeatedly executed at a predetermined frame rate (for example, 60 fps). Fig.13 The processing in S301 to S314 shown. Note that the processing in S301 to S314 can be repeatedly performed at a variable frame rate.

[0166] First, the data receiving unit 40 receives the image data and the feature amount data associated with the image data transmitted from the cloud server 10 in the process of S108 ( S301 ).

[0167] Then, the decoding processing unit 42 performs a decoding process on the image data received in the process of S301 to generate a decoded image ( S302 ).

[0168] Then, the amplification unit 44 checks whether the reception of all the image data and the feature data in the frame is successful (S303). In the case where the terminal 12 has not received the image data and the feature data, or in the case where the terminal 12 has received only the damaged image data and the feature data, the failure to receive all the image data and the feature data in the frame is confirmed in the process of S303. In addition, in the case where the decoded image has not been generated in the process of S302, the failure to receive all the image data and the feature data in the frame can be confirmed in the process of S303. In addition, here, in the case where the reception of all the image data and the feature data fails, the data receiving unit 40 can notify the amplification unit 44 of the failure to receive all the image data and the feature data. Then, when the amplification unit 44 receives the notification, the failure to receive all the image data in the frame can be confirmed in the process of S303.

[0169] Assume that it is confirmed in the process of S203 that the reception of all the image data and feature amount data in the frame has been successful (S303: Y). In this case, the enlargement unit 44 performs an enlargement process on the decoded image generated in the process of S302 to generate an enlarged image (S304). Here, the enlargement process using the enlargement model 68 is performed. The enlargement model 62 is Fig.14 Here, the amplified model 68 may adopt deep learning. The amplified model 68 may include coupling topology information and coupling strength parameter information about the nodes included in the neural network.

[0170] In the process of S304, for example, the decoded image generated in the process of S302 and the feature amount data received in the process of S301 are input to the enlargement model 68. Then, the enlargement model 68 outputs an enlarged image based on the input. Here, for example, the enlargement model 68 is included in the enlargement unit 44.

[0171] Fig.15 It is schematically shown Fig.141 is a diagram of exemplary learning of the enlarged model 68 shown. For example, learning using a plurality of pieces of learning data is performed by the enlarged model 68. The learning data includes, for example, learning input data including a learning input image and learning input feature amount data, and an enlarged training image as training data to be compared with the enlarged image output from the enlarged model 68.

[0172] For example, by performing the above-mentioned reduction processing, the above-mentioned encoding processing, and the above-mentioned decoding processing on a high-resolution image (e.g., a 4K image), a learning input image as a low-resolution image (e.g., a 2K image) can be generated. Then, by processing similar to the above-mentioned processing, learning input feature quantity data as feature quantity data corresponding to the high-resolution image can be generated.

[0173] Then, learning data including the learning input data including the above-mentioned learning input image and the above-mentioned learning input feature amount data and the enlarged training image as the above-mentioned high-resolution image can be generated. Then, by supervised learning using a plurality of pieces of learning data generated in this manner based on a plurality of high-resolution images, learning of the enlargement model 68 can be performed. For example, an enlarged image output when the learning input data included in the learning data is input to the enlargement model 68 and an enlarged training image included in the learning data in question can be compared with each other. Then, based on the result of the comparison, the parameters of the enlargement model 68 can be updated by back propagation so that learning of the enlargement model 68 can be performed.

[0174] Note that the learning of the amplification model 68 is not limited to the above method. For example, the learning of the amplification model 68 can be performed by unsupervised learning or reinforcement learning.

[0175] When the process of S304 ends, the substitute image generation unit 48 starts to generate a substitute image of the next frame based on the basic image and the feature amount data corresponding to the basic image (S305). Here, for example, the enlargement unit 44 may store the basic image and the feature amount data corresponding to the basic image in the image storage unit 46 in association with each other. Here, the basic image is, for example, an image to be used for substitute image generation generated based on the image data received in the process of S301. Here, for example, the enlarged image generated in the process of S304 may be stored in the image storage unit 46 as the basic image. In addition, for example, the decoded image generated in the process of S302 may be stored in the image storage unit 46 as the basic image. In addition, the feature amount data stored in the image storage unit 46 may be, for example, feature amount data associated with the image data received in the process of S301. Then, the substitute image generation unit 48 may start to generate a substitute image based on the basic image stored in the image storage unit 46 and the feature amount data corresponding to the basic image.

[0176] Here, the substitute image generation process is started using the substitute image generation model 70. The substitute image generation model 64 is Fig.16 Here, the substitute image generation model 70 may adopt deep learning. The substitute image generation model 70 may include coupling topology information and coupling strength parameter information about the nodes included in the neural network.

[0177] In the process of S305, for example, the basic image in the frame stored in the image storage unit 46 and the feature amount data that has been sent to the frame and is associated with the basic image are input to the substitute image generation model 70. Then, the substitute image generation model 70 outputs the substitute image based on the input. Here, the substitute image generation model 70 is included in the substitute image generation unit 48, for example.

[0178] Fig.17 It is schematically shown Fig.16 1 is a diagram of exemplary learning of the substitute image generation model 70 shown. For example, learning using a plurality of pieces of learning data is performed by the substitute image generation model 70. The learning data includes, for example, learning input data including a learning input image and learning input feature amount data, and a substitute training image as training data to be compared with the substitute image output from the substitute image generation model 70.

[0179] For example, it is assumed that the basic image is a decoded image. In this case, the above-mentioned reduction processing, the above-mentioned encoding processing, and the above-mentioned decoding processing can be performed on a high-resolution image (for example, a 4K image) as a frame image included in a given motion image. Thereby, a learning input image as a low-resolution image (for example, a 2K image) can be generated. In addition, for example, it is assumed that the basic image is an enlarged image. In this case, the above-mentioned reduction processing, the above-mentioned encoding processing, the above-mentioned decoding processing, and the above-mentioned enlargement processing can be performed on a high-resolution image (for example, a 4K image) as a frame image included in a motion image. Thereby, a learning input image as a high-resolution image can be generated. Then, through processing similar to the above processing, learning input feature quantity data as feature quantity data corresponding to the high-resolution image can be generated.

[0180] Then, learning data can be generated, which includes the learning input data including the above-mentioned learning input image and the above-mentioned learning input feature quantity data, and a substitute training image, which is a frame image in a frame next to a frame image corresponding to the learning input image in the above-mentioned given motion image. Then, by using supervised learning of a plurality of learning data generated in this manner based on a plurality of motion images and a plurality of frame images, learning of the substitute image generation model 70 can be performed. For example, a substitute image output when the learning input data included in the learning data is input to the substitute image generation model 70 and a substitute training image included in the learning data in question can be compared with each other. Then, based on the result of the comparison, the parameters of the substitute image generation model 70 can be updated by back propagation so that learning of the substitute image generation model 70 can be performed.

[0181] Note that the learning of the substitute image generation model 70 is not limited to the above method. For example, the learning of the substitute image generation model 70 can be performed by unsupervised learning or reinforcement learning.

[0182] Then, the substitute image generation unit 48 stores the substitute image of the frame next to the one generated in the process started in S305 in the image storage unit 46 (S306). In addition, in the process of S306, the substitute image generation unit 48 also stores the feature amount data used for substitute image generation in the process of S305 in the image storage unit 46 as feature amount data of the next frame.

[0183] Then, the display control unit 50 causes the display unit 12f to display the enlarged image generated in the process of S304 (S307), and the process returns to the process of S301.

[0184] Assume that the reception failure of all the image data and feature data in the frame is confirmed in the process of S203 (S303: Y). In this case, the substitute image generation unit 48 checks whether the reception of part of the feature data in the frame is successful (S308). In the case where part of the feature data received by the terminal 12 is damaged, the reception failure of part of the feature data in the frame is confirmed in the process of S308. In addition, in the case where the decoded image generated in the process of S302 has a missing part, the reception failure of part of the feature data in the frame can be confirmed in the process of S308. In addition, here, in the case of the reception failure of part of the feature data, the data receiving unit 40 can notify the substitute image generation unit 48 of the reception failure of part of the feature data. Then, when the substitute image generation unit 48 receives the notification, the reception failure of part of the feature data in the frame can be confirmed in the process of S308.

[0185] Assume that it is confirmed in the process of S308 that the reception of the partial feature amount data in the frame has been successful (S308: Y). In this case, the substitute image generation unit 48 updates the feature amount data of the frame based on the partial feature amount data received in the frame (S309). In the process of S309, the feature amount data received in the frame is reflected in the feature amount data of the frame. Here, the feature amount data update process using the feature amount data update model 72 is performed, and the feature amount data update model 72 is Fig.18 Here, the feature quantity data update model 72 may adopt deep learning. The feature quantity data update model 72 may include coupling topology information and coupling strength parameter information about the nodes included in the neural network.

[0186] In the process of S309, for example, the first feature quantity data stored in the process of S306 of the immediately preceding frame and the partial feature quantity data as a part of the feature quantity data received in the frame are input to the feature quantity data update model 72. Then, the feature quantity data update model 72 outputs the second feature quantity data based on the input. In this way, the first feature quantity data stored in the process of S306 in the immediately preceding frame is updated to the second feature quantity data. Here, the feature quantity data update model 72 is included in the alternative image generation unit 48, for example.

[0187] Fig.19 It is schematically shown Fig.18 1 is a diagram of exemplary learning of the feature quantity data update model 72 shown. For example, learning using a plurality of pieces of learning data is performed by the feature quantity data update model 72. The learning data includes, for example, learning input data including learning input feature quantity data and learning input partial feature quantity data, and feature quantity training data as training data to be compared with the feature quantity data output from the feature quantity data update model 72.

[0188] For example, by processing similar to the above processing, learning input feature quantity data can be generated, which is feature quantity data corresponding to a high-resolution image (e.g., a 4K image) as a first frame image included in a given motion image. Then, by processing similar to the above processing, learning input partial feature quantity data can be generated, which is a part of feature quantity data corresponding to a second frame image that is a frame image in a frame next to the first frame image in the above given motion image.

[0189] Then, learning data can be generated, which includes learning input data including the above-mentioned learning input feature quantity data and the above-mentioned learning input partial feature quantity data and feature quantity training data as feature quantity data corresponding to the second frame image generated by processing similar to the above-mentioned processing. Then, by using supervised learning of multiple learning data generated in this way based on multiple motion images and multiple frame images, learning of the feature quantity data update model 72 can be performed. For example, feature quantity data output when the learning input data included in the learning data is input to the feature quantity data update model 72 and feature quantity training data included in the learning data in question can be compared with each other. Then, based on the result of the comparison, the parameters of the feature quantity data update model 72 can be updated by back propagation so that learning of the feature quantity data update model 72 can be performed.

[0190] Note that the learning of the feature amount data updating model 72 is not limited to the above method. For example, the learning of the feature amount data updating model 72 can be performed by unsupervised learning or reinforcement learning.

[0191] It is assumed that the processing of S309 ends, or that the failure to receive the partial feature data in the frame is confirmed in the processing of S308 (S308: Y). In these cases, the substitute image generation unit 48 checks whether the reception of the partial image data in the frame is successful (S310). In the case where the partial image data received by the terminal 12 is damaged, the failure to receive the partial image data in the frame is confirmed in the processing of S310. In addition, in the case where the decoded image generated in the processing of S302 has a missing portion, the failure to receive the partial image data in the frame can be confirmed in the processing of S310. In addition, here, in the case of the failure to receive the partial image data, the data receiving unit 40 can notify the substitute image generation unit 48 of the failure to receive the partial image data. Then, when the substitute image generation unit 48 receives the notification, the failure to receive the partial image data in the frame can be confirmed in the processing of S310.

[0192] Assume that it is confirmed in the process of S310 that the reception of the partial image data in the frame has been successful (S310: Y). In this case, the substitute image generation unit 48 updates the substitute image of the frame based on the partial image data received in the frame (S311). Here, for example, the partial image data received in the frame is reflected in the substitute image of the frame. Here, the substitute image update process using the substitute image update model 74 is performed, and the substitute image update model 74 is Fig. 20 Here, the alternative image update model 74 may adopt deep learning. The alternative image update model 74 may include coupling topology information and coupling strength parameter information about the nodes included in the neural network.

[0193] In the processing of S311, for example, the first substitute image stored in the processing of S306 in the immediately preceding frame, the partial image data and the feature quantity data which are part of the image data received in the frame in question are input to the substitute image update model 74. Here, in the case where the processing in S309 is performed, the updated feature quantity data of the frame is input to the substitute image update model 74. On the other hand, in the case where the processing in S309 is not performed, the feature quantity data of the frame stored in the immediately preceding frame is input to the substitute image update model 74. Then, the substitute image update model 74 outputs the second substitute image based on the input. Thereby, the first substitute image stored in the processing of S306 in the immediately preceding frame is updated to the second substitute image. Here, the substitute image update model 74 is included in the substitute image generation unit 48, for example.

[0194] Fig.21 It is schematically shown Fig. 20 1 is a diagram of exemplary learning of the substitute image update model 74 shown. For example, learning using a plurality of pieces of learning data is performed by the substitute image update model 74. The learning data includes, for example, learning input data including a learning input image, learning input partial image data, and learning input feature amount data, and a substitute training image as training data to be compared with the substitute image output from the substitute image update model 74.

[0195] For example, it is assumed that the basic image is a decoded image. In this case, the above-mentioned reduction processing, the above-mentioned encoding processing, and the above-mentioned decoding processing can be performed on a high-resolution image (for example, a 4K image) as a frame image included in a given motion image. Thus, a learning input image as a low-resolution image (for example, a 2K image) can be generated. In addition, for example, it is assumed that the basic image is an enlarged image. In this case, the above-mentioned reduction processing, the above-mentioned encoding processing, the above-mentioned decoding processing, and the above-mentioned enlargement processing can be performed on a high-resolution image (for example, a 4K image) as a frame image included in a motion image. Thus, a learning input image as a high-resolution image can be generated.

[0196] Then, learning input partial image data can be generated as a part of image data generated by performing the above-mentioned reduction processing and the above-mentioned encoding processing on the second frame image, which is a frame image in a frame next to the first frame image corresponding to the learning input image in the above-mentioned given motion image. Then, by processing similar to the above-mentioned processing, learning input feature quantity data can be generated as feature quantity data corresponding to the first frame image. Note that feature quantity data obtained by updating feature quantity data corresponding to the first frame image based on feature quantity data corresponding to the second frame image can be used as learning input feature quantity data.

[0197] Then, learning data can be generated, which includes the learning input data including the above-mentioned learning input image, the above-mentioned learning input partial image data and the above-mentioned learning input feature quantity data, and the alternative training image as the above-mentioned second frame image. Then, by using supervised learning of multiple learning data generated in this way based on multiple motion images and multiple frame images, learning of the alternative image update model 74 can be performed. For example, the alternative image output when the learning input data included in the learning data is input to the alternative image update model 74 and the alternative training image included in the learning data in question can be compared with each other. Then, based on the result of the comparison, the parameters of the alternative image update model 74 can be updated by back propagation so that learning of the alternative image update model 74 can be performed.

[0198] Note that the learning of the substitute image update model 74 is not limited to the above method. For example, the learning of the substitute image update model 74 can be performed by unsupervised learning or reinforcement learning.

[0199] When the process of S311 is completed or it is confirmed in the process of S310 that the reception of the partial image data in the frame has not been successful (S310: N), the substitute image generation unit 48 starts to generate the substitute image of the next frame (S312). Here, when the process of S311 is completed, the substitute image generation unit 48 can store the substitute image updated in the process of S311 as the basic image in the image storage unit 46. In addition, in the process of S310, when the reception of the partial image data in the frame has not been successful, the substitute image stored in the process of S306 in the immediately previous frame can be stored in the image storage unit 46 as the basic image. In addition, when the process of S309 is performed, the substitute image generation unit 48 can store the updated feature amount data in the image storage unit 46 in association with the basic image. In addition, when the process of S309 is not performed, the substitute image generation unit 48 can store the feature amount data of the frame in association with the basic image in the image storage unit 46. Then, the substitute image generation unit 48 may start generating a substitute image of the next frame based on the base image and the feature amount data associated with the base image.

[0200] Here, the substitute image generation process is started using the substitute image generation model 70. Fig.16 Here, the alternative image generation model 70 may employ deep learning.

[0201] Then, the substitute image generation unit 48 stores the substitute image of the frame next to the one generated in the process started in S312 in the image storage unit 46 (S313). Here, in the case where the process of S309 is executed, the substitute image generation unit 48 stores the updated feature amount data in association with the substitute image in the image storage unit 46 as feature amount data of the next frame. In addition, in the case where the process of S309 is not executed, the substitute image generation unit 48 stores the feature amount data of the frame in question in association with the substitute image in the image storage unit 46 as feature amount data of the next frame.

[0202] Then, the display control unit 50 causes the display unit 12f to display the substitute image generated in the frame (S314), and the process returns to the process of S301.

[0203] In this processing example, it is assumed that reception of all image data and feature amount data in the nth frame is confirmed to have been successful in the nth frame (S303: Y). In this case, an enlarged image generated based on the image data and feature amount data received in the nth frame is displayed (S307). Furthermore, in this case, in the nth frame, a substitute image of the (n+1)th frame is generated based on the enlarged image and feature amount data (S305 and S306).

[0204] Assume that in the nth frame, the reception of all the image data in the nth frame fails (S303: N) and it is confirmed that the reception of part of the image data in the Nth frame has been successful (S310: Y). In this case, in the nth frame, the replacement image of the nth frame generated in the (n-1)th frame is updated (S311). Then, the updated replacement image of the nth frame is displayed (S314). In addition, in this case, in the nth frame, based on the updated replacement image of the nth frame and the feature amount data of the nth frame, the replacement image of the (n+1)th frame is generated (S312 and S313).

[0205] Assume that in the nth frame, reception of all image data in the nth frame fails (S303: N) and it is confirmed that reception of part of the image data in the Nth frame has not been successful (S310: Y). In this case, in the nth frame, the replacement image of the nth frame generated in the (n-1)th frame is displayed (S314). Furthermore, in this case, in the nth frame, based on the replacement image of the nth frame generated in the (n-1)th frame and the feature amount data of the nth frame, a replacement image of the (n+1)th frame is generated (S312 and S313).

[0206] Furthermore, in the case where part of the feature amount data has been successfully received in the n-th frame ( S308 : Y), the feature amount data of the n-th frame generated in the (n−1)-th frame is updated ( S309 ).

[0207] In the case where the partial feature amount data includes ME information data, optical flow data, etc., a substitute image based on the prediction of the motion of the object in the image can be generated. At this time, other feature amounts indicated by the partial feature amount data are also used to improve the prediction accuracy. By using the partial feature amount data, not only the quality of the enlarged image can be improved, but also the reproducibility of the change of the object in the image in the time direction immediately before the previous frame in the substitute image can be improved.

[0208] Furthermore, in this processing example, before starting to generate a substitute image for the next frame and ending to store the substitute image for the next frame, display of an enlarged image or display of a substitute image for the frame in question may be performed.

[0209] Furthermore, in some cases, reception of image data fails, but reception of all feature amount data associated with the image data succeeds. For example, in this case, the substitute image generation unit 48 may generate a substitute image based on feature amount data associated with the first image data and the second image data.

[0210] Furthermore, in the case where the transmission control unit 30 performs control to prevent the transmission of the second image data, a notification indicating that the image data is not to be transmitted may be transmitted to the terminal 12. In this case, the data transmission unit 32 transmits the notification indicating that the image data is not to be transmitted to the terminal 12. Furthermore, in this case, the amplification unit 44 may check whether all the image data in the communication frame are successfully received in the process of S203 or S303 based on the notification indicating that the image data is not to be transmitted received by the data reception unit 40.

[0211] In addition, the substitute image update model 74 may output a determination value indicating whether an erroneous image different from the image that should be displayed is output as the second substitute image. For example, in the case where an erroneous image different from the image that should be displayed is output as the second substitute image, a determination value of "1" may be output, and in the case where the image that should be displayed is output as the second substitute image, a determination value of "0" may be output. In this case, the learning of the substitute image update model 74 may be performed using training data including a given learning determination value based on learning input data, which is compared with a determination value output from the substitute image update model 74 when the learning input data is received. In this case, in the process of S311, the substitute image update model 74 may also output a determination value.

[0212] Then, based on the determination value, in the process of S314, the display control unit 50 can control whether to make the display unit 12f display the substitute image generated in the frame. For example, in the case where the substitute image update model 74 outputs the determination value "1", the display control unit 50 can perform control to prevent the display of the substitute image. In addition, in the case where the substitute image update model 74 outputs the determination value "0", the display control unit 50 can perform control to display the substitute image.

[0213] In addition, in a similar manner, the substitute image generation model 70 may output a determination value indicating whether an erroneous image different from the image that should be displayed is output as a substitute image. For example, in the case where an erroneous image different from the image that should be displayed is output as a substitute image, a determination value "1" may be output, and in the case where the image that should be displayed is output as a substitute image, a determination value "0" may be output. In this case, the learning of the substitute image generation model 70 may be performed using training data including a given learning determination value based on learning input data, which is compared with a determination value output from the substitute image generation model 70 when the learning input data is received. In this case, in the processing of S305 or S312, the substitute image generation model 70 may also output a determination value.

[0214] Then, based on the determination value, in the process of S314 of the next frame, the display control unit 50 can control whether to make the display unit 12f display the substitute image generated in the frame in question. For example, in the case where the substitute image generation model 70 outputs the determination value "1", the display control unit 50 can perform control to prevent the display of the substitute image. In addition, in the case where the substitute image generation model 70 outputs the determination value "0", the display control unit 50 can perform control to display the substitute image.

[0215] Here, for example, in the case where any one of the substitute image generation model 70 and the substitute image update model 74 outputs a determination value of “1”, the display control unit 50 may perform control to prevent display of the substitute image.

[0216] As described above, in this embodiment, the cloud server 10 performs image data transmission control based on the possibility that the terminal 12 successfully estimates the image to be generated for display. Therefore, according to this embodiment, the image data communication volume can be reduced while reducing the risk of displaying an erroneous moving image that is different from the image that should be displayed.

[0217] Note that the present invention is not limited to the above-described embodiments.

[0218] For example, based on the time sequence of feature amount data including the above-mentioned first feature amount data and the above-mentioned second feature amount data, the possibility of successfully estimating the second image to be displayed based on the first image data by the substitute image generating unit 48 can be determined.

[0219] Here, for example, the success estimation possibility assessment model 60 may be implemented as a recurrent neural network (RNN).

[0220] In this case, in the process of S106, the success estimation possibility evaluation model 60 may receive as input the time sequence of feature amount data including feature amount data in the frame and stored in the feature amount data storage unit 28. Then, the success estimation possibility evaluation model 60 may output success estimation possibility data based on the input.

[0221] Here, in the learning of the success estimation possibility evaluation model 60, a time sequence of learning input feature quantity data can be generated, and the time sequence of learning input feature quantity data is the time sequence of feature quantity data corresponding to a series of high-resolution images (for example, 4K images) (the time sequence of high-resolution images).

[0222] Then, the user can check with his / her eyes the comparison between a series of high-resolution images or the comparison between a series of high-resolution images and the substitute image generated by the substitute image generating unit 48, thereby determining the value of the successful estimation possibility training data. In addition, for example, the user can decide the value of the successful estimation possibility training data according to an empirical rule.

[0223] Then, learning data including time-sequential learning input data including the above-mentioned learning input feature quantity data and the above-mentioned success estimation possibility training data can be generated.

[0224] Then, by using supervised learning of a plurality of pieces of learning data generated in this manner based on a plurality of motion images and a plurality of frame images, learning of the success estimation possibility evaluation model 60 can be performed. For example, the success estimation possibility data output when the learning input data included in the learning data is input to the success estimation possibility evaluation model 60 and the success estimation possibility training data included in the learning data in question can be compared with each other. Then, based on the result of the comparison, the parameters of the success estimation possibility evaluation model 60 can be updated by back propagation so that learning of the success estimation possibility evaluation model 60 can be performed.

[0225] In addition, for example, the enlargement model 62 may be implemented as an RNN. In this case, the enlargement unit 44 of the terminal 12 may store the decoded image generated in the process of S202.

[0226] In this case, in the process of S204, the time sequence of the decoded images including the latest decoded image may be input to the enlargement model 62. Then, the enlargement model 62 may output the enlarged image based on the input.

[0227] Here, in the learning of the magnification model 62, by performing the above-mentioned reduction processing, encoding processing and decoding processing on a series of high-resolution images (e.g., 4K images) (time sequence of high-resolution images), a time sequence of learning input images as low-resolution images (e.g., 2K images) can be generated.

[0228] Then, learning data including the learning input data including the time sequence of the above-mentioned learning input images and the enlarged training image which is the last image in the time sequence of the above-mentioned high-resolution image can be generated. Then, by supervised learning using a plurality of pieces of learning data generated in this way based on the time sequence of a plurality of high-resolution images, learning of the enlargement model 62 can be performed. For example, an enlarged image output when the learning input data included in the learning data is input to the enlargement model 62 and an enlarged training image included in the learning data in question can be compared with each other. Then, based on the result of the comparison, the parameters of the enlargement model 62 can be updated by back propagation so that learning of the enlargement model 62 can be performed.

[0229] In addition, for example, the substitute image generation model 64 may be implemented as an RNN. In this case, the substitute image generation unit 48 of the terminal 12 may store the basic image.

[0230] In this case, in the process of S205 or S210, the time sequence of the base images including the latest base image may be input to the substitute image generation model 64. Then, the substitute image generation model 64 may output the substitute image based on the input.

[0231] For example, it is assumed that the basic image is a decoded image. In this case, in the learning of the alternative image generation model 64, by performing the above-mentioned reduction processing, encoding processing, and decoding processing on a series of high-resolution images (e.g., 4K images) included in a given moving image (time sequence of high-resolution images), a time sequence of learning input images that are low-resolution images (e.g., 2K images) can be generated. In addition, for example, it is assumed that the basic image is an enlarged image.

[0232] In this case, the above-mentioned reduction processing, encoding processing, decoding processing, and enlargement processing may be performed on a series of high-resolution images (e.g., 4K images) included in a given moving image in the learning of the alternative image generation model 64. Thus, a time sequence of learning input images as high-resolution images may be generated.

[0233] Then, learning data can be generated, which includes learning input data containing the time sequence of the above-mentioned learning input images and alternative training images that are frame images in frames next to the above-mentioned series of high-resolution images in the above-mentioned given motion image. Then, by using supervised learning of multiple learning data generated in this way based on multiple motion images and multiple frame images, learning of the alternative image generation model 64 can be performed. For example, the alternative image output when the learning input data included in the learning data is input to the alternative image generation model 64 and the alternative training image included in the learning data in question can be compared with each other. Then, based on the result of the comparison, the parameters of the alternative image generation model 64 can be updated by back propagation so that learning of the alternative image generation model 64 can be performed.

[0234] In addition, for example, the substitute image update model 66 may be implemented as an RNN. In this case, the substitute image generation unit 48 of the terminal 12 may store the substitute image.

[0235] In this case, the substitute image update model 66 may receive as input a temporal sequence of substitute images including a substitute image in the immediately preceding frame and partial image data that is a portion of the image data received in the frame in question. The substitute image update model 66 may then output a second substitute image based on the input.

[0236] For example, it is assumed that the basic image is a decoded image. In this case, in the learning of the alternative image update model 66, by performing the above-mentioned reduction processing, encoding processing, and decoding processing on a series of high-resolution images (e.g., 4K images) included in a given motion image (the time sequence of high-resolution images), a time sequence of learning input images as low-resolution images (e.g., 2K images) can be generated. In addition, for example, it is assumed that the basic image is an enlarged image. In this case, in the learning of the alternative image update model 66, the above-mentioned reduction processing, encoding processing, decoding processing, and enlargement processing can be performed on a series of high-resolution images (e.g., 4K images) included in a given motion image. Thereby, a time sequence of learning input images as high-resolution images can be generated.

[0237] Then, learning input partial image data may be generated as a part of image data generated by performing the above-mentioned reduction processing and encoding processing on frame images in frames beside the above-mentioned series of high-resolution images in the above-mentioned given moving image.

[0238] Then, learning data can be generated, which includes learning input data containing the time sequence of the above-mentioned learning input image and the learning input partial image data and a substitute training image that is a frame image in a frame next to a series of high-resolution images in the above-mentioned given motion image. Then, by using supervised learning of a plurality of learning data generated in this manner based on a plurality of motion images and a plurality of frame images, learning of the substitute image update model 66 can be performed. For example, a substitute image output when the learning input data included in the learning data is input to the substitute image update model 66 and a substitute training image included in the learning data in question can be compared with each other. Then, based on the result of the comparison, the parameters of the substitute image update model 66 can be updated by back propagation so that learning of the substitute image update model 66 can be performed.

[0239] Furthermore, for example, the enlargement model 68 may be implemented as an RNN. In this case, the enlargement unit 44 of the terminal 12 may store the decoded image generated in the process of S302 and the feature amount data received in the process of S301.

[0240] In this case, in the process of S304, the enlargement model 68 may receive as input the time sequence of decoded images including the latest decoded image and the time sequence of feature quantity data including the latest feature quantity data. Then, the enlargement model 68 may output an enlarged image based on the input.

[0241] Here, in the learning of the enlargement model 68, by performing the above-mentioned reduction processing, encoding processing, and decoding processing on a series of high-resolution images (for example, 4K images) (time sequence of high-resolution images), a time sequence of learning input images as low-resolution images (for example, 2K images) can be generated. Then, by processing similar to the above-mentioned processing, a time sequence of learning input feature amount data as a time sequence of feature amount data corresponding to a series of high-resolution images can be generated.

[0242] Then, learning data can be generated, which includes learning input data including the time sequence of the above-mentioned learning input images and the time sequence of the learning input feature amount data and an enlarged training image which is the last image in the time sequence of the above-mentioned high-resolution image. Then, by supervised learning using a plurality of learning data generated in this way based on the time sequence of a plurality of high-resolution images, learning of the enlargement model 68 can be performed. For example, an enlarged image output when the learning input data included in the learning data is input to the enlargement model 68 and an enlarged training image included in the learning data in question can be compared with each other. Then, based on the result of the comparison, the parameters of the enlargement model 68 can be updated by back propagation so that learning of the enlargement model 68 can be performed.

[0243] Furthermore, for example, the substitute image generation model 70 may be implemented as an RNN. In this case, the substitute image generation unit 48 of the terminal 12 may store a basic image and feature amount data associated with the basic image.

[0244] In this case, the substitute image generation model 70 may receive as input a temporal sequence of base images including the latest base image and a temporal sequence of feature quantity data including the latest feature quantity data. The substitute image generation model 70 may then output a substitute image based on the input.

[0245] For example, it is assumed that the basic image is a decoded image. In this case, in the learning of the alternative image generation model 64, by performing the above-mentioned reduction processing, encoding processing, and decoding processing on a series of high-resolution images (e.g., 4K images) included in a given motion image (the time sequence of high-resolution images), a time sequence of learning input images as low-resolution images (e.g., 2K images) can be generated. In addition, for example, it is assumed that the basic image is an enlarged image. In this case, in the learning of the alternative image generation model 70, the above-mentioned reduction processing, encoding processing, decoding processing, and enlargement processing can be performed on a series of high-resolution images (e.g., 4K images) included in a given motion image. Thus, a time sequence of learning input images as high-resolution images can be generated. Then, through processing similar to the above-mentioned processing, a time sequence of learning input feature quantity data corresponding to the time sequence of high-resolution images can be generated.

[0246] Then, learning data can be generated, which includes learning input data including the time sequence of the above-mentioned learning input images and the time sequence of the learning input feature quantity data and a substitute training image that is a frame image in a frame next to a series of high-resolution images. Then, by using supervised learning of multiple learning data generated in this way based on multiple motion images and multiple frame images, learning of the substitute image generation model 70 can be performed. For example, a substitute image output when the learning input data included in the learning data is input to the substitute image generation model 70 and a substitute training image included in the learning data in question can be compared with each other. Then, based on the result of the comparison, the parameters of the substitute image generation model 70 can be updated by back propagation so that learning of the substitute image generation model 70 can be performed.

[0247] Furthermore, for example, the feature amount data update model 72 may be implemented as an RNN. In this case, the substitute image generation unit 48 of the terminal 12 may store the feature amount data.

[0248] In this case, the feature quantity data updating model 72 can receive the time sequence of feature quantity data as input, and the feature quantity data includes the feature quantity data in the immediately previous frame and the partial feature quantity data as a part of the feature quantity data received in the frame in question. Then, the feature quantity data updating model 72 can output the second feature quantity data based on the input.

[0249] In the learning of the feature quantity data update model 72, through processing similar to the above processing, the time sequence of the learning input feature quantity data can be generated, which is the time sequence of the feature quantity data corresponding to a series of high-resolution images (for example, 4K images) included in a given motion image. Then, through processing similar to the above processing, the learning input partial feature quantity data can be generated, which is a part of the feature quantity data corresponding to the second frame image, which is a frame image in the frame next to the series of high-resolution images.

[0250] Then, learning data can be generated, which includes learning input data containing the time sequence of the above-mentioned learning input feature quantity data and the learning input partial feature quantity data and feature quantity training data as feature quantity data corresponding to the above-mentioned second frame image. Then, by using supervised learning of multiple learning data generated in this way based on multiple motion images and multiple frame images, learning of the feature quantity data update model 72 can be performed. For example, the feature quantity data output when the learning input data included in the learning data is input to the feature quantity data update model 72 and the feature quantity training data included in the learning data in question can be compared with each other. Then, based on the result of the comparison, the parameters of the feature quantity data update model 72 can be updated by back propagation so that learning of the feature quantity data update model 72 can be performed.

[0251] Furthermore, for example, the substitute image update model 74 may be implemented as an RNN. In this case, the substitute image generation unit 48 of the terminal 12 may store the substitute image and the feature amount data.

[0252] In this case, the substitute image update model 74 may receive as input a temporal sequence of substitute images including a substitute image in the immediately preceding frame, a temporal sequence of feature quantity data corresponding to the temporal sequence of substitute images including a substitute image in the immediately preceding frame, and partial image data that is a portion of the image data received in the frame in question. The substitute image update model 74 may then output a second substitute image based on the input.

[0253] For example, it is assumed that the basic image is a decoded image. In this case, in the learning of the alternative image update model 74, by performing the above-mentioned reduction processing, encoding processing, and decoding processing on a series of high-resolution images (e.g., 4K images) included in a given motion image (the time sequence of the high-resolution images), a time sequence of learning input images as low-resolution images (e.g., 2K images) can be generated. In addition, for example, it is assumed that the basic image is an enlarged image. In this case, in the learning of the alternative image update model 74, the above-mentioned reduction processing, encoding processing, decoding processing, and enlargement processing can be performed on a series of high-resolution images (e.g., 4K images) included in a given motion image. Thereby, a time sequence of learning input images as high-resolution images can be generated. In addition, through processing similar to the above-mentioned processing, a time sequence of learning input feature quantity data corresponding to the time sequence of high-resolution images can be generated.

[0254] Then, learning input partial image data may be generated as a part of image data generated by performing the above-mentioned reduction processing and encoding processing on frame images in frames beside the above-mentioned series of high-resolution images in the above-mentioned given moving image.

[0255] Then, learning data can be generated, which includes learning input data including the time sequence of the above-mentioned learning input images, the time sequence of the above-mentioned learning input feature data, and the learning input partial image data, and a substitute training image of a frame image in a frame next to the above-mentioned series of high-resolution images. Then, by using supervised learning of a plurality of learning data generated in this manner based on a plurality of motion images and a plurality of frame images, learning of the substitute image update model 74 can be performed. For example, a substitute image output when the learning input data included in the learning data is input to the substitute image update model 74 and a substitute training image included in the learning data in question can be compared with each other. Then, based on the result of the comparison, the parameters of the substitute image update model 74 can be updated by back propagation so that learning of the substitute image update model 74 can be performed.

[0256] As described above, in this embodiment, by using a machine learning model configured to receive the time sequence of data as input, the machine learning model outputs data that takes into account the prediction of frame changes based on the time sequence of past frame data. Therefore, the accuracy of the output result of the machine learning model can be further improved.

[0257] In addition, in the present embodiment, for example, the success estimation possibility assessment model 60 can be implemented as a generative adversarial network (GAN). In addition, for example, the amplification model 62 can be implemented as a GAN. In addition, for example, the substitute image generation model 64 can be implemented as a GAN. In addition, for example, the substitute image update model 66 can be implemented as a GAN. In addition, for example, the amplification model 68 can be implemented as a GAN. In addition, for example, the substitute image generation model 70 can be implemented as a GAN. In addition, for example, the feature quantity data update model 72 can be implemented as a GAN. In addition, for example, the substitute image update model 74 can be implemented as a GAN.

[0258] In addition, for example, the enlargement unit 44 of the terminal 12 may decide whether to perform the super-resolution process based on the data indicating whether to perform the super-resolution process contained in the feature amount data. Then, in the case where the enlargement unit 44 decides not to perform the super-resolution process, the enlargement unit 44 may perform control to prevent the enlargement process from being performed. In addition, in the case where the enlargement unit 44 decides not to perform the super-resolution process, the enlargement unit 44 may control the substitute image generation unit 48 not to perform the substitute image generation process. In this case, the display control unit 50 may cause the display unit 12f to display the decoded image or the image displayed in the immediately previous frame.

[0259] In addition, for example, the alternative image generation unit 48 of the terminal 12 may decide whether to generate an alternative image based on the basic image. Here, a trained alternative image generation model 64 may be used to decide whether to generate an alternative image, and the alternative image generation model 64 is configured to output information indicating whether to generate an alternative image, which information has been learned by a method similar to the above method. In addition, the alternative image generation unit 48 may decide whether to generate an alternative image based on a basic image and feature quantity data associated with the basic image. Here, a trained alternative image generation model 70 may be used to decide whether to generate an alternative image, and the alternative image generation model 64 is configured to output information indicating whether to generate an alternative image, which information has been learned by a method similar to the above method. Then, when it is decided not to generate an alternative image, the display control unit 50 may cause the display unit 12f to display the decoded image or the image displayed in the immediately previous frame.

[0260] In addition, for example, the game image to be reduced can be divided into a plurality of partial images. Here, for example, the game image can be divided into a plurality of partial images, each of which corresponds to a rectangular area of, for example, 8×8 pixels or 32×32 pixels in size. In addition, for example, the game image can be divided horizontally so that the game image can be divided into a plurality of strip-shaped (strip-shaped) partial images including a plurality of rows (for example, 32 rows). In this case, the partial images correspond to the original image described above. On the original image including the partial images, Figure 3 S102 to S108 shown and Figure 6 Alternatively, on the original image including the partial image, a series of processing stages from S201 to S212 may be performed. Figure 3 S102 to S108 shown and Fig.13 A series of processing stages from S301 to S314 are shown.

[0261] In addition, regardless of whether the played image corresponds to the original image or the partial image corresponds to the original image, the processing in S204, S206, S209, S211, S304, S306, S311 and S313 can be performed with a smaller granularity than the original image. For example, the processing in S204, S206, S304 and S306 can be performed for each 8×8 pixel image or each 32×32 pixel image. Here, when the decoded image has a resolution of 1920×1088 pixels and the above processing is performed on each 32×32 pixel image, 60×24 processing is performed. In this case, the learning input image used for learning the machine learning model can be an image having the same shape and size as the image input to the machine learning model in the process.

[0262] In addition, the processing in S304, S309, S311 and S313 can be performed at a smaller granularity than the feature quantity data. In this case, the learning input part feature quantity data or the learning input feature quantity data used for learning the machine learning model can be data having the same size as the data input to the machine learning model in the processing.

[0263] Furthermore, the enlargement unit 44 may perform super-resolution processing or extrapolation to generate an enlarged image. Furthermore, the substitute image generation unit 48 may perform extrapolation to generate a substitute image.

[0264] In addition, the magnification model 62, the substitute image generation model 64, the substitute image update model 66, the magnification model 68, the substitute image generation model 70, the feature data update model 72 and the substitute image update model 74 can be separate machine learning models, or can be partially or fully integrated into an integrated machine learning model.

[0265] In addition, the scope of the present invention is not limited to cloud gaming services. The present invention is generally applicable to transmission / reception of image data.

[0266] Here, for example, image data or image data and feature data may be repeatedly sent from the cloud server 10 to a transmission server configured to transmit moving images depicting situations in a video game. The transmission server may then store image data or image data and feature data corresponding to a series of play images and having sequence information. Later, in response to a request from a terminal used by a user viewing moving images depicting situations in a video game, the transmission server may then send the image data or image data and feature data stored in the transmission server to the terminal according to a sequence associated with the data. Then, in the terminal, the processing associated with the image data may be performed. Figure 1 The processing in the terminal 12 shown is similar to the processing.

[0267] Furthermore, in the video game streaming, the above-mentioned transmission server can transmit the image data or the image data and the feature amount data in real time to a terminal used by a user who is watching a moving image depicting a situation in the video game when receiving the image data or the image data and the feature amount data. Then, in the terminal, the same processing as that in the video game can be performed. Figure 1 The processing in the terminal 12 shown is similar to the processing.

[0268] Here, the cloud server 10 can also be used as the above-mentioned delivery server configured to send image data or image data and feature data to the terminal. In addition, in the terminal 12, it is possible to perform the same Figure 1 The processing in the cloud server 10 shown is similar to the processing, and image data or image data and feature amount data corresponding to a series of playing images can be repeatedly sent from the terminal 12 to the transmission server.

[0269] Furthermore, the specific character strings and numerical values ​​described above and those in the drawings are illustrative, and the present invention is not limited to these character strings and numerical values.

Claims

1. An image sending / receiving system, comprising: Image sending device; as well as an image receiving device, The image sending device comprises: an image generating unit configured to generate a first image and generate a second image after generating the first image, an image data sending unit configured to send first image data generated based on the first image, and a transmission control unit configured to control whether to transmit second image data generated based on the second image, The image receiving device comprises: an image data receiving unit, configured to receive the first image data, a to-be-displayed image generating unit, configured to generate a first to-be-displayed image based on the first image data, and to generate a second to-be-displayed image based on the second image data, a substitute image generating unit configured to generate a substitute image by estimating the second to-be-displayed image based on the first image data, and a display control unit configured to, after displaying the first to-be-displayed image, display any one of the second to-be-displayed image and the substitute image generated based on the second image data, depending on whether the reception of the second image data is successful, The sending control unit controls whether to send the second image data based on the possibility that the alternative image generation unit successfully estimates the second image to be displayed based on the first image data, and the possibility is determined based on the feature value of the first image and the feature value of the second image.

2. An image sending / receiving system according to claim 1, wherein the sending control unit controls whether to send the second image data based on the possibility that the alternative image generation unit successfully estimates the second image to be displayed based on the first image data, and the possibility is determined based on the feature value of the first image and the feature value of the second image.

3. An image sending / receiving system according to claim 2, wherein the sending control unit controls whether to send the second image data based on the possibility that the alternative image generation unit successfully estimates the second image to be displayed based on the first image data, and the possibility is determined based on the time sequence of feature quantities including the feature quantity of the first image and the feature quantity of the second image.

4. An image sending / receiving system according to claim 1 or 2, wherein the sending control unit controls whether to send the second image data indicating a reduced image based on the possibility that the alternative image generation unit successfully estimates the second image to be displayed based on the first image data, and the possibility is determined based on a feature quantity including parameters for encoding the reduced image obtained by reducing the second image.

5. The image transmission / reception system according to claim 1 or 2, in, the image data transmission unit transmits the first image data indicating a first reduced image obtained by reducing the first image, and The sending control unit controls whether to send the second image data indicating the second reduced image based on the possibility that the alternative image generation unit successfully estimates the second image to be displayed based on the first image data, and the possibility is determined based on a feature quantity including parameters for encoding the first reduced image and a feature quantity including parameters for encoding the second reduced image obtained by reducing the second image.

6. An image sending / receiving system according to claim 2, wherein the feature quantity indicates at least any one of a plurality of information: the plurality of information indicates feature points of an image, edge strength of the image, depth of pixels included in the image, texture type of the image, optical flow in the image, and moving direction and speed of a rectangular area in the image.

7. The image transmission / reception system according to claim 2, wherein the transmission control unit performs control of transmitting the feature amount data indicating the feature amount to the image reception device in priority to transmitting the image data indicating the image. 8 . The image transmission / reception system according to claim 1 , wherein the transmission control unit controls whether to transmit the second image data based on whether the second image includes a key frame. 9 . The image transmission / reception system according to claim 1 , wherein the transmission control unit controls whether to transmit the second image data based on whether a scene in the second image is switched from a scene in the first image. 10 . The image transmission / reception system according to claim 1 , wherein the transmission control unit controls whether to transmit the second image data based on whether the second image includes an image of an object that does not appear in the first image.

11. The image transmission / reception system according to claim 1, wherein: The substitute image generation unit starts generating the substitute image based on the first image data before checking whether the reception of the second image data is successful.

12. The image transmission / reception system according to claim 1, wherein: In a case where a transmission speed of the image transmission device is higher than a predetermined speed, the transmission control unit performs control to transmit the second image data regardless of the possibility that the substitute image generation unit successfully estimates the second image to be displayed based on the first image data.

13. An image sending device, comprising: an image generating unit configured to generate a first image and generate a second image after generating the first image; an image data transmitting unit configured to transmit first image data generated based on the first image to an image receiving device, The image receiving device comprises: an image data receiving unit, configured to receive the first image data, a to-be-displayed image generating unit configured to generate a first to-be-displayed image based on the first image data, and to generate a second to-be-displayed image based on second image data generated according to the second image, a substitute image generating unit configured to generate a substitute image by estimating the second to-be-displayed image based on the first image data, and a display control unit configured to, after displaying the first to-be-displayed image, display any one of the second to-be-displayed image and the substitute image generated based on the second image data, depending on whether the reception of the second image data is successful, A sending control unit is configured to control whether to send the second image data generated based on the second image to the image receiving device based on the possibility that the alternative image generating unit successfully estimates the second image to be displayed based on the first image data, wherein the possibility is determined based on a feature value of the first image and a feature value of the second image.

14. An image receiving device, comprising: an image data receiving unit configured to receive first image data from an image sending device; a to-be-displayed image generating unit, configured to generate a first to-be-displayed image based on the first image data, and to generate a second to-be-displayed image based on the second image data; a substitute image generating unit configured to generate a substitute image by estimating the second image to be displayed based on the first image data; as well as A display control unit is configured to, after displaying the first image to be displayed, control whether to send the second image data by the image sending device, depending on whether the reception of the second image data is successful and the possibility that the alternative image generating unit successfully estimates the second image to be displayed based on the first image data, and display any one of the second image to be displayed and the alternative image generated based on the second image data, wherein the possibility is determined based on the feature quantity of the first image and the feature quantity of the second image.

15. An image sending / receiving method, comprising: A first image generating step of generating a first image by an image sending device; A second image generating step of generating a second image after generating the first image by the image sending device; An image data sending step of sending, by the image sending device, first image data generated based on the first image; A sending control step of controlling, by the image sending device, whether to send second image data generated based on the second image; An image data receiving step of receiving the first image data by an image receiving device; An image generating step of generating a first image to be displayed based on the first image data and generating a second image to be displayed based on the second image data by the image receiving device; a substitute image generating step of generating a substitute image by estimating the second image to be displayed based on the first image data by the image receiving device; as well as a display control step of displaying, by the image receiving device, after displaying the first image to be displayed, any one of the second image to be displayed and the substitute image generated based on the second image data, depending on whether reception of the second image data is successful, Wherein, in the sending control step, the image sending device controls whether to send the second image data based on the possibility of successfully estimating the second image to be displayed based on the first image data in the alternative image generation step, and the possibility is determined based on the feature value of the first image and the feature value of the second image.

16. A non-transitory computer-readable storage medium having a program stored thereon, which, when executed by a computer, causes the computer to execute: A first image generating process, generating a first image; A second image generating process, generating a second image after generating the first image; The image data sending process includes sending the first image data generated based on the first image to an image receiving device, The image receiving device comprises: an image data receiving unit, configured to receive the first image data, a to-be-displayed image generating unit configured to generate a first to-be-displayed image based on the first image data, and to generate a second to-be-displayed image based on second image data generated according to the second image, a substitute image generating unit configured to generate a substitute image by estimating the second to-be-displayed image based on the first image data, and a display control unit configured to, after displaying the first to-be-displayed image, display any one of the second to-be-displayed image and the substitute image generated based on the second image data, depending on whether the reception of the second image data is successful, A sending control process, which controls whether to send the second image data generated based on the second image to the image receiving device based on the possibility that the alternative image generation unit successfully estimates the second image to be displayed based on the first image data, and the possibility is determined based on the feature value of the first image and the feature value of the second image.

17. A non-transitory computer-readable storage medium having a program stored thereon, which, when executed by a computer, causes the computer to execute: An image data receiving process, receiving first image data from an image sending device; a process of generating an image to be displayed, generating a first image to be displayed based on the first image data, and generating a second image to be displayed based on the second image data; a substitute image generating process, generating a substitute image by estimating the second to-be-displayed image based on the first image data; as well as A display control process, after displaying the first image to be displayed, depends on whether the reception of the second image data is successful and based on the possibility of successfully estimating the second image to be displayed based on the first image data in the replacement image generation process, and controls whether to send the second image data by the image sending device, and displays any one of the second image to be displayed and the replacement image generated based on the second image data, wherein the possibility is determined based on the feature value of the first image and the feature value of the second image.

Images