Image processing system, image processing method, and program

Abstract

An image processing system (1) including at least one processor, wherein the at least one processor acquires an input frame (22) having an input pixel number greater than or equal to an initial pixel number based on a frame (20) to be processed that indicates, from a predetermined viewpoint (C), a virtual space (VS) in which one or more game objects (O) represented by three-dimensional data are arranged, and that has a predetermined initial pixel number, acquires virtual space information (27) related to a virtual space (VS) that can be used to determine the pixel value of each pixel of the input frame (22), and acquires an estimation frame (24) having a greater number of estimation pixels than the number of input pixels, on the basis of the input frame (22), the virtual space information (27), and a machine learning model (200).

Description

Image processing system, image processing method and program

[0001] The present invention relates to an image processing system, an image processing method, and a program.

[0002] Conventionally, a technique (super-resolution) for estimating a high-quality image based on a low-quality image using a machine learning model has been known (see Non-Patent Document 1 below).

[0003] Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang. Learning a Deep Convolutional Network for Image Super-Resolution, in Proceedings of European Conference on Computer Vision (ECCV), 2014

[0004] The inventors of the present application are considering applying the above-described super-resolution to virtual images. A virtual image is an image showing a virtual space in which one or more objects represented by three-dimensional data are arranged, from a predetermined viewpoint. Super-resolution of a virtual image can be considered to be the task of estimating an image that more precisely and accurately reproduces the appearance of the virtual space, based on the original virtual image.

[0005] A virtual image is generated by rendering three-dimensional data. The rendering is performed based on information about the virtual space that can be used to determine the pixel value of each pixel in the virtual image (hereinafter referred to as "virtual space information"). The virtual space information includes, for example, information about the viewpoint from which the virtual space is viewed, information about the depth of an object, information about the movement of an object, information about the color and texture of an object, and information about the intensity, color, and illumination direction of a light source.

[0006] However, since the virtual image obtained as a result of rendering may not contain sufficient information about the virtual space, there is a limit to the accuracy of super-resolution based solely on the virtual image. That is, although the virtual space information can be used to determine the pixel value of each pixel of the virtual image, the virtual space information itself does not remain in the virtual image. For example, if there are originally C pieces of information about the virtual space, in the process of determining the pixel value (RGB value) of each pixel of the virtual image, the C pieces of information are reduced to three pieces of information (RGB).

[0007] The present invention aims to provide an image processing system, an image processing method, and a program that can effectively utilize virtual space information to estimate a high-quality estimated image with high accuracy based on a low-quality input image, which is a virtual image.

[0008] The image processing system of the present invention is an image processing system including at least one processor, wherein the at least one processor acquires an input image having a number of input pixels equal to or greater than an initial number of pixels based on a processing target image that shows, from a predetermined viewpoint, a virtual space in which one or more objects represented by three-dimensional data are arranged and that has a predetermined initial number of pixels; acquires virtual space information that is information about the virtual space that can be used to determine pixel values ​​of each pixel of the input image; and acquires an estimated image having an estimated number of pixels greater than the input number of pixels based on the input image, the virtual space information, and a machine learning model; the machine learning model has been trained using multiple training data, each of which includes a training input image, training virtual space information, and a training estimated image; the training input image is an image having the input number of pixels based on a training processing target image that shows, from a predetermined viewpoint, a training virtual space in which one or more training objects represented by training three-dimensional data are arranged and that has the initial number of pixels; the training virtual space information is information about the training virtual space that can be used to determine pixel values ​​of each pixel of the training input image; and the training estimated image is an image having the estimated number of pixels.

[0009] 1 is a diagram illustrating an example of a hardware configuration of an image processing system; FIG. 2 is a diagram illustrating an overview of the image processing system; FIG. 3 is a diagram illustrating a schematic diagram of processing in the image processing system; FIG. 4 is a functional block diagram illustrating an example of functions realized in the image processing system; FIG. 5 is a diagram illustrating processing in a rendering unit; FIG. 6 is a diagram illustrating processing in an input frame acquisition unit; FIG. 7 is a flow diagram illustrating an example of the flow of processing executed in the image processing system; FIG. 8 is a flow diagram illustrating an example of the flow of processing executed in the image processing system.

[0010] Hereinafter, an example of an embodiment of an image processing system according to the present invention will be described with reference to the drawings.

[0011] 1 is a diagram showing an example of the hardware configuration of an image processing system 1. The image processing system 1 is a computer such as a game console. As shown in FIG. 1, the image processing system 1 includes a control unit 10, a storage unit 12, a communication unit 14, an operation unit 16, a display unit 18, and an audio output unit 19.

[0012] The control unit 10 includes a program control device such as a CPU that operates according to a program installed in the image processing system 1. The control unit 10 also includes a GPU (Graphics Processing Unit) that draws an image in a frame buffer based on graphics commands and data supplied from the CPU.

[0013] The storage unit 12 includes, for example, a main storage device such as a ROM or a RAM, and an auxiliary storage device such as a HDD or an SSD. The storage unit 12 stores programs executed by the control unit 10. The storage unit 12 stores, for example, game programs (game software) in addition to programs for realizing various functions of the image processing system 1 described below. The storage unit 12 also reserves an area for a frame buffer in which images are drawn by the GPU.

[0014] The communication unit 14 is a communication interface such as an Ethernet (registered trademark) module or a wireless LAN module.

[0015] The operation unit 16 is a user interface such as a keyboard, mouse, or game console controller, and receives operation inputs from the user and outputs signals indicating the contents of the inputs to the control unit 10 .

[0016] The display unit 18 is a display device such as a liquid crystal display or an organic EL display, and displays various images according to instructions from the control unit 10 .

[0017] The audio output unit 19 is, for example, a speaker, and outputs audio represented by audio data generated by the image processing system 1 .

[0018] In addition to the devices described above, the image processing system 1 may also include an optical disc drive that reads optical discs such as DVD-ROMs and Blu-ray (registered trademark) discs, a USB (Universal Serial Bus) port, and the like.

[0019] 2. Overview of Image Processing System Fig. 2 is a diagram showing an overview of the image processing system 1. Fig. 3 is a diagram showing a schematic diagram of processing in the image processing system 1. Here, an example is shown in which the image processing system 1 is used to improve the image quality of gameplay video in a game. Gameplay video is video generated in response to a game program executed by the control unit 10 and user input accepted by the operation unit 16, and is composed of a plurality of still images (frames) that are time-series data. The processing performed by the image processing system 1 is mainly as follows:

[0020] (1) Generation of Processing Frames First, the image processing system 1 generates an image (processing frame) depicting one or more game objects by rendering three-dimensional data representing the game objects as viewed from a predetermined viewpoint. This processing frame is an image having a predetermined number of pixels (initial pixel count) and a predetermined image quality (initial image quality). The processing frame can be considered an image representing the virtual space VS in which the one or more game objects represented by three-dimensional data are placed, from a predetermined viewpoint (see FIG. 5 ). Processing frames are generated at predetermined time intervals. The number of pixels of the processing frame is, for example, 1920 x 1080 (1080p). Each generated processing frame is not displayed directly on the display unit 18, but is temporarily stored in the storage unit 12 for subsequent processing. Note that the following description mainly illustrates processing for the nth processing frame 20_n, but similar processing is also performed for other processing frames (i.e., n = 2, 3, ..., N).

[0021] (2) Acquisition of Input Frame Based on the acquired processing target frame 20_n, the image processing system 1 acquires a frame (input frame) 22_n having a number of pixels (number of input pixels) greater than the initial number of pixels. The number of input pixels is, for example, 3840 × 2160 (4K). Specifically, the processing target frame 20_n is subjected to enlargement and interpolation processing to generate the input frame 22_n.

[0022] It should be noted that although the input frame 22_n has a larger number of pixels than the target frame 20_n, its image quality is not necessarily sufficiently improved. In other words, the image quality of a frame does not simply refer to the number of pixels (high resolution). The image quality of a frame may be evaluated based on, for example, a high signal-to-noise ratio, high spatial frequency reproducibility, high temporal stability (low artifacts and flicker when multiple frames are displayed consecutively), etc., when compared to a reference frame, or a combination of these.

[0023] (3) Obtaining an Estimated Frame The image processing system 1 inputs the input frame 22_n into the machine learning model 200 and obtains the estimated frame 24_n. The estimated frame 24_n is an image having the same number of pixels (estimated pixel count) as the number of input pixels and an image quality (estimated image quality) that is equal to or higher than the initial image quality. Here, in addition to the input frame 22_n, the machine learning model 200 receives input of the nth virtual space information 27_n and information based on the nth variation information 29_n (the nth variation position information 29a_n and the nth ordinal information 29b_n) (see Figures 2 and 3). Details of the nth virtual space information 27_n and the nth variation information 29_n will be described later.

[0024] The machine learning model 200 is a model trained using multiple training data, each of which includes a learning input frame having an input pixel count, learning virtual space information, information based on learning variation information, and a learning estimated frame having an estimated pixel count and estimated image quality.

[0025] (4) Acquisition of Cumulative Feature Information The machine learning model 200 has an accumulated feature information output layer 202 that receives input of an input frame 22_n, the nth virtual space information 27_n, the nth variation position information 29a_n, and the nth ordinal information 29b_n, and outputs nth accumulated feature information 26_n that indicates the features of the first to nth input frames 22 (see FIG. 2). The image processing system 1 acquires the nth accumulated feature information 26_n.

[0026] The acquired n-th cumulative feature information 26_n is input to the estimated frame output layer 204, which outputs the n-th estimated frame 24_n (see FIG. 2 ). The acquired n-th cumulative feature information 26_n is also stored in the storage unit 12 and is used to estimate an estimated frame 24_n+1 corresponding to the next processing target frame (the (n+1)th processing target frame) 20_n+1.

[0027] As described above, the image processing system 1 estimates the estimated frame 24 using the accumulated feature information 26, which stores past information, in addition to the input frame 22 corresponding to the current processing target frame 20. This increases the amount of information available for estimation, making it possible to obtain a high-quality estimated frame 24_n.

[0028] (5) Input of Virtual Space Information to Machine Learning Model As described above, the processing target frame 20, which is a virtual image, is generated by rendering three-dimensional data. The rendering is performed based on virtual space information 27, which is information about the virtual space VS that can be used to determine the pixel value of each pixel of the processing target frame 20. The virtual space information 27 includes, for example, information about the viewpoint C from which the virtual space VS is viewed, information about the depth of the game object O, information about the movement of the game object O, information about the color and texture of the game object, and information about the intensity, color, and illumination direction of the light source (see FIG. 5 ).

[0029] However, since the processing target frame 20 obtained as a result of rendering may not contain sufficient information about the virtual space VS, there is a limit to the accuracy of estimation based only on the processing target frame 20. In other words, although the virtual space information 27 can be used when determining the pixel value of each pixel of the processing target frame 20, the virtual space information 27 itself does not remain in the processing target frame 20. For example, if there are originally C pieces of information about the virtual space VS, the C pieces of information are reduced to three pieces of information (RGB) in the process of determining the pixel values ​​(RGB values) of each pixel of the processing target frame 20.

[0030] Therefore, in the image processing system 1 according to this embodiment, in addition to the input frame 22_n, n-th virtual space information 27_n, which is information about the virtual space VS that can be used to determine the pixel value of each pixel of the n-th processing target frame 20_n, is input to the machine learning model 200 (see FIGS. 2 and 3 ). This makes it possible to effectively utilize the information about the virtual space VS, thereby obtaining a high-quality estimated frame 24 with high accuracy. Details of the image processing system 1 are described below.

[0031] 4 is a functional block diagram showing an example of functions realized by the image processing system 1. As shown in FIG. 4, the image processing system 1 realizes a game processing unit 400, a rendering unit 402, a rendering information storage unit 404, a processing target frame acquisition unit 406, a variation information acquisition unit 408, an input frame acquisition unit 410, a virtual space information acquisition unit 412, a variation position information acquisition unit 414, an ordinal number information acquisition unit 416, a machine learning model storage unit 418, an estimated frame acquisition unit 420, and a cumulative feature information acquisition unit 422. The game processing unit 400, the rendering unit 402, the processing target frame acquisition unit 406, the variation information acquisition unit 408, the input frame acquisition unit 410, the virtual space information acquisition unit 412, the variation position information acquisition unit 414, the ordinal number information acquisition unit 416, the estimated frame acquisition unit 420, and the cumulative feature information acquisition unit 422 are mainly realized by the control unit 10. The rendering information storage unit 404 and the machine learning model storage unit 418 are mainly realized by the storage unit 12. Note that the game processing unit 400, the rendering unit 402, and the rendering information storage unit 404 are functions provided by the game software.

[0032] [Game Processing Unit] The game processing unit 400 executes various processes related to the game. For example, in response to a game program executed by the control unit 10 or user input received by the operation unit 16, the game processing unit 400 executes processes such as placing a game object O in the virtual space VS, operating or moving the game object O, and changing the viewpoint C from which the virtual space VS is viewed (see FIG. 5 ). The game object O is composed of primitives such as polygons represented by three-dimensional data. The three-dimensional data includes geometric information indicating the positions of vertices, topological information indicating how the vertices are connected, and attribute information such as color.

[0033] [Rendering Unit] FIG. 5 is a diagram illustrating the processing of the rendering unit 402. The rendering unit 402 generates first to Nth (N is a natural number equal to or greater than 2) processing frames 20 by rendering (drawing) three-dimensional data representing one or more game objects O as viewed from a predetermined viewpoint C. The processing frames 20 can also be considered images representing a virtual space VS in which one or more game objects O represented by three-dimensional data are placed, from the predetermined viewpoint C. The processing frames 20 have a predetermined initial number of pixels. The rendering unit 402 performs rendering based on the results of various processes executed by the game processing unit 400. Specifically, the rendering unit 402 performs vertex processing (vertex shading) and pixel processing (pixel shading) based on the three-dimensional data representing the game objects O placed in the virtual space VS. The vertex processing includes a coordinate transformation process (perspective projection) from the view coordinate system to the screen coordinate system, and a value related to the change in the viewpoint C is added to the perspective projection matrix (camera matrix) used in the coordinate transformation process, as described below. The rendering unit 402 may perform rendering based on light source information, depth information (depth buffer), texture information, normal information, etc. In addition to the above processes, the rendering unit 402 may also perform processes that apply effects such as depth of field (DoF) and motion blur. The processing of the rendering unit 402 may be set appropriately by a game software developer, etc. Here, the game software developer, etc. may adjust the MIP of the texture according to the estimated number of pixels of the estimated frame 24, etc. This makes it possible to suppress the occurrence of noise such as moiré in the estimated frame 24.

[0034] Here, the rendering unit 402 generates each processing frame 20 by performing rendering such that the viewpoint C varies for each processing frame 20. Here, even if the game processing unit 400 fixes the viewpoint C at a predetermined position, the rendering unit 402 varies the viewpoint C for each processing frame 20. As a result, as shown in FIG. 5 , the position of the displayed game object O varies in each of the processing frames 20_n, 20_n+1, and 20_n+2. In other words, the rendering unit 402 applies jitter when generating each processing frame 20. Specifically, the rendering unit 402 varies the viewpoint C for each processing frame 20 by adding a numerical value corresponding to a fractional pixel size that differs for each processing frame 20 to the perspective projection matrix.

[0035] The rendering unit 402 performs rendering of the three-dimensional data in accordance with a predetermined sequence so that the viewpoint C varies for each processing target frame 20. The predetermined sequence is a sequence with a period k consisting of first to k-th variation vectors (k is a natural number equal to or greater than 2), each indicating the amount and direction of variation of the viewpoint C. For example, a Halton sequence can be used as such a sequence. In this embodiment, as an example, the rendering unit 402 performs rendering of the three-dimensional data in accordance with a Halton sequence with a period of 32 (i.e., k=32) so that the viewpoint C varies for each processing target frame 20.

[0036] [Rendering Information Storage Unit] The rendering information storage unit 404 stores information necessary for the rendering process in the rendering unit 402 and information obtained as a result of the rendering process. For example, the rendering information storage unit 404 stores the processing target frame 20. The rendering information storage unit 404 also stores virtual space information 27 and variation information 29. Details of the virtual space information 27 and variation information 29 will be described later.

[0037] [Processing Frame Acquisition Unit] The processing frame acquisition unit 406 acquires the first to Nth processing frames 20. Specifically, the processing frame acquisition unit 406 acquires the first to Nth processing frames 20 stored in the rendering information storage unit 404.

[0038] [Variation Information Acquisition Unit] The variation information acquisition unit 408 acquires first to Nth variation information 29, which is information related to the variation of the viewpoint C for each of the first to Nth processing frames 20 during rendering. The variation information acquisition unit 408 acquires the first to Nth variation information 29 stored in the rendering information storage unit 404. The nth variation information 29 includes a variation vector corresponding to the nth processing frame 20_n and an ordinal number in the series. If the variation vector corresponding to the nth processing frame 20_n is the ith variation vector (i is a natural number greater than or equal to 1 and less than or equal to k), the ordinal number corresponding to the nth processing frame 20_n is i. In other words, the ordinal number corresponding to the nth processing frame 20_n is a value indicating the ordinal number of the variation vector corresponding to the nth processing frame 20_n in the series.

[0039] [Input Frame Acquisition Unit] The input frame acquisition unit 410 acquires the first to Nth input frames 22 by generating, based on each processing target frame 20, an input frame 22 that corresponds to the processing target frame 20 and has a number of input pixels equal to or greater than the initial number of pixels. In this embodiment, each input frame 22 has a number of input pixels greater than the initial number of pixels. That is, in this embodiment, each input frame 22 is an enlarged image of the processing target frame 20 that corresponds to the input frame 22.

[0040] Specifically, the input frame acquisition unit 410 interpolates pixel values ​​at positions in the processing frame 20 that correspond to the pixels before the fluctuation, based on the fluctuation information 29 and each pixel in each processing frame 20, to generate each input frame 22. Fig. 6 is a diagram illustrating the processing in the input frame acquisition unit 410. Fig. 6 illustrates an example of obtaining the n-th input frame 22_n. For example, as shown in Fig. 6, if the pixel center of a pixel in the input frame 22_n to be obtained is P １，０ In this case, the input frame acquisition unit 410 calculates P １，０ The pixel centers P' of the four pixels closest to ０，０ , P' １，０ , P' ０，１ , P' １，１ Based on the coordinates and pixel values ​​of P １，０ The pixel value of P' is calculated by bilinear interpolation. １，０ is increased by the amount of fluctuation indicated by the fluctuation information 29. １，０ The pixel values ​​of the pixels newly generated by the enlargement process are calculated in a similar manner. Note that, in addition to bilinear interpolation, various other well-known interpolation methods such as bicubic interpolation and Lanczos interpolation can also be used as the interpolation method.

[0041] When rendering is performed so that the viewpoint C varies for each processing frame 20, the amount of time-series information increases. However, by using each processing frame 20 obtained in this way (hereinafter referred to as a "variable processing frame") for estimation, an estimated frame 24 with higher image quality can be obtained.

[0042] On the other hand, if the frame to be subjected to variation processing (or an enlarged image thereof) is input directly to the machine learning model 200, the influence of the variation in viewpoint C described above may result in a decrease in the accuracy of estimation.

[0043] Therefore, as described above, in the image processing system 1, pixel values ​​at positions in the processing frame 20 corresponding to each pixel before the fluctuation are obtained by interpolation based on the fluctuation information 29 and each pixel in each processing frame 20, and each input frame 22 is generated and input to the machine learning model 200. This corrects for the influence of fluctuations in the viewpoint C, making it possible to prevent a decrease in estimation accuracy.

[0044] [Virtual Space Information Acquisition Unit] The virtual space information acquisition unit 412 acquires nth virtual space information 27_n, which is information about the virtual space VS that can be used to determine the pixel values ​​of each pixel in the nth processing target frame 20_n. The virtual space information acquisition unit 412 acquires the nth virtual space information 27_n stored in the rendering information storage unit 404. The virtual space information 27 includes, for example, information about the viewpoint C from which the virtual space VS is viewed, information about the depth of the game object O, information about the movement of the game object O, information about the color and texture of the game object, and information about the intensity, color, and illumination direction of the light source. The nth virtual space information 27_n can also be said to be information that can be used for rendering the nth processing target frame 20_n. Note that the nth virtual space information 27_n is not limited to information actually used for rendering the nth processing target frame 20_n.

[0045] The n-th virtual space information 27_n is information having the same number of pixels as the number of input pixels. That is, since the n-th virtual space information 27_n has the same number of pixels as the input frame 22_n, the n-th virtual space information 27_n can be input to the machine learning model 200 together with the input frame 22_n.

[0046] In this embodiment, the nth virtual space information 27_n includes depth information 27a_n indicating the depth in the virtual space VS of the game object O displayed in each pixel of the nth input frame 22_n ( FIG. 3 ). The depth information 27a is also called a depth buffer, Z-buffer, or depth buffer. Specifically, the virtual space information acquisition unit 412 acquires original depth information having the same number of pixels as the initial number of pixels, and performs enlargement and interpolation processing on the original depth information to acquire depth information 27a having the same number of pixels as the number of input pixels.

[0047] In this embodiment, the nth virtual space information 27_n includes texture information 27b_n that indicates the texture of the game object O displayed in each pixel of the nth input frame 22_n. The texture information 27b includes, for example, a normal map or an albedo map. In this embodiment, as an example, the texture information 27b is a normal map. Specifically, the virtual space information acquisition unit 412 acquires original texture information having the same number of pixels as the initial pixel count, and performs enlargement and interpolation processing on the original texture information to acquire texture information 27b having the same number of pixels as the input pixel count.

[0048] Furthermore, in this embodiment, the nth virtual space information 27_n includes nth motion information 27c_n, which indicates the amount and direction of movement of the game object O displayed in each pixel of the n-1th input frame 22_n-1 from the n-1st input frame 22_n-1 to the nth input frame 22_n. The pixel value of each pixel of the nth motion information 27c_n is a two-dimensional vector indicating the amount and direction of movement of the game object O displayed in each pixel of the n-1st input frame 22_n-1 from the n-1st input frame 22_n-1 to the nth input frame 22_n. The motion information 27c is also referred to as a motion vector. Specifically, the virtual space information acquisition unit 412 acquires original motion information having the same number of pixels as the initial pixel count, and performs enlargement and interpolation processing on the original motion information to acquire motion information 27c having the same number of pixels as the input pixel count.

[0049] It goes without saying that the virtual space information 27 may include information other than the depth information 27a, the texture information 27b, and the movement information 27c.

[0050] [Variation Position Information Acquisition Unit] Based on the nth variation information 29_n, the variation position information acquisition unit 414 acquires nth variation position information 29a_n, in which the value of one element included in the variation vector corresponding to the nth processing target frame 20_n is used as the pixel value of each pixel. The nth variation position information 29a_n is information having the same number of pixels as the number of input pixels. Specifically, the nth variation position information 29a_n is information in which the value of a first element included in the variation vector corresponding to the nth processing target frame 20_n is used as the pixel value of each pixel, and information in which the value of a second element included in the variation vector corresponding to the nth processing target frame 20_n is used as the pixel value of each pixel. Here, the first element and the second element correspond to the amount of variation in the width direction and the amount of variation in the height direction of the processing target frame 20, respectively. Note that the nth variation position information 29a_n may be either information in which the value of the first element is used as the pixel value of each pixel, or information in which the value of the second element is used as the pixel value of each pixel.

[0051] [Ordinal Information Acquisition Unit] The ordinal information acquisition unit 416 acquires nth ordinal information 29b_n, which indicates an ordinal number in the sequence, based on the nth variation information 29_n. The nth ordinal information 29b_n is information having the same number of pixels as the number of input pixels. Specifically, the ordinal information acquisition unit 416 applies positional encoding to the ordinal number indicated by the nth variation information 29_n to generate information having the same number of pixels as the number of input pixels, and acquires this information as the nth ordinal information 29b_n. For example, the ordinal information acquisition unit 416 may acquire the nth ordinal information 29b_n by applying positional encoding to the ordinal number indicated by the nth variation information 29_n according to the following equation (1): In the following equation 1, PE(pos, x, y) is the pixel value of the pixel located at coordinates (x, y) in the n-th ordinal number information 29b_n. In equation 1, pos is the ordinal number indicated by the n-th variation information 29_n (0≦pos≦31), and width and height are the width and height of the n-th ordinal number information 29b_n (i.e., the width and height of the input frame 22), respectively. Here, 0≦x≦width−1, 0≦y≦height−1.

[0052]

[0053] [Machine Learning Model] The machine learning model 200 is a model that estimates the nth estimated frame 24_n based on the nth input frame 22_n. Specifically, the machine learning model 200 is a model that estimates the nth estimated frame 24_n based on the nth input frame 22_n, the nth virtual space information 27_n, and the nth movement information 29_n. Specifically, the machine learning model 200 is a convolutional neural network (CNN). As the machine learning model 200, for example, a well-known model such as a multi-layered ResNet having a residual connection mechanism or a so-called encoder-decoder type U-Net can be used. The model described in Non-Patent Document 1 may also be used as the machine learning model 200.

[0054] The machine learning model 200 is a model trained using multiple training data, each of which includes a training input frame having an input pixel count, training virtual space information, training variation information, and a training estimation frame having an estimated pixel count. The training input frame is an image based on a training processing target frame that shows, from a predetermined viewpoint, a training virtual space in which one or more training game objects represented by training three-dimensional data are arranged. The training input frame is obtained by rendering the training three-dimensional data so that the viewpoint varies for each training processing target frame according to the sequence. Specifically, the machine learning model 200 is trained based on the loss between the output when the nth training input frame, the n-1th training cumulative feature information indicating the features of the first to n-1th training input frames, the nth training virtual space information (the nth depth information, the nth texture information, and the nth motion information), and information based on the nth training variation information (the nth training variation position information and the nth learning ordinal information) are input, and the nth training estimation frame. Here, the nth training virtual space information is information about the training virtual space that can be used to determine the pixel values ​​of each pixel of the nth training processing target frame, and the nth training variation information is information related to the variation of the viewpoint of the nth training processing target frame in rendering the training three-dimensional data. During training, in addition to the nth training virtual space information, the (n-1)th training virtual space information is also input to the machine learning model 200. The machine learning model 200 is trained so as to minimize the loss. Various known techniques, such as backpropagation, can be used to train the machine learning model 200.

[0055] Specifically, the machine learning model 200 includes an accumulated feature information output layer 202, an estimated frame output layer 204, and a convolution layer 206 (see FIG. 2).

[0056] The cumulative feature information output layer 202 receives the nth input frame 22_n, the (n-1)th cumulative feature information 26_n-1 indicating the features of the first to (n-1)th input frames 22, the nth virtual space information 27_n, and information based on the nth movement information 29_n, and outputs the nth cumulative feature information 26_n indicating the features of the first to nth input frames 22_n. Specifically, the cumulative feature information output layer 202 receives the nth input frame 22_n, the (n-1)th cumulative feature information 26_n-1, the nth depth information 27a_n, the nth texture information 27b_n, the nth movement information 27c_n, the nth movement position information 29a_n, and the nth ordinal information 29b_n, and outputs the nth cumulative feature information 26_n. The cumulative feature information output layer 202 may be composed of, for example, one or more convolutional layers. The cumulative feature information 26_n-1 is image information (bitmap information) having the same number of pixels as the number of input pixels. The cumulative feature information 26_n-1 can also be considered a feature map that indicates the features of the first to (n-1)th input frames 22.

[0057] Furthermore, in this embodiment, in addition to the n-th virtual space information 27_n, the (n-1)th virtual space information 27_n-1 is further input to the cumulative feature information output layer 202. Specifically, the (n-1)th depth information 27a_n-1 is input to the cumulative feature information output layer 202. Note that the (n-1)th virtual space information 27_n-1 other than the (n-1)th depth information 27a_n-1 may also be input to the cumulative feature information output layer 202.

[0058] The cumulative feature information output layer 202 receives the first input frame 22_1, the given feature information, the first virtual space information 27_1, and the first variation information, and outputs the first cumulative feature information 26_1. When n=1, there is no previous cumulative feature information 26, and therefore the given feature information prepared in advance is input to the cumulative feature information output layer 202 together with the first input frame 22_1.

[0059] The estimated frame output layer 204 receives the n-th accumulated feature information 26_n and outputs the n-th estimated frame 24_n. The estimated frame output layer 204 may be configured, for example, by one or more convolutional layers, similar to the accumulated feature information output layer 202. Alternatively, the estimated frame output layer 204 may be configured by one or more transposed convolutional layers (deconvolutional layers).

[0060] The convolution layer 206 is a layer that reduces the number of channels of the accumulated feature information 26 while maintaining the number of pixels of the accumulated feature information 26. The convolution layer 206 reduces the dimension of the accumulated feature information 26, thereby reducing calculation costs. The convolution layer 206 is, for example, a convolution layer with a kernel size of 1×1, but is not limited to this.

[0061] [Machine Learning Model Storage Unit] The machine learning model storage unit 418 stores the machine learning model 200. Specifically, the machine learning model storage unit 418 stores parameters of the machine learning model 200 (such as the number of convolutional layers, the number of nodes used in each convolutional layer, and the weight of each node).

[0062] [Estimated Frame Acquisition Unit] The estimated frame acquisition unit 420 acquires first to Nth estimated frames 24, each having an estimated pixel number that is greater than the initial pixel number and equal to or greater than the input pixel number, based on the first to Nth input frames 22, the first to Nth virtual space information 27, the first to Nth variation information 29, and the machine learning model 200. In this embodiment, the estimated frames 24 have the same number of estimated pixels as the input pixel number. More specifically, the estimated frame acquisition unit 420 inputs the nth input frame 22_n, the n-1th cumulative feature information 26_n-1, the nth depth information 27a_n, the n-1th depth information 27a_n-1, the nth texture information 27b_n, the nth movement information 27c_n, the nth fluctuation position information 29a_n, and the nth ordinal information 29b_n into the machine learning model 200 to acquire the nth estimated frame 24_n. In this embodiment, the estimated frame acquisition unit 420 acquires a combined feature 30_n by concatenating the nth input frame 22_n, the (n-1)th cumulative feature information 26_n-1, the nth depth information 27a_n, the nth texture information 27b_n, the nth movement information 27c_n, the nth fluctuation position information 29a_n, and the nth ordinal information 29b_n, and inputs this combined feature 30_n to the machine learning model 200 (see Figure 3).

[0063] [Cumulative feature information acquisition unit] The cumulative feature information acquisition unit 422 inputs the nth input frame 22_n, the (n-1)th cumulative feature information 26_n-1, the nth depth information 27a_n, the (n-1)th depth information 27a_n-1, the nth texture information 27b_n, the nth movement information 27c_n, the nth fluctuation position information 29a_n, and the nth ordinal information 29b_n to the machine learning model 200 to acquire the nth cumulative feature information 26_n.

[0064] 7A and 7B are flow diagrams showing an example of the flow of processing executed in the image processing system 1. The processing shown in Fig. 7A and 7B is executed by the control unit 10 operating in accordance with a program stored in the storage unit 12.

[0065] (1) Processing for n=1 First, as shown in FIG. 7A , the control unit 10 acquires a first processing frame 20_1 (S700). The control unit 10 acquires a first input frame 22_1 based on the first processing frame 20_1 (S702). The control unit 10 acquires first virtual space information 27_1 (S704), and acquires first variable position information 29 a_1 and first ordinal information 29 b_1 (S706). In this embodiment, as described above, the control unit 10 acquires first depth information 27 a_1, first texture information 27 b_1, and first motion information 27 c_1 as the first virtual space information 27_1. The control unit 10 inputs the first input frame 22_1, the given feature information, the first virtual space information 27_1, the first variable position information 29a_1, and the first ordinal information 29b_1 into the machine learning model 200, and obtains the first estimated frame 24_1 and the first cumulative feature information 26_1 (S708).

[0066] 7B, the control unit 10 acquires the nth processing target frame 20_n (S710). The control unit 10 acquires the nth input frame 22_n based on the nth processing target frame 20_n (S712).

[0067] Next, the control unit 10 acquires the nth virtual space information 27_n and the n-1th virtual space information 27_n-1 (S714). In this embodiment, as described above, the control unit 10 acquires the n-1th depth information 27a_n-1 as the n-1th virtual space information 27_n-1. The control unit 10 also acquires the nth variable position information 29a_n and the nth ordinal information 29b_n (S716). Then, the control unit 10 inputs the nth input frame 22_n, the n-1th cumulative feature information 26_n-1, the nth virtual space information 27_n, the n-1th virtual space information 27_n-1, the nth variable position information 29a_n, and the nth ordinal information 29b_n to the machine learning model 200, and acquires the nth estimated frame 24_n and the nth cumulative feature information 26_n (S718). The control unit 10 determines whether or not a next frame exists (S720), and if it determines that a next frame exists (S720; Y), it increments n to n+1 and repeats the processes of S706 to S718. If the control unit 10 determines that a next frame does not exist (S720; N), it ends this process. Note that if it determines that a next frame does not exist (S720; N), the control unit 10 may cause the display unit 18 to display the first to Nth estimated frames 24 as they are.

[0068] [5. Summary] According to the image processing system 1 according to the present embodiment described above, the nth estimated frame 24_n is estimated using the (n-1)th accumulated feature information 26_n-1 indicating the features of the first to (n-1)th input frames 22. That is, in addition to the information of the nth processing target frame 20_n, the information of the first to (n-1)th processing target frames 20 can be used for estimation, so that the amount of information available for estimation increases, and a high-quality estimated frame 24_n can be obtained.

[0069] Furthermore, according to the image processing system 1, the nth estimated frame 24_n is obtained based on the nth input frame 22_n, the nth virtual space information 27_n, and the machine learning model 200, thereby making it possible to effectively utilize information about the virtual space VS, and as a result, to obtain an estimated frame 24 with high accuracy and high image quality.

[0070] Furthermore, the image processing system 1 is configured to acquire the nth estimated frame 24_n further based on the nth variation information 29_n. That is, as described above, if the frame to be subjected to variation processing (or an enlarged image thereof) is input as is to the machine learning model 200, there is a risk that the accuracy of the estimation will decrease due to the influence of variation in the viewpoint C. However, in this embodiment, the variation in the viewpoint C is taken into account when making an estimation by the machine learning model 200, and therefore it is possible to more reliably suppress the decrease in the accuracy of the estimation.

[0071] Note that the variation information 29 itself is data in a format different from that of the input frame 22, and therefore the variation information 29 itself cannot be input to the machine learning model 200. Therefore, the image processing system 1 is configured to acquire variation position information 29a and ordinal number information 29b, which are information having the same number of pixels as the number of input pixels, based on the variation information 29. This makes it possible to acquire an estimated frame 24 based on the variation information 29.

[0072] Furthermore, in the image processing system 1 , the n-th motion information 27 c_n is input to the cumulative feature information output layer 202 .

[0073] If there is movement of the game object O between the nth processing frame 20_n and the n-1th processing frame 20_n-1, when obtaining the nth estimated frame 24_n, if the nth input frame 22_n and the n-1th accumulated feature information 26_n-1 are input directly into the machine learning model 200, a ghost phenomenon may occur in which an afterimage of the game object O that was displayed in the nth input frame 22_n is displayed in the output nth estimated frame 24_n.

[0074] In the image processing system 1, as described above, the nth movement information 27c_n is input to the cumulative feature information output layer 202, so that when making an estimation by the machine learning model 200, the movement of the game object O between the nth input frame 22_n and the n-1th input frame 22_n-1 is taken into consideration, thereby suppressing the above-mentioned ghost phenomenon.

[0075] Furthermore, in the image processing system 1 , the (n−1)th virtual space information 27 _n−1 (particularly the (n−1)th depth information 27 a _n−1) is input to the cumulative feature information output layer 202 .

[0076] If all or part of a game object O that is not displayed in the n-1st processing frame 20_n-1 is displayed in the nth processing frame 20_n, when acquiring the nth estimated frame 24_n, if the nth input frame 22_n and the n-1st cumulative feature information 26_n-1 are input directly into the machine learning model 200, the aforementioned ghost phenomenon may occur in the output nth estimated frame 24_n.

[0077] In the image processing system 1, as described above, the n-1th depth information 27a_n-1 is input to the cumulative feature information output layer 202, so that when making an estimation by the machine learning model 200, the depth of the game object O indicated by the n-1th input frame 22_n-1, i.e., the previous frame, is taken into consideration, thereby suppressing the above-mentioned ghost phenomenon.

[0078] It should be noted that the present invention is not limited to the above-described embodiment. Furthermore, the specific character strings and numerical values ​​described above and the specific character strings and numerical values ​​in the drawings are merely examples, and the present invention is not limited to these character strings and numerical values.

[0079] For example, in the present embodiment, the number of input pixels is greater than the initial number of pixels, and the number of input pixels is equal to the estimated number of pixels, but the number of input pixels may be equal to the initial number of pixels, and the estimated number of pixels may be greater than the number of input pixels. In other words, the input frame 22 does not necessarily have to be an enlarged version of the processing target frame 20.

[0080] Furthermore, the processing target frame 20 may be input directly to the machine learning model 200 .

[0081] Furthermore, in the present embodiment, a case has been illustrated in which information based on the virtual space information 27 and the variation information 29 is input to the cumulative feature information output layer 202, but this is not limiting. That is, the cumulative feature information output layer 202 may be input with the nth input frame 22_n and the (n-1)th cumulative feature information 26_n-1, and output the nth cumulative feature information 26_n. In this case, the estimated frame output layer 204 may be input with the nth cumulative feature information 26_n, the nth virtual space information 27_n, the (n-1)th virtual space information 27_n-1, the nth variation position information 29a_n, and the nth ordinal information 29b_n, which are output from the cumulative feature information output layer 202, and output the nth estimated frame 24_n.

[0082] In addition, in this embodiment, an example is given of a case where both information based on virtual space information 27 and information based on variation information 29 are input to the machine learning model 200, but it is also possible to input only one of the information based on virtual space information 27 or variation information 29 to the machine learning model 200.

[0083] In addition, in order to more reliably suppress the occurrence of ghost phenomena, processing may be performed on the n-1th cumulative feature information 26_n-1 based on the nth depth information 27a_n, the n-1th depth information 27a_n-1, and the nth motion information 27c_n.

[0084] For example, the n-1th auxiliary information may be obtained by applying motion compensation to the n-1th cumulative feature information 26_n-1 based on the nth motion information 27c_n, and this n-1th auxiliary information may be input into the machine learning model 200 instead of the n-1th cumulative feature information 26_n-1.

[0085] Furthermore, for example, based on the nth depth information 27a_n and the n-1th depth information 27a_n-1, the nth appearance pixel, which is a pixel among the pixels of the nth input frame 22_n in which all or part of the game object O that is not displayed in the n-1th input frame 22_n-1 is displayed, may be identified, and the n-1th auxiliary information may be obtained by replacing the pixel value of the nth appearance pixel in the n-1th cumulative feature information 26_n-1 with a predetermined value.

[0086] Furthermore, in the present embodiment, the image processing system 1 is applied to a moving image, but the image processing system 1 may also be applied to a still image.

[0087] [6. Supplementary Notes] (1) An image processing system including at least one processor, wherein the at least one processor: acquires an input image, the input image having a number of input pixels equal to or greater than an initial number of pixels, based on a processing target image that shows, from a predetermined viewpoint, a virtual space in which one or more objects represented by three-dimensional data are arranged, and that has a predetermined initial number of pixels; acquires virtual space information that is information about the virtual space that can be used to determine a pixel value of each pixel of the processing target image; acquires an estimated image having an estimated number of pixels that is greater than the input number of pixels, based on the input image, the virtual space information, and a machine learning model; the machine learning model is trained using a plurality of training data, each of which includes a learning input image, learning virtual space information, and a learning estimated image; the learning input image is an image having the input number of pixels, based on a learning processing target image that shows, from a predetermined viewpoint, a learning virtual space in which one or more learning objects represented by learning three-dimensional data are arranged, and that has the initial number of pixels; the learning virtual space information is information about the learning virtual space that can be used to determine a pixel value of each pixel of the learning processing target image; and the learning estimated image is an image having the estimated number of pixels. (2) The image processing system according to (1), wherein the virtual space information is information having the same number of pixels as the number of input pixels. (3) The image processing system according to (1) or (2), wherein the virtual space information includes depth information indicating a depth in the virtual space of the object displayed in each pixel of the input image. (4) The image processing system according to any of (1) to (3), wherein the virtual space information includes texture information indicating a texture of the object displayed in each pixel of the input image. (5) The at least one processor processes first to Nth images to be processed (N is a natural number equal to or greater than 2) arranged in chronological order., N), acquires the first to N-th input images based on a set of input image information, acquires the first to N-th virtual space information corresponding to the first to N-th input images, respectively, acquires the first to N-th estimated images based on the first to N-th input images, the first to N-th virtual space information, and the machine learning model. (6) The image processing system according to (5), wherein the machine learning model includes an cumulative feature information output layer and an estimated image output layer, wherein the cumulative feature information output layer receives the n-th input image (n=2, 3, ..., N), the n-th virtual space information, and n-1-th cumulative feature information indicating features of the first to n-1-th input images, and outputs the n-th cumulative feature information indicating the features of the first to n-th input images, and the estimated image output layer receives the n-th cumulative feature information and outputs the n-th estimated image. (7) The image processing system according to (6), wherein the nth virtual space information includes nth motion information indicating an amount and direction of motion of the object displayed in each pixel of the n-1th input image from the n-1th input image to the nth input image. (8) The image processing system according to any of (6) to (7), wherein the cumulative feature information output layer receives the nth input image, the nth virtual space information, the n-1th virtual space information, and the n-1th cumulative feature information as inputs, and outputs the nth cumulative feature information. (9) The image processing system according to any one of (5) to (8), wherein each of the processing target images is an image obtained by rendering the three-dimensional data so that the viewpoint varies for each of the processing target images according to a predetermined series, the predetermined series being a series of first to k-th variation vectors (k is a natural number equal to or greater than 2) with a period of k, each of which indicates an amount and a direction of variation of the viewpoint, and the at least one processor acquires first to N-th variation information which is information related to the variation of the viewpoint for each of the first to N-th processing target images in the rendering, and acquires the first to N-th estimated images based on the n-th input image, the n-th virtual space information, the n-th variation information, and the machine learning model.(10) The image processing system according to (9), wherein the at least one processor acquires, based on the nth variation information, nth variation position information having the same number of pixels as the number of input pixels and having the pixel value of each pixel as the value of one element included in the variation vector corresponding to the nth image to be processed, and acquires the first to Nth estimated images based on the nth input image, the nth virtual space information, the nth variation position information, and the machine learning model. (11) The image processing system according to (9) or (10), wherein the at least one processor acquires, based on the nth variation information, nth ordinal information having the same number of pixels as the number of input pixels and indicating an ordinal number in the series, and acquires the first to Nth estimated images based on the nth input image, the nth virtual space information, the nth ordinal information, and the machine learning model.

Claims

1. An image processing system including at least one processor, wherein the at least one processor: acquires an input image having a number of input pixels equal to or greater than an initial number of pixels based on a target image for processing that shows, from a predetermined viewpoint, a virtual space in which one or more learning objects represented by three-dimensional data are arranged, and has a predetermined initial number of pixels; acquires virtual space information that is information about the virtual space that can be used to determine pixel values ​​of each pixel of the target image for processing; acquires an estimated image having an estimated number of pixels greater than the input number of pixels based on the input image, the virtual space information, and a machine learning model; the machine learning model is trained using a plurality of training data each including a learning input image, learning virtual space information, and a learning estimated image; the learning input image is an image having the input number of pixels based on a target image for processing that shows, from a predetermined viewpoint, a learning virtual space in which one or more learning objects represented by learning three-dimensional data are arranged, and has the initial number of pixels; the learning virtual space information is information about the learning virtual space that can be used to determine pixel values ​​of each pixel of the target image for processing; and the learning estimated image is an image having the estimated number of pixels.

2. The image processing system according to claim 1, wherein the virtual space information is information having the same number of pixels as the number of input pixels.

3. An image processing system as described in claim 1 or 2, wherein the virtual space information includes depth information indicating a depth in the virtual space of the object displayed at each pixel of the input image.

4. The image processing system according to claim 1 or 2, wherein the virtual space information includes texture information indicating the texture of the object displayed in each pixel of the input image.

5. The image processing system of claim 1 or 2, wherein the at least one processor: acquires the first to Nth input images based on the first to Nth processing target images (N is a natural number greater than or equal to 2) arranged in chronological order; acquires the first to Nth virtual space information corresponding to the first to Nth input images, respectively; and acquires the first to Nth estimated images based on the first to Nth input images, the first to Nth virtual space information, and the machine learning model.

6. The image processing system of claim 5, wherein the machine learning model includes an cumulative feature information output layer and an estimated image output layer, wherein the cumulative feature information output layer receives the nth input image (n = 2, 3, ..., N), the nth virtual space information, and the n-1th cumulative feature information indicating features of the 1st to n-1st input images, and outputs the nth cumulative feature information indicating the features of the 1st to nth input images, and the estimated image output layer receives the nth cumulative feature information, and outputs the nth estimated image.

7. The image processing system of claim 6, wherein the nth virtual space information includes nth motion information indicating the amount and direction of motion of the object displayed at each pixel of the n-1th input image from the n-1th input image to the nth input image.

8. The image processing system of claim 6, wherein the cumulative feature information output layer receives the nth input image, the nth virtual space information, the n-1th virtual space information, and the n-1th cumulative feature information, and outputs the nth cumulative feature information.

9. The image processing system of claim 6, wherein each of the processing target images is an image obtained by rendering the three-dimensional data so that the viewpoint varies for each of the processing target images in accordance with a predetermined series, the predetermined series being a series with a period k consisting of 1st to kth variation vectors (k is a natural number equal to or greater than 2) each indicating an amount and direction of the variation of the viewpoint, and the at least one processor obtains 1st to Nth variation information which is information related to the variation of the viewpoint for each of the 1st to Nth processing target images in the rendering, and obtains the 1st to Nth estimated images based on the nth input image, the nth virtual space information, the nth variation information, and the machine learning model.

10. The image processing system of claim 9, wherein the at least one processor acquires, based on the nth variation information, nth variation position information having the same number of pixels as the number of input pixels and with the pixel value of each pixel being the value of an element included in the variation vector corresponding to the nth image to be processed, and acquires the 1st to Nth estimated images based on the nth input image, the nth virtual space information, the nth variation position information, and the machine learning model.

11. The image processing system of claim 9, wherein the at least one processor: acquires, based on the nth variation information, nth ordinal information having the same number of pixels as the number of input pixels and indicating an ordinal number in the series; and acquires the first to Nth estimated images based on the nth input image, the nth virtual space information, the nth ordinal information, and the machine learning model.

12. An image processing system including at least one processor, wherein the at least one processor acquires first to Nth input images having a number of input pixels equal to or greater than an initial number of pixels based on first to Nth processing target images (N is a natural number equal to or greater than 2) which show a virtual space in which one or more objects represented by three-dimensional data are arranged from a predetermined viewpoint and which have a predetermined initial number of pixels and are arranged in chronological order, each of the processing target images being images obtained by rendering the three-dimensional data such that the viewpoint varies for each of the processing target images in accordance with a predetermined sequence, and the predetermined sequence includes first to kth variation vectors (k is a natural number equal to or greater than 2), which respectively indicate an amount and direction of variation of the viewpoint., N), the nth variation information, and n-1th cumulative feature information indicating features of the first to n-1th input images are input to the cumulative feature information output layer, and the nth cumulative feature information output layer outputs the nth cumulative feature information indicating features of the first to nth input images; and the nth estimated image output layer receives the nth input image (n=2, 3, ..., N), the nth variation information, and n-1th cumulative feature information indicating features of the first to n-1th input images, and outputs the nth cumulative feature information indicating features of the first to nth input images. The nth estimated image output layer receives the nth cumulative feature information, and outputs the nth estimated image. An image processing system in which the machine learning model is trained using a plurality of training data each including 1st to Nth training input images, 1st to Nth training variation information, and 1st to Nth training estimated images, the nth training input image being an image having the input pixel count based on the nth training process target image which shows a training virtual space in which one or more training objects represented by training three-dimensional data are arranged from a predetermined viewpoint and has the initial pixel count, each of the training process target images being images obtained by rendering the training three-dimensional data such that the viewpoint varies for each of the training process target images according to a predetermined series, the nth training variation information being information relating to the variation of the viewpoint of the nth training process target image in the rendering of the training three-dimensional data, and the nth training estimated image being an image having the estimated pixel count.

13. An image processing method, comprising: a processor: acquiring an input image having a number of input pixels equal to or greater than an initial number of pixels, based on a target image for processing that shows, from a specified viewpoint, a virtual space in which one or more learning objects represented by three-dimensional data are arranged, and that has a specified initial number of pixels; acquiring virtual space information that is information about the virtual space that can be used to determine a pixel value of each pixel of the target image for processing; acquiring an estimated image having an estimated number of pixels greater than the input number of pixels, based on the input image, the virtual space information, and a machine learning model; the machine learning model being trained using a plurality of training data each including a learning input image, learning virtual space information, and a learning estimated image; the learning input image being an image having the input number of pixels, based on a target image for processing that shows, from a specified viewpoint, a learning virtual space in which one or more learning objects represented by learning three-dimensional data are arranged, and that has the initial number of pixels; the learning virtual space information being information about the learning virtual space that can be used to determine a pixel value of each pixel of the target image for processing; and the learning estimated image being an image having the estimated number of pixels.

14. A program for causing a computer to function as: an input image acquisition means for acquiring an input image having a number of input pixels equal to or greater than an initial number of pixels, based on a processing target image which shows, from a specified viewpoint, a virtual space in which one or more learning objects represented by three-dimensional data are arranged, and which has a specified initial number of pixels; a virtual space information acquisition means for acquiring virtual space information which is information about the virtual space that can be used to determine pixel values ​​of each pixel of the processing target image; and an estimated image acquisition means for acquiring an estimated image having a number of estimated pixels greater than the input number of pixels, based on the input image, the virtual space information, and a machine learning model, wherein the machine learning model has been trained using a plurality of training data each including a learning input image, learning virtual space information, and a learning estimated image, wherein the learning input image is an image having the input number of pixels, based on a learning processing target image which shows, from a specified viewpoint, a learning virtual space in which one or more learning objects represented by learning three-dimensional data are arranged, and which has the initial number of pixels; the learning virtual space information is information about the learning virtual space that can be used to determine pixel values ​​of each pixel of the learning processing target image; and the learning estimated image is an image having the estimated number of pixels.

Images