Image processing system, image processing method, and program

Abstract

An objective of the present invention is to improve the estimation accuracy of an estimation frame (24) by effectively using virtual space information (27). An image processing system (1) includes at least one processor. The processor acquires an n-th processing target frame (20), acquires an n-th input frame (22) on the basis of the n-th processing target frame (20), acquires n-th virtual space information (27), which is information about a virtual space that can be used to determine the pixel value of each pixel of the n-th processing target frame, specifies an n-th appearance pixel on the basis of n-1-th depth information and n-th depth information, and acquires n-th auxiliary information (30) on the basis of at least the n-th appearance pixel, the n-th virtual space information (27), the n-th cumulative feature information (28), and a machine learning model (422a).

Description

Image processing system, image processing method and program

[0001] The present invention relates to an image processing system, an image processing method, and a program.

[0002] Conventionally, a technique (super-resolution) for estimating a high-quality image based on a low-quality image using a machine learning model has been known (see Non-Patent Document 1 below).

[0003] Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang. Learning a Deep Convolutional Network for Image Super-Resolution, in Proceedings of European Conference on Computer Vision (ECCV), 2014

[0004] The inventors of the present application are considering using a machine learning model with a recursive structure that utilizes information about past frames to achieve super-resolution of moving images such as game screens. The inventors are also considering applying the super-resolution to frames to be processed that show, from a predetermined viewpoint, a virtual space in which one or more objects represented by three-dimensional data are arranged.

[0005] The frame to be processed is generated by rendering the three-dimensional data. The rendering is performed based on information about the virtual space that can be used to determine the pixel value of each pixel in the virtual image (hereinafter referred to as "virtual space information"). The virtual space information includes, for example, information about the viewpoint from which the virtual space is viewed, information about the movement of objects, information about the color and texture of objects, and information about the intensity, color, and illumination direction of light sources.

[0006] The inventors of the present application have been studying ways to improve the estimation accuracy of estimated frames output from machine learning models by using the virtual space information. However, the reliability of the virtual space information is sometimes insufficient, and there is room for improvement in its use.

[0007] An object of the present invention is to provide an image processing system, an image processing method, and a program that improve the estimation accuracy of an estimated frame by effectively utilizing virtual space information.

[0008] An image processing system according to the present invention is an image processing system including at least one processor, wherein the at least one processor acquires an nth (n is a natural number of 2 or more) processing frame showing, from a predetermined viewpoint, a virtual space in which one or more objects represented by three-dimensional data are arranged, acquires an nth input frame based on the nth processing frame, inputs the nth input frame and n-1th auxiliary information relating to features of the first to n-1th input frames to a first machine learning model, thereby acquiring nth cumulative feature information indicating features of the first to nth input frames output from the first machine learning model, and an nth estimated frame, and calculates the nth cumulative feature information for each image of the nth processing frame. The method acquires nth virtual space information, which is information about the virtual space that can be used to determine raw pixel values; acquires n-1th depth information indicating the depth of each pixel of the n-1th frame to be processed and nth depth information indicating the depth of each pixel of the nth frame to be processed; identifies an nth appearance pixel, which is a pixel of the nth frame to be processed that displays all or part of the object that is not displayed in the n-1th frame to be processed, based on the n-1st depth information and the nth depth information; and acquires the nth auxiliary information based on at least the nth appearance pixel, the nth virtual space information, the nth cumulative feature information, and a second machine learning model.

[0009] FIG. 1 is a diagram illustrating an example of a hardware configuration of an image processing system; FIG. 2 is a diagram illustrating an overview of the image processing system; FIG. 3 is a functional block diagram illustrating an example of functions realized by the image processing system; FIG. 4 is a diagram illustrating processing by a rendering unit; FIG. 5 is a diagram illustrating processing by an input frame acquisition unit; FIG. 6 is a flow diagram illustrating an example of the flow of processing executed by the image processing system; FIG. 7 is a functional block diagram illustrating an example of functions realized by an image processing system according to a modified example;

[0010] Hereinafter, an example of an embodiment of an image processing system according to the present invention will be described with reference to the drawings.

[0011] 1 is a diagram showing an example of the hardware configuration of an image processing system 1. The image processing system 1 is a computer such as a game console. As shown in FIG. 1, the image processing system 1 includes a control unit 10, a storage unit 12, a communication unit 14, an operation unit 16, a display unit 18, and an audio output unit 19.

[0012] The control unit 10 includes a program control device such as a CPU that operates according to a program installed in the image processing system 1. The control unit 10 also includes a GPU (Graphics Processing Unit) that draws an image in a frame buffer based on graphics commands and data supplied from the CPU.

[0013] The storage unit 12 includes, for example, a main storage device such as a ROM or a RAM, and an auxiliary storage device such as a HDD or an SSD. The storage unit 12 stores programs executed by the control unit 10. The storage unit 12 stores, for example, game programs (game software) in addition to programs for realizing various functions of the image processing system 1 described below. The storage unit 12 also reserves an area for a frame buffer in which images are drawn by the GPU.

[0014] The communication unit 14 is a communication interface such as an Ethernet (registered trademark) module or a wireless LAN module.

[0015] The operation unit 16 is a user interface such as a keyboard, mouse, or game console controller, and receives operation inputs from the user and outputs signals indicating the contents of the inputs to the control unit 10 .

[0016] The display unit 18 is a display device such as a liquid crystal display or an organic EL display, and displays various images according to instructions from the control unit 10 .

[0017] The audio output unit 19 is, for example, a speaker, and outputs audio represented by audio data generated by the image processing system 1 .

[0018] In addition to the devices described above, the image processing system 1 may also include an optical disc drive that reads optical discs such as DVD-ROMs and Blu-ray (registered trademark) discs, a USB (Universal Serial Bus) port, and the like.

[0019] 2. Overview of Image Processing System Fig. 2 is a diagram showing an overview of the image processing system 1. Here, an example is shown in which the image processing system 1 is used to improve the image quality of gameplay video in a game. Gameplay video is video generated in response to a game program executed by the control unit 10 and user input accepted by the operation unit 16, and is composed of a plurality of still images (frames) that are time-series data. The processes performed by the image processing system 1 are mainly as follows:

[0020] (1) Generation of Processing Frames First, the image processing system 1 generates an image (processing frame) depicting one or more game objects by rendering three-dimensional data representing the game objects as viewed from a predetermined viewpoint. This processing frame is an image having a predetermined number of pixels (initial pixel count) and a predetermined image quality (initial image quality). The processing frame can be considered an image representing the virtual space VS in which the one or more game objects represented by three-dimensional data are placed, from a predetermined viewpoint (see FIG. 4 ). Processing frames are generated at predetermined time intervals. The number of pixels of the processing frame is, for example, 1920 x 1080 (1080p). Each generated processing frame is not displayed directly on the display unit 18, but is temporarily stored in the storage unit 12 for subsequent processing. Note that the following description mainly illustrates processing for the nth processing frame 20_n, but similar processing is also performed for other processing frames (i.e., n = 2, 3, ..., N).

[0021] (2) Acquisition of Input Frame Based on the acquired processing target frame 20_n, the image processing system 1 acquires a frame (input frame) 22_n having a number of pixels (number of input pixels) greater than the initial number of pixels. The number of input pixels is, for example, 3840 × 2160 (4K). Specifically, the processing target frame 20_n is subjected to enlargement and interpolation processing to generate the input frame 22_n.

[0022] It should be noted that although the input frame 22_n has a larger number of pixels than the target frame 20_n, its image quality is not necessarily sufficiently improved. In other words, the image quality of a frame does not simply refer to the number of pixels (high resolution). The image quality of a frame may be evaluated based on, for example, a high signal-to-noise ratio, high spatial frequency reproducibility, high temporal stability (low artifacts and flicker when multiple frames are displayed consecutively), etc., when compared to a reference frame, or a combination of these.

[0023] (3) Obtaining an Estimated Frame The image processing system 1 inputs the input frame 22_n into a machine learning model 200 (first machine learning model), which is a machine learning model, and obtains an estimated frame 24_n. The estimated frame 24_n is an image having the same number of pixels (estimated pixel count) as the number of input pixels and an image quality (estimated image quality) that is equal to or higher than the initial image quality. Here, in addition to the input frame 22_n, the (n-1)th auxiliary information 30_n-1 is input to the machine learning model 200. Details of the (n-1)th auxiliary information 30_n-1 will be described later.

[0024] The machine learning model 200 is a model trained using a plurality of training data sets, each of which includes a training input frame having an input pixel count and a training estimation frame having an estimated pixel count and estimated image quality.

[0025] (4) Acquisition of Cumulative Feature Information The machine learning model 200 has an accumulated feature information output layer 202 that receives the nth input frame 22_n and the (n-1)th auxiliary information 30_n-1 as input and outputs nth accumulated feature information 26_n that indicates the features of the first to nth input frames 22 (see FIG. 2). The image processing system 1 acquires the nth accumulated feature information 26_n. The acquired nth accumulated feature information 26_n is used to generate the nth auxiliary information 30_n.

[0026] The acquired nth cumulative feature information 26_n is also stored in the memory unit 12 and is used to estimate the estimated frame 24_n+1 corresponding to the next frame to be processed (the n+1th frame to be processed) 20_n+1.

[0027] As described above, the image processing system 1 estimates the estimated frame 24 using the accumulated feature information 26, which stores past information, in addition to the input frame 22 corresponding to the current processing target frame 20. This increases the amount of information available for estimation, making it possible to obtain a high-quality estimated frame 24_n.

[0028] (5) Acquisition of Auxiliary Information As described above, the (n-1)th accumulated feature information 26_n-1 is information that indicates the features of the first to (n-1)th input frames 22 (and thus the first to (n-1)th processing target frames 20). If the (n-1)th accumulated feature information 26_n-1, in which information on past processing target frames 20 is accumulated, is used to estimate the nth estimated frame 24_n, the amount of information available for estimation increases, and therefore a high-quality estimated frame 24_n can be obtained.

[0029] However, if there is movement or the like in the displayed game object between the (n-1)th processing frame 20_n-1 and the nth processing frame 20_n, when the nth input frame 22_n and the (n-1)th cumulative feature information 26_n-1 are input as is to the machine learning model 200, a phenomenon (the so-called ghost phenomenon) may occur in which an afterimage of the game object that was displayed in the (n-1)th processing frame 20_n-1 is displayed.

[0030] Therefore, the image processing system 1 obtains the (n-1)th auxiliary information 30_n-1 for the (n-1)th cumulative feature information 26_n-1 based on information obtained during rendering (motion vectors, depth buffer, etc.) (see Figure 2).

[0031] As described above, the processing target frame 20 is generated by rendering three-dimensional data. Rendering is performed based on virtual space information 27, which is information about the virtual space VS that can be used to determine the pixel value of each pixel in the processing target frame 20. The virtual space information 27 includes, for example, information about the viewpoint C from which the virtual space VS is viewed and information about the movement of the game object O. The virtual space information 27 may also include information about the color and texture of the game object, as well as information about the intensity, color, and illumination direction of a light source. The virtual space information 27 can also be considered information that can be used for rendering the processing target frame 20. Note that the virtual space information 27 is not limited to information actually used for rendering the processing target frame 20. In addition, in this embodiment, the virtual space information 27 is described separately from depth information, which will be described later, but the depth information may also be treated as information included in the virtual space information 27.

[0032] Here, the processing target frame 20 obtained as a result of rendering may not contain sufficient information about the virtual space VS, and therefore there is a limit to the accuracy of estimation based only on the processing target frame 20. In other words, although the virtual space information 27 can be used when determining the pixel value of each pixel of the processing target frame 20, the virtual space information 27 itself does not remain in the processing target frame 20. For example, if there are originally C pieces of information about the virtual space VS, then in the process of determining the pixel values ​​(RGB values) of each pixel of the processing target frame 20, the C pieces of information are reduced to three pieces of information (RGB).

[0033] Therefore, the image processing system 1 according to this embodiment employs a configuration in which the nth auxiliary information 30_n is acquired using the nth virtual space information 27_n, which is information about the virtual space VS that can be used to determine the pixel value of each pixel in the nth processing target frame 20_n. This makes it possible to effectively utilize the information about the virtual space VS, thereby enabling an estimated frame 24 of high image quality to be obtained with high accuracy.

[0034] However, there are cases where the reliability of the virtual space information 27 is insufficient, and when such virtual space information 27 is used, it may not be possible to accurately acquire the appearance pixels described below. For example, when a fast-moving object is displayed in the processing target frame 20, the positional accuracy of the appearance pixels generated based on the motion information described below may decrease, and as a result, artifacts may remain in the estimated frame 24. Therefore, the image processing system 1 according to this embodiment employs a configuration in which estimated appearance pixel information 37 described below is generated using the virtual space information 27, and auxiliary information 30, which is information related to past frames, is generated based on the estimated appearance pixel information 37. The functions realized by the image processing system 1 will be described in detail below.

[0035] 3. Functions Realized in the Image Processing System Figure 3 is a functional block diagram showing an example of functions realized in the image processing system 1. As shown in Figure 3, the image processing system 1 realizes a game processing unit 400, a rendering unit 402, a rendering information storage unit 404, a processing target frame acquisition unit 406, a variation information acquisition unit 408, an input frame acquisition unit 410, a machine learning model storage unit 412, an estimated frame acquisition unit 414, a motion information acquisition unit 416, a depth information acquisition unit 418, an appearance pixel identification unit 420, an appearance pixel information acquisition unit 422, and an auxiliary information acquisition unit 424. The game processing unit 400, the rendering unit 402, the processing target frame acquisition unit 406, the variation information acquisition unit 408, the input frame acquisition unit 410, the estimated frame acquisition unit 414, the motion information acquisition unit 416, the depth information acquisition unit 418, the appearance pixel identification unit 420, the appearance pixel information acquisition unit 422, and the auxiliary information acquisition unit 424 are mainly realized by the control unit 10. The rendering information storage unit 404 and the machine learning model storage unit 412 are mainly realized by the storage unit 12. Note that the game processing unit 400, the rendering unit 402, and the rendering information storage unit 404 are functions provided by the game software.

[0036] [Game Processing Unit] The game processing unit 400 executes various processes related to the game. For example, in response to a game program executed by the control unit 10 or user input received by the operation unit 16, the game processing unit 400 executes processes such as placing a game object O in the virtual space VS, operating or moving the game object O, and changing the viewpoint C from which the virtual space VS is viewed (see FIG. 4 ). The game object O is composed of primitives such as polygons represented by three-dimensional data. The three-dimensional data includes geometric information indicating the positions of vertices, topological information indicating how the vertices are connected, and attribute information such as color.

[0037] [Rendering Unit] FIG. 4 is a diagram illustrating the processing of the rendering unit 402. The rendering unit 402 generates first to Nth (N is a natural number equal to or greater than 2) processing frames 20 by rendering (drawing) three-dimensional data representing one or more game objects O as viewed from a predetermined viewpoint C. The processing frames 20 can also be considered images representing a virtual space VS in which one or more game objects O represented by three-dimensional data are placed, from the predetermined viewpoint C. The processing frames 20 have a predetermined initial number of pixels. The rendering unit 402 performs rendering based on the results of various processes executed by the game processing unit 400. Specifically, the rendering unit 402 performs vertex processing (vertex shading) and pixel processing (pixel shading) based on the three-dimensional data representing the game objects O placed in the virtual space VS. The vertex processing includes a coordinate transformation process (perspective projection) from the view coordinate system to the screen coordinate system, and a value related to the change in the viewpoint C is added to the perspective projection matrix (camera matrix) used in the coordinate transformation process, as described below. The rendering unit 402 may perform rendering based on light source information, depth information (depth buffer), texture information, normal information, etc. In addition to the above processes, the rendering unit 402 may also perform processes that apply effects such as depth of field (DoF) and motion blur. The processing of the rendering unit 402 may be set appropriately by a game software developer, etc. Here, the game software developer, etc. may adjust the MIP of the texture according to the estimated number of pixels of the estimated frame 24, etc. This makes it possible to suppress the occurrence of noise such as moiré in the estimated frame 24.

[0038] Here, the rendering unit 402 generates each processing frame 20 by performing rendering such that the viewpoint C varies for each processing frame 20. Here, even if the game processing unit 400 fixes the viewpoint C at a predetermined position, the rendering unit 402 varies the viewpoint C for each processing frame 20. As a result, as shown in FIG. 5 , the position of the displayed game object O varies in each of the processing frames 20_n, 20_n+1, and 20_n+2. In other words, the rendering unit 402 applies jitter when generating each processing frame 20. Specifically, the rendering unit 402 varies the viewpoint C for each processing frame 20 by adding a numerical value corresponding to a fractional pixel size that differs for each processing frame 20 to the perspective projection matrix.

[0039] [Rendering Information Storage Unit] The rendering information storage unit 404 stores information necessary for the rendering process in the rendering unit 402 and information obtained as a result of the rendering process. For example, the rendering information storage unit 404 stores the processing target frame 20. The rendering information storage unit 404 also stores virtual space information 27 and variation information 29. Details of the virtual space information 27 and variation information 29 will be described later.

[0040] [Processing Frame Acquisition Unit] The processing frame acquisition unit 406 acquires the first to Nth processing frames 20. Specifically, the processing frame acquisition unit 406 acquires the first to Nth processing frames 20 stored in the rendering information storage unit 404.

[0041] The variation information acquisition unit 408 acquires first to Nth variation information 29, which is information related to variation of the viewpoint C of each of the first to Nth processing target frames 20 during rendering. The variation information acquisition unit 408 acquires the first to Nth variation information 29 stored in the rendering information storage unit 404.

[0042] [Input Frame Acquisition Unit] The input frame acquisition unit 410 acquires the first to Nth input frames 22 by generating, based on each processing target frame 20, an input frame 22 that corresponds to the processing target frame 20 and has a number of input pixels equal to or greater than the initial number of pixels. In this embodiment, each input frame 22 has a number of input pixels greater than the initial number of pixels. That is, in this embodiment, each input frame 22 is an enlarged image of the processing target frame 20 that corresponds to the input frame 22.

[0043] Specifically, the input frame acquisition unit 410 interpolates pixel values ​​at positions in the processing frame 20 that correspond to the pixels before the fluctuation, based on the fluctuation information 29 and each pixel in each processing frame 20, to generate each input frame 22. Fig. 5 is a diagram illustrating the processing in the input frame acquisition unit 410. Fig. 5 illustrates an example of obtaining the n-th input frame 22_n. For example, as shown in Fig. 5, if the pixel center of a pixel in the input frame 22_n to be obtained is P １，０ In this case, the input frame acquisition unit 410 calculates P １，０ The pixel centers P' of the four pixels closest to ０，０ , P' １，０ , P' ０，１ , P' １，１ Based on the coordinates and pixel values ​​of P １，０ The pixel value of P' is calculated by bilinear interpolation. １，０ is increased by the amount of fluctuation indicated by the fluctuation information 29. １，０ The pixel values ​​of the pixels newly generated by the enlargement process are calculated in the same manner. Note that, in addition to bilinear interpolation, various other well-known interpolation methods such as bicubic interpolation and Lanczos interpolation can also be used as the interpolation method.

[0044] When rendering is performed so that the viewpoint C varies for each processing frame 20, the amount of time-series information increases. However, by using each processing frame 20 obtained in this way (hereinafter referred to as a "variable processing frame") for estimation, an estimated frame 24 with higher image quality can be obtained.

[0045] On the other hand, if the frame to be subjected to variation processing (or an enlarged image thereof) is input directly to the machine learning model 200, the influence of the variation in viewpoint C described above may result in a decrease in the accuracy of estimation.

[0046] Therefore, as described above, in the image processing system 1, pixel values ​​at positions in the processing frame 20 corresponding to each pixel before the fluctuation are obtained by interpolation based on the fluctuation information 29 and each pixel in each processing frame 20, and each input frame 22 is generated and input to the machine learning model 200. This corrects for the influence of fluctuations in the viewpoint C, making it possible to prevent a decrease in estimation accuracy.

[0047] [Machine Learning Model Storage Unit] The machine learning model storage unit 412 stores the machine learning model 200, which is a first machine learning model, and the machine learning model 422a, which is a second machine learning model. Specifically, the machine learning model storage unit 412 stores parameters of the machine learning models (such as the number of convolutional layers, the number of nodes used in each convolutional layer, and the weight of each node).

[0048] [First Machine Learning Model] The machine learning model 200 is a model that estimates the nth estimated frame 24_n based on the nth input frame 22_n. Specifically, the machine learning model 200 is a model that estimates the nth estimated frame 24_n based on the nth input frame 22_n and the nth auxiliary information 30_n. Specifically, the machine learning model 200 is a convolutional neural network (CNN). As the machine learning model 200, for example, a well-known model such as a multi-layered ResNet having a residual connection mechanism or a so-called encoder-decoder U-Net can be used. As the machine learning model 200, the model described in Non-Patent Document 1 may be used.

[0049] The machine learning model 200 is a model trained using a plurality of training data sets, each of which includes a training input frame having an input pixel count and a training estimation frame having an estimated pixel count. The machine learning model 200 includes an accumulated feature information output layer 202, an estimated frame output layer 204, and a convolutional layer 206 (see FIG. 2 ).

[0050] The cumulative feature information output layer 202 receives the nth input frame 22_n and the (n-1)th auxiliary information 30_n-1 as input, and outputs nth cumulative feature information 26_n indicating the features of the first to nth input frames 22_n. The cumulative feature information output layer 202 may be configured, for example, with one or more convolutional layers. The cumulative feature information 26_n-1 is image information (bitmap information) having the same number of pixels as the number of input pixels. The cumulative feature information 26_n-1 can also be considered a feature map indicating the features of the first to n-1th input frames 22.

[0051] The cumulative feature information output layer 202 receives the first input frame 22_1 and given auxiliary information, and outputs first cumulative feature information 26_1. When n=1, there is no previous cumulative feature information 26, so the given feature information prepared in advance is input to the cumulative feature information output layer 202 together with the first input frame 22_1.

[0052] The estimated frame output layer 204 receives the n-th accumulated feature information 26_n and outputs the n-th estimated frame 24_n. The estimated frame output layer 204 may be configured, for example, by one or more convolutional layers, similar to the accumulated feature information output layer 202. Alternatively, the estimated frame output layer 204 may be configured by one or more transposed convolutional layers (deconvolutional layers).

[0053] The convolution layer 206 is a layer that reduces the number of channels of the accumulated feature information 26 while maintaining the number of pixels of the accumulated feature information 26. The convolution layer 206 reduces the dimension of the accumulated feature information 26, thereby reducing calculation costs. The convolution layer 206 is, for example, a convolution layer with a kernel size of 1×1, but is not limited to this.

[0054] [Estimated Frame Acquisition Unit] The estimated frame acquisition unit 414 acquires first to Nth estimated frames 24, each having an estimated pixel number that is greater than the initial pixel number and equal to or greater than the input pixel number, based on the first to Nth input frames 22, the first to Nth auxiliary information 30, and the machine learning model 200. In this embodiment, the estimated frames 24 have the same number of estimated pixels as the input pixel number.

[0055] [Motion Information Acquisition Unit] The motion information acquisition unit 416 acquires n-1th motion information, which is information indicating the amount and direction of motion from the n-1th processing target frame 20_n-1 to the nth processing target frame 20_n. Specifically, the n-1th motion information is image information indicating the amount and direction of motion of each pixel between the n-1th processing target frame 20_n-1 and the nth processing target frame 20_n. The motion information is also called a motion vector. The motion information is information having the same number of pixels as the number of input pixels. Specifically, the motion information acquisition unit 416 acquires original motion information having the same number of pixels as the initial number of pixels, and performs enlargement and interpolation processing on the original motion information to acquire motion information having the same number of pixels as the number of input pixels.

[0056] [Depth Information Acquisition Unit] The depth information acquisition unit 418 acquires n-1th depth information indicating the depth of each pixel in the n-1th processing target frame 20_n-1 and nth depth information indicating the depth of each pixel in the nth processing target frame 20_n. Depth information is also called a depth buffer, Z buffer, or depth buffer. Depth information is information having the same number of pixels as the number of input pixels. Specifically, the depth information acquisition unit 418 acquires original depth information having the same number of pixels as the initial number of pixels, and performs enlargement and interpolation processing on the original depth information to acquire depth information having the same number of pixels as the number of input pixels.

[0057] [Appearance Pixel Identification Unit] Based on the (n-1)th depth information and the (n)th depth information, the appearance pixel identification unit 420 identifies an nth appearance pixel, which is a pixel in the nth input frame 22_n where all or part of the game object O, which is not displayed in the (n-1)th input frame 22_n-1, is displayed. Specifically, the appearance pixel identification unit 420 identifies the nth appearance pixel based on the difference between the (n-1)th depth information and the nth depth information. Note that the appearance pixel identification unit 420 may also identify the nth appearance pixel based on the (n-1)th perspective projection matrix associated with the (n-1)th input frame 22_n-1 and the nth perspective projection matrix associated with the nth input frame 22_n. Alternatively, the appearance pixel identification unit 420 may identify the nth appearance pixel using the (n-1)th motion information.

[0058] The appearance pixel information acquisition unit 422 acquires estimated appearance pixel information 37 based on the input frame 22, the virtual space information 27, appearance pixel information that is image information indicating the positions of appearance pixels, and the machine learning model 422 a. The estimated appearance pixel information 37 is information that is more detailed than the appearance pixel information acquired by the appearance pixel identification unit 420.

[0059] The machine learning model 422a is a model that estimates the estimated occurrence pixel information 37 based on the input frame 22, the virtual space information 27, and the occurrence pixel information. The machine learning model 422a may be, for example, a convolutional neural network (CNN). As the machine learning model 422a, for example, a known model such as a multi-layered ResNet having a residual connection mechanism or a so-called encoder-decoder U-Net can be used.

[0060] 2, the machine learning model 422a has a multi-layer network structure in which a convolutional layer 4221, a layer 4222 of a ReLU function (activation function), etc. are connected, and parameters of filters and functions used in these layers are the targets of learning. Note that each layer, such as the convolutional layer 4221, may be provided with two or more layers.

[0061] The machine learning model 422a may be a model trained using a plurality of training data sets, including, for example, a training input frame having a number of input pixels, training virtual space information (such as training motion information) having the same number of pixels as the number of input pixels, training occurrence pixel information having the same number of pixels as the number of input pixels, and training estimated occurrence pixel information having the same number of pixels as the number of input pixels. The machine learning model 422a may be trained based on the loss between the output when each piece of training information is input and the training estimated occurrence pixel information. Here, the training input frame is an image showing, from a predetermined viewpoint, a training virtual space in which one or more training objects represented by training three-dimensional data are arranged. The training virtual space information is information about the training virtual space that can be used to determine the pixel value of each pixel in the frame to be trained. The training occurrence pixel information is image information indicating the position of the training occurrence pixel.

[0062] The machine learning model 422a may output estimated appearance pixel information 37 having 4K pixels and one channel based on, for example, the input frame 22 (4K, 3ch), motion information (4K, 2ch) which is virtual space information, depth information (4K, 1ch), and appearance pixel information (4K, 1ch). In other words, the appearance pixel information acquisition unit 422 may acquire an output in which the number of pixels is maintained while the number of channels is reduced for the input. This reduces the calculation cost in subsequent processing.

[0063] Auxiliary Information Acquisition Unit The auxiliary information acquisition unit 424 acquires auxiliary information 30 based on the virtual space information 27 , the cumulative feature information 26 , and the estimated appearing pixel information 37 .

[0064] Specifically, the auxiliary information acquisition unit 424 applies motion compensation to the (n-1)th cumulative feature information 26_n-1 based on the (n-1)th motion information, and acquires the (n-1)th auxiliary information 30_n-1 using the nth estimated-appearing pixel information 37_n. Motion compensation refers to, for example, a process of moving a pixel at position x in the (n-1)th cumulative feature information 26_n-1 to position x' when a pixel at position x in the (n-1)th processing target frame 20_n-1 has moved to position x' in the nth processing target frame 20_n. That is, the auxiliary information acquisition unit 424 acquires the (n-1)th auxiliary information 30_n-1 based on the (n-1)th motion information by setting the pixel values ​​of one or more pixels in the (n-1)th cumulative feature information 26_n-1 to pixels at positions moved in accordance with the amount and direction of the pixel's movement.

[0065] If the game object O moves between the nth processing frame 20_n and the n-1th processing frame 20_n-1, and the nth input frame 22_n and the n-1th cumulative feature information 26_n-1 are input to the machine learning model 200 as is when acquiring the nth estimated frame 24_n without taking the object's movement into consideration, a ghost phenomenon may occur in the output nth estimated frame 24_n, in which an afterimage of the game object O that was displayed in the previous frame is displayed. Therefore, as described above, the image processing system 1 applies motion compensation to the n-1th cumulative feature information 26_n-1 based on the n-1th motion information to acquire the n-1th auxiliary information 30_n-1. This makes it possible to suppress the ghost phenomenon.

[0066] Furthermore, the auxiliary information acquisition unit 424 acquires the (n-1)th auxiliary information 30_n-1 based on the nth estimated appearance pixel information 37_n by replacing the pixel value of the nth estimated appearance pixel information 37_n in the (n-1)th accumulated feature information 26_n-1 with a predetermined value. The predetermined value may be a constant value such as 0 (black), or the pixel value of the nth appearance pixel in the nth input frame 22_n.

[0067] 6A and 6B are flow diagrams showing an example of the flow of processing executed in the image processing system 1. The processing shown in Fig. 6A and 6B is executed by the control unit 10 operating in accordance with a program stored in the storage unit 12.

[0068] (1) Processing for n=1 First, the control unit 10 acquires a first processing frame 20_1 (S100). The control unit 10 acquires a first input frame 22_1 based on the first processing frame 20_1 (S102). Furthermore, the control unit 10 inputs the input frame 22_1 and given auxiliary information to the machine learning model 200, and acquires a first estimated frame 24_1 and first cumulative feature information 26_1 (S104).

[0069] (2) Processing when n≧2 The control unit 10 acquires the nth processing target frame 20_n (S106). The control unit 10 acquires the nth input frame 22_n based on the nth processing target frame 20_n (S108).

[0070] Next, the control unit 10 acquires the nth motion information (S110). The control unit 10 also acquires the (n-1)th depth information and the nth depth information (S112), and identifies the nth appearance pixel based on the (n-1)th depth information and the nth depth information (S114). The control unit 10 also acquires the nth estimated appearance pixel information 37_n based on the nth input frame 22, the nth motion information, the nth appearance pixel, and the machine learning model 422a (S116). The control unit 10 acquires the (n-1)th auxiliary information 30_n-1 based on the (n-1)th cumulative feature information 26_n-1, the (n-1)th motion information, and the nth estimated appearance pixel information 37_n (S118).

[0071] Then, the control unit 10 inputs the nth input frame 22_n and the n-1th auxiliary information 30_n-1 into the machine learning model 200, and obtains the nth estimated frame 24_n and the nth cumulative feature information 26_n (S120).

[0072] The control unit 10 determines whether or not the next frame exists (S122), and if it determines that the next frame exists (S122; Y), it increments n to n+1 and repeats the processes of S108 to S120. If it determines that the next frame does not exist (S122; N), the control unit 10 ends this process.

[0073] [5. Modification] Next, a modification of this embodiment will be described with reference to FIG. 7. FIG. 7 is a functional block diagram showing an example of functions implemented in an image processing system of the modification. Note that components having similar functions to those described with reference to FIG. 4 are designated by the same reference numerals, and detailed description thereof will be omitted. In the modification, the auxiliary information acquisition unit 1424 includes a machine learning model 1422a, which is a second machine learning model. The auxiliary information acquisition unit 1424 acquires auxiliary information 30 output from the machine learning model 1422a by inputting the input frame 22, virtual space information 27, occurring pixel information, and cumulative feature information 26 to the machine learning model 1422a.

[0074] The machine learning model 1422a may be a model trained using a plurality of training data including, for example, a training input frame having a number of input pixels, training virtual space information (such as training motion information) having the same number of pixels as the number of input pixels, training appearance pixel information having the same number of pixels as the number of input pixels, and training auxiliary information having the same number of pixels as the number of input pixels. The machine learning model 1422a may be trained based on the loss between the output when each piece of training information is input and the training auxiliary information. Note that the training auxiliary information may be information obtained by applying motion compensation to the training cumulative feature information.

[0075] [6. Summary] According to the image processing system 1 according to the present embodiment and the modified examples described above, the nth estimated frame 24_n is estimated using the (n-1)th accumulated feature information 26_n-1 indicating the features of the first to (n-1)th input frames 22. That is, in addition to the information of the nth processing target frame 20_n, the information of the first to (n-1)th processing target frames 20 can be used for estimation, so that the amount of information available for estimation increases, and a high-quality estimated frame 24_n can be obtained.

[0076] Furthermore, according to the image processing system 1 of this embodiment and its modified examples, the nth estimated frame 24_n is obtained based on the nth input frame 22_n, the nth virtual space information 27_n, and the machine learning model 200, thereby making it possible to effectively utilize information about the virtual space VS, and as a result, to obtain an estimated frame 24 of high image quality with high accuracy.

[0077] Furthermore, according to the image processing system 1 of this embodiment and its modified examples, the reliability of the information on the past frame is improved by generating the auxiliary information 30 based on the input frame 22, the virtual space information 27, and the machine learning model 422a, thereby improving the estimation accuracy of the estimated frame 24.

[0078] The present invention is not limited to the above-described embodiment and modified examples. For example, in the present embodiment and modified examples, the number of input pixels is greater than the initial number of pixels, and the number of input pixels is equal to the estimated number of pixels. However, the number of input pixels may be equal to the initial number of pixels, and the estimated number of pixels may be greater than the number of input pixels. In other words, the input frame 22 does not necessarily have to be an enlarged version of the processing target frame 20.

Claims

1. An image processing system including at least one processor, wherein the at least one processor: acquires an nth (n is a natural number equal to or greater than 2) processing frame showing, from a predetermined viewpoint, a virtual space in which one or more objects represented by three-dimensional data are arranged; acquires an nth input frame based on the nth processing frame; inputs the nth input frame and n-1th auxiliary information related to features of the 1st to nth input frames to a first machine learning model to acquire nth cumulative feature information indicating features of the 1st to nth input frames output from the first machine learning model and an nth estimated frame; acquires nth virtual space information which is information related to the virtual space that can be used to determine a pixel value of each pixel of the nth processing frame; acquires n-1th depth information indicating a depth of each pixel of the n-1th processing frame and nth depth information indicating the depth of each pixel of the nth processing frame; An image processing system which identifies an nth appearance pixel, which is a pixel in the nth processing frame where all or part of the object not displayed in the n-1th processing frame is displayed, among the pixels of the nth processing frame based on the n-1th depth information and the nth depth information; and obtains the nth auxiliary information based on at least the nth appearance pixel, the nth virtual space information, the nth cumulative feature information, and a second machine learning model.

2. The image processing system of claim 1, wherein the at least one processor: obtains the nth estimated occurring pixel information output from the second machine learning model by inputting the nth occurring pixel and the nth virtual space information into the second machine learning model; and obtains the nth auxiliary information based on the nth estimated occurring pixel information and the nth accumulated feature information.

3. The image processing system of claim 2, wherein the at least one processor inputs the nth input frame in addition to the nth occurring pixel and the nth virtual space information to the second machine learning model, thereby obtaining the nth estimated occurring pixel information output from the second machine learning model.

4. The image processing system of claim 1, wherein the at least one processor obtains the nth auxiliary information output from the second machine learning model by inputting the nth occurrence pixel, the nth virtual space information, and the nth cumulative feature information into the second machine learning model.

5. The image processing system of claim 4, wherein the at least one processor obtains the nth auxiliary information output from the second machine learning model by inputting the nth input frame in addition to the nth occurring pixel, the nth virtual space information, and the nth accumulated feature information to the second machine learning model.

6. An image processing system as described in any one of claims 1 to 5, wherein the nth virtual space information includes nth motion information indicating an amount and direction of motion of the object displayed in each pixel of the n-1th input frame from the n-1st input frame to the nth input frame.

7. A processor acquires an nth (n is a natural number equal to or greater than 2) processing frame showing, from a specific viewpoint, a virtual space in which one or more objects represented by three-dimensional data are arranged; acquires an nth input frame based on the nth processing frame; inputs the nth input frame and n-1th auxiliary information related to features of the 1st to nth input frames into a first machine learning model to acquire nth cumulative feature information indicating features of the 1st to nth input frames output from the first machine learning model and an nth estimated frame; acquires nth virtual space information which is information related to the virtual space that can be used to determine pixel values ​​of each pixel of the nth processing frame; acquires n-1th depth information indicating the depth of each pixel of the n-1th processing frame and nth depth information indicating the depth of each pixel of the nth processing frame; An image processing method comprising: identifying an nth appearance pixel, which is a pixel in the nth processing frame where all or part of the object not displayed in the n-1th processing frame is displayed, among the pixels of the nth processing frame based on the n-1th depth information and the nth depth information; and acquiring the nth auxiliary information based on at least the nth appearance pixel, the nth virtual space information, the nth cumulative feature information, and a second machine learning model.

8. A means for acquiring an nth (n is a natural number equal to or greater than 2) processing frame showing, from a predetermined viewpoint, a virtual space in which one or more objects represented by three-dimensional data are arranged; a means for acquiring an nth input frame based on the nth processing frame; a means for acquiring nth cumulative feature information showing features of the first to nth input frames output from a first machine learning model by inputting the nth input frame and n-1th auxiliary information related to features of the first to n-1th input frames into the first machine learning model, and an nth estimated frame; a means for acquiring nth virtual space information which is information related to the virtual space usable for determining a pixel value of each pixel of the nth processing frame; a means for acquiring n-1th depth information showing a depth of each pixel of the n-1th processing frame and nth depth information showing the depth of each pixel of the nth processing frame; A program that causes a computer to function as: a means for identifying an nth appearance pixel, which is a pixel in which all or part of the object not displayed in the n-1th processing frame is displayed, among the pixels of the nth processing frame based on the n-1th depth information and the nth depth information; and a means for acquiring the nth auxiliary information based on at least the nth appearance pixel, the nth virtual space information, the nth cumulative feature information, and a second machine learning model.

Images