Image processing system, image processing method, and program

Abstract

An estimation frame acquisition unit (314) inputs intermediate frames (22) to a machine learning model (200) and acquires first to Nth estimation frames (24), each of which has an estimated pixel count that is greater than an inputted pixel count and equal to or greater than an intermediate pixel count. The machine learning model (200) includes: a cumulative feature information output layer (202) to which an nth intermediate frame (22_n) (n = 2, 3, ..., N) and n–1th auxiliary information (28_n–1) that is based on n–1th cumulative feature information (26_n–1) indicating the features of first to n–1th intermediate frames (22) are inputted, and which outputs nth cumulative feature information (26_n) indicating the features of the first to nth intermediate frames (22); and an estimation frame output layer (204) to which the nth cumulative feature information (26_n) is inputted, and which outputs a nth estimation frame (24_n). The machine learning model (200) is trained using a plurality of training data, each of which includes a training intermediate frame that has the intermediate pixel count and is generated on the basis of a training input frame having the inputted pixel count, and a training estimation frame having the estimated pixel count.

Description

Image processing system, image processing method and program

[0001] The present invention relates to an image processing system, an image processing method, and a program.

[0002] Conventionally, a technique (super-resolution) for estimating a high-quality still image based on a low-quality still image using a machine learning model is known (see Non-Patent Document 1 below).

[0003] Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang. Learning a Deep Convolutional Network for Image Super Resolution, in Proceedings of European Conference on Computer Vision (ECCV), 2014

[0004] The inventors of the present application are considering applying the above-mentioned super-resolution to moving images such as game screens. Here, in super-resolution of moving images, it is believed that by taking into account not only information about each frame to be processed but also information about past frames of the frame, it is possible to estimate moving images with higher image quality. However, as described above, conventional super-resolution is targeted at still images, and therefore, even if this technology is applied directly to moving images, information about past frames is not sufficiently taken into account when estimating high-quality moving images.

[0005] An object of the present invention is to provide an image processing system, an image processing method, and a program that make it possible to estimate a high-quality moving image based on a low-quality moving image by utilizing information on past frames.

[0006] An image processing system according to the present invention is an image processing system including at least one processor, wherein the at least one processor acquires first to Nth input frames (N is a natural number equal to or greater than 2) each having a predetermined number of input pixels, and generates intermediate frames corresponding to the input frames and having an intermediate pixel number equal to or greater than the number of input pixels based on the input frames, thereby acquiring first to Nth intermediate frames, inputting each intermediate frame to a machine learning model, and acquiring first to Nth estimated frames each having an estimated pixel number greater than the number of input pixels and equal to or greater than the intermediate pixel number, and The learning model includes an accumulated feature information output layer that receives the nth intermediate frame (n = 2, 3, ..., N) and n-1th auxiliary information based on the n-1th cumulative feature information indicating the features of the 1st to n-1th intermediate frames, and outputs the nth cumulative feature information indicating the features of the 1st to nth intermediate frames, and an estimated frame output layer that receives the nth cumulative feature information and outputs the nth estimated frame, and is learned using a plurality of training data, each of which includes a learning intermediate frame having the intermediate number of pixels generated based on a learning input frame having the input number of pixels, and a learning estimated frame having the estimated number of pixels.

[0007] FIG. 1 is a diagram illustrating an example of a hardware configuration of an image processing system; FIG. 2 is a diagram illustrating an overview of the image processing system; FIG. 3 is a diagram illustrating a schematic diagram of processing of the image processing system; FIG. 4 is a functional block diagram illustrating an example of functions realized by the image processing system; FIG. 5 is a diagram illustrating processing of a rendering unit; FIG. 6 is a diagram illustrating processing in an intermediate frame acquisition unit; and FIG. 7 is a flow diagram illustrating an example of a flow of processing executed in the image processing system.

[0008] Hereinafter, an example of an embodiment of an image processing system according to the present invention will be described with reference to the drawings.

[0009] 1 is a diagram showing an example of the hardware configuration of an image processing system 1. The image processing system 1 is a computer such as a game console. As shown in FIG. 1, the image processing system 1 includes a control unit 10, a storage unit 12, a communication unit 14, an operation unit 16, a display unit 18, and an audio output unit 19.

[0010] The control unit 10 includes a program control device such as a CPU that operates according to a program installed in the image processing system 1. The control unit 10 also includes a GPU (Graphics Processing Unit) that draws an image in a frame buffer based on graphics commands and data supplied from the CPU.

[0011] The storage unit 12 includes, for example, a main storage device such as a ROM or a RAM, and an auxiliary storage device such as a HDD or an SSD. The storage unit 12 stores programs executed by the control unit 10. The storage unit 12 stores, for example, a game program (game software) in addition to programs for realizing various functions of the image processing system 1, which will be described later. The storage unit 12 also reserves an area for a frame buffer in which images are drawn by the GPU.

[0012] The communication unit 14 is a communication interface such as an Ethernet (registered trademark) module or a wireless LAN module.

[0013] The operation unit 16 is a user interface such as a keyboard, mouse, or game console controller, and receives operation inputs from the user and outputs signals indicating the contents of the inputs to the control unit 10 .

[0014] The display unit 18 is a display device such as a liquid crystal display or an organic EL display, and displays various images according to instructions from the control unit 10 .

[0015] The audio output unit 19 is, for example, a speaker, and outputs audio represented by audio data generated by the image processing system 1 .

[0016] In addition to the devices described above, the image processing system 1 may also include an optical disc drive that reads optical discs such as DVD-ROMs and Blu-ray (registered trademark) discs, a USB (Universal Serial Bus) port, and the like.

[0017] 2. Overview of the Image Processing System Fig. 2 is a diagram showing an overview of the image processing system 1. Fig. 3 is a diagram showing the processing of the image processing system 1. In this embodiment, an example is shown in which the image processing system 1 is used to improve the image quality of gameplay video in a game. The gameplay video is a video generated in response to a game program executed by the control unit 10 or user input accepted by the operation unit 16, and is composed of a plurality of still images (frames) that are time-series data. The processing performed by the image processing system 1 is mainly as follows:

[0018] (1) Generation of Input Frames First, the image processing system 1 generates an image (input frame) depicting one or more game objects by rendering three-dimensional data representing the game objects as viewed from a predetermined viewpoint. This input frame is an image having a predetermined number of pixels (input pixel count) and a predetermined image quality (input image quality) (see FIG. 3 ). The input frame is generated at predetermined time intervals. The number of pixels of the input frame is, for example, 1920 x 1080 (1080p). Each generated input frame is not displayed directly on the display unit 18, but is temporarily stored in the storage unit 12 for subsequent processing. Note that the following description mainly illustrates processing for the nth (nth) input frame 20_n, but similar processing is also performed for other input frames (i.e., n = 2, 3, ..., N).

[0019] (2) Acquisition of Intermediate Frame Based on the acquired input frame 20_n, the image processing system 1 acquires a frame (intermediate frame) 22_n having a number of pixels (intermediate pixel number) greater than the number of input pixels. The number of intermediate pixels is, for example, 3840 x 2160 (4K). Specifically, the input frame 20_n is subjected to enlargement and interpolation processing to generate the intermediate frame 22_n (see FIG. 3 ).

[0020] It should be noted that although the intermediate frame 22_n has a larger number of pixels than the input frame 20_n, its image quality is not necessarily improved sufficiently. In other words, the image quality of a frame does not simply refer to the number of pixels (high resolution). The image quality of a frame may be evaluated based on, for example, a higher signal-to-noise ratio, a higher reproducibility of spatial frequencies, or a higher temporal stability (fewer artifacts or flicker when multiple frames are displayed consecutively) compared to a reference frame, or a combination of these.

[0021] (3) Obtaining an Estimated Frame The image processing system 1 inputs the intermediate frame 22_n to the machine learning model 200 and obtains an estimated frame 24_n. The estimated frame 24_n is an image having the same number of pixels (estimated pixel number) as the number of intermediate pixels and image quality (estimated image quality) that is equal to or higher than the input image quality (see FIG. 3 ).

[0022] Here, in addition to the intermediate frame 22_n, the (n-1)th auxiliary information 28_n-1 is input to the machine learning model 200 (see FIGS. 2 and 3). The auxiliary information 28_n-1 is information based on the (n-1)th cumulative feature information 26_n-1 that indicates the features of the first to (n-1)th intermediate frames 22. Details of the cumulative feature information 26 and the auxiliary information 28 will be described later.

[0023] The machine learning model 200 is a model trained using a plurality of training data sets, each of which includes a training intermediate frame having an intermediate number of pixels generated based on a training input frame having an input pixel number and input image quality, and a training estimated frame having an estimated pixel number and estimated image quality. Details of the machine learning model 200 will be described later.

[0024] (4) Acquisition of Cumulative Feature Information The machine learning model 200 has a cumulative feature information output layer 202 that receives the intermediate frame 22_n and auxiliary information 28_n-1 and outputs n-th cumulative feature information 26_n that indicates the features of the first to n-th intermediate frames 22 (see FIG. 2). The image processing system 1 acquires the n-th cumulative feature information 26_n.

[0025] The acquired n-th cumulative feature information 26_n is input to the estimated frame output layer 204, which outputs the n-th estimated frame 24_n (see FIG. 2 ). The acquired n-th cumulative feature information 26_n is also stored in the storage unit 12 and is used to estimate an estimated frame 24_n+1 corresponding to the next input frame (the (n+1)th input frame) 20_n+1.

[0026] (5) Acquisition of Auxiliary Information As described above, the (n-1)th accumulated feature information 26_n-1 is information that indicates the features of the first to (n-1)th intermediate frames 22 (and thus the first to (n-1)th input frames 20). If the accumulated feature information 26_n-1, which accumulates information on past input frames 20, is used to estimate the nth estimated frame 24_n, the amount of information available for estimation increases, making it possible to obtain a high-quality estimated frame 24_n.

[0027] However, if there is movement or the like in the displayed game object between the (n-1)th input frame 20_n-1 and the nth input frame 20_n, when the nth intermediate frame 22_n and the cumulative feature information 26_n-1 are input as is to the machine learning model 200, a phenomenon (the so-called ghost phenomenon) may occur in which an afterimage of the game object that was displayed in the (n-1)th input frame 20_n-1 is displayed.

[0028] Therefore, the image processing system 1 acquires the (n-1)th auxiliary information 28_n-1 by applying various corrections (described later) to the cumulative feature information 26_n-1 based on information obtained during rendering (motion vectors, depth buffers, etc.) (see FIGS. 2 and 3). As described above, the acquired (n-1)th auxiliary information 28_n-1 is input to the machine learning model 200 together with the nth intermediate frame 22_n, and is used to estimate the nth estimated frame 24_n.

[0029] As described above, the image processing system 1 according to this embodiment estimates the estimated frame 24 using auxiliary information 28, which stores past information, in addition to the intermediate frame 22 corresponding to the current input frame 20. This increases the amount of information available for estimation, making it possible to obtain a high-quality estimated frame 24_n. The image processing system 1 will now be described in detail.

[0030] 4 is a functional block diagram showing an example of functions realized by the image processing system 1. As shown in FIG. 4, the image processing system 1 realizes a game processing unit 300, a rendering unit 302, a rendering information storage unit 304, an input frame acquisition unit 306, a variation information acquisition unit 308, an intermediate frame acquisition unit 310, a machine learning model storage unit 312, an estimated frame acquisition unit 314, a motion information acquisition unit 316, a depth information acquisition unit 318, an appearance pixel identification unit 320, and an auxiliary information acquisition unit 322. The game processing unit 300, the rendering unit 302, the input frame acquisition unit 306, the variation information acquisition unit 308, the intermediate frame acquisition unit 310, the estimated frame acquisition unit 314, the motion information acquisition unit 316, the depth information acquisition unit 318, the appearance pixel identification unit 320, and the auxiliary information acquisition unit 322 are mainly realized by the control unit 10. The rendering information storage unit 304 and the machine learning model storage unit 312 are mainly realized by the storage unit 12. Note that the game processing unit 300, the rendering unit 302, and the rendering information storage unit 304 are functions provided by the game software.

[0031] [Game Processing Unit] The game processing unit 300 executes various processes related to the game. For example, in response to a game program executed by the control unit 10 or user input received by the operation unit 16, the game processing unit 300 executes processes such as placing a game object O in a virtual three-dimensional space VS, operating or moving the game object O, and changing the viewpoint C from which the virtual three-dimensional space VS is viewed (see FIG. 5 ). The game object O is composed of primitives such as polygons represented by three-dimensional data. The three-dimensional data includes geometric information indicating the positions of vertices, topological information indicating how the vertices are connected, and attribute information such as color.

[0032] [Rendering Unit] FIG. 5 is a diagram illustrating the processing of the rendering unit 302. The rendering unit 302 generates first to Nth (N is a natural number equal to or greater than 2) input frames 20 by rendering (drawing) three-dimensional data representing one or more game objects O viewed from a predetermined viewpoint C. The rendering unit 302 performs rendering based on the results of various processes executed by the game processing unit 300. Specifically, the rendering unit 302 performs vertex processing (vertex shading) and pixel processing (pixel shading) based on three-dimensional data representing game objects O placed in the virtual three-dimensional space VS. The vertex processing includes a coordinate transformation process (perspective projection) from the view coordinate system to the screen coordinate system, and a numerical value related to a change in the viewpoint C is added to the perspective projection matrix (camera matrix) used in the coordinate transformation process, as described below. The rendering unit 302 may perform rendering based on light source information, depth information (depth buffer), texture information, normal information, and the like. In addition to the above processes, the rendering unit 302 may also perform processes that apply effects such as depth of field (DoF) and motion blur. The processes performed by the rendering unit 302 may be set as appropriate by game software developers, etc. Here, the game software developers, etc. may adjust the MIP of the texture according to the estimated number of pixels of the estimated frame 24, etc. This makes it possible to suppress the occurrence of noise such as moiré in the estimated frame 24.

[0033] Here, the rendering unit 302 generates each input frame 20 by performing rendering such that the viewpoint C varies for each input frame 20. Here, even if the game processing unit 300 fixes the viewpoint C at a predetermined position, the rendering unit 302 varies the viewpoint C for each input frame 20. As a result, as shown in FIG. 5 , the position of the displayed game object O varies in each of input frames 20_n, 20_n+1, and 20_n+2. In other words, the rendering unit 302 applies jitter when generating each input frame 20. Specifically, the rendering unit 302 varies the viewpoint C for each input frame 20 by adding a numerical value corresponding to a sub-pixel size that differs for each input frame 20 to the perspective projection matrix. The rendering unit 302 varies the viewpoint C for each input frame 20 according to a predetermined rule. For example, a Halton sequence can be used as such a rule.

[0034] [Rendering Information Storage Unit] The rendering information storage unit 304 stores information necessary for the rendering process in the rendering unit 302 and information obtained as a result of the rendering process. For example, the rendering information storage unit 304 stores the input frame 20. The rendering information storage unit 304 also stores variation information, motion information, and depth information. Details of the variation information, motion information, and depth information will be described later. In addition, the rendering information storage unit 304 may store parameters used for coordinate transformation, light source information, texture information, normal information, etc.

[0035] [Input Frame Acquisition Unit] The input frame acquisition unit 306 acquires the first to Nth input frames 20. Specifically, the input frame acquisition unit 306 acquires the first to Nth input frames 20 stored in the rendering information storage unit 304.

[0036] [Variation Information Acquisition Unit] The variation information acquisition unit 308 acquires variation information. The variation information acquisition unit 308 acquires variation information stored in the rendering information storage unit 304. The variation information is information related to variation of the viewpoint C for each input frame 20 during rendering. Specifically, the variation information is information indicating the amount of variation of the viewpoint C between before and after the variation. The information indicating the amount of variation can also be said to be a variation vector indicating the direction and distance of the variation. For example, since the above-mentioned Halton sequence includes information indicating the amount of variation of the viewpoint C, this information may be used as the variation information.

[0037] [Intermediate Frame Acquisition Unit] The intermediate frame acquisition unit 310 acquires first to Nth intermediate frames 22, respectively, based on each input frame 20 by generating an intermediate frame 22 that corresponds to the input frame 20 and has a number of intermediate pixels equal to or greater than the number of input pixels. In this embodiment, each intermediate frame 22 has a number of intermediate pixels greater than the number of input pixels. That is, in this embodiment, each intermediate frame 22 is an enlarged image of the input frame 20 corresponding to that intermediate frame 22.

[0038] Specifically, the intermediate frame acquisition unit 310 interpolates pixel values ​​at positions in the input frame 20 corresponding to the pixels before the fluctuation based on the fluctuation information and each pixel of each input frame 20, and generates each intermediate frame 22. Fig. 6 is a diagram for explaining the processing in the intermediate frame acquisition unit 310. Fig. 6 illustrates an example of obtaining the n-th intermediate frame 22_n. For example, as shown in Fig. 6, if the pixel center of a pixel in the intermediate frame 22_n to be obtained is P １，０ Then, the intermediate frame acquisition unit 310 calculates P １，０ The pixel centers P' of the four pixels closest to ０，０ , P' １，０ , P' ０，１ , P' １，１ Based on the coordinates and pixel values ​​of P １，０ The pixel value of P' is calculated by bilinear interpolation. １，０ is the amount of fluctuation indicated by the fluctuation information. １，０The pixel values ​​of the pixels newly generated by the enlargement process are calculated in a similar manner. Note that, in addition to bilinear interpolation, various other well-known interpolation methods such as bicubic interpolation and Lanczos interpolation can also be used as the interpolation method.

[0039] When rendering is performed so that the viewpoint C varies for each input frame 20, the amount of time-series information increases. However, by using each input frame 20 obtained in this way (hereinafter referred to as a "variable input frame") for estimation, an estimated frame 24 with higher image quality can be obtained.

[0040] On the other hand, if the varying input frame (or an enlarged image thereof) is input directly to the machine learning model 200, the influence of the variation in viewpoint C described above may result in a decrease in the accuracy of estimation.

[0041] Therefore, as described above, in the image processing system 1, pixel values ​​at positions in the input frame 20 corresponding to the pixels before the fluctuation are obtained by interpolation based on the fluctuation information and each pixel of the input frame 20, and each intermediate frame 22 is generated and input to the machine learning model 200. This corrects the influence of fluctuations in the viewpoint C, thereby preventing a decrease in the accuracy of estimation.

[0042] [Machine Learning Model] The machine learning model 200 is a model that estimates the nth estimated frame 24_n based on the nth intermediate frame 22_n. Specifically, the machine learning model 200 is a model that estimates the nth estimated frame 24_n based on the nth intermediate frame 22_n and the n-1th auxiliary information 28_n-1. Specifically, the machine learning model 200 is a convolutional neural network (CNN). As the machine learning model 200, for example, a well-known model such as a multi-layered ResNet having a residual connection mechanism or a so-called encoder-decoder U-Net can be used. The model described in Non-Patent Document 1 may also be used as the machine learning model 200.

[0043] The machine learning model 200 is a model trained using a plurality of training data sets, each of which includes a training intermediate frame having an intermediate number of pixels generated based on a training input frame having an input pixel number, and a training estimation frame having an estimated pixel number. Various known techniques, such as backpropagation, can be used to train the machine learning model 200.

[0044] Specifically, the machine learning model 200 includes an accumulated feature information output layer 202, an estimated frame output layer 204, and a convolution layer 206 (see FIG. 2).

[0045] The cumulative feature information output layer 202 receives the nth intermediate frame 22_n and the n-1th auxiliary information 28_n-1 based on the n-1th cumulative feature information 26_n-1 indicating the features of the first to n-1th intermediate frames 22, and outputs the nth cumulative feature information 26_n indicating the features of the first to n-th intermediate frames 22_n. The cumulative feature information output layer 202 may be composed of, for example, one or more convolutional layers. The cumulative feature information 26_n-1 is image information (bitmap information) having the same number of pixels as the number of intermediate pixels. The cumulative feature information 26_n-1 can also be considered a feature map indicating the features of the first to n-1th intermediate frames 22.

[0046] The cumulative feature information output layer 202 receives the first intermediate frame 22_1 and given auxiliary information, and outputs first cumulative feature information 26_1. When n=1, there is no previous cumulative feature information 26 or auxiliary information 28, so the given auxiliary information prepared in advance is input to the cumulative feature information output layer 202 together with the first intermediate frame 22_1.

[0047] The estimated frame output layer 204 receives the n-th accumulated feature information 26_n and outputs the n-th estimated frame 24_n. The estimated frame output layer 204 may be configured, for example, by one or more convolutional layers, similar to the accumulated feature information output layer 202. Alternatively, the estimated frame output layer 204 may be configured by one or more transposed convolutional layers (deconvolutional layers).

[0048] The convolution layer 206 is a layer that reduces the number of channels of the accumulated feature information 26 while maintaining the number of pixels of the accumulated feature information 26. The accumulated feature information 26 output from the convolution layer 206 is subjected to processing in the auxiliary information acquisition unit 322. The convolution layer 206 reduces the dimensions of the accumulated feature information 26, thereby reducing calculation costs. The convolution layer 206 is, for example, a convolution layer with a kernel size of 1×1, but is not limited to this.

[0049] [Machine Learning Model Storage Unit] The machine learning model storage unit 312 stores the machine learning model 200. Specifically, the machine learning model storage unit 312 stores parameters of the machine learning model 200 (such as the number of convolutional layers, the number of nodes used in each convolutional layer, and the weight of each node).

[0050] [Estimated Frame Acquisition Unit] The estimated frame acquisition unit 314 inputs each intermediate frame 22 into the machine learning model 200 to acquire first to Nth estimated frames 24, each having an estimated pixel count greater than the input pixel count and equal to or greater than the intermediate pixel count. In this embodiment, the estimated frame 24 has an estimated pixel count equal to the intermediate pixel count. More specifically, the estimated frame acquisition unit 314 inputs the nth intermediate frame 22_n and the (n-1)th auxiliary information 28_n-1 into the machine learning model 200 to acquire the nth estimated frame 24_n.

[0051] [Motion Information Acquisition Unit] The motion information acquisition unit 316 acquires n-1th motion information, which is information indicating the amount and direction of motion from the n-1th input frame 20_n-1 to the nth input frame 20_n. Specifically, the n-1th motion information has the same number of pixels as the number of intermediate pixels, and is image information (bitmap format information) indicating the amount and direction of motion of each pixel between the n-1th input frame 20_n-1 and the nth input frame 20_n. The motion information is also called a motion vector. Specifically, the motion information acquisition unit 316 acquires original motion information having the same number of pixels as the number of input pixels, and performs enlargement and interpolation processing on the original motion information to acquire motion information having the same number of pixels as the number of intermediate pixels.

[0052] [Depth Information Acquisition Unit] The depth information acquisition unit 318 acquires n-1th depth information indicating the depth of each pixel in the n-1th input frame 20_n-1 and nth depth information indicating the depth of each pixel in the nth input frame 20_n. Specifically, the depth information is image information having the same number of pixels as the intermediate pixel count (bitmap format information). The depth information is also called a depth buffer, Z buffer, or depth buffer. Specifically, the depth information acquisition unit 318 acquires original depth information having the same number of pixels as the input pixel count, and performs enlargement and interpolation processing on the original depth information to acquire depth information having the same number of pixels as the intermediate pixel count.

[0053] [Appearance Pixel Identification Unit] Based on the (n-1)th depth information and the (n)th depth information, the appearance pixel identification unit 320 identifies the nth appearance pixel 222_n, which is a pixel in the nth intermediate frame 22_n where all or part of the game object O, which is not displayed in the (n-1)th intermediate frame 22_n-1, is displayed (see FIG. 3 ). Specifically, the appearance pixel identification unit 320 identifies the nth appearance pixel 222_n based on the difference between the (n-1)th depth information and the nth depth information. Note that the appearance pixel identification unit 320 may also identify the nth appearance pixel 222_n based on the (n-1)th perspective projection matrix associated with the (n-1)th intermediate frame 22_n-1 and the nth perspective projection matrix associated with the nth intermediate frame 22_n. Alternatively, the appearance pixel identification unit 320 may identify the nth appearance pixel 222_n using the (n-1)th motion information. More specifically, the appearance pixel specifying section 320 specifies the nth appearance pixel 222_n and generates nth appearance pixel information, which is image information indicating the position of the nth appearance pixel 222_n.

[0054] [Auxiliary Information Acquisition Unit] The auxiliary information acquisition unit 322 acquires the (n-1)th auxiliary information 28_n-1 by applying motion compensation to the (n-1)th cumulative feature information 26_n-1 based on the (n-1)th motion information. Motion compensation refers to, for example, a process of moving a pixel at position x in the (n-1)th cumulative feature information 26_n to position x' in the nth intermediate frame 22_n, assuming that the pixel at position x in the (n-1)th intermediate frame 22_n-1 has moved to position x' (see FIG. 3 ). That is, the auxiliary information acquisition unit 322 acquires the (n-1)th auxiliary information 28_n-1 based on the (n-1)th motion information by setting the pixel values ​​of one or more pixels in the (n-1)th cumulative feature information 26_n-1 to pixels at positions moved in accordance with the amount and direction of the pixel's movement.

[0055] If there is movement of the game object O between the nth input frame 20_n and the n-1th input frame 20_n-1, if the nth intermediate frame 22_n and the n-1th cumulative feature information 26_n-1 are input directly into the machine learning model 200 when obtaining the nth estimated frame 24_n, a ghost phenomenon may occur in which an afterimage of the game object O that was displayed in the nth intermediate frame 22_n is displayed in the output nth estimated frame 24_n.

[0056] Therefore, in the image processing system 1, as described above, motion compensation is applied to the (n-1)th accumulated feature information 26_n-1 based on the (n-1)th motion information to obtain the (n-1)th auxiliary information 28_n-1, and when obtaining the nth estimated frame 24_n, this (n-1)th auxiliary information 28_n-1 is input to the machine learning model 200. This makes it possible to suppress the above-described ghost phenomenon.

[0057] Furthermore, the auxiliary information acquisition unit 322 acquires the (n-1)th auxiliary information 28_n-1 by replacing the pixel value of the nth appearance pixel 222_n in the (n-1)th cumulative feature information 26_n-1 with a predetermined value. Specifically, the auxiliary information acquisition unit 322 acquires the (n-1)th auxiliary information 28_n-1 by replacing the pixel value of the nth appearance pixel 222_n in the (n-1)th cumulative feature information 26_n-1 with a predetermined value based on the nth appearance pixel information. The predetermined value may be a constant value such as 0 (black), or may be the pixel value of the nth appearance pixel 222_n in the nth intermediate frame 22_n.

[0058] If all or part of a game object O that is not displayed in the n-1th input frame 20_n-1 is displayed in the nth input frame 20_n, and the nth intermediate frame 22_n and the n-1th cumulative feature information 26_n-1 are input directly into the machine learning model 200 when obtaining the nth estimated frame 24_n, the aforementioned ghost phenomenon may occur in the output nth estimated frame 24_n.

[0059] Therefore, as described above, the image processing system 1 identifies the nth appearance pixel 222_n, which is a pixel in the nth intermediate frame 22_n where all or part of the game object O that is not displayed in the (n-1)th intermediate frame 22_n-1 is displayed, and then replaces the pixel value of the nth appearance pixel 222_n in the (n-1)th cumulative feature information 26_n-1 with a predetermined value to obtain the (n-1)th auxiliary information 28_n-1. This makes it possible to suppress the ghost phenomenon described above.

[0060] 7 is a flow diagram showing an example of the flow of processing executed in the image processing system 1. The processing shown in FIG. 7 is executed by the control unit 10 operating in accordance with a program stored in the storage unit 12.

[0061] (1) Processing for n=1 First, the control unit 10 acquires a first input frame 20_1 (S700). The control unit 10 acquires a first intermediate frame 22_1 based on the first input frame 20_1 (S702). Then, the control unit 10 inputs the first intermediate frame 22_1 and given auxiliary information to the machine learning model 200, and acquires a first estimated frame 24_1 and first cumulative feature information 26_1 (S704).

[0062] (2) Processing when n≧2 The control unit 10 acquires the n-th input frame 20_n (S706).The control unit 10 acquires the n-th intermediate frame 22_n based on the n-th input frame 20_n (S708).

[0063] Next, the control unit 10 acquires the (n-1)th motion information (S710). The control unit 10 also acquires the (n-1)th depth information and the nth depth information (S712) and identifies the nth appearance pixel 222_n based on the (n-1)th depth information and the nth depth information (S714). The control unit 10 acquires the (n-1)th auxiliary information 28_n-1 based on the (n-1)th cumulative feature information 26_n-1, the (n-1)th motion information, and the nth appearance pixel 222_n (S716). The control unit 10 then inputs the nth intermediate frame 22_n and the (n-1)th auxiliary information 28_n-1 into the machine learning model 200, and acquires the nth estimated frame 24_n and the nth cumulative feature information 26_n (S718). The control unit 10 determines whether or not a next frame exists (S720), and if it determines that a next frame exists (S720; Y), it increments n to n+1 and repeats the processes of S706 to S718. If the control unit 10 determines that a next frame does not exist (S720; N), it ends this process. Note that if it determines that a next frame does not exist (S720; N), the control unit 10 may cause the display unit 18 to display the first to Nth estimated frames 24 as they are.

[0064] [5. Summary] According to the image processing system 1 according to the present embodiment described above, the nth estimated frame 24_n is estimated using the (n-1)th accumulated feature information 26_n-1 indicating the features of the first to (n-1)th intermediate frames 22. That is, in addition to the information of the nth input frame 20_n, the information of the first to (n-1)th input frames 20 can be used for estimation, increasing the amount of information available for estimation and enabling the acquisition of a high-quality estimated frame 24_n.

[0065] It should be noted that the present invention is not limited to the above-described embodiment. Furthermore, the specific character strings and numerical values ​​described above and the specific character strings and numerical values ​​in the drawings are merely examples, and the present invention is not limited to these character strings and numerical values.

[0066] For example, in the present embodiment, the number of intermediate pixels is greater than the number of input pixels, and the number of intermediate pixels is equal to the number of estimated pixels. However, the number of intermediate pixels may be equal to the number of input pixels, and the number of estimated pixels may be greater than the number of intermediate pixels. In other words, the intermediate frame 22 does not necessarily have to be an enlarged version of the input frame 20.

[0067] [6. Supplementary Note] (1) An image processing system including at least one processor, wherein the at least one processor: acquires first to Nth (N is a natural number equal to or greater than 2) input frames each having a predetermined number of input pixels; generates, based on each input frame, an intermediate frame corresponding to the input frame and having an intermediate pixel number equal to or greater than the input pixel number, thereby acquiring first to Nth intermediate frames; inputs each intermediate frame to a machine learning model to acquire first to Nth estimated frames each having an estimated pixel number greater than the input pixel number and equal to or greater than the intermediate pixel number; the machine learning model includes: an accumulated feature information output layer that receives the nth intermediate frame (n=2, 3, ..., N) and n-1th auxiliary information based on n-1th accumulated feature information indicating features of the first to n-1th intermediate frames, and outputs the nth accumulated feature information indicating features of the first to nth intermediate frames; and an estimated frame output layer that receives the nth accumulated feature information, and outputs the nth estimated frame; An image processing system trained using a plurality of training data, each of which includes a training intermediate frame having the intermediate number of pixels generated based on a training input frame having the input pixel number, and a training estimated frame having the estimated number of pixels. (2) The image processing system according to (1), wherein each of the input frames is an image obtained by rendering three-dimensional data showing one or more objects viewed from a predetermined viewpoint. (3) The image processing system according to (2), wherein each of the input frames is an image obtained by performing the rendering so that the viewpoint varies for each of the input frames, and the at least one processor acquires variation information related to the variation of the viewpoint for each of the input frames in the rendering, and interpolates pixel values ​​in the input frames at positions corresponding to each pixel before the variation based on the variation information and each pixel of each of the input frames, to generate each of the intermediate frames.(4) The image processing system according to (2) or (3), wherein the at least one processor acquires (n-1)th motion information, which is information indicating the amount and direction of motion from the (n-1)th input frame to the nth input frame, and acquires the (n-1)th auxiliary information by applying motion compensation to the (n-1)th accumulated feature information based on the (n-1)th motion information. (5) The image processing system according to (4), wherein the at least one processor: acquires (n-1)th depth information indicating a depth of each pixel of the (n-1)th input frame and nth depth information indicating a depth of each pixel of the nth input frame; identifies an nth appearance pixel, among the pixels of the nth intermediate frame, which is a pixel in which all or part of the object not displayed in the (n-1)th intermediate frame is displayed, based on the (n-1)th depth information and the nth depth information; and acquires the (n-1)th auxiliary information by replacing the pixel value of the nth appearance pixel in the (n-1)th cumulative feature information with a predetermined value. (6) The image processing system according to (1) to (5), wherein the cumulative feature information output layer receives the first intermediate frame and given auxiliary information as input, and outputs the first cumulative feature information. (7) The image processing system according to any one of (1) to (6), wherein the cumulative feature information is image information having the same number of pixels as the intermediate pixel number.

Claims

1. An image processing system comprising at least one processor, The at least one processor is The first to Nth input frames (where N is a natural number greater than or equal to 2) each have a predetermined number of input pixels, Based on each of the aforementioned input frames, an intermediate frame is generated that corresponds to the input frame and has an intermediate pixel count equal to or greater than the number of input pixels, thereby obtaining the first to Nth intermediate frames. Each of the aforementioned intermediate frames is input to a machine learning model to obtain the first to Nth estimated frames, each having an estimated number of pixels greater than or equal to the number of intermediate pixels and greater than or equal to the number of input pixels. The aforementioned machine learning model, A cumulative feature information output layer receives the nth intermediate frame (n = 2, 3, ..., N) and the nth-1st auxiliary information based on the nth-1st cumulative feature information indicating the characteristics of the first to nth intermediate frames, and outputs the nth cumulative feature information indicating the characteristics of the first to nth intermediate frames. It includes an estimated frame output layer that receives the nth cumulative feature information and outputs the nth estimated frame, A learning intermediate frame having the intermediate number of pixels, generated based on a learning input frame having the aforementioned input number of pixels, and a learning estimated frame having the estimated number of pixels, respectively, are included. It was learned using multiple training datasets. Image processing system.

2. Each of the aforementioned input frames is an image obtained by rendering three-dimensional data showing one or more objects viewed from a predetermined viewpoint. The image processing system according to claim 1.

3. Each of the aforementioned input frames is an image obtained by performing the rendering such that the viewpoint changes with each input frame. The at least one processor is The following information is obtained, which is information relating to the change in the viewpoint for each input frame in the rendering: Based on the aforementioned variation information and each pixel of each input frame, the pixel values ​​at the positions corresponding to each pixel before the variation in the input frame are interpolated to generate each intermediate frame. The image processing system according to claim 2.

4. The at least one processor is The (n-1)th motion information is obtained, which is information indicating the amount and direction of movement from the (n-1)th input frame to the nth input frame. Based on the n-1 motion information, motion compensation is applied to the n-1 cumulative feature information to obtain the n-1 auxiliary information. The image processing system according to claim 2 or 3.

5. The at least one processor is The (n-1)th depth information indicating the depth of each pixel in the (n-1)th input frame and the nth depth information indicating the depth of each pixel in the (n)th input frame are obtained. Based on the n-1 depth information and the n depth information, the nth occurrence pixel is identified, which is a pixel in the nth intermediate frame in which all or part of the object not displayed in the n-1 intermediate frame is displayed. The n-th auxiliary information is obtained by replacing the pixel value of the nth occurrence pixel in the n-th cumulative feature information with a predetermined value. The image processing system according to claim 4.

6. The cumulative feature information output layer receives the first intermediate frame and given auxiliary information as input and outputs the first cumulative feature information. The image processing system according to claim 1 or 2.

7. The cumulative feature information is image information having the same number of pixels as the number of intermediate pixels. The image processing system according to claim 1 or 2.

8. The processor, The first to Nth input frames (where N is a natural number greater than or equal to 2) each have a predetermined number of input pixels, Based on each of the aforementioned input frames, an intermediate frame is generated that corresponds to the input frame and has an intermediate pixel count equal to or greater than the number of input pixels, thereby obtaining the first to Nth intermediate frames. Each of the aforementioned intermediate frames is input to a machine learning model to obtain the first to Nth estimated frames, each having an estimated number of pixels greater than or equal to the number of intermediate pixels and greater than or equal to the number of input pixels. The aforementioned machine learning model, A cumulative feature information output layer receives the nth intermediate frame (n = 2, 3, ..., N) and the nth-1st auxiliary information based on the nth-1st cumulative feature information indicating the characteristics of the first to nth intermediate frames, and outputs the nth cumulative feature information indicating the characteristics of the first to nth intermediate frames. It includes an estimated frame output layer that receives the nth cumulative feature information and outputs the nth estimated frame, A learning intermediate frame having the intermediate number of pixels, generated based on a learning input frame having the aforementioned input number of pixels, and a learning estimated frame having the estimated number of pixels, respectively, are included. It was learned using multiple training datasets. Image processing methods.

9. Input frame acquisition means for acquiring first to Nth (where N is a natural number of 2 or more) input frames, each having a predetermined number of input pixels. Intermediate frame acquisition means that acquires the first to Nth intermediate frames by generating an intermediate frame that corresponds to the input frame and has an intermediate pixel count equal to or greater than the number of input pixels, based on each of the aforementioned input frames. Estimated frame acquisition means inputting each of the aforementioned intermediate frames into a machine learning model to acquire first to N estimated frames, each having an estimated number of pixels greater than or equal to the number of intermediate pixels and greater than or equal to the number of input pixels. It is a program that makes a computer function as such. The aforementioned machine learning model, A cumulative feature information output layer receives the nth intermediate frame (n = 2, 3, ..., N) and the nth-1st auxiliary information based on the nth-1st cumulative feature information indicating the characteristics of the first to nth intermediate frames, and outputs the nth cumulative feature information indicating the characteristics of the first to nth intermediate frames. It includes an estimated frame output layer that receives the nth cumulative feature information and outputs the nth estimated frame, A learning intermediate frame having the intermediate number of pixels, generated based on a learning input frame having the aforementioned input number of pixels, and a learning estimated frame having the estimated number of pixels, respectively, are included. It was learned using multiple training datasets. program.