Image processing system, image processing method, and program

Abstract

The present invention makes it possible to apply super-resolution processing with high accuracy to a moving image generated from an object with which the texture changes, without depending on motion information. In an image processing system according to the present invention, a processor acquires first to N-th input frames having a number of pixels input, acquires first to N-th intermediate frames from the individual input frames, acquires first to N-th inferred frames from the individual intermediate frames, specifies n-th color change pixels including color information that changes regardless of the motion of an object in the n-th intermediate frame on the basis of texture information of the object, and acquires n-th auxiliary information by replacing the pixel values of the color change pixels in n-th cumulative feature information with predetermined values. Furthermore, in the image processing system, a machine learning model includes an output layer that outputs the n-th cumulative feature information and an output layer that outputs the n-th inferred frame, and the machine learning model is trained with a plurality of items of training data including training intermediate frames, auxiliary information in which the color change pixels have been replaced with predetermined values, and training inferred frames.

Description

Image processing system, image processing method and program

[0001] The present disclosure relates to an image processing system, an image processing method, and a program.

[0002] Conventionally, a technique (super-resolution) for estimating a high-quality still image based on a low-quality still image using a machine learning model is known (see Non-Patent Document 1 below).

[0003] Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang. Learning a Deep Convolutional Network for Image Super-Resolution, in Proceedings of European Conference on Computer Vision (ECCV), 2014

[0004] The inventors of the present application are considering applying the above-described super-resolution to video images such as game screens. Here, in video super-resolution, it is believed that higher-quality video images can be estimated by considering not only information about each frame to be processed but also information about previous frames of the frame. In particular, by considering information representing the movement of an object, such as a motion vector, it is possible to avoid degradation of image quality due to ghosting. However, there are cases where the texture of an object changes regardless of the object's motion information, such as when the object is a mirror or has an animated texture. Performing super-resolution processing that takes motion information into account on video images generated from such objects can actually result in a degradation of image quality.

[0005] The present disclosure aims to provide an image processing system, an image processing method, and a program that can perform high-precision super-resolution processing on moving images generated from objects with changing textures without relying on motion information, in an image processing means that estimates high-quality moving images based on low-quality moving images using motion information and information from past frames.

[0006] An image processing system according to the present disclosure is an image processing system including at least one processor, wherein the at least one processor acquires first to Nth input frames (N is a natural number equal to or greater than 2) each having a predetermined number of input pixels, and generates, based on each input frame, an intermediate frame corresponding to the input frame and having an intermediate pixel number equal to or greater than the input pixel number, thereby acquiring first to Nth intermediate frames, and inputting each intermediate frame to a machine learning model to acquire first to Nth estimated frames each having an estimated pixel number greater than the input pixel number and equal to or greater than the intermediate pixel number, wherein the machine learning model receives as input the nth intermediate frame (n=2, 3, ..., N) and n-1th auxiliary information based on n-1th accumulated feature information indicating features of the first to n-1th intermediate frames, and generates the first to nth estimated frames. The machine learning model includes an accumulated feature information output layer that outputs the nth accumulated feature information indicating the features of the intermediate frame, and an estimated frame output layer that receives the nth accumulated feature information and outputs the nth estimated frame, wherein the at least one processor identifies the nth color-changing pixel including color information that changes regardless of the movement of the object in the nth intermediate frame based on texture information of the object, and obtains the nth auxiliary information by replacing the pixel value of the color-changing pixel in the nth accumulated feature information with a predetermined value, and the machine learning model is trained using a plurality of training data, each of which includes a training intermediate frame having the intermediate number of pixels generated based on a training input frame having the input number of pixels, the auxiliary information in which the color-changing pixel has been replaced with a predetermined value, and a training estimation frame having the estimated number of pixels.

[0007] FIG. 1 is a diagram illustrating an example of a hardware configuration of an image processing system. FIG. 2 is a diagram illustrating an overview of the image processing system. FIG. 3 is a diagram illustrating a schematic diagram of processing of the image processing system. FIG. 4 is a functional block diagram illustrating an example of functions realized by the image processing system. FIG. 5 is a diagram illustrating processing of a rendering unit. FIG. 6 is a diagram illustrating processing in an intermediate frame acquisition unit. FIG. 7 is a diagram illustrating a schematic diagram of processing for identifying color-change pixels. FIG. 8 is a flow chart illustrating an example of a flow of processing executed in the image processing system.

[0008] Hereinafter, an example of an embodiment of an image processing system according to the present disclosure will be described with reference to the drawings.

[0009] 1 is a diagram showing an example of the hardware configuration of an image processing system 1. The image processing system 1 is a computer such as a game console. As shown in FIG. 1, the image processing system 1 includes a control unit 10, a storage unit 12, a communication unit 14, an operation unit 16, a display unit 18, and an audio output unit 19.

[0010] The control unit 10 includes a program control device such as a CPU that operates according to a program installed in the image processing system 1. The control unit 10 also includes a GPU (Graphics Processing Unit) that draws an image in a frame buffer based on graphics commands and data supplied from the CPU.

[0011] The storage unit 12 includes, for example, a main storage device such as a ROM or a RAM, and an auxiliary storage device such as a HDD or an SSD. The storage unit 12 stores programs executed by the control unit 10. The storage unit 12 stores, for example, a game program (game software) in addition to programs for realizing various functions of the image processing system 1, which will be described later. The storage unit 12 also reserves an area for a frame buffer in which images are drawn by the GPU.

[0012] The communication unit 14 is a communication interface such as an Ethernet (registered trademark) module or a wireless LAN module.

[0013] The operation unit 16 is a user interface such as a keyboard, mouse, or game console controller, and receives operation inputs from the user and outputs signals indicating the contents of the inputs to the control unit 10 .

[0014] The display unit 18 is a display device such as a liquid crystal display or an organic EL display, and displays various images according to instructions from the control unit 10 .

[0015] The audio output unit 19 is, for example, a speaker, and outputs audio represented by audio data generated by the image processing system 1 .

[0016] In addition to the devices described above, the image processing system 1 may also include an optical disc drive that reads optical discs such as DVD-ROMs and Blu-ray (registered trademark) discs, a USB (Universal Serial Bus) port, and the like.

[0017] 2. Overview of the Image Processing System Fig. 2 is a diagram showing an overview of the image processing system 1. Fig. 3 is a diagram showing the processing of the image processing system 1. In this embodiment, an example is shown in which the image processing system 1 is used to improve the image quality of gameplay video in a game. The gameplay video is a video generated in response to a game program executed by the control unit 10 or user input accepted by the operation unit 16, and is composed of a plurality of still images (frames) that are time-series data. The processing performed by the image processing system 1 is mainly as follows:

[0018] (1) Generation of Input Frames First, the image processing system 1 generates an image (input frame) depicting one or more game objects by rendering three-dimensional data representing the game objects as viewed from a predetermined viewpoint. This input frame is an image having a predetermined number of pixels (input pixel count) and a predetermined image quality (input image quality) (see FIG. 3 ). The input frame is generated at predetermined time intervals. The number of pixels of the input frame is, for example, 1920 x 1080 (1080p). Each generated input frame is not displayed directly on the display unit 18, but is temporarily stored in the storage unit 12 for subsequent processing. Note that the following description mainly illustrates processing for the nth (nth) input frame 20_n, but similar processing is also performed for other input frames (i.e., n = 2, 3, ..., N).

[0019] (2) Acquisition of Intermediate Frame Based on the acquired input frame 20_n, the image processing system 1 acquires a frame (intermediate frame) 22_n having a number of pixels (intermediate pixel number) greater than the number of input pixels. The number of intermediate pixels is, for example, 3840 x 2160 (4K). Specifically, the input frame 20_n is subjected to enlargement and interpolation processing to generate the intermediate frame 22_n (see FIG. 3 ).

[0020] It should be noted that although the intermediate frame 22_n has a larger number of pixels than the input frame 20_n, its image quality is not necessarily improved sufficiently. In other words, the image quality of a frame does not simply refer to the number of pixels (high resolution). The image quality of a frame may be evaluated based on, for example, a higher signal-to-noise ratio, a higher reproducibility of spatial frequencies, or a higher temporal stability (fewer artifacts or flicker when multiple frames are displayed consecutively) compared to a reference frame, or a combination of these.

[0021] (3) Obtaining an Estimated Frame The image processing system 1 inputs the intermediate frame 22_n to the machine learning model 200 and obtains an estimated frame 24_n. The estimated frame 24_n is an image having the same number of pixels (estimated pixel number) as the number of intermediate pixels and image quality (estimated image quality) that is equal to or higher than the input image quality (see FIG. 3 ).

[0022] Here, in addition to the intermediate frame 22_n, the (n-1)th auxiliary information 28_n-1 is input to the machine learning model 200 (see FIGS. 2 and 3). The auxiliary information 28_n-1 is information based on the (n-1)th cumulative feature information 26_n-1 that indicates the features of the first to (n-1)th intermediate frames 22. Details of the cumulative feature information 26 and the auxiliary information 28 will be described later.

[0023] The machine learning model 200 is a model trained using a plurality of training data sets, each of which includes a training intermediate frame having an intermediate number of pixels generated based on a training input frame having an input pixel number and input image quality, and a training estimated frame having an estimated pixel number and estimated image quality. Details of the machine learning model 200 will be described later.

[0024] (4) Acquisition of Cumulative Feature Information The machine learning model 200 has a cumulative feature information output layer 202 that receives the intermediate frame 22_n and auxiliary information 28_n-1 and outputs n-th cumulative feature information 26_n that indicates the features of the first to n-th intermediate frames 22 (see FIG. 2). The image processing system 1 acquires the n-th cumulative feature information 26_n.

[0025] The acquired n-th cumulative feature information 26_n is input to the estimated frame output layer 204, which outputs the n-th estimated frame 24_n (see FIG. 2 ). The acquired n-th cumulative feature information 26_n is also stored in the storage unit 12 and is used to estimate an estimated frame 24_n+1 corresponding to the next input frame (the (n+1)th input frame) 20_n+1.

[0026] (5) Acquisition of Auxiliary Information As described above, the (n-1)th accumulated feature information 26_n-1 is information that indicates the features of the first to (n-1)th intermediate frames 22 (and thus the first to (n-1)th input frames 20). If the accumulated feature information 26_n-1, which accumulates information on past input frames 20, is used to estimate the nth estimated frame 24_n, the amount of information available for estimation increases, making it possible to obtain a high-quality estimated frame 24_n.

[0027] However, if there is movement or the like in the displayed game object between the (n-1)th input frame 20_n-1 and the nth input frame 20_n, when the nth intermediate frame 22_n and the cumulative feature information 26_n-1 are input as is to the machine learning model 200, a phenomenon (the so-called ghost phenomenon) may occur in which an afterimage of the game object that was displayed in the (n-1)th input frame 20_n-1 is displayed.

[0028] Therefore, the image processing system 1 acquires the (n-1)th auxiliary information 28_n-1 by applying various corrections (described later) to the cumulative feature information 26_n-1 based on information obtained during rendering (motion vectors, depth buffer, information indicating the texture type, etc.) (see FIGS. 2 and 3). As described above, the acquired (n-1)th auxiliary information 28_n-1 is input to the machine learning model 200 together with the nth intermediate frame 22_n, and is used to estimate the nth estimated frame 24_n.

[0029] As described above, the image processing system 1 according to this embodiment estimates the estimated frame 24 using auxiliary information 28, which stores past information, in addition to the intermediate frame 22 corresponding to the current input frame 20. This increases the amount of information available for estimation, making it possible to obtain a high-quality estimated frame 24_n. The image processing system 1 will now be described in detail.

[0030] 4 is a functional block diagram showing an example of functions realized by the image processing system 1. As shown in FIG. 4, the image processing system 1 realizes a game processing unit 300, a rendering unit 302, a rendering information storage unit 304, an input frame acquisition unit 306, a variation information acquisition unit 308, an intermediate frame acquisition unit 310, a machine learning model storage unit 312, an estimated frame acquisition unit 314, a motion information acquisition unit 316, a depth information acquisition unit 318, an appearance pixel identification unit 320, an auxiliary information acquisition unit 322, and a color-change pixel information acquisition unit 324. The game processing unit 300, the rendering unit 302, the input frame acquisition unit 306, the variation information acquisition unit 308, the intermediate frame acquisition unit 310, the estimated frame acquisition unit 314, the motion information acquisition unit 316, the depth information acquisition unit 318, the appearance pixel identification unit 320, the auxiliary information acquisition unit 322, and the color-change pixel information acquisition unit 324 are mainly realized by the control unit 10. The rendering information storage unit 304 and the machine learning model storage unit 312 are mainly realized by the storage unit 12. Note that the game processing unit 300, the rendering unit 302, and the rendering information storage unit 304 are functions provided by the game software.

[0031] [Game Processing Unit] The game processing unit 300 executes various processes related to the game. For example, in response to a game program executed by the control unit 10 or user input received by the operation unit 16, the game processing unit 300 executes processes such as placing a game object O in a virtual three-dimensional space VS, operating or moving the game object O, and changing the viewpoint C from which the virtual three-dimensional space VS is viewed (see FIG. 5 ). The game object O is composed of primitives such as polygons represented by three-dimensional data. The three-dimensional data includes geometric information indicating the positions of vertices, topological information indicating how the vertices are connected, and attribute information such as color.

[0032] [Rendering Unit] FIG. 5 is a diagram illustrating the processing of the rendering unit 302. The rendering unit 302 generates first to Nth (N is a natural number equal to or greater than 2) input frames 20 by rendering (drawing) three-dimensional data representing one or more game objects O viewed from a predetermined viewpoint C. The rendering unit 302 performs rendering based on the results of various processes executed by the game processing unit 300. Specifically, the rendering unit 302 performs vertex processing (vertex shading) and pixel processing (pixel shading) based on three-dimensional data representing game objects O placed in the virtual three-dimensional space VS. The vertex processing includes a coordinate transformation process (perspective projection) from the view coordinate system to the screen coordinate system, and a numerical value related to a change in the viewpoint C is added to the perspective projection matrix (camera matrix) used in the coordinate transformation process, as described below. The rendering unit 302 may perform rendering based on light source information, depth information (depth buffer), texture information, normal information, and the like. The texture information of the game object may be an animation texture (animation information), which is a moving image, or color information (mirror map information) that reflects the game object as a mirror surface at the viewpoint C. The texture information of the game object may also be color information (ray tracing information) calculated from the light intensity at the first point on the object surface that is hit by extending a line connecting the viewpoint C and each pixel on the drawing surface into space, while reflecting transmission and refraction. In addition to the above processing, the rendering unit 302 may also perform processing that applies effects such as depth of field (DoF) and motion blur. The processing of the rendering unit 302 may be appropriately set by a game software developer or the like. Here, the game software developer or the like may adjust the MIP of the texture depending on the estimated number of pixels of the estimated frame 24, etc. This can suppress the occurrence of noise such as moiré in the estimated frame 24.

[0033] Here, the rendering unit 302 generates each input frame 20 by performing rendering such that the viewpoint C varies for each input frame 20. Here, even if the game processing unit 300 fixes the viewpoint C at a predetermined position, the rendering unit 302 varies the viewpoint C for each input frame 20. As a result, as shown in FIG. 5 , the position of the displayed game object O varies in each of input frames 20_n, 20_n+1, and 20_n+2. In other words, the rendering unit 302 applies jitter when generating each input frame 20. Specifically, the rendering unit 302 varies the viewpoint C for each input frame 20 by adding a numerical value corresponding to a sub-pixel size that differs for each input frame 20 to the perspective projection matrix. The rendering unit 302 varies the viewpoint C for each input frame 20 according to a predetermined rule. For example, a Halton sequence can be used as such a rule.

[0034] [Rendering Information Storage Unit] The rendering information storage unit 304 stores information necessary for the rendering process in the rendering unit 302 and information obtained as a result of the rendering process. For example, the rendering information storage unit 304 stores the input frame 20. The rendering information storage unit 304 also stores variation information, motion information, and depth information. Details of the variation information, motion information, and depth information will be described later. Furthermore, when the texture information of the game object is animation information, mirror map information, or ray tracing information, the rendering information storage unit 304 may store color-change pixel information that represents the distribution of pixels generated based on this information among the pixels of the input frame 20. The rendering information storage unit 304 may also store parameters used for coordinate transformation, light source information, texture information, normal information, etc.

[0035] [Input Frame Acquisition Unit] The input frame acquisition unit 306 acquires the first to Nth input frames 20. Specifically, the input frame acquisition unit 306 acquires the first to Nth input frames 20 stored in the rendering information storage unit 304.

[0036] [Variation Information Acquisition Unit] The variation information acquisition unit 308 acquires variation information. The variation information acquisition unit 308 acquires variation information stored in the rendering information storage unit 304. Specifically, the variation information is information indicating the amount of variation of the viewpoint C between before and after the variation. The information indicating the amount of variation can also be said to be a variation vector indicating the direction and distance of the variation. For example, since the above-mentioned Halton sequence includes information indicating the amount of variation of the viewpoint C, this information may be used as the variation information.

[0037] [Intermediate Frame Acquisition Unit] The intermediate frame acquisition unit 310 acquires first to Nth intermediate frames 22, respectively, based on each input frame 20 by generating an intermediate frame 22 that corresponds to the input frame 20 and has a number of intermediate pixels equal to or greater than the number of input pixels. In this embodiment, each intermediate frame 22 has a number of intermediate pixels greater than the number of input pixels. That is, in this embodiment, each intermediate frame 22 is an enlarged image of the input frame 20 corresponding to that intermediate frame 22.

[0038] Specifically, the intermediate frame acquisition unit 310 interpolates pixel values ​​at positions in each input frame 20 corresponding to the pixel values ​​before the fluctuation based on the fluctuation information and each pixel in the input frame 20, thereby generating each intermediate frame 22. FIG. 6 is a diagram illustrating the processing performed by the intermediate frame acquisition unit 310. FIG. 6 illustrates an example of obtaining the nth intermediate frame 22_n. For example, as shown in FIG. 6 , if the pixel center of a pixel in the intermediate frame 22_n to be acquired is P1,0, the intermediate frame acquisition unit 310 uses bilinear interpolation to obtain the pixel value of P1,0 based on the coordinates and pixel values ​​of the pixel centers P'0,0, P'1,0, P'0,1, and P'1,1 of the four pixels closest to P1,0 in the input frame 20_n. Here, P'1,0 is shifted from P1,0 by the amount of fluctuation indicated by the fluctuation information. The pixel values ​​of pixels newly generated by the enlargement process are similarly obtained. As the interpolation method, various known methods such as bicubic interpolation and Lanczos interpolation can be used in addition to bilinear interpolation.

[0039] When rendering is performed so that the viewpoint C varies for each input frame 20, the amount of time-series information increases. However, by using each input frame 20 obtained in this way (hereinafter referred to as a "variable input frame") for estimation, an estimated frame 24 with higher image quality can be obtained.

[0040] On the other hand, if the varying input frame (or an enlarged image thereof) is input directly to the machine learning model 200, the influence of the variation in viewpoint C described above may result in a decrease in the accuracy of estimation.

[0041] Therefore, as described above, the image processing system 1 interpolates pixel values ​​at positions in the input frame 20 corresponding to the pixels before the fluctuation based on the fluctuation information and each pixel of the input frame 20, generates each intermediate frame 22, and inputs this to the machine learning model 200. This corrects the influence of fluctuations in the viewpoint C, thereby preventing a decrease in the accuracy of estimation.

[0042] [Machine Learning Model] The machine learning model 200 is a model that estimates the nth estimated frame 24_n based on the nth intermediate frame 22_n. Specifically, the machine learning model 200 is a model that estimates the nth estimated frame 24_n based on the nth intermediate frame 22_n and the n-1th auxiliary information 28_n-1. Specifically, the machine learning model 200 is a convolutional neural network (CNN). As the machine learning model 200, for example, a well-known model such as a multi-layered ResNet having a residual connection mechanism or a so-called encoder-decoder U-Net can be used. The model described in Non-Patent Document 1 may also be used as the machine learning model 200.

[0043] The machine learning model 200 is a model trained using a plurality of training data sets, each set including a training intermediate frame having an intermediate number of pixels generated based on a training input frame having an input pixel count, auxiliary information 28 in which color-change pixels are replaced with predetermined values, and a training estimation frame having an estimated pixel count. Various known techniques, such as backpropagation, can be used to train the machine learning model 200. The auxiliary information 28 included in the training data has color-change pixels replaced with predetermined values. In other words, the machine learning model 200 has been trained using the auxiliary information 28 to which motion compensation (described below) has not been applied for the color-change pixels.

[0044] Specifically, the machine learning model 200 includes an accumulated feature information output layer 202, an estimated frame output layer 204, and a convolution layer 206 (see FIG. 2).

[0045] The cumulative feature information output layer 202 receives the nth intermediate frame 22_n and the n-1th auxiliary information 28_n-1 based on the n-1th cumulative feature information 26_n-1 indicating the features of the first to n-1th intermediate frames 22, and outputs the nth cumulative feature information 26_n indicating the features of the first to n-th intermediate frames 22_n. The cumulative feature information output layer 202 may be composed of, for example, one or more convolutional layers. The cumulative feature information 26_n-1 is image information (bitmap information) having the same number of pixels as the number of intermediate pixels. The cumulative feature information 26_n-1 can also be considered a feature map indicating the features of the first to n-1th intermediate frames 22.

[0046] The cumulative feature information output layer 202 receives the first intermediate frame 22_1 and given auxiliary information, and outputs first cumulative feature information 26_1. When n=1, there is no previous cumulative feature information 26 or auxiliary information 28, so the given auxiliary information prepared in advance is input to the cumulative feature information output layer 202 together with the first intermediate frame 22_1.

[0047] The estimated frame output layer 204 receives the n-th accumulated feature information 26_n and outputs the n-th estimated frame 24_n. The estimated frame output layer 204 may be configured, for example, by one or more convolutional layers, similar to the accumulated feature information output layer 202. Alternatively, the estimated frame output layer 204 may be configured by one or more transposed convolutional layers (deconvolutional layers).

[0048] The convolution layer 206 is a layer that reduces the number of channels of the accumulated feature information 26 while maintaining the number of pixels of the accumulated feature information 26. The accumulated feature information 26 output from the convolution layer 206 is subjected to processing in the auxiliary information acquisition unit 322. The convolution layer 206 reduces the dimensions of the accumulated feature information 26, thereby reducing calculation costs. The convolution layer 206 is, for example, a convolution layer with a kernel size of 1×1, but is not limited to this.

[0049] [Machine Learning Model Storage Unit] The machine learning model storage unit 312 stores the machine learning model 200. Specifically, the machine learning model storage unit 312 stores parameters of the machine learning model 200 (such as the number of convolutional layers, the number of nodes used in each convolutional layer, and the weight of each node).

[0050] [Estimated Frame Acquisition Unit] The estimated frame acquisition unit 314 inputs each intermediate frame 22 into the machine learning model 200 to acquire first to Nth estimated frames 24, each having an estimated pixel count greater than the input pixel count and equal to or greater than the intermediate pixel count. In this embodiment, the estimated frame 24 has an estimated pixel count equal to the intermediate pixel count. More specifically, the estimated frame acquisition unit 314 inputs the nth intermediate frame 22_n and the (n-1)th auxiliary information 28_n-1 into the machine learning model 200 to acquire the nth estimated frame 24_n.

[0051] [Motion Information Acquisition Unit] The motion information acquisition unit 316 acquires n-1th motion information, which is information indicating the amount and direction of motion from the n-1th input frame 20_n-1 to the nth input frame 20_n. Specifically, the n-1th motion information has the same number of pixels as the number of intermediate pixels, and is image information (bitmap format information) indicating the amount and direction of motion of each pixel between the n-1th input frame 20_n-1 and the nth input frame 20_n. Motion information is also called a motion vector. Specifically, the motion information acquisition unit 316 acquires motion information having the same number of pixels as the number of input pixels, and performs enlargement and interpolation processing on the motion information to acquire motion information having the same number of pixels as the number of intermediate pixels. The motion information acquisition unit 316 acquires information representing stationary motion (e.g., a value of 0) as motion information for pixels generated from game objects that are not moving by rendering. Here, when the texture information of the game object is animation information, mirror map information, or ray tracing information, the motion information acquisition unit 316 may acquire information indicating that the pixel is a color-change pixel instead of information indicating that the pixel is stationary. A color-change pixel is represented by information indicating infinity (INF) or not a number (NaN), for example (see FIG. 7 ).

[0052] [Depth Information Acquisition Unit] The depth information acquisition unit 318 acquires n-1th depth information indicating the depth of each pixel in the n-1th input frame 20_n-1 and nth depth information indicating the depth of each pixel in the nth input frame 20_n. Specifically, the depth information is image information having the same number of pixels as the intermediate pixel count (bitmap format information). The depth information is also called a depth buffer, Z buffer, or depth buffer. Specifically, the depth information acquisition unit 318 acquires depth information having the same number of pixels as the input pixel count and performs enlargement and interpolation processing on the depth information to acquire depth information having the same number of pixels as the intermediate pixel count.

[0053] [Appearance Pixel Identification Unit] Based on the (n-1)th depth information and the (n)th depth information, the appearance pixel identification unit 320 identifies the nth appearance pixel 222_n, which is a pixel in the nth intermediate frame 22_n where all or part of the game object O, which is not displayed in the (n-1)th intermediate frame 22_n-1, is displayed (see FIG. 3 ). Specifically, the appearance pixel identification unit 320 identifies the nth appearance pixel 222_n based on the difference between the (n-1)th depth information and the nth depth information. Note that the appearance pixel identification unit 320 may also identify the nth appearance pixel 222_n based on the (n-1)th perspective projection matrix associated with the (n-1)th intermediate frame 22_n-1 and the nth perspective projection matrix associated with the nth intermediate frame 22_n. Alternatively, the appearance pixel identification unit 320 may identify the nth appearance pixel 222_n using the (n-1)th motion information. More specifically, the appearance pixel specifying section 320 specifies the nth appearance pixel 222_n and generates nth appearance pixel information, which is image information indicating the position of the nth appearance pixel 222_n.

[0054] [Color-Changing Pixel Information Acquisition Unit] Based on the texture information of the object, the color-changing pixel information acquisition unit 324 identifies the nth color-changing pixel, which contains color information that changes regardless of the movement of the object in the nth intermediate frame 22, and acquires the color-changing pixel information. Specifically, the process of identifying color-changing pixels will be described using the diagram in FIG. 7 . The upper part of FIG. 7 schematically illustrates the rendering of a scene reflected on the windshield from the driver's seat viewpoint in a racing game or the like. The diagram shows multiple trees lined up against the sky, with a rearview mirror 700 positioned near the center. Each tree is a game object consisting of leaves and a trunk, and color information such as green or brown is added to the tree game object as a texture.

[0055] On the other hand, rearview mirror 700 is a game object made up of a mirror surface portion and non-mirror portions such as a frame. The scenery reflected by rearview mirror 700 as seen from viewpoint C is attached as a texture to the mirror surface portion. For example, an image generated by rendering in a direction symmetrical to viewpoint C (the direction of specular reflection) with the position of rearview mirror 700 as a new viewpoint is attached as the texture of the game object called rearview mirror.

[0056] The motion information acquisition unit 316 acquires the motion information shown in the lower left of FIG. 7 as nth motion information, which indicates the amount and direction of movement from the (n-1)th input frame 20_n-1 to the nth input frame 20_n. Specifically, because viewpoint C is located inside the vehicle, the scenery reflected on the windshield changes according to the movement of the game object, the car. For example, the motion information acquisition unit 316 acquires motion information representing a pixel representing a tree that moves 0.0f in the x direction and 0.1f in the y direction. The motion information acquisition unit 316 also acquires motion information for a pixel representing a rearview mirror. Here, the rearview mirror 700 is fixed inside the vehicle and appears stationary when viewed from viewpoint C. However, the motion information acquisition unit 316 acquires information representing a color-changing pixel (e.g., NaN in the x direction and NaN in the y direction) rather than motion information representing the rearview mirror 700's inherent stationary state (i.e., motion information representing a movement of 0 in the x direction and 0 in the y direction).

[0057] The color-changing pixel information acquisition unit 324 identifies an nth color-changing pixel in the motion information, which contains color information that changes regardless of the object's movement in the nth intermediate frame 22, and acquires the color-changing pixel information. That is, the color-changing pixel information acquisition unit 324 identifies pixels in the motion information that contain information indicating that they are color-changing pixels (here, NaN) and generates the color-changing pixel information shown in the lower right of FIG. 7 . The color-changing pixel information is image information (bitmap format information) in which the identified color-changing pixel is set to 0 and pixels other than the color-changing pixel are set to Refer RFM. Note that Refer RFM indicates that no calculation is performed on that pixel in the processing performed by the auxiliary information acquisition unit using the color-changing pixel information. The color-changing pixel information acquisition unit 324 acquires color-changing pixel information having the same number of pixels as the number of input pixels, and then performs enlargement and interpolation processing on the color-changing pixel information to acquire color-changing pixel information having the same number of pixels as the number of intermediate pixels.

[0058] In addition, if the rendering information storage unit 304 stores color-changing pixel information representing the distribution of pixels generated based on animation information, mirror map information, or ray tracing information, the color-changing pixel information acquisition unit 324 may acquire the color-changing pixel information from the rendering information storage unit 304 without using motion information.

[0059] [Auxiliary Information Acquisition Unit] The auxiliary information acquisition unit 322 acquires the (n-1)th auxiliary information 28_n-1 by applying motion compensation to the (n-1)th cumulative feature information 26_n-1 based on the (n-1)th motion information. Motion compensation refers to, for example, a process of moving a pixel at position x in the (n-1)th cumulative feature information 26_n to position x' in the nth intermediate frame 22_n, assuming that the pixel at position x in the (n-1)th intermediate frame 22_n-1 has moved to position x' (see FIG. 3 ). That is, the auxiliary information acquisition unit 322 acquires the (n-1)th auxiliary information 28_n-1 based on the (n-1)th motion information by setting the pixel values ​​of one or more pixels in the (n-1)th cumulative feature information 26_n-1 to pixels at positions moved in accordance with the amount and direction of the pixel's movement.

[0060] If there is movement of the game object O between the nth input frame 20_n and the n-1th input frame 20_n-1, if the nth intermediate frame 22_n and the n-1th cumulative feature information 26_n-1 are input directly into the machine learning model 200 when obtaining the nth estimated frame 24_n, a ghost phenomenon may occur in which an afterimage of the game object O that was displayed in the nth intermediate frame 22_n is displayed in the output nth estimated frame 24_n.

[0061] Therefore, as described above, the image processing system 1 applies motion compensation to the (n-1)th accumulated feature information 26_n-1 based on the (n-1)th motion information to acquire the (n-1)th auxiliary information 28_n-1, and when acquiring the nth estimated frame 24_n, this (n-1)th auxiliary information 28_n-1 is input to the machine learning model 200. This makes it possible to suppress the ghost phenomenon described above.

[0062] Furthermore, the auxiliary information acquisition unit 322 acquires the (n-1)th auxiliary information 28_n-1 by replacing the pixel value of the nth appearance pixel 222_n in the (n-1)th cumulative feature information 26_n-1 with a predetermined value. Specifically, the auxiliary information acquisition unit 322 acquires the (n-1)th auxiliary information 28_n-1 by replacing the pixel value of the nth appearance pixel 222_n in the (n-1)th cumulative feature information 26_n-1 with a predetermined value based on the nth appearance pixel information. The predetermined value may be a constant value such as 0 (black), or may be the pixel value of the nth appearance pixel 222_n in the nth intermediate frame 22_n.

[0063] If all or part of a game object O that is not displayed in the n-1th input frame 20_n-1 is displayed in the nth input frame 20_n, and the nth intermediate frame 22_n and the n-1th cumulative feature information 26_n-1 are input directly into the machine learning model 200 when obtaining the nth estimated frame 24_n, the aforementioned ghost phenomenon may occur in the output nth estimated frame 24_n.

[0064] Therefore, as described above, the image processing system 1 identifies the nth appearance pixel 222_n, which is a pixel in the nth intermediate frame 22_n where all or part of the game object O that is not displayed in the (n-1)th intermediate frame 22_n-1 is displayed, and then replaces the pixel value of the nth appearance pixel 222_n in the (n-1)th cumulative feature information 26_n-1 with a predetermined value to obtain the (n-1)th auxiliary information 28_n-1. This makes it possible to suppress the ghost phenomenon described above.

[0065] Furthermore, the auxiliary information acquisition unit 322 acquires the nth auxiliary information by replacing the pixel value of the color-changing pixel in the nth cumulative feature information with a predetermined value. Specifically, the auxiliary information acquisition unit 322 acquires the nth auxiliary information 28_n by replacing the pixel value of the nth color-changing pixel in the nth cumulative feature information 26_n with a predetermined value based on the nth color-changing pixel information. The predetermined value may be a constant value such as 0 (black), or the pixel value of the nth color-changing pixel n in the nth intermediate frame 22_n. The auxiliary information acquisition unit 322 does not perform the above-described replacement for pixels other than color-changing pixels.

[0066] The color-change pixels identified as described above are pixels whose color information is obtained based on a game object whose texture changes, regardless of the game object's motion information. Even if the game object is moving, the motion information representing that motion is unrelated to the game object's texture. Applying the above-described motion compensation to such color-change pixels would actually result in a loss of image quality.

[0067] Therefore, as described above, the image processing system 1 identifies the nth color-changing pixel containing color information that changes independently of the object's movement in the nth intermediate frame based on the object's texture information, and acquires the nth auxiliary information by replacing the pixel value of the color-changing pixel in the nth cumulative feature information with a predetermined value. This makes it possible to suppress the degradation of image quality.

[0068] In addition, the motion compensation, replacement processing based on appearance pixel information, and replacement processing based on color change pixel information performed by the auxiliary information acquisition unit 322 may be performed in whole or in part on one cumulative feature information 26.

[0069] 8 is a flow diagram showing an example of the flow of processing executed in the image processing system 1. The processing shown in FIG. 7 is executed by the control unit 10 operating in accordance with a program stored in the storage unit 12.

[0070] (1) Processing for n=1 First, the control unit 10 acquires a first input frame 20_1 (S700). The control unit 10 acquires a first intermediate frame 22_1 based on the first input frame 20_1 (S702). Then, the control unit 10 inputs the first intermediate frame 22_1 and given auxiliary information to the machine learning model 200, and acquires a first estimated frame 24_1 and first cumulative feature information 26_1 (S704).

[0071] (2) Processing when n≧2 The control unit 10 acquires the n-th input frame 20_n (S706).The control unit 10 acquires the n-th intermediate frame 22_n based on the n-th input frame 20_n (S708).

[0072] Next, the control unit 10 acquires the (n-1)th motion information (S710). The control unit 10 also acquires the (n-1)th depth information and the nth depth information (S712) and identifies the nth appearance pixel 222_n based on the (n-1)th depth information and the nth depth information (S714). The control unit 10 acquires the (n-1)th auxiliary information 28_n-1 based on the (n-1)th cumulative feature information 26_n-1, the (n-1)th motion information, and the nth appearance pixel 222_n (S716). The control unit 10 then inputs the nth intermediate frame 22_n and the (n-1)th auxiliary information 28_n-1 into the machine learning model 200, and acquires the nth estimated frame 24_n and the nth cumulative feature information 26_n (S718). The control unit 10 determines whether or not a next frame exists (S720), and if it determines that a next frame exists (S720; Y), it increments n to n+1 and repeats the processes of S706 to S718. If the control unit 10 determines that a next frame does not exist (S720; N), it ends this process. Note that if it determines that a next frame does not exist (S720; N), the control unit 10 may cause the display unit 18 to display the first to Nth estimated frames 24 as they are.

[0073] [5. Summary] According to the image processing system 1 of the present embodiment described above, an nth color-changing pixel containing color information that changes independently of the object's movement in the nth intermediate frame is identified based on the object's texture information, and the pixel value of the color-changing pixel in the nth cumulative feature information is replaced with a predetermined value to obtain the nth auxiliary information. In other words, the image processing means that estimates a high-quality video based on a low-quality video using motion information and information from past frames can perform high-precision super-resolution processing on a video generated from an object whose texture changes independently of motion information.

[0074] The invention according to the present disclosure is not limited to the above-described embodiments. Furthermore, the specific character strings and numerical values ​​described above and in the drawings are merely examples, and the invention is not limited to these character strings and numerical values.

[0075] For example, in the present embodiment, the number of intermediate pixels is greater than the number of input pixels, and the number of intermediate pixels is equal to the number of estimated pixels. However, the number of intermediate pixels may be equal to the number of input pixels, and the number of estimated pixels may be greater than the number of intermediate pixels. In other words, the intermediate frame 22 does not necessarily have to be an enlarged version of the input frame 20.

[0076] [6. Supplementary Note] (1) An image processing system including at least one processor, wherein the at least one processor: acquires first to Nth (N is a natural number equal to or greater than 2) input frames each having a predetermined number of input pixels; generates, based on each input frame, an intermediate frame corresponding to the input frame and having an intermediate pixel number equal to or greater than the number of input pixels, thereby acquiring the first to Nth intermediate frames; inputs each intermediate frame to a machine learning model to acquire first to Nth estimated frames each having an estimated pixel number greater than the number of input pixels and equal to or greater than the intermediate pixel number; the machine learning model includes: an accumulated feature information output layer that receives the nth intermediate frame (n=2, 3, ..., N) and n-1th auxiliary information based on n-1th accumulated feature information indicating features of the first to n-1th intermediate frames, and outputs the nth accumulated feature information indicating features of the first to nth intermediate frames; and an estimated frame output layer that receives the nth accumulated feature information, and outputs the nth estimated frame; and the at least one processor: an image processing system according to claim 1, wherein each of the input frames is an image obtained by rendering three-dimensional data representing one or more objects viewed from a predetermined viewpoint, the image processing system comprising: a learning intermediate frame having the intermediate number of pixels generated based on a learning input frame having the input number of pixels, the auxiliary information in which the color-changing pixels have been replaced with a predetermined value, and a learning estimated frame having the estimated number of pixels;(3) The image processing system according to (2), wherein each of the input frames is an image obtained by performing the rendering so that the viewpoint varies for each of the input frames, and the at least one processor acquires variation information that is information related to variation of the viewpoint for each of the input frames in the rendering, and calculates pixel values ​​in the input frames at positions corresponding to each pixel before the variation by interpolation based on the variation information and each pixel of each of the input frames, thereby generating each of the intermediate frames. (4) The image processing system according to (2) or (3), wherein the at least one processor acquires (n-1)th motion information that is information indicating an amount and direction of motion from the (n-1)th input frame to the nth input frame, and acquires the (n-1)th auxiliary information by applying motion compensation to the (n-1)th accumulated feature information based on the (n-1)th motion information. (5) The image processing system according to (4), wherein the at least one processor acquires (n-1)th depth information indicating a depth of each pixel of the (n-1)th input frame and nth depth information indicating a depth of each pixel of the nth input frame, identifies an nth appearance pixel, among the pixels of the nth intermediate frame, which is a pixel in which all or part of the object not displayed in the (n-1)th intermediate frame is displayed, based on the (n-1)th depth information and the nth depth information, and acquires the (n-1)th auxiliary information by replacing the pixel value of the nth appearance pixel in the (n-1)th cumulative feature information with a predetermined value. (6) The image processing system according to (1) or (2), wherein the cumulative feature information output layer receives the first intermediate frame and given auxiliary information as input, and outputs the first cumulative feature information. (7) The image processing system according to (1) or (2), wherein the cumulative feature information is image information having the same number of pixels as the number of intermediate pixels. (8) The image processing system according to (1) or (2), wherein the color-change pixels are represented by information representing infinity or non-number. (9) The processor:), and based on each input frame, generate an intermediate frame that corresponds to the input frame and has a number of intermediate pixels equal to or greater than the number of input pixels, thereby obtaining first to Nth intermediate frames, and input each intermediate frame to a machine learning model to obtain first to Nth estimated frames each having a number of estimated pixels greater than the number of input pixels and equal to or greater than the number of intermediate pixels, wherein the machine learning model includes: an accumulated feature information output layer that receives the nth intermediate frame (n=2, 3, ..., N) and n-1th auxiliary information based on n-1th accumulated feature information that indicates features of the first to n-1th intermediate frames, and outputs the nth accumulated feature information that indicates the features of the first to nth intermediate frames; and an estimated frame output layer that receives the nth accumulated feature information, and outputs the nth estimated frame, wherein the processor identifies an nth color-changing pixel that includes color information that changes regardless of movement of the object in the nth intermediate frame, based on texture information of the object, An image processing method, comprising: acquiring the nth auxiliary information by replacing the pixel values ​​of the color-change pixels in the nth cumulative feature information with a predetermined value; and the machine learning model being trained using a plurality of training data, each of which includes a training intermediate frame having the intermediate number of pixels generated based on a training input frame having the input pixel number, the auxiliary information in which the color-change pixels have been replaced with a predetermined value, and a training estimation frame having the estimated pixel number. (10) An image processing method, comprising: acquiring the nth auxiliary information by replacing the pixel values ​​of the color-change pixels in the nth cumulative feature information with a predetermined value;, N) based on each input frame, thereby acquiring first to Nth intermediate frames respectively; intermediate frame acquisition means for generating, based on each input frame, intermediate frames that correspond to the input frame and have a number of intermediate pixels equal to or greater than the number of input pixels, thereby acquiring first to Nth intermediate frames respectively; estimated frame acquisition means for inputting each intermediate frame to a machine learning model to acquire first to Nth estimated frames respectively having a number of estimated pixels that is greater than the number of input pixels and equal to or greater than the number of intermediate pixels, wherein the machine learning model includes: an accumulated feature information output layer that receives as input the nth intermediate frame (n=2, 3, ..., N) and n-1th auxiliary information based on n-1th accumulated feature information that indicates features of the first to n-1th intermediate frames, and outputs the nth accumulated feature information that indicates features of the first to nth intermediate frames; and an estimated frame output layer that receives as input the nth accumulated feature information, and outputs the nth estimated frame; and the program further comprises: A program that causes the computer to function as: an identification means that identifies an nth color-changing pixel including color information that changes regardless of the movement of the object in the nth intermediate frame based on texture information of the object; and an auxiliary information acquisition means that acquires the nth auxiliary information by replacing the pixel value of the color-changing pixel in the nth cumulative feature information with a predetermined value; and the machine learning model is trained using a plurality of training data that respectively include: a training intermediate frame having the intermediate number of pixels generated based on a training input frame having the input number of pixels, the auxiliary information in which the color-changing pixel has been replaced with a predetermined value, and a training estimation frame having the estimated number of pixels.

Claims

1. An image processing system including at least one processor, wherein the at least one processor: acquires first to Nth input frames (N is a natural number equal to or greater than 2) each having a predetermined number of input pixels; generates an intermediate frame based on each input frame, which corresponds to the input frame and has an intermediate number of pixels equal to or greater than the input number of pixels, thereby acquiring the first to Nth intermediate frames; inputs each intermediate frame to a machine learning model to acquire first to Nth estimated frames each having an estimated number of pixels greater than the input number of pixels and equal to or greater than the intermediate number of pixels; the machine learning model includes: an accumulated feature information output layer that receives the nth intermediate frame (n=2, 3, ..., N) and n-1th auxiliary information based on n-1th accumulated feature information indicating features of the first to n-1th intermediate frames, and outputs the nth accumulated feature information indicating features of the first to nth intermediate frames; and an estimated frame output layer that receives the nth accumulated feature information, and outputs the nth estimated frame; and the at least one processor: an image processing system comprising: identifying an nth color-changing pixel including color information that changes regardless of the movement of the object in the nth intermediate frame based on texture information of the object; and obtaining the nth auxiliary information by replacing the pixel value of the color-changing pixel in the nth cumulative feature information with a predetermined value; and the machine learning model being trained using a plurality of training data each including a learning intermediate frame having the intermediate number of pixels generated based on a learning input frame having the input number of pixels, the auxiliary information in which the color-changing pixel has been replaced with a predetermined value, and a learning estimated frame having the estimated number of pixels.

2. The image processing system of claim 1, wherein each of the input frames is an image obtained by performing a rendering of three-dimensional data showing one or more objects as viewed from a predetermined viewpoint.

3. The image processing system of claim 2, wherein each of the input frames is an image obtained by performing the rendering so that the viewpoint varies for each of the input frames, and the at least one processor acquires variation information which is information relating to the variation of the viewpoint for each of the input frames in the rendering, and based on the variation information and each pixel of each of the input frames, calculates by interpolation pixel values ​​at positions in the input frame which correspond to each pixel before the variation, thereby generating each of the intermediate frames.

4. The image processing system of claim 2 or 3, wherein the at least one processor obtains (n-1)th motion information, which is information indicating the amount and direction of motion from the (n-1)th input frame to the nth input frame, and obtains the (n-1)th auxiliary information by applying motion compensation to the (n-1)th accumulated feature information based on the (n-1)th motion information.

5. The image processing system of claim 4, wherein the at least one processor acquires n-1th depth information indicating the depth of each pixel of the n-1th input frame and nth depth information indicating the depth of each pixel of the nth input frame, identifies an nth appearance pixel among the pixels of the nth intermediate frame based on the n-1th depth information and the nth depth information, which is a pixel in which all or part of the object not displayed in the n-1th intermediate frame is displayed, and acquires the n-1th auxiliary information by replacing the pixel value of the nth appearance pixel in the n-1th cumulative feature information with a predetermined value.

6. The image processing system according to claim 1 or 2, wherein the cumulative feature information output layer receives the first intermediate frame and given auxiliary information, and outputs the first cumulative feature information.

7. The image processing system according to claim 1 or 2, wherein the cumulative feature information is image information having the same number of pixels as the intermediate number of pixels.

8. The image processing system according to claim 1 or 2, wherein the color-changing pixels are represented by information representing infinity or non-number.

9. A processor obtains first to Nth (N is a natural number equal to or greater than 2) input frames each having a predetermined number of input pixels, and generates, based on each input frame, an intermediate frame corresponding to the input frame and having an intermediate number of pixels equal to or greater than the input number of pixels, thereby obtaining first to Nth intermediate frames, and inputting each intermediate frame to a machine learning model to obtain first to Nth estimated frames each having an estimated number of pixels greater than the input number of pixels and equal to or greater than the intermediate number of pixels, the machine learning model including: an accumulated feature information output layer that receives the nth intermediate frame (n=2, 3, ..., N) and n-1th auxiliary information based on n-1th accumulated feature information indicating features of the first to n-1th intermediate frames, and outputs the nth accumulated feature information indicating features of the first to nth intermediate frames; and an estimated frame output layer that receives the nth accumulated feature information, and outputs the nth estimated frame, An image processing method comprising: identifying an nth color-changing pixel including color information that changes regardless of the movement of the object in the nth intermediate frame based on texture information of the object; and obtaining the nth auxiliary information by replacing the pixel value of the color-changing pixel in the nth cumulative feature information with a predetermined value; and the machine learning model is trained using a plurality of training data each including a learning intermediate frame having the intermediate number of pixels generated based on a learning input frame having the input number of pixels, the auxiliary information in which the color-changing pixel has been replaced with a predetermined value, and a learning estimated frame having the estimated number of pixels.

10. A program for causing a computer to function as: input frame acquisition means for acquiring first to Nth (N is a natural number equal to or greater than 2) input frames each having a predetermined number of input pixels; intermediate frame acquisition means for acquiring first to Nth intermediate frames each by generating, based on each input frame, an intermediate frame that corresponds to the input frame and has an intermediate pixel number equal to or greater than the input pixel number; estimated frame acquisition means for inputting each intermediate frame to a machine learning model to acquire first to Nth estimated frames each having an estimated pixel number greater than the input pixel number and equal to or greater than the intermediate pixel number; wherein the machine learning model includes: an accumulated feature information output layer that receives the nth intermediate frame (n=2, 3, ..., N) and n-1th auxiliary information based on n-1th accumulated feature information indicating features of the first to n-1th intermediate frames, and outputs the nth accumulated feature information indicating features of the first to nth intermediate frames; and an estimated frame output layer that receives the nth accumulated feature information, and outputs the nth estimated frame; and the program further comprises: A program that causes the computer to function as: an identification means for identifying an nth color-changing pixel including color information that changes regardless of the movement of the object in the nth intermediate frame based on texture information of the object; and an auxiliary information acquisition means for acquiring the nth auxiliary information by replacing the pixel value of the color-changing pixel in the nth cumulative feature information with a predetermined value; and the machine learning model is trained using a plurality of training data each including: a learning intermediate frame having the intermediate number of pixels generated based on a learning input frame having the input number of pixels, the auxiliary information in which the color-changing pixel has been replaced with a predetermined value, and a learning estimated frame having the estimated number of pixels.

Images