Image processing system, image processing method, and program

Abstract

The present invention provides an image processing system that maintains spatial accuracy and improves time-series stability. The image processing system comprises at least one processor which inputs each of first to n-th input frames (n is a natural number of 2 or greater) to a machine-learning model and acquires each of first to n-th estimation frames (26). The at least one processor: acquires each of first to n-th frames-to-be-processed (20); acquires (n-1)-th cumulative feature information (28) indicating the features, output from the machine-learning model, of the first to (n-1)-th input frames (24); acquires (n-1)-th auxiliary information (30) on the basis of the (n-1)-th cumulative feature information (28); and acquires the n-th input frame (24) having per-pixel information elements more than per-pixel information elements included in the n-th frame-to-be-processed (20) on the basis of the n-th frame-to-be-processed (20) and the (n-1)-th auxiliary information 30.

Description

Image processing system, image processing method and program

[0001] The present invention relates to an image processing system, an image processing method, and a program.

[0002] Conventionally, a technique (super-resolution) for estimating a high-quality image based on a low-quality image using a machine learning model has been known (see Non-Patent Document 1 below).

[0003] Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang. Learning a Deep Convolutional Network for Image Super-Resolution, in Proceedings of European Conference on Computer Vision (ECCV), 2014

[0004] In order to achieve super-resolution of moving images such as game screens, the inventors of the present application are considering a system with a recursive configuration in which information indicating the characteristics of the current frame (the nth frame) and past frames (frames 1 to n-1) is input into a machine learning model to improve the image quality of the nth frame.

[0005] In the above system, by increasing the number of information elements per pixel in the frames input to the machine learning model, time-series stability is improved and multiple frames can be displayed smoothly and continuously. However, if the number of information elements in the input frames is increased, the number of pixels must be reduced to reduce the processing load. If the number of pixels in the input frames is small, the spatial accuracy of the output frames may be reduced.

[0006] An object of the present invention is to provide an image processing system, an image processing method, and a program that maintain spatial accuracy while improving time-series stability.

[0007] The image processing system of the present invention is an image processing system that includes at least one processor that inputs first to nth input frames (n is a natural number greater than or equal to 2) into a machine learning model and obtains first to nth estimated frames, respectively, wherein the at least one processor obtains the first to nth processing target frames, obtains n-1th cumulative feature information that indicates the features of the first to n-1th input frames output from the machine learning model, obtains n-1th auxiliary information based on the n-1st cumulative feature information, and obtains the nth input frame that has a greater number of information elements per pixel than the number of information elements per pixel that the nth processing target frame has, based on the nth processing target frame and the n-1st auxiliary information.

[0008] 1 is a diagram illustrating an example of a hardware configuration of an image processing system; FIG. 2 is a diagram illustrating an overview of the image processing system; FIG. 3 is a functional block diagram illustrating an example of functions realized by the image processing system; FIG. 4 is a diagram illustrating processing by a rendering unit; FIG. 5 is a diagram illustrating processing by an intermediate frame acquisition unit; FIG. 6 is a diagram schematically illustrating generation of an input frame by an input frame acquisition unit; FIG. 7 is a flow diagram illustrating an example of the flow of processing executed by the image processing system; FIG. 8 is a flow diagram illustrating an example of the flow of processing executed by the image processing system.

[0009] Hereinafter, an example of an embodiment of an image processing system according to the present invention will be described with reference to the drawings.

[0010] 1 is a diagram showing an example of the hardware configuration of an image processing system 1. The image processing system 1 is a computer such as a game console. As shown in FIG. 1, the image processing system 1 includes a control unit 10, a storage unit 12, a communication unit 14, an operation unit 16, a display unit 18, and an audio output unit 19.

[0011] The control unit 10 includes a program control device such as a CPU that operates according to a program installed in the image processing system 1. The control unit 10 also includes a GPU (Graphics Processing Unit) that draws an image in a frame buffer based on graphics commands and data supplied from the CPU.

[0012] The storage unit 12 includes, for example, a main storage device such as a ROM or a RAM, and an auxiliary storage device such as a HDD or an SSD. The storage unit 12 stores programs executed by the control unit 10. The storage unit 12 stores, for example, a game program (game software) in addition to programs for realizing various functions of the image processing system 1, which will be described later. The storage unit 12 also reserves an area for a frame buffer in which images are drawn by the GPU.

[0013] The communication unit 14 is a communication interface such as an Ethernet (registered trademark) module or a wireless LAN module.

[0014] The operation unit 16 is a user interface such as a keyboard, mouse, or game console controller, and receives operation inputs from the user and outputs signals indicating the contents of the inputs to the control unit 10 .

[0015] The display unit 18 is a display device such as a liquid crystal display or an organic EL display, and displays various images according to instructions from the control unit 10 .

[0016] The audio output unit 19 is, for example, a speaker, and outputs audio represented by audio data generated by the image processing system 1 .

[0017] In addition to the devices described above, the image processing system 1 may also include an optical disc drive that reads optical discs such as DVD-ROMs and Blu-ray (registered trademark) discs, a USB (Universal Serial Bus) port, and the like.

[0018] 2. Overview of Image Processing System Fig. 2 is a diagram showing an overview of the image processing system 1. Here, an example is shown in which the image processing system 1 is used to improve the image quality of gameplay video in a game. Gameplay video is video generated in response to a game program executed by the control unit 10 and user input accepted by the operation unit 16, and is composed of a plurality of still images (frames) that are time-series data. The processes performed by the image processing system 1 are mainly as follows:

[0019] (1) Generation of Processing Frame First, the image processing system 1 generates an image (processing frame) in which one or more game objects are drawn by rendering three-dimensional data representing the game objects as seen from a predetermined viewpoint. This processing frame is an image having a predetermined initial number of pixels and a predetermined initial image quality. The initial number of pixels is, for example, 3840 x 2160 (hereinafter, this number of pixels may be simply referred to as "4K").

[0020] The processing target frame can be considered to be an image showing the virtual space VS in which the one or more game objects represented by three-dimensional data are placed, from a predetermined viewpoint (see FIG. 4). The processing target frame is generated at predetermined time intervals. Each generated processing target frame is not displayed directly on the display unit 18, but is temporarily stored in the storage unit 12 and used for subsequent processing. Note that, in the following explanation, processing for the nth processing target frame 20_n will be mainly illustrated, but similar processing is also performed on other processing target frames (i.e., n=2, 3, ..., N).

[0021] In this embodiment, the image quality of a frame does not simply refer to the number of pixels (high resolution). The image quality of a frame may be evaluated based on, for example, a high signal-to-noise ratio, high reproducibility of spatial frequencies, high temporal stability (low artifacts and flicker when multiple frames are displayed consecutively), or a combination of these, when compared with a reference frame.

[0022] (2) Acquisition of Intermediate Frame Based on the target frame 20_n, the image processing system 1 acquires an intermediate frame 22_n having a number of intermediate pixels greater than the initial number of pixels. The number of intermediate pixels is, for example, 7680 x 4320 (hereinafter, this number of pixels may be simply referred to as "8K"). The intermediate frame 22_n is generated by performing enlargement and interpolation processing on the target frame 20_n.

[0023] (3) Acquisition of Input Frame Based on the intermediate frame 22_n and auxiliary information 30_n (described later), the image processing system 1 acquires an input frame 24_n having a number of channels per pixel greater than the number of channels (number of information elements) per pixel in the intermediate frame 22_n, but having a number of input pixels less than the number of intermediate pixels. The number of input pixels is, for example, 3840 x 2160 (4K). Note that a channel refers to various information elements that define the pixel value of each pixel in the frame. For example, a frame with three channels is an image containing color information (RGB) for each pixel.

[0024] (4) Obtaining an Estimated Frame The image processing system 1 inputs the input frame 24_n to the machine learning model 200 and obtains an estimated frame 26_n. The estimated frame 26_n is an image having the same number of estimated pixels as the number of input pixels and an estimated image quality that is equal to or higher than the initial image quality.

[0025] The machine learning model 200 is a model trained using a plurality of training data sets, each of which includes a training input frame and a training estimation frame.

[0026] (5) Obtaining cumulative feature information The machine learning model 200 has a cumulative feature information output layer 202 that receives the (n-1)th input frame 24_n-1 and outputs the (n-1)th cumulative feature information 28_n-1 that indicates the features of the first to (n-1)th input frames 24 (see Figure 2).

[0027] The image processing system 1 acquires the (n-1)th accumulated feature information 28_n-1. The acquired (n-1)th accumulated feature information 28_n-1 is input to the estimated frame output layer 204, which outputs the (n-1)th estimated frame 26_n-1 (see FIG. 2). The acquired (n-1)th accumulated feature information 28_n-1 is also stored in the storage unit 12 and is used to estimate the estimated frame 26_n corresponding to the next frame to be processed (the nth frame to be processed) 20_n.

[0028] (6) Acquisition of Auxiliary Information As described above, the (n-1)th accumulated feature information 28_n-1 is information that indicates the features of the first to (n-1)th input frames 24 (and thus the first to (n-1)th processing target frames 20). If the (n-1)th accumulated feature information 28_n-1, in which information on past processing target frames 20 is accumulated, is used to estimate the nth estimated frame 26_n, the amount of information available for estimation increases, and therefore a high-quality estimated frame 26_n can be obtained.

[0029] However, if there is movement or the like in the displayed game object between the (n-1)th processing frame 20_n-1 and the nth processing frame 20_n, when the nth input frame 24_n and the (n-1)th cumulative feature information 28_n-1 are input as is to the machine learning model 200, a phenomenon (the so-called ghost phenomenon) may occur in which an afterimage of the game object that was displayed in the (n-1)th processing frame 20_n-1 is displayed.

[0030] Therefore, the image processing system 1 obtains the n-1th auxiliary information 30_n-1 by applying various corrections, described below, to the n-1th cumulative feature information 28_n-1 based on information obtained during rendering (motion vectors, depth buffers, etc.) (see Figure 2).

[0031] As described above, the image processing system 1 estimates the estimated frame 26 using auxiliary information 30 based on cumulative feature information 28 that stores past information, in addition to the input frame 24 that corresponds to the current processing target frame 20. This increases the amount of information available for estimation, making it possible to obtain a high-quality estimated frame 26_n.

[0032] 3. Functions Realized by the Image Processing System Fig. 3 is a functional block diagram showing an example of functions realized by the image processing system 1. Note that Fig. 3 shows an increase or decrease in the number of pixels and the number of channels corresponding to blocks representing each function. For example, Fig. 3 shows that the input frame acquisition unit 412 performs processing to change from 8K pixels and 3 channels (3ch) to 4K pixels and 20 channels (20ch).

[0033] As shown in Figure 3, the image processing system 1 includes a game processing unit 400, a rendering unit 402, a rendering information storage unit 404, a processing target frame acquisition unit 406, a variation information acquisition unit 408, an intermediate frame acquisition unit 410, an input frame acquisition unit 412, an estimated frame acquisition unit 414, a machine learning model storage unit 415, a pixel number increase unit 416, an output frame acquisition unit 418, a channel number reduction unit 420, an auxiliary information acquisition unit 424, a motion information acquisition unit 426, a depth information acquisition unit 428, and an appearance pixel identification unit 430.

[0034] The game processing unit 400, rendering unit 402, processing target frame acquisition unit 406, variation information acquisition unit 408, intermediate frame acquisition unit 410, input frame acquisition unit 412, estimated frame acquisition unit 414, pixel number increase unit 416, output frame acquisition unit 418, channel number decrease unit 420, auxiliary information acquisition unit 424, motion information acquisition unit 426, depth information acquisition unit 428, and appearance pixel identification unit 430 are mainly realized by the control unit 10. The rendering information storage unit 404 and machine learning model storage unit 415 are mainly realized by the storage unit 12. Note that the game processing unit 400, rendering unit 402, and rendering information storage unit 404 are functions provided by the game software.

[0035] [Game Processing Unit] The game processing unit 400 executes various processes related to the game. For example, in response to a game program executed by the control unit 10 or user input received by the operation unit 16, the game processing unit 400 executes processes such as placing a game object O in the virtual space VS, operating or moving the game object O, and changing the viewpoint C from which the virtual space VS is viewed (see FIG. 4 ). The game object O is composed of primitives such as polygons represented by three-dimensional data. The three-dimensional data includes geometric information indicating the positions of vertices, topological information indicating how the vertices are connected, and attribute information such as color.

[0036] 4 is a diagram illustrating the processing of the rendering unit 402. The rendering unit 402 generates first to Nth (N is a natural number equal to or greater than 2) processing frames 20 by performing rendering (drawing processing) of three-dimensional data representing one or more game objects O as viewed from a predetermined viewpoint C. The processing frame 20 can also be said to be an image representing, from the predetermined viewpoint C, a virtual space VS in which one or more game objects O represented by three-dimensional data are arranged. The processing frame 20 has a predetermined initial number of pixels.

[0037] The rendering unit 402 performs rendering based on the results of various processes executed by the game processing unit 400. Specifically, the rendering unit 402 performs vertex processing (vertex shading) and pixel processing (pixel shading) based on three-dimensional data representing a game object O placed in the virtual space VS.

[0038] The vertex processing includes a coordinate transformation process (perspective projection) from the view coordinate system to the screen coordinate system, and a perspective projection matrix (camera matrix) used in the coordinate transformation process is added with a numerical value related to a change in the viewpoint C, as will be described later. The rendering unit 402 may perform rendering based on light source information, depth information (depth buffer), texture information, normal information, etc.

[0039] Here, the rendering unit 402 generates each processing frame 20 by performing rendering so that the viewpoint C varies for each processing frame 20. Here, even if the game processing unit 400 fixes the viewpoint C at a predetermined position, the rendering unit 402 varies the viewpoint C for each processing frame 20. As a result, as shown in FIG. 4 , the position of the displayed game object O varies in each processing frame 20_n, 20_n+1, and 20_n+2. In other words, the rendering unit 402 applies jitter when generating each processing frame 20. Specifically, the rendering unit 402 varies the viewpoint C for each processing frame 20 by adding a numerical value corresponding to a fractional pixel size that differs for each processing frame 20 to the perspective projection matrix. As such a rule, for example, a Halton sequence can be used.

[0040] [Rendering Information Storage Unit] The rendering information storage unit 404 stores information necessary for the rendering process in the rendering unit 402 and information obtained as a result of the rendering process.

[0041] [Processing Frame Acquisition Unit] The processing frame acquisition unit 406 acquires the first to Nth processing frames 20. Specifically, the processing frame acquisition unit 406 acquires the first to Nth processing frames 20 stored in the rendering information storage unit 404.

[0042] [Variation Information Acquisition Unit] The variation information acquisition unit 408 acquires variation information stored in the rendering information storage unit 404. The variation information is information indicating the amount of variation of the viewpoint C between before and after the variation. The information indicating the amount of variation can also be said to be a variation vector indicating the direction and distance of the variation. For example, since the above-mentioned Halton sequence includes information indicating the amount of variation of the viewpoint C, this information may be used as the variation information.

[0043] [Intermediate Frame Acquisition Unit] The intermediate frame acquisition unit 410 acquires the first to Nth intermediate frames 22, based on each processing target frame 20, by generating an intermediate frame 22 that corresponds to the processing target frame 20 and has a number of intermediate pixels greater than the initial number of pixels. In other words, each intermediate frame 22 is an enlarged image of the processing target frame 20 that corresponds to the intermediate frame 22. Specifically, the number of intermediate pixels is 8K. Details of the generation of the intermediate frames 22 will be described later.

[0044] [Input Frame Acquisition Unit] The input frame acquisition unit 412 acquires the first to Nth input frames 24 by generating input frames 24 that correspond to the intermediate frames 22 and have a number of input pixels that is less than the number of intermediate pixels, based on each intermediate frame 22 and each auxiliary information 30. Each input frame 24 is a reduced image of the intermediate frame 22 that corresponds to the input frame 24. Specifically, the number of input pixels is 4K. Details of the generation of the input frames 24 will be described later.

[0045] [Motion Information Acquisition Unit] The motion information acquisition unit 426 acquires n-1th motion information, which is information indicating the amount and direction of motion from the n-1th processing target frame 20_n-1 to the nth processing target frame 20_n. Specifically, the n-1th motion information is image information indicating the amount and direction of motion of each pixel between the n-1th processing target frame 20_n-1 and the nth processing target frame 20_n. The motion information is also called a motion vector.

[0046] The number of pixels in the motion information may be increased using the above-described variation information in a manner similar to that used when enlarging the processing target frame 20 to obtain the intermediate frame 22. Therefore, the motion information may be image information (bitmap format information) having 8K pixels, similar to the number of intermediate pixels. Specifically, the motion information acquisition unit 426 acquires original motion information having the same number of pixels as the initial pixel number, and then performs enlargement and interpolation processing on the original motion information using variation information having the same number of pixels as the intermediate pixel number, thereby acquiring motion information having the same number of pixels as the intermediate pixel number.

[0047] [Depth Information Acquisition Unit] The depth information acquisition unit 428 acquires (n-1)th depth information indicating the depth of each pixel in the (n-1)th processing target frame 20_n-1 and (n)th depth information indicating the depth of each pixel in the nth processing target frame 20_n. The depth information is also called a depth buffer, Z buffer, or depth buffer.

[0048] The depth information may have its number of pixels increased based on the variation information described above using a method similar to that used when enlarging the processing target frame 20 to obtain the intermediate frame 22. Therefore, the depth information may be image information (bitmap format information) having 8K pixels, similar to the intermediate pixel count. Specifically, the depth information acquisition unit 428 acquires original depth information having the same number of pixels as the initial pixel count, and then performs enlargement and interpolation processing on the original depth information using variation information having the same number of pixels as the intermediate pixel count, thereby acquiring depth information having the same number of pixels as the intermediate pixel count.

[0049] [Appearance Pixel Identification Unit] Based on the (n-1)th depth information and the (n)th depth information, the appearance pixel identification unit 430 identifies an nth appearance pixel, which is a pixel of the nth processing frame 20_n where all or part of a game object O that is not displayed in the (n-1)th processing frame 20_n-1 is displayed. Specifically, the appearance pixel identification unit 430 identifies the nth appearance pixel based on the difference between the (n-1)th depth information and the (n)th depth information. Note that the appearance pixel identification unit 430 may also identify the nth appearance pixel based on the (n-1)th perspective projection matrix for the (n-1)th processing frame 20_n-1 and the nth perspective projection matrix for the nth processing frame 20_n. Alternatively, the appearance pixel identification unit 430 may identify the nth appearance pixel using variation information and motion information. More specifically, the appearance pixel specifying section 430 specifies the nth appearance pixel and generates nth appearance pixel information, which is image information indicating the position of the nth appearance pixel.

[0050] [Machine Learning Model] The machine learning model 200 is a model that estimates the nth estimated frame 26_n based on the nth input frame 24_n. Specifically, the machine learning model 200 is a convolutional neural network (CNN). As the machine learning model 200, for example, a known model such as a multi-layered ResNet with a residual connection mechanism or a so-called encoder-decoder U-Net can be used. The model described in Non-Patent Document 1 may also be used as the machine learning model 200.

[0051] The machine learning model 200 is a model trained using a plurality of training data sets, each of which includes a training input frame having an input pixel count and a training estimation frame having an estimated pixel count. Various known techniques, such as backpropagation, can be used to train the machine learning model 200.

[0052] Specifically, the machine learning model 200 includes an accumulated feature information output layer 202, an estimated frame output layer 204, and a convolution layer 206 (see FIG. 2).

[0053] The cumulative feature information output layer 202 receives the n-th input frame 24_n and the (n-1)th auxiliary information 30_n-1 based on the (n-1)th cumulative feature information 28_n-1 indicating the features of the first to n-1th input frames 24, and outputs the n-th cumulative feature information 28_n indicating the features of the first to n-th input frames 24_n. The cumulative feature information output layer 202 may be composed of, for example, one or more convolutional layers.

[0054] The cumulative feature information 28_n-1 is image information (bitmap information) having the same number of pixels as the number of input pixels. The cumulative feature information 28_n-1 can also be considered a feature map that indicates the features of the first to (n-1)th input frames 24.

[0055] The cumulative feature information output layer 202 receives the first input frame 24_1 and given auxiliary information, and outputs first cumulative feature information 28_1. When n=1, there is no previous cumulative feature information 28 or auxiliary information 30, so the given auxiliary information prepared in advance is input to the cumulative feature information output layer 202 together with the first input frame 24_1.

[0056] The estimated frame output layer 204 receives the n-th accumulated feature information 28_n and outputs the n-th estimated frame 26_n. The estimated frame output layer 204 may be composed of, for example, one or more convolutional layers, similar to the accumulated feature information output layer 202. Alternatively, the estimated frame output layer 204 may be composed of one or more transposed convolutional layers (deconvolutional layers).

[0057] The convolutional layer 206 is a layer that reduces the number of channels of the accumulated feature information 28 while maintaining the number of pixels of the accumulated feature information 28. The accumulated feature information 28 output from the convolutional layer 206 is subjected to processing by the auxiliary information acquisition unit 424. The convolutional layer 206 reduces the dimensions of the accumulated feature information 28, thereby reducing calculation costs. The convolutional layer 206 is, for example, a convolutional layer with a kernel size of 1×1, but is not limited to this.

[0058] [Machine Learning Model Storage Unit] The machine learning model storage unit 415 stores the machine learning model 200. Specifically, the machine learning model storage unit 415 stores parameters of the machine learning model 200 (such as the number of convolutional layers, the number of nodes used in each convolutional layer, and the weight of each node).

[0059] [Estimated Frame Acquisition Unit] The estimated frame acquisition unit 414 inputs the nth input frame 24_n to the machine learning model 200 to acquire the nth estimated frame 26_n. In this embodiment, the estimated frame 26 has the same number of estimated pixels as the number of input pixels. Specifically, the number of estimated pixels is 4K.

[0060] [Pixel Number Increasing Unit] The pixel number increasing unit 416 increases the number of pixels included in the estimated frame 26 and the accumulated feature information 28 output from the machine learning model 200. Note that the increase in the number of pixels may be performed by a method such as bilinear interpolation.

[0061] In this embodiment, since the input frame 24 has 4K input pixels, the estimated frame 26 and the cumulative feature information 28 are also acquired as information having 4K pixels. The pixel number increasing unit 416 increases the number of pixels of the estimated frame 26 and the cumulative feature information 28 from 4K to 8K. By increasing the number of pixels of the estimated frame 26 to 8K, it becomes possible to display high-quality images corresponding to the display, as described below. Furthermore, by increasing the number of pixels of the cumulative feature information 28 to 8K, it becomes possible to acquire auxiliary information 30 having 8K pixels, as described below.

[0062] The pixel number increasing unit 416 needs to reduce the number of channels to compensate for the increase in the amount of information that accompanies an increase in the number of pixels. Increasing the number of pixels from 4K to 8K results in a fourfold increase in the number of pixels. Therefore, in this embodiment, the information having N channels output from the machine learning model 200 is reduced to one-fourth. In other words, as the pixel number increasing unit 416 increases the number of pixels, the number of channels that the estimated frame 26 and the accumulated feature information 28 have is reduced to N / 4.

[0063] [Output Frame Acquisition Unit] The output frame acquisition unit 418 acquires an output frame based on the estimated frame 26 whose pixel count has been increased by the pixel count increase unit 416. The output frame is an image that corresponds to the display format of the display unit 18. The output frame acquisition unit 418 may include, for example, a convolution layer and reduce the number of channels in the estimated frame 26 through convolution processing. Specifically, the output frame acquisition unit 418 may generate and acquire an output frame whose pixel count is 8K and whose channel count is 3.

[0064] [Number of Channels Reduction Unit] The number of channels reduction unit 420 reduces the number of channels of the accumulated feature information 28, the number of pixels of which has been increased by the pixel number increase unit 416. Specifically, the number of channels reduction unit 420 may include, for example, a convolution layer, and reduce the number of channels of the accumulated feature information 28 from N / 4 to 2 by convolution processing.

[0065] [Auxiliary Information Acquisition Unit] The auxiliary information acquisition unit 424 acquires the (n-1)th auxiliary information 30_n-1 by applying motion compensation to the (n-1)th cumulative feature information 28_n-1 based on the (n-1)th motion information. Motion compensation refers to, for example, a process of moving a pixel at position x in the (n-1)th cumulative feature information 28_n-1 to position x' in the (n-1)th processing frame 20_n-1 (see FIG. 4 ). That is, the auxiliary information acquisition unit 424 acquires the (n-1)th auxiliary information 30_n-1 based on the (n-1)th motion information by setting the pixel values ​​of one or more pixels in the (n-1)th cumulative feature information 28_n-1 to pixels at positions moved in accordance with the amount and direction of the pixel's movement.

[0066] If the game object O moves between the nth processing frame 20_n and the n-1th processing frame 20_n-1, and the nth input frame 24_n and the n-1th cumulative feature information 28_n-1 are input to the machine learning model 200 as is when acquiring the nth estimated frame 26_n without taking the object's movement into consideration, a ghost phenomenon may occur in the output nth estimated frame 26_n, in which an afterimage of the game object O that was displayed in the previous frame is displayed. Therefore, as described above, the image processing system 1 applies motion compensation to the n-1th cumulative feature information 28_n-1 based on the n-1th motion information to acquire the n-1th auxiliary information 30_n-1. This makes it possible to suppress the ghost phenomenon.

[0067] Furthermore, the auxiliary information acquisition unit 424 generates and acquires auxiliary information 30 having an 8K pixel count based on the motion information, appearance pixels, and cumulative feature information 28 having an 8K pixel count. In this way, generating auxiliary information 30 based on high-resolution information makes it possible to effectively utilize spatial information of past frames.

[0068] [Generation of Intermediate Frames and Input Frames] Fig. 5 is a diagram for explaining the processing in the intermediate frame acquisition unit 410. Fig. 5 illustrates an example of obtaining the n-th intermediate frame 22_n. For example, as shown in Fig. 5, if the pixel center of a certain pixel in the intermediate frame 22_n to be acquired is P １，０ In this case, the intermediate frame acquisition unit 410 obtains P １，０ The pixel centers P' of the four pixels closest to ０，０， P' １，０ , P' ０，１ , P' １，１ Based on the coordinates and pixel values ​​of P １，０ The pixel value of P' is calculated by bilinear interpolation. １，０ is the amount of fluctuation indicated by the fluctuation information. １，０ The pixel values ​​of the pixels newly generated by the enlargement process are calculated in the same manner. Note that, in addition to bilinear interpolation, various other well-known interpolation methods such as bicubic interpolation and Lanczos interpolation can also be used as the interpolation method.

[0069] When rendering is performed so that the viewpoint C varies for each processing frame 20, the amount of time-series information increases. However, if an enlarged frame of each processing frame 20 obtained in this way is used for estimation, an estimated frame 26 with higher image quality can be obtained.

[0070] On the other hand, if an enlarged frame of the processing target frame 20 obtained by performing rendering so that the viewpoint C varies is input directly into the machine learning model 200, there is a risk that the accuracy of the estimation will decrease due to the influence of the variation in the viewpoint C.

[0071] Therefore, as described above, the image processing system 1 generates each intermediate frame 22 by interpolating pixel values ​​at positions in the processing frame 20 corresponding to each pixel before the fluctuation based on the fluctuation information and each pixel of each processing frame 20, and inputs each input frame 24 generated based on each intermediate frame 22 to the machine learning model 200. This corrects for the influence of fluctuations in the viewpoint C, thereby preventing a decrease in estimation accuracy.

[0072] 6 is a diagram illustrating the generation of an input frame by the input frame acquisition unit 412. The input frame acquisition unit 412 generates and acquires an input frame 24 based on the intermediate frame 22 and auxiliary information 30. As described above, the intermediate frame 22 is an 8K image. The intermediate frame 22 generated based on the processing target frame 20 has three channels.

[0073] Furthermore, the cumulative feature information 28 has been processed by the pixel number increasing unit 416 and the channel number decreasing unit 420 as described above, and has become information with 8K pixels and 2 channels. Therefore, the auxiliary information acquiring unit 424 acquires auxiliary information 30 with 8K pixels and 2 channels based on the motion information, the number of pixels that appear, and the cumulative feature information 28.

[0074] The input frame acquisition unit 412 generates a frame with 8K pixels and 5 channels (2 + 3) based on the auxiliary information 30 and the intermediate frame 22. Furthermore, the input frame acquisition unit 412 reduces the number of pixels in the generated frame. The reduced number of pixels allows information to be allocated to the number of channels. In other words, the number of channels can be increased without increasing or decreasing the overall amount of information. Specifically, the input frame acquisition unit 412 acquires an input frame 24 with 4K pixels and 20 channels.

[0075] 3, when an input frame 24 having 4K pixels and 20 channels is input to the machine learning model 200, cumulative feature information 28 having 4K pixels and N channels (N is a natural number less than 20) is output. By processing this cumulative feature information 28 using the pixel number increasing unit 416 and the channel number decreasing unit 420, auxiliary information 30 having 8K pixels and 2 channels can be generated as described above.

[0076] 7A and 7B are flow diagrams showing an example of the flow of processing executed in the image processing system 1. The processing shown in Fig. 7A and 7B is executed by the control unit 10 operating in accordance with a program stored in the storage unit 12. Note that Fig. 7A shows processing when n=1, and Fig. 7B shows processing when n=2 and thereafter.

[0077] (1) Processing for n=1 First, the control unit 10 acquires a first processing target frame 20_1 (S100). The control unit 10 acquires a first intermediate frame 22_1 based on the first processing target frame 20_1 (S102). At this time, the number of pixels of the frame is increased. Specifically, the first intermediate frame 22_1 having 8K pixels is acquired based on the first processing target frame 20_1 having 4K pixels.

[0078] The control unit 10 also acquires a first input frame 24_1 based on the first intermediate frame 22_1 and the given auxiliary information (S104). At this time, the number of pixels of the frame is reduced. Specifically, the control unit 10 acquires a first input frame 24_1 having 4K pixels based on the first intermediate frame 22_1 having 8K pixels and the given auxiliary information.

[0079] Furthermore, the control unit 10 inputs the first input frame 24_1 to the machine learning model 200, and obtains a first estimated frame 26_1 and first accumulated feature information 28_1 (S106).

[0080] (2) Processing for n≧2 The control unit 10 acquires the nth processing target frame 20_n (S108). Next, the control unit 10 acquires the nth variation information (S110). The control unit 10 also acquires the nth motion information (S112). The control unit 10 also acquires the (n-1)th depth information and the nth depth information (S114), and identifies the nth appearance pixel based on the (n-1)th depth information and the nth depth information (S116).

[0081] The control unit 10 also increases the number of pixels in the (n-1)th cumulative feature information that has already been acquired (S118). Specifically, the control unit 10 increases the number of pixels in the (n-1)th cumulative feature information from 4K to 8K. Furthermore, the control unit 10 decreases the number of channels in the (n-1)th cumulative feature information (S120).

[0082] The control unit 10 also acquires an nth intermediate frame based on the nth processing target frame 20_n and the nth variation information (S122). At this time, the control unit 10 increases the number of pixels of the frame. Specifically, the control unit 10 acquires an nth intermediate frame 22_n having 8K pixels based on the nth processing target frame 20_n having 4K pixels.

[0083] Furthermore, the control unit 10 acquires the (n-1)th auxiliary information 30_n-1 based on the (n-1)th accumulated feature information, the nth motion information, and the nth appearance pixel (S124).

[0084] The control unit 10 also acquires the nth input frame 24_n based on the nth intermediate frame 22_n and the (n-1)th auxiliary information 30_n-1 (S126). At this time, the control unit 10 reduces the number of pixels in the frame. Specifically, the control unit 10 acquires the nth input frame 24_n, which has 4K pixels, based on the nth intermediate frame 22_n, which has 8K pixels, and the (n-1)th auxiliary information 30_n-1.

[0085] Then, the control unit 10 inputs the n-th input frame 24_n to the machine learning model 200, and acquires the n-th estimated frame 26_n and the n-th accumulated feature information 28_n (S128).

[0086] The control unit 10 determines whether or not the next frame exists (S130), and if it determines that the next frame exists (S130; Y), it increments n to n+1 and repeats the processes of S108 to S128. If it determines that the next frame does not exist (S130; N), the control unit 10 ends this process.

[0087] [5. Summary] According to the image processing system 1 according to the present embodiment described above, the nth estimated frame 26_n is estimated using the (n-1)th accumulated feature information 28_n-1 indicating the features of the first to (n-1)th input frames 24. That is, in addition to the information of the nth processing target frame 20_n, the information of the first to (n-1)th processing target frames 20 can be used for estimation, increasing the amount of information available for estimation and enabling the acquisition of a high-quality estimated frame 26_n.

[0088] Furthermore, in this embodiment, the auxiliary information 30 related to past frames is used to increase the number of channels in the input frames 24 input to the machine learning model 200, thereby improving time-series stability. Furthermore, the number of pixels in the auxiliary information 30 is set to 8K, allowing for effective use of spatial information of past frames. As described above, the image processing system 1 can maintain spatial accuracy while improving time-series stability.

[0089] In this embodiment, an example has been described in which the number of pixels in the processing target frame 20 and the input frame 24 is 4K, and the number of pixels in the intermediate frame 22 is 8K, but this is not limiting. For example, the number of pixels in the processing target frame 20 and the input frame 24 may be 2K, and the number of pixels in the intermediate frame 22 may be 4K. In this case, the variation information, motion information, and appearance pixels may also be information having the number of pixels (4K) corresponding to the intermediate frame 22.

[0090] The number of channels in each frame described with reference to Fig. 3 etc. is also an example and is not limited to the number described in this embodiment. At least, it is preferable that the number of channels in the auxiliary information 30 is used to increase the number of channels in the input frame 24.

[0091] [6. Supplementary Note] For example, the image processing system may have the following configuration: (1) An image processing system including at least one processor that inputs first to n-th input frames (n is a natural number greater than or equal to 2) to a machine learning model and obtains first to n-th estimated frames, respectively, wherein the at least one processor: obtains each of the first to n-th processing target frames, obtains n-1th cumulative feature information indicating features of the first to n-1th input frames output from the machine learning model, obtains n-1th auxiliary information based on the n-1th cumulative feature information, and obtains the n-th input frame that has a greater number of information elements per pixel than the number of information elements per pixel of the n-th processing target frame, based on the n-th processing target frame and the n-1th auxiliary information. (2) The image processing system described in (1), wherein the n-th input frame has a number of input pixels that is smaller than the number of pixels included in the n-1th auxiliary information. (3) The image processing system according to (1) or (2), wherein the at least one processor acquires, based on the nth frame to be processed having a predetermined initial number of pixels, an nth intermediate frame having a number of intermediate pixels greater than the initial number of pixels, and acquires the nth input frame based on the nth intermediate frame and the (n-1)th auxiliary information. (4) The image processing system according to (3), wherein the at least one processor acquires the nth frame to be processed by rendering a three-dimensional virtual space so that a predetermined viewpoint varies, acquires nth variation information related to the variation of the viewpoint of the nth frame to be processed in the rendering, the nth variation information having a number of pixels greater than the initial number of pixels, and acquires the nth intermediate frame based on the nth frame to be processed and the nth variation information. (5) The image processing system according to (4), wherein the at least one processor acquires the (n-1)th auxiliary information based at least on the nth variation information and the (n-1)th cumulative feature information.(6) The image processing system described in (5), wherein the at least one processor: obtains nth motion information indicating the amount and direction of motion from the n-1th processing frame to the nth processing frame based on the n-1th processing frame, the nth processing frame, and the nth variation information, and having the same number of pixels as the number of pixels contained in the nth variation information; and obtains the n-1th auxiliary information based on at least the nth motion information and the n-1th cumulative feature information. (7) The image processing system described in (5) or (6), wherein the at least one processor: acquires, based on the (n-1)th frame to be processed and the (n-1)th variation information, n-1th depth information indicating the depth of each pixel of the (n-1)th frame to be processed, and having the same number of pixels as the number of pixels in the (n-1)th variation information; acquires, based on the nth frame to be processed and the nth variation information, n-1st depth information indicating the depth of each pixel of the nth frame to be processed, and having the same number of pixels as the number of pixels in the nth variation information; identifies, based on the (n-1)th depth information and the nth depth information, an n-th appearance pixel, which is a pixel in which all or part of an object not displayed in the (n-1)th frame to be processed is displayed, among the pixels of the nth frame to be processed; and acquires the n-1st auxiliary information by replacing the pixel value of the n-1st appearance pixel in the (n-1)th cumulative feature information with a predetermined value. (8) The image processing system according to any one of (2) to (7), wherein the at least one processor obtains an estimated frame having the same number of estimated pixels as the number of input pixels, which is output by inputting the input frame into the machine learning model, and obtains an output frame by converting the estimated number of pixels and the number of information elements per pixel in the estimated frame to correspond to a display format of a display means.

Claims

1. An image processing system including at least one processor that inputs first to nth input frames (n is a natural number greater than or equal to 2) into a machine learning model to obtain first to nth estimated frames, respectively, wherein the at least one processor: obtains the first to nth processing target frames, respectively; obtains n-1th cumulative feature information indicating features of the first to n-1th input frames output from the machine learning model; obtains n-1th auxiliary information based on the n-1st cumulative feature information; and obtains the nth input frame having a greater number of information elements per pixel than the number of information elements per pixel in the nth processing target frame, based on the nth processing target frame and the n-1th auxiliary information.

2. The image processing system according to claim 1, wherein the nth input frame has a number of input pixels that is less than the number of pixels contained in the n-1th auxiliary information.

3. The image processing system of claim 1 or 2, wherein the at least one processor: obtains an nth intermediate frame having a number of intermediate pixels greater than a predetermined number of initial pixels based on the nth frame to be processed having a predetermined number of initial pixels; and obtains the nth input frame based on the nth intermediate frame and the n-1th auxiliary information.

4. The image processing system of claim 3, wherein the at least one processor obtains the nth processing frame by rendering a three-dimensional virtual space so as to vary a predetermined viewpoint; obtains nth variation information relating to the variation of the viewpoint of the nth processing frame in the rendering, the variation information having a number of pixels greater than the initial number of pixels; and obtains the nth intermediate frame based on the nth processing frame and the nth variation information.

5. The image processing system according to claim 4, wherein the at least one processor acquires the (n-1)th auxiliary information based on at least the nth fluctuation information and the (n-1)th accumulated feature information.

6. The image processing system of claim 5, wherein the at least one processor: obtains nth motion information indicating the amount and direction of movement from the n-1st processing frame to the nth processing frame based on the n-1st processing frame, the nth processing frame, and the nth movement information, and having the same number of pixels as the nth movement information; and obtains the n-1st auxiliary information based on at least the nth motion information and the n-1st cumulative feature information.

7. The image processing system according to claim 5 or 6, wherein the at least one processor: acquires n-1th depth information based on the n-1th frame to be processed and the n-1th variation information, the n-1th depth information indicating the depth of each pixel of the n-1th frame to be processed and having the same number of pixels as the number of pixels contained in the n-1st variation information; acquires nth depth information based on the nth frame to be processed and the nth variation information, the nth depth information indicating the depth of each pixel of the nth frame to be processed and having the same number of pixels as the number of pixels contained in the nth variation information; identifies an nth appearance pixel among the pixels of the nth frame to be processed, the nth appearance pixel being a pixel in which all or part of an object not displayed in the n-1st frame to be processed is displayed, based on the n-1st depth information and the nth depth information; and acquires the n-1st auxiliary information by replacing the pixel value of the nth appearance pixel in the n-1st cumulative feature information with a predetermined value.

8. The image processing system of claim 2, wherein the at least one processor obtains an estimated frame having the same number of estimated pixels as the number of input pixels, which is output by inputting the input frame into the machine learning model, and obtains an output frame by converting the estimated number of pixels and the number of information elements per pixel in the estimated frame to correspond to the display format of a display means.

9. An image processing method for inputting first to n-th input frames (n is a natural number greater than or equal to 2) to a machine learning model to obtain first to n-th estimated frames, respectively, wherein a processor: obtains the first to n-th processing target frames, respectively; obtains n-1st cumulative feature information indicating features of the first to n-1st input frames output from the machine learning model; obtains n-1st auxiliary information based on the n-1st cumulative feature information; and obtains the nth input frame having a number of information elements per pixel that is greater than the number of information elements per pixel that the nth processing target frame has, based on the nth processing target frame and the n-1st auxiliary information.

10. A program that causes a computer to function as an image processing system that inputs first to nth input frames (n is a natural number of 2 or greater) to a machine learning model and obtains first to nth estimated frames, respectively, comprising: a means for respectively obtaining the first to nth processing target frames; a means for obtaining n-1th cumulative feature information indicating features of the first to n-1th input frames output from the machine learning model; a means for obtaining n-1th auxiliary information based on the n-1st cumulative feature information; and a means for obtaining the nth input frame having a number of information elements per pixel that is greater than the number of information elements per pixel that the nth processing target frame has, based on the nth processing target frame and the n-1th auxiliary information.

Images