Image processing system, image processing method, and program

Abstract

In the present invention, at least one processor acquires (n-1)th auxiliary information (48_n-1) according to the product of at least one numerical value that is included in (n-1)th cumulative feature information (46_n-1) and an attenuation coefficient that is greater than 0 but less than 1.

Description

Image processing system, image processing method and program

[0001] The present invention relates to an image processing system, an image processing method, and a program.

[0002] Conventionally, a technique (super-resolution) for estimating a high-quality still image based on a low-quality still image using a machine learning model is known (see Non-Patent Document 1 below).

[0003] Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang. Learning a Deep Convolutional Network for Image Super-Resolution, in Proceedings of European Conference on Computer Vision (ECCV), 2014

[0004] The inventors of the present application have been studying a system (hereinafter sometimes referred to as the "reference technology") having the following recursive configuration in order to achieve super-resolution of moving images such as game screens. That is, this system inputs the current frame, i.e., the nth (nth) frame, and information on past frames, i.e., information indicating the characteristics of frames 1 to n-1 (1st to n-1th), into a machine learning model to improve the image quality of the nth frame (see FIG. 2). Generally speaking, by using information on past frames in addition to the current frame in this way, it is expected that the estimation performance of the machine learning model will improve.

[0005] However, according to the study of the inventors of the present application, when a moving image showing a scene with no or very little movement (hereinafter, sometimes referred to as a "still image") is input, it has been found that using information from past frames actually leads to a decrease in estimation performance. Specifically, when a still image is input to the above system, artifacts may occur in the moving image obtained as a result of estimation.

[0006] However, according to the study of the inventors of the present application, when a moving image showing a scene in which at least a part of the scene does not move for a long time or the movement is slight (hereinafter, sometimes referred to as a "still image") is input, it has been found that using information of past frames actually leads to a deterioration in estimation performance. Specifically, when a still image is input to the above system, artifacts may occur in the moving image obtained as a result of estimation.

[0007] One of the reasons for this is thought to be the lack of learning of machine learning models for situations in which parts of a video with no or very little movement remain in the same position for a long time. That is, from the perspective of the time and cost required for learning, there is a limit to the length of video that can be used for training a machine learning model. Furthermore, since video basically represents scenes with movement, there tends to be a shortage of still images like those described above in the training data used for training a machine learning model. For these reasons, it is difficult to train a machine learning model sufficiently on still images.

[0008] Furthermore, it is generally known that if the same information is continuously input excessively to a machine learning model having a recursive configuration, artifacts in the output are amplified. The above-mentioned reference technology also employs a recursive configuration, and when a still image is input, the same information is continuously input excessively, resulting in amplified artifacts being observed in the output.

[0009] An object of the present invention is to provide an image processing system, an image processing method, and a program that make it possible to estimate a high-quality still image with fewer artifacts based on a low-quality still image.

[0010] An image processing system according to the present invention is an image processing system including at least one processor, wherein the at least one processor acquires first to Nth input frames each having a predetermined number of input pixels and corresponding to first to Nth (N is a natural number of 2 or more) frames to be processed, inputs the input frames to a machine learning model to acquire first to Nth estimated frames each having an estimated pixel number equal to or greater than the number of input pixels, and the machine learning model calculates a feature based on the nth input frame (n=2, 3, ..., N) and n-1th accumulated feature information indicating features of the first to n-1th input frames and including one or more numerical values. and an estimated frame output layer to which the n-th cumulative feature information is input and outputs the n-th estimated frame, the n-th cumulative feature information being input and the n-1th auxiliary information being based on the n-1th cumulative feature information, and the n-1th cumulative feature information being input and outputting the n-th estimated frame, the n-th cumulative feature information being input and the n-1th estimated frame being output, the n-th cumulative feature information being learned using a plurality of training data, each of which includes a learning input frame having the input pixel count and a learning estimated frame having the estimated pixel count, and the at least one processor further obtains the n-1th auxiliary information according to a product of at least one numerical value included in the n-1th cumulative feature information and an attenuation coefficient greater than 0 and less than 1.

[0011] FIG. 1 is a diagram illustrating an example of the hardware configuration of an image processing system. FIG. 2 is a diagram illustrating an overview of a reference technology. FIG. 3 is a diagram illustrating a schematic diagram of processing in the reference technology. FIG. 4 is a diagram illustrating an overview of an image processing system. FIG. 5 is a diagram illustrating a schematic diagram of processing in the image processing system. FIG. 6 is a diagram illustrating a still screen. FIG. 7 is a functional block diagram illustrating an example of functions realized in the image processing system. FIG. 8 is a diagram illustrating processing in a rendering unit. FIG. 9 is a diagram illustrating processing in an input frame acquisition unit. FIG. 10 is a flow diagram illustrating an example of the flow of processing executed in the image processing system.

[0012] Hereinafter, an example of an embodiment of an image processing system according to the present invention will be described with reference to the drawings.

[0013] 1 is a diagram showing an example of the hardware configuration of an image processing system 1. The image processing system 1 is a computer such as a game console. As shown in FIG. 1, the image processing system 1 includes a control unit 10, a storage unit 12, a communication unit 14, an operation unit 16, a display unit 18, and an audio output unit 19.

[0014] The control unit 10 includes a program control device such as a CPU that operates according to a program installed in the image processing system 1. The control unit 10 also includes a GPU (Graphics Processing Unit) that draws an image in a frame buffer based on graphics commands and data supplied from the CPU.

[0015] The storage unit 12 includes, for example, a main storage device such as a ROM or a RAM, and an auxiliary storage device such as a HDD or an SSD. The storage unit 12 stores programs executed by the control unit 10. The storage unit 12 stores, for example, a game program (game software) in addition to programs for realizing various functions of the image processing system 1, which will be described later. The storage unit 12 also reserves an area for a frame buffer in which images are drawn by the GPU.

[0016] The communication unit 14 is a communication interface such as an Ethernet (registered trademark) module or a wireless LAN module.

[0017] The operation unit 16 is a user interface such as a keyboard, mouse, or game console controller, and receives operation inputs from the user and outputs signals indicating the contents of the inputs to the control unit 10 .

[0018] The display unit 18 is a display device such as a liquid crystal display or an organic EL display, and displays various images according to instructions from the control unit 10 .

[0019] The audio output unit 19 is, for example, a speaker, and outputs audio represented by audio data generated by the image processing system 1 .

[0020] In addition to the devices described above, the image processing system 1 may also include an optical disc drive that reads optical discs such as DVD-ROMs and Blu-ray (registered trademark) discs, a USB (Universal Serial Bus) port, and the like.

[0021] [2. Overview of the Reference Technology] Before describing the image processing system 1 according to this embodiment, a reference technology that is the premise of the image processing system 1 according to this embodiment will be described using Figures 2 and 3. Figure 2 is a diagram showing an overview of the reference technology. Figure 3 is a diagram showing a schematic diagram of the processing in the reference technology. Here, an example is shown in which the reference technology is used to improve the image quality of gameplay video in a game. Gameplay video is video generated in response to a game program executed by a control unit or user input accepted by an operation unit, and is composed of a plurality of still images (frames) that are time-series data. The processing performed by the reference technology is mainly as follows:

[0022] (1) Generation of Processing Frames First, the system according to the reference technology generates an image (processing frame) depicting one or more game objects by rendering three-dimensional data representing the game objects as viewed from a predetermined viewpoint. This processing frame is an image having a predetermined number of pixels (initial pixel count) and a predetermined image quality (initial image quality) (see FIG. 3 ). Processing frames are generated at predetermined time intervals. The number of pixels of the processing frame is, for example, 1920 x 1080 (1080p). Each generated processing frame is not displayed directly on the display unit 18, but is temporarily stored in the storage unit 12 for subsequent processing. Note that the following description mainly illustrates processing for the nth processing frame 20_n, but similar processing is also performed for other processing frames (i.e., n = 2, 3, ..., N).

[0023] (2) Obtaining an Input Frame The system according to the reference technology obtains a frame (input frame) 22_n having a larger number of pixels (input pixel count) than the initial pixel count based on the obtained processing target frame 20_n. The input pixel count is, for example, 3840 x 2160 (4K). Specifically, the processing target frame 20_n is subjected to enlargement and interpolation processing to generate the input frame 22_n (see FIG. 3 ).

[0024] It should be noted that although the input frame 22_n has a larger number of pixels than the target frame 20_n, its image quality is not necessarily sufficiently improved. In other words, the image quality of a frame does not simply refer to the number of pixels (high resolution). The image quality of a frame may be evaluated based on, for example, a high signal-to-noise ratio, high spatial frequency reproducibility, high temporal stability (low artifacts and flicker when multiple frames are displayed consecutively), etc., when compared to a reference frame, or a combination of these.

[0025] (3) Obtaining an Estimated Frame The system according to the reference technology inputs the input frame 22_n to the machine learning model 200 and obtains the estimated frame 24_n. The estimated frame 24_n is an image having the same number of pixels (estimated number of pixels) as the number of input pixels and an image quality (estimated image quality) that is equal to or higher than the initial image quality (see FIG. 3 ).

[0026] Here, in addition to the input frame 22_n, the (n-1)th auxiliary information 28_n-1 is input to the machine learning model 200 (see FIGS. 2 and 3). The auxiliary information 28_n-1 is information based on the (n-1)th cumulative feature information 26_n-1 that indicates the features of the first to (n-1)th input frames 22. Details of the cumulative feature information 26 and the auxiliary information 28 will be described later.

[0027] The machine learning model 200 is a model trained using a plurality of training data sets, each of which includes a training input frame having an input pixel count and a training estimation frame having an estimated pixel count and estimated image quality.

[0028] (4) Acquisition of Cumulative Feature Information The machine learning model 200 has an cumulative feature information output layer 202 that receives input frame 22_n and auxiliary information 28_n-1 and outputs n-th cumulative feature information 26_n that indicates the features of the first to n-th input frames 22 (see FIG. 2). The system according to the reference technology acquires the n-th cumulative feature information 26_n.

[0029] The acquired n-th cumulative feature information 26_n is input to the estimated frame output layer 204, which outputs the n-th estimated frame 24_n (see FIG. 2 ). The acquired n-th cumulative feature information 26_n is also stored in the storage unit 12 and is used to estimate an estimated frame 24_n+1 corresponding to the next processing target frame (the (n+1)th processing target frame) 20_n+1.

[0030] (5) Acquisition of Auxiliary Information As described above, the (n-1)th accumulated feature information 26_n-1 is information that indicates the features of the first to (n-1)th input frames 22 (and thus the first to (n-1)th processing target frames 20). If the accumulated feature information 26_n-1, which accumulates information on past processing target frames 20, is used to estimate the nth estimated frame 24_n, the amount of information available for estimation increases, making it possible to obtain a high-quality estimated frame 24_n.

[0031] However, if there is movement or the like in the displayed game object between the (n-1)th processing frame 20_n-1 and the nth processing frame 20_n, when the nth input frame 22_n and the cumulative feature information 26_n-1 are input as is to the machine learning model 200, a phenomenon (the so-called ghost phenomenon) may occur in which an afterimage of the game object that was displayed in the (n-1)th processing frame 20_n-1 is displayed.

[0032] Therefore, the system according to the reference technology acquires the (n-1)th auxiliary information 28_n-1 by applying various corrections (described later) to the cumulative feature information 26_n-1 based on information obtained during rendering (such as a motion vector or a depth buffer) (see FIGS. 2 and 3). As described above, the acquired (n-1)th auxiliary information 28_n-1 is input to the machine learning model 200 together with the nth input frame 22_n, and is used to estimate the nth estimated frame 24_n.

[0033] As described above, according to the reference technique of this embodiment, the estimated frame 24 is estimated using auxiliary information 28 that stores past information in addition to the input frame 22 that corresponds to the current processing target frame 20. This increases the amount of information available for estimation, making it possible to obtain a high-quality estimated frame 24_n.

[0034] 3. Overview of Image Processing System Next, an overview of the image processing system 1 will be described with reference to FIGS. 4 and 5. FIG. 4 is a diagram illustrating an overview of the image processing system 1. FIG. 5 is a diagram illustrating a schematic diagram of processing in the image processing system 1. In particular, the image processing system 1 employs a configuration in which the auxiliary information generation unit 716 includes an attenuation unit 7163 in order to enable estimation of a high-quality still image with fewer artifacts based on a low-quality still image. Note that, hereinafter, a description of the same configuration as that of the reference technology will be omitted.

[0035] 6 is a diagram illustrating a still screen. For example, as shown in FIG. 6, if there is no movement of a displayed object in input frames 42_n, 42_n+1, and 42_n+2, that is, if input frames 42_n, 42_n+1, and 42_n+2 are frames related to a still screen, artifacts may occur in the resulting estimated frame 44.

[0036] One of the reasons for this is thought to be the lack of learning of machine learning models for situations in which parts of a video with no or very little movement remain in the same position for a long time. That is, from the perspective of the time and cost required for learning, there is a limit to the length of video that can be used for training a machine learning model. Furthermore, since video basically represents scenes with movement, there tends to be a shortage of still images like those described above in the training data used for training a machine learning model. For these reasons, it is difficult to train a machine learning model sufficiently on still images.

[0037] Furthermore, it is generally known that if the same information is continuously input excessively to a machine learning model having a recursive configuration, artifacts in the output are amplified. The machine learning model 500 according to this embodiment also employs a recursive configuration, and when a still image is input, the same information is continuously input excessively, resulting in amplified artifacts being observed in the output estimated frame 44.

[0038] Therefore, in the image processing system 1, the (n-1)th auxiliary information 48_n-1 is acquired according to the product of at least one pixel value included in the (n-1)th cumulative feature information 46_n-1 and an attenuation coefficient greater than 0 and less than 1. This prevents the same information from being excessively input to the machine learning model 500 even when a still image is being estimated, thereby suppressing the occurrence of artifacts in the resulting estimated frame 44. The image processing system 1 will be described in detail below.

[0039] 7 is a functional block diagram showing an example of functions realized in the image processing system 1. As shown in Fig. 7, the image processing system 1 realizes a game processing unit 700, a rendering unit 702, a rendering information storage unit 704, a processing target frame acquisition unit 706, a variation information acquisition unit 708, an input frame acquisition unit 710, a machine learning model storage unit 712, an estimated frame acquisition unit 714, and an auxiliary information generation unit 716. The auxiliary information generation unit 716 includes a motion information acquisition unit 7160, a depth information acquisition unit 7161, an appearance pixel identification unit 7162, an attenuation unit 7163, and an auxiliary information acquisition unit 7164. The game processing unit 700, rendering unit 702, processing target frame acquisition unit 706, variation information acquisition unit 708, input frame acquisition unit 710, estimated frame acquisition unit 714, motion information acquisition unit 7160, depth information acquisition unit 7161, appearance pixel identification unit 7162, attenuation unit 7163, and auxiliary information acquisition unit 7164 are mainly realized by the control unit 10. The rendering information storage unit 704 and machine learning model storage unit 712 are mainly realized by the storage unit 12. Note that the game processing unit 700, rendering unit 702, and rendering information storage unit 704 are functions provided by game software.

[0040] [Game Processing Unit] The game processing unit 700 executes various processes related to the game. For example, in response to a game program executed by the control unit 10 or user input received by the operation unit 16, the game processing unit 700 executes processes such as placing a game object O in a virtual three-dimensional space VS, operating or moving the game object O, and changing the viewpoint C from which the virtual three-dimensional space VS is viewed (see FIG. 8 ). The game object O is composed of primitives such as polygons represented by three-dimensional data. The three-dimensional data includes geometric information indicating the positions of vertices, topological information indicating how the vertices are connected, and attribute information such as color.

[0041] [Rendering Unit] FIG. 8 is a diagram illustrating the processing of the rendering unit 702. The rendering unit 702 generates first to Nth (N is a natural number equal to or greater than 2) processing frames 40 by rendering (drawing) three-dimensional data representing one or more game objects O viewed from a predetermined viewpoint C. The rendering unit 702 performs rendering based on the results of various processes executed by the game processing unit 700. Specifically, the rendering unit 702 performs vertex processing (vertex shading) and pixel processing (pixel shading) based on three-dimensional data representing game objects O placed in the virtual three-dimensional space VS. The vertex processing includes a coordinate transformation process (perspective projection) from the view coordinate system to the screen coordinate system, and a numerical value related to a change in the viewpoint C is added to the perspective projection matrix (camera matrix) used in the coordinate transformation process, as described below. The rendering unit 702 may perform rendering based on light source information, depth information (depth buffer), texture information, normal information, etc. In addition to the above processing, the rendering unit 702 may also execute processing to apply effects such as depth of field (DoF) and motion blur. The processing of the rendering unit 702 may be set as appropriate by a game software developer or the like. Here, the game software developer or the like may adjust the MIP of the texture according to the estimated number of pixels of the estimated frame 44 or the like. This makes it possible to suppress the occurrence of noise such as moiré in the estimated frame 44.

[0042] Here, the rendering unit 702 generates each processing frame 40 by performing rendering such that the viewpoint C varies for each processing frame 40. Here, even if the game processing unit 700 fixes the viewpoint C at a predetermined position, the rendering unit 702 varies the viewpoint C for each processing frame 40. As a result, as shown in FIG. 8 , the position of the displayed game object O varies in each processing frame 40_n, 40_n+1, and 40_n+2. In other words, the rendering unit 702 applies jitter when generating each processing frame 40. Specifically, the rendering unit 702 varies the viewpoint C for each processing frame 40 by adding a numerical value corresponding to a sub-pixel size that differs for each processing frame 40 to the perspective projection matrix. The rendering unit 702 varies the viewpoint C for each processing frame 40 according to a predetermined rule. For example, a Halton sequence can be used as such a rule.

[0043] [Rendering Information Storage Unit] The rendering information storage unit 704 stores information necessary for the rendering process in the rendering unit 702 and information obtained as a result of the rendering process. For example, the rendering information storage unit 704 stores the processing target frame 40. The rendering information storage unit 704 also stores variation information, motion information, and depth information. Details of the variation information, motion information, and depth information will be described later. In addition, the rendering information storage unit 704 may store parameters used for coordinate transformation, light source information, texture information, normal information, etc.

[0044] [Processing Frame Acquisition Unit] The processing frame acquisition unit 706 acquires each of the first to Nth processing frames 40. Specifically, the processing frame acquisition unit 706 acquires each of the first to Nth processing frames 40 stored in the rendering information storage unit 704.

[0045] [Variation Information Acquisition Unit] The variation information acquisition unit 708 acquires variation information. The variation information acquisition unit 708 acquires variation information stored in the rendering information storage unit 704. Specifically, the variation information is information indicating the amount of variation of the viewpoint C between before and after the variation. The information indicating the amount of variation can also be said to be a variation vector indicating the direction and distance of the variation. For example, since the above-mentioned Halton sequence includes information indicating the amount of variation of the viewpoint C, this information may be used as the variation information.

[0046] [Input Frame Acquisition Unit] The input frame acquisition unit 710 acquires the first to Nth input frames 42, respectively, based on each processing target frame 40 by generating an input frame 42 that corresponds to the processing target frame 40 and has a number of input pixels equal to or greater than the initial number of pixels. In this embodiment, each input frame 42 has a number of input pixels greater than the initial number of pixels. That is, in this embodiment, each input frame 42 is an enlarged image of the processing target frame 40 corresponding to that input frame 42.

[0047] Specifically, the input frame acquisition unit 710 interpolates pixel values ​​at positions in the processing frame 40 that correspond to the positions of the pixels before the fluctuation, based on the fluctuation information and each pixel of each processing frame 40, to generate each input frame 42. Fig. 9 is a diagram illustrating the processing in the input frame acquisition unit 710. Fig. 9 illustrates an example of obtaining the n-th input frame 42_n. For example, as shown in Fig. 9, if the pixel center of a pixel in the input frame 42_n to be obtained is P １，０ In this case, the input frame acquisition unit 710 calculates P １，０ The pixel centers P' of the four pixels closest to ０，０ , P' １，０ , P' ０，１ , P' １，１ Based on the coordinates and pixel values ​​of P １，０ The pixel value of P' is calculated by bilinear interpolation. １，０ is the amount of fluctuation indicated by the fluctuation information. １，０The pixel values ​​of the pixels newly generated by the enlargement process are calculated in a similar manner. Note that, in addition to bilinear interpolation, various other well-known interpolation methods such as bicubic interpolation and Lanczos interpolation can also be used as the interpolation method.

[0048] When rendering is performed so that the viewpoint C varies for each processing frame 40, the amount of time-series information increases. However, by using each processing frame 40 obtained in this way (hereinafter referred to as a "variable processing frame") for estimation, an estimated frame 44 with higher image quality can be obtained.

[0049] On the other hand, if the frame to be subjected to variation processing (or an enlarged image thereof) is input directly into the machine learning model 500, the influence of the variation in viewpoint C described above may result in a decrease in the accuracy of estimation.

[0050] Therefore, as described above, in the image processing system 1, pixel values ​​at positions in each processing frame 40 corresponding to each pixel before the fluctuation are obtained by interpolation based on the fluctuation information and each pixel in each processing frame 40, and each input frame 42 is generated and input to the machine learning model 500. This corrects for the influence of fluctuations in the viewpoint C, thereby preventing a decrease in the accuracy of estimation.

[0051] [Machine Learning Model] The machine learning model 500 is a model that estimates the nth estimated frame 44_n based on the nth input frame 42_n. Specifically, the machine learning model 500 is a model that estimates the nth estimated frame 44_n based on the nth input frame 42_n and the n-1th auxiliary information 48_n-1. Specifically, the machine learning model 500 is a convolutional neural network (CNN). As the machine learning model 500, for example, a well-known model such as a multi-layered ResNet having a residual connection mechanism or a so-called encoder-decoder U-Net can be used. The model described in Non-Patent Document 1 may also be used as the machine learning model 500.

[0052] The machine learning model 500 is a model trained using a plurality of training data sets, each of which includes a training input frame having an input pixel count and a training estimation frame having an estimated pixel count. Various known techniques, such as backpropagation, can be used to train the machine learning model 500.

[0053] Specifically, the machine learning model 500 includes an accumulated feature information output layer 502, an estimated frame output layer 504, and a convolution layer 506 (see FIG. 4).

[0054] The cumulative feature information output layer 502 receives the nth input frame 42_n and the n-1th auxiliary information 48_n-1 based on the n-1th cumulative feature information 46_n-1 indicating the features of the first to n-1th input frames 42, and outputs the nth cumulative feature information 46_n indicating the features of the first to n-th input frames 42_n. The cumulative feature information output layer 502 may be composed of, for example, one or more convolutional layers. The cumulative feature information 46_n-1 is image information (bitmap information) having the same number of pixels as the number of input pixels. The cumulative feature information 46_n-1 can also be considered a feature map indicating the features of the first to n-1th input frames 42.

[0055] The cumulative feature information output layer 502 receives the first input frame 42_1 and given auxiliary information, and outputs the first cumulative feature information 46_1. When n=1, there is no previous cumulative feature information 46 or auxiliary information 48, and therefore the given auxiliary information prepared in advance is input to the cumulative feature information output layer 502 together with the first input frame 42_1.

[0056] The estimated frame output layer 504 receives the n-th accumulated feature information 46_n and outputs the n-th estimated frame 44_n. The estimated frame output layer 504 may be composed of, for example, one or more convolutional layers, similar to the accumulated feature information output layer 502. Alternatively, the estimated frame output layer 504 may be composed of one or more transposed convolutional layers (deconvolutional layers).

[0057] The convolutional layer 506 is a layer that reduces the number of channels of the accumulated feature information 46 while maintaining the number of pixels of the accumulated feature information 46. The accumulated feature information 46 output from the convolutional layer 506 is subjected to processing by the auxiliary information acquisition unit 7164. The convolutional layer 506 reduces the dimensions of the accumulated feature information 46, thereby reducing calculation costs. The convolutional layer 506 is, for example, a convolutional layer with a kernel size of 1×1, but is not limited to this.

[0058] [Machine Learning Model Storage Unit] The machine learning model storage unit 712 stores the machine learning model 500. Specifically, the machine learning model storage unit 712 stores parameters of the machine learning model 500 (such as the number of convolutional layers, the number of nodes used in each convolutional layer, and the weight of each node).

[0059] [Estimated Frame Acquisition Unit] The estimated frame acquisition unit 714 inputs each input frame 42 to the machine learning model 500 and acquires first to Nth estimated frames 44, each having an estimated pixel count greater than the initial pixel count and equal to or greater than the input pixel count. In this embodiment, the estimated frame 44 has the same number of estimated pixels as the number of input pixels. More specifically, the estimated frame acquisition unit 714 inputs the nth input frame 42_n and the (n-1)th auxiliary information 48_n-1 to the machine learning model 500 to acquire the nth estimated frame 44_n.

[0060] The auxiliary information generation unit 716 generates the (n-1)th auxiliary information 48_n-1 based on the (n-1)th cumulative feature information 46_n-1. The auxiliary information generation unit 716 includes a motion information acquisition unit 7160, a depth information acquisition unit 7161, an appearance pixel identification unit 7162, an attenuation unit 7163, and an auxiliary information acquisition unit 7164.

[0061] [Motion Information Acquisition Unit] The motion information acquisition unit 7160 acquires n-1th motion information, which is information indicating the amount and direction of motion from the n-1th processing frame 40_n-1 to the nth processing frame 40_n. Specifically, the n-1th motion information has the same number of pixels as the number of input pixels, and is image information (bitmap format information) indicating the amount and direction of motion of each pixel between the n-1th processing frame 40_n-1 and the nth processing frame 40_n. Motion information is also called a motion vector. Specifically, the motion information acquisition unit 7160 acquires original motion information having the same number of pixels as the initial number of pixels, and performs enlargement and interpolation processing on the original motion information to acquire motion information having the same number of pixels as the input pixel number.

[0062] [Depth Information Acquisition Unit] The depth information acquisition unit 7161 acquires n-1th depth information indicating the depth of each pixel in the n-1th processing target frame 40_n-1 and nth depth information indicating the depth of each pixel in the nth processing target frame 40_n. Specifically, the depth information is image information having the same number of pixels as the number of input pixels (information in bitmap format). The depth information is also called a depth buffer, Z buffer, or depth buffer. Specifically, the depth information acquisition unit 7161 acquires original depth information having the same number of pixels as the initial number of pixels, and performs enlargement and interpolation processing on the original depth information to acquire depth information having the same number of pixels as the number of input pixels.

[0063] [Appearance Pixel Identification Unit] The appearance pixel identification unit 7162 identifies the nth appearance pixel 422_n, which is a pixel of the nth input frame 42_n where all or part of the game object O, which is not displayed in the n-1th input frame 42_n-1, is displayed, based on the (n-1)th depth information and the nth depth information (see FIG. 5 ). Specifically, the appearance pixel identification unit 7162 identifies the nth appearance pixel 422_n based on the difference between the (n-1)th depth information and the nth depth information. Note that the appearance pixel identification unit 7162 may also identify the nth appearance pixel 422_n based on the (n-1)th perspective projection matrix associated with the (n-1)th input frame 42_n-1 and the nth perspective projection matrix associated with the nth input frame 42_n. Alternatively, the appearance pixel identification unit 7162 may identify the nth appearance pixel 422_n using the (n-1)th motion information. More specifically, the appearance pixel specifying unit 7162 specifies the nth appearance pixel 422_n and generates nth appearance pixel information, which is image information indicating the position of the nth appearance pixel 422_n.

[0064] [Attenuation Unit] The attenuation unit 7163 obtains the product of at least one numerical value included in the (n-1)th cumulative feature information 46_n-1 and an attenuation coefficient greater than 0 and less than 1. Here, since the cumulative feature information 46 is image information having the number of input pixels, the numerical value included in the cumulative feature information 46 is specifically a pixel value. In this embodiment, the attenuation unit 7163 obtains the product of the pixel values ​​of all pixels in the (n-1)th cumulative feature information 46_n-1 and the attenuation coefficient. Note that the attenuation unit 7163 may also obtain the product of the pixel values ​​of some of the pixels in the (n-1)th cumulative feature information 46_n-1 and the attenuation coefficient.

[0065] [Auxiliary Information Acquisition Unit] (1) The auxiliary information acquisition unit 7164 acquires the (n-1)th auxiliary information 48_n-1 in accordance with the product of at least one pixel value included in the (n-1)th cumulative feature information 46_n-1 and an attenuation coefficient. In this embodiment, the auxiliary information acquisition unit 7164 acquires the (n-1)th auxiliary information 48_n-1 in accordance with the product of the attenuation coefficient and the pixel value of each pixel of the (n-1)th cumulative feature information 46_n-1. Here, the (n-1)th auxiliary information 48_n-1 includes pixel information whose pixel value is the product of the pixel value of the pixel in the (n-1)th cumulative feature information 46_n-1 and the attenuation coefficient.

[0066] As described above, in the image processing system 1 according to this embodiment, the auxiliary information acquisition unit 7164 acquires the (n-1)th auxiliary information 48_n-1 according to the product of at least one numerical value included in the (n-1)th cumulative feature information 46_n-1 and an attenuation coefficient greater than 0 and less than 1. This prevents the same information from being excessively input into the machine learning model 500 even when a still image is being estimated, thereby suppressing the occurrence of artifacts in the resulting estimated frame 44.

[0067] (2) Furthermore, the auxiliary information acquisition unit 7164 acquires the (n-1)th auxiliary information 48_n-1 by applying motion compensation to the (n-1)th cumulative feature information 46_n-1 based on the (n-1)th motion information. Motion compensation refers to, for example, a process of moving a pixel at position x in the (n-1)th cumulative feature information 46_n to position x' when a pixel at position x in the (n-1)th input frame 42_n-1 has moved to position x' in the nth input frame 42_n (see FIG. 5 ). That is, the auxiliary information acquisition unit 7164 acquires the (n-1)th auxiliary information 48_n-1 based on the (n-1)th motion information by setting the pixel values ​​of one or more pixels in the (n-1)th cumulative feature information 46_n-1 to pixels at positions moved in accordance with the amount and direction of the pixel's movement.

[0068] If there is movement of the game object O between the nth processing frame 40_n and the n-1th processing frame 40_n-1, when obtaining the nth estimated frame 44_n, if the nth input frame 42_n and the n-1th accumulated feature information 46_n-1 are input directly into the machine learning model 500, a ghost phenomenon may occur in which an afterimage of the game object O that was displayed in the nth input frame 42_n is displayed in the output nth estimated frame 44_n.

[0069] Therefore, in the image processing system 1, as described above, motion compensation is applied to the (n-1)th accumulated feature information 46_n-1 based on the (n-1)th motion information to obtain the (n-1)th auxiliary information 48_n-1, and when obtaining the nth estimated frame 44_n, this (n-1)th auxiliary information 48_n-1 is input to the machine learning model 500. This makes it possible to suppress the ghost phenomenon described above.

[0070] (3) Furthermore, the auxiliary information acquisition unit 7164 acquires the (n-1)th auxiliary information 48_n-1 by replacing the pixel value of the nth appearance pixel 422_n in the (n-1)th cumulative feature information 46_n-1 with a predetermined value. Specifically, the auxiliary information acquisition unit 7164 acquires the (n-1)th auxiliary information 48_n-1 by replacing the pixel value of the nth appearance pixel 422_n in the (n-1)th cumulative feature information 46_n-1 with a predetermined value based on the nth appearance pixel information. The predetermined value may be a constant value such as 0 (black), or may be the pixel value of the nth appearance pixel 422_n in the nth input frame 42_n.

[0071] If all or part of a game object O that is not displayed in the n-1st processing frame 40_n-1 is displayed in the nth processing frame 40_n, and the nth input frame 42_n and the n-1st cumulative feature information 46_n-1 are input directly into the machine learning model 500 when obtaining the nth estimated frame 44_n, the aforementioned ghost phenomenon may occur in the output nth estimated frame 44_n.

[0072] Therefore, as described above, the image processing system 1 identifies the nth appearance pixel 422_n, which is a pixel in the nth input frame 42_n where all or part of the game object O that is not displayed in the (n-1)th input frame 42_n-1 is displayed, and then acquires the (n-1)th auxiliary information 48_n-1 by replacing the pixel value of the nth appearance pixel 422_n in the (n-1)th cumulative feature information 46_n-1 with a predetermined value. This makes it possible to suppress the ghost phenomenon described above.

[0073] In this embodiment, the auxiliary information acquisition unit 7164 performs the processes in the order of (1) to (3) above, but the order of the processes performed by the auxiliary information acquisition unit 7164 is not limited to this example.

[0074] 10 is a flow diagram showing an example of the flow of processing executed in the image processing system 1. The processing shown in FIG. 10 is executed by the control unit 10 operating in accordance with a program stored in the storage unit 12.

[0075] (1) Processing for n=1 First, the control unit 10 acquires a first processing frame 40_1 (S1000). The control unit 10 acquires a first input frame 42_1 based on the first processing frame 40_1 (S1002). Then, the control unit 10 inputs the first input frame 42_1 and given auxiliary information to the machine learning model 500, and acquires a first estimated frame 44_1 and first cumulative feature information 46_1 (S1004).

[0076] (2) Processing when n≧2 The control unit 10 acquires the nth processing target frame 40_n (S1006). The control unit 10 acquires the nth input frame 42_n based on the nth processing target frame 40_n (S1008).

[0077] Next, the control unit 10 acquires the (n-1)th motion information (S1010). The control unit 10 also acquires the (n-1)th depth information and the nth depth information (S1012), and identifies the nth appearance pixel 422_n based on the (n-1)th depth information and the nth depth information (S1014). The control unit 10 acquires the product of at least one pixel value included in the (n-1)th cumulative feature information and an attenuation coefficient (S1015). The control unit 10 acquires the (n-1)th auxiliary information 48_n-1 based on the (n-1)th cumulative feature information 46_n-1, the (n-1)th motion information, and the nth appearance pixel 422_n (S1016). The control unit 10 then inputs the nth input frame 42_n and the (n-1)th auxiliary information 48_n-1 into the machine learning model 500, and obtains the nth estimated frame 44_n and the nth accumulated feature information 46_n (S1018). The control unit 10 determines whether a next frame exists (S1020). If it determines that a next frame exists (S1020; Y), it increments n to n+1 and repeats the processes of S1006 to S1018. If it determines that a next frame does not exist (S1020; N), the control unit 10 terminates this process. Note that if it determines that a next frame does not exist (S1020; N), the control unit 10 may cause the display unit 18 to display the first to Nth estimated frames 44 as they are.

[0078] [5. Summary] According to the image processing system 1 according to the present embodiment described above, the nth estimated frame 44_n is estimated using the (n-1)th accumulated feature information 46_n-1 indicating the features of the first to (n-1)th input frames 42. That is, in addition to the information of the nth processing target frame 40_n, the information of the first to (n-1)th processing target frames 40 can be used for estimation, increasing the amount of information available for estimation and enabling the acquisition of a high-quality estimated frame 44_n.

[0079] Furthermore, according to the image processing system 1 of this embodiment, the (n-1)th auxiliary information 48_n-1 is acquired according to the product of at least one pixel value included in the (n-1)th cumulative feature information 46_n-1 and the attenuation coefficient. In other words, even when a still image is being estimated, it is possible to prevent the same information from being excessively input into the machine learning model 500, thereby suppressing the occurrence of artifacts in the resulting estimated frame 44.

[0080] It should be noted that the present invention is not limited to the above-described embodiment. Furthermore, the specific character strings and numerical values ​​described above and the specific character strings and numerical values ​​in the drawings are merely examples, and the present invention is not limited to these character strings and numerical values.

[0081] For example, in the present embodiment, the number of input pixels is greater than the initial number of pixels, and the number of input pixels is equal to the estimated number of pixels, but the number of input pixels may be equal to the initial number of pixels, and the estimated number of pixels may be greater than the number of input pixels. In other words, the input frame 42 does not necessarily have to be an enlarged version of the processing target frame 40.

[0082] Furthermore, the processing target frame 40 may be input directly to the machine learning model 500 .

[0083] [6. Supplementary Note] (1) An image processing system including at least one processor, wherein the at least one processor: acquires first to Nth input frames each having a predetermined number of input pixels and corresponding to first to Nth (N is a natural number equal to or greater than 2) frames to be processed; inputs each of the input frames to a machine learning model to acquire first to Nth estimated frames each having an estimated number of pixels equal to or greater than the number of input pixels; the machine learning model includes: an accumulated feature information output layer that receives the nth input frame (n=2, 3, ..., N) and n-1th auxiliary information based on n-1th accumulated feature information that indicates features of the first to n-1th input frames and includes one or more numerical values, and outputs the nth accumulated feature information that indicates the features of the first to nth input frames; and an estimated frame output layer that receives the nth accumulated feature information and outputs the nth estimated frame, and is trained using a plurality of training data each including a training input frame having the said number of input pixels and a training estimated frame having the said estimated number of pixels, The at least one processor further acquires the (n-1)th auxiliary information according to a product of at least one of the numerical values ​​included in the (n-1)th cumulative feature information and an attenuation coefficient greater than 0 and less than 1. (2) The image processing system according to (1), wherein the cumulative feature information and the auxiliary information are each image information having the number of input pixels, and the numerical value is a pixel value. (3) The image processing system according to (2), wherein the (n-1)th auxiliary information includes pixel information whose pixel value is the product of the pixel value of a pixel in the (n-1)th cumulative feature information and the attenuation coefficient. (4) The image processing system according to any of (1) to (3), wherein each of the processing target frames is an image obtained by rendering three-dimensional data showing one or more objects seen from a predetermined viewpoint.(5) The image processing system according to any one of (1) to (4), wherein the at least one processor acquires (n-1)th motion information, which is information indicating the amount and direction of motion from the (n-1)th frame to be processed to the nth frame to be processed, and acquires the (n-1)th auxiliary information by applying motion compensation to the (n-1)th accumulated feature information based on the (n-1)th motion information. (6) The image processing system according to any one of (1) to (5), wherein the at least one processor acquires (n-1)th depth information indicating a depth of each pixel of the (n-1)th frame to be processed and nth depth information indicating a depth of each pixel of the nth frame to be processed, identifies an nth appearance pixel, among the pixels of the nth input frame, which is a pixel in which all or part of the object not displayed in the (n-1)th input frame is displayed, based on the (n-1)th depth information and the nth depth information, and acquires the (n-1)th auxiliary information by replacing a pixel value of the nth appearance pixel in the (n-1)th cumulative feature information with a predetermined value. (7) The image processing system according to any one of (1) to (6), wherein the cumulative feature information output layer receives the first input frame and given auxiliary information as input, and outputs the first cumulative feature information.

Claims

1. An image processing system including at least one processor, wherein the at least one processor: acquires first to Nth input frames each having a predetermined number of input pixels and corresponding to first to Nth (N is a natural number equal to or greater than 2) frames to be processed; inputs each of the input frames to a machine learning model to acquire first to Nth estimated frames each having an estimated number of pixels equal to or greater than the number of input pixels; the machine learning model includes: an accumulated feature information output layer that receives the nth input frame (n=2, 3, ..., N) and n-1th auxiliary information based on n-1th accumulated feature information indicating features of the first to n-1th input frames and including one or more numerical values, and outputs the nth accumulated feature information indicating features of the first to nth input frames; and an estimated frame output layer that receives the nth accumulated feature information and outputs the nth estimated frame, and is trained using a plurality of training data each including a learning input frame having the input pixel number and a learning estimated frame having the estimated number of pixels; and the at least one processor further: acquiring the (n-1)th auxiliary information according to a product of at least one of the numerical values ​​included in the (n-1)th accumulated feature information and an attenuation coefficient greater than 0 and less than 1.

2. The image processing system according to claim 1, wherein the cumulative feature information and the auxiliary information are image information having the number of input pixels, and the numerical value is a pixel value.

3. The image processing system according to claim 2, wherein the (n-1)th auxiliary information includes pixel information whose pixel value is the product of the pixel value of the (n-1)th pixel of the accumulated feature information and the attenuation coefficient.

4. An image processing system according to claim 1 or 2, wherein each of the frames to be processed is an image obtained by rendering three-dimensional data showing one or more objects as seen from a predetermined viewpoint.

5. The image processing system of claim 4, wherein the at least one processor obtains (n-1)th motion information which is information indicating the amount and direction of motion from the (n-1)th frame to be processed to the nth frame to be processed, and obtains the (n-1)th auxiliary information by applying motion compensation to the (n-1)th accumulated feature information based on the (n-1)th motion information.

6. The image processing system of claim 5, wherein the at least one processor: acquires n-1th depth information indicating the depth of each pixel in the n-1th frame to be processed and nth depth information indicating the depth of each pixel in the nth frame to be processed; identifies an nth appearance pixel among the pixels of the nth input frame based on the n-1th depth information and the nth depth information, which is a pixel in which all or part of the object not displayed in the n-1th input frame is displayed; and acquires the n-1th auxiliary information by replacing the pixel value of the nth appearance pixel in the n-1th cumulative feature information with a predetermined value.

7. The image processing system according to claim 1 or 2, wherein the cumulative feature information output layer receives the first input frame and given auxiliary information, and outputs the first cumulative feature information.

8. A processor acquires first to Nth input frames each having a predetermined number of input pixels and corresponding to first to Nth (N is a natural number equal to or greater than 2) frames to be processed, and inputs each of the input frames to a machine learning model to acquire first to Nth estimated frames each having an estimated number of pixels equal to or greater than the number of input pixels, the machine learning model including: an accumulated feature information output layer that receives the nth input frame (n=2, 3, ..., N) and n-1th auxiliary information based on n-1th accumulated feature information indicating features of the first to n-1th input frames and including one or more numerical values, and outputs the nth accumulated feature information indicating features of the first to nth input frames; and an estimated frame output layer that receives the nth accumulated feature information and outputs the nth estimated frame, and is trained using a plurality of training data each including a learning input frame having the input pixel number and a learning estimated frame having the estimated number of pixels, the processor further comprising: The image processing method includes acquiring the (n-1)th auxiliary information according to a product of at least one of the numerical values ​​included in the (n-1)th accumulated feature information and an attenuation coefficient greater than 0 and less than 1.

9. A program for causing a computer to function as: a means for acquiring first to Nth input frames each having a predetermined number of input pixels and corresponding to first to Nth (N is a natural number equal to or greater than 2) frames to be processed; and a means for inputting each of the input frames to a machine learning model to acquire first to Nth estimated frames each having an estimated number of pixels equal to or greater than the number of input pixels, wherein the machine learning model includes: an accumulated feature information output layer that receives the nth input frame (n=2, 3, ..., N) and n-1th auxiliary information based on n-1th accumulated feature information indicating features of the 1st to n-1th input frames and including one or more numerical values, and outputs the nth accumulated feature information indicating features of the 1st to nth input frames; and an estimated frame output layer that receives the nth accumulated feature information and outputs the nth estimated frame, and has been trained using a plurality of training data each including a learning input frame having the said number of input pixels and a learning estimated frame having the said estimated number of pixels, and the program further includes: a means for acquiring the (n-1)th auxiliary information in accordance with a product of at least one of the numerical values ​​included in the (n-1)th cumulative characteristic information and an attenuation coefficient greater than 0 and less than 1.

Images