Image processing system, image processing method, and program

Abstract

This makes it possible to estimate a high-resolution still image with fewer artifacts based on a low-resolution still image. [Solution] At least one processor obtains n-1 motion information, which is information indicating the amount and direction of movement of each pixel between the n-1st processing frame 22_n-1 and the nth processing frame 22_n. Based on the n-1 motion information, the processor obtains n-1 auxiliary information such that the pixel value of one or more pixels of the n-1st cumulative feature information 46_n-1 is set to the pixel at the position moved according to a pseudo-random number.

Description

[Technical Field]

[0001] This invention relates to an image processing system, an image processing method, and a program. [Background technology]

[0002] Conventionally, a technique (super-resolution) has been known that uses machine learning models to estimate high-resolution still images based on low-resolution still images (see Non-Patent Document 1 below). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang. Learning a Deep Convolutional Network for Image Super-Resolution, in Proceedings of European Conference on Computer Vision (ECCV), 2014

[0004] The inventors of this application are investigating a system with the following recursive configuration (hereinafter sometimes referred to as "reference technology") in order to achieve super-resolution of moving images such as game screens. That is, a system that inputs the current frame, i.e., the nth frame, and information from past frames, i.e., information indicating the characteristics of the 1st to n-1st frames, into a machine learning model to improve the image quality of the nth frame (see Figure 2). In general, by using information from past frames in addition to the current frame in this way, it is expected that the estimation performance of the machine learning model will improve.

[0005] However, the inventors of this application have found that when using video footage (hereinafter sometimes referred to as "still images") showing scenes with little or no movement for extended periods is input, using information from past frames actually leads to a decrease in estimation performance. Specifically, when still images are input to the above system, artifacts may occur in the resulting video footage.

[0006] This is likely due in part to the lack of training of machine learning models on situations where parts of a video remain in the same position for extended periods, often with little or no movement. In other words, there are limitations on the length of video footage that can be used to train machine learning models, considering the time and cost involved. Furthermore, since video footage inherently represents scenes with movement, still images like those described above tend to be scarce in the training data used to train machine learning models. For these reasons, it is difficult to adequately train machine learning models on still images.

[0007] Furthermore, it is generally known that if the same information is repeatedly input into a machine learning model with a recursive structure, artifacts in the output will be amplified. The reference technology mentioned above also employs a recursive structure, and when a static image is used as input, the same information is repeatedly input, resulting in amplified artifacts being observed in the output.

[0008] The present invention aims to provide an image processing system, an image processing method, and a program that enable the estimation of a high-resolution still image with fewer artifacts based on a low-resolution still image. [Overview of the Initiative] [Means for solving the problem]

[0009] The image processing system according to the present invention is an image processing system including at least one processor, the at least one processor having a predetermined number of input pixels and acquiring the first to the nth input frames corresponding to the first to the nth frames to be processed, inputting each of the input frames into a machine learning model to acquire the first to the nth estimated frames having an estimated number of pixels equal to or greater than the number of input pixels, the machine learning model receives the nth input frame (n=2,3,...,N) and the nth-1st auxiliary information based on the nth-1st cumulative feature information which represents the characteristics of the first to the nth input frames and has the same number of pixels as the number of input pixels, and the first to the nth input frames The processor includes a cumulative feature information output layer that outputs the nth cumulative feature information that shows the features, and an estimated frame output layer that receives the nth cumulative feature information as input and outputs the nth estimated frame, and is trained using a plurality of training data, each including a training input frame having the number of input pixels and a training estimated frame having the number of estimated pixels, wherein at least one processor further acquires n-th motion information, which is information indicating the amount and direction of movement of each pixel between the n-th processing target frame and the nth processing target frame, and acquires n-th auxiliary information based on the n-th motion information such that the pixel value of one or more pixels of the n-th cumulative feature information is set to the pixel at the position moved according to a pseudo-random number. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an example of the hardware configuration of an image processing system. [Figure 2] This is a diagram illustrating the overview of the reference technology. [Figure 3] This diagram schematically illustrates the processing in the reference technology. [Figure 4] This is a diagram illustrating the overview of the image processing system. [Figure 5] This diagram schematically illustrates the processing in an image processing system. [Figure 6]This diagram illustrates the process of adding pseudo-motion to cumulative feature information. [Figure 7] This is a functional block diagram showing an example of the functions implemented in the image processing system. [Figure 8] This diagram explains the rendering process. [Figure 9] This diagram illustrates the processing in the input frame acquisition section. [Figure 10] This is a flowchart illustrating an example of the processing flow performed by an image processing system. [Modes for carrying out the invention]

[0011] Hereinafter, an example of an embodiment of the image processing system according to the present invention will be described with reference to the drawings.

[0012] [1. Hardware configuration of the image processing system] Figure 1 shows an example of the hardware configuration of image processing system 1. Image processing system 1 is a computer such as a game console (game machine). As shown in Figure 1, image processing system 1 includes a control unit 10, a storage unit 12, a communication unit 14, an operation unit 16, a display unit 18, and an audio output unit 19.

[0013] The control unit 10 includes a program control device such as a CPU that operates according to a program installed in the image processing system 1, for example. The control unit 10 also includes a GPU (Graphics Processing Unit) that draws images to a frame buffer based on graphics commands and data supplied by the CPU.

[0014] The storage unit 12 includes, for example, main storage devices such as ROM and RAM, and auxiliary storage devices such as HDD and SSD. The storage unit 12 stores programs and the like executed by the control unit 10. In addition to programs for implementing various functions of the image processing system 1 described later, the storage unit 12 stores, for example, a game program (game software). Further, a frame buffer area in which an image is drawn by a GPU is secured in the storage unit 12.

[0015] The communication unit 14 is a communication interface such as an Ethernet (registered trademark) module or a wireless LAN module, for example.

[0016] The operation unit 16 is a user interface such as a keyboard, a mouse, and a controller of a game console, and accepts a user's operation input and outputs a signal indicating the content of the operation input to the control unit 10.

[0017] The display unit 18 is a display device such as a liquid crystal display or an organic EL display, and displays various images according to instructions from the control unit 10.

[0018] The audio output unit 19 is, for example, a speaker or the like, and outputs audio indicated by audio data generated by the image processing system 1.

[0019] Note that, in addition to the above-described devices, the image processing system 1 may include an optical disc drive that reads optical discs such as DVD-ROM and Blu-ray (registered trademark) discs, a USB (Universal Serial Bus) port, and the like.

[0020] [2. Outline of Reference Technology] First, before describing the image processing system 1 according to this embodiment, we will explain the reference technology that serves as a prerequisite for the image processing system 1 according to this embodiment using Figures 2 and 3. Figure 2 is a diagram showing an overview of the reference technology. Figure 3 is a diagram schematically showing the processing in the reference technology. Here, we will illustrate the case in which the reference technology is used to improve the image quality of gameplay videos in a game. Gameplay videos are videos generated in response to the game program executed by the control unit or user input received by the operation unit, and consist of multiple still images (frames) which are time-series data. The processing performed in the reference technology is mainly as follows.

[0021] (1) Generation of frames to be processed First, the system relating to the reference technology generates an image (processing frame) in which a game object is drawn by rendering 3D data representing one or more game objects viewed from a predetermined viewpoint. This processing frame is an image having a predetermined number of pixels (initial number of pixels) and a predetermined image quality (initial image quality) (see Figure 3). Processing frames are generated at predetermined time intervals. The number of pixels in a processing frame is, for example, 1920 × 1080 (1080p). Each generated processing frame is not displayed directly on the display unit 18, but is temporarily stored in the storage unit 12 and used for subsequent processing. In the following explanation, processing targeting the nth processing frame 20_n will be mainly used as an example, but similar processing can be performed on other processing frames as well (i.e., n = 2, 3, ..., N).

[0022] (2) Obtaining the input frame The system related to the reference technology acquires a frame (input frame) 22_n having a greater number of pixels (input pixel count) than the initial number of pixels, based on the acquired processing target frame 20_n. The input pixel count is, for example, 3840 × 2160 (4K). Specifically, the input frame 22_n is generated by performing scaling and interpolation processing on the processing target frame 20_n (see Figure 3).

[0023] It should be noted that although the input frame 22_n has more pixels than the processing frame 20_n, its image quality is not necessarily sufficiently improved. In other words, the image quality of a frame is not simply determined by the number of pixels (high resolution). The image quality of a frame may be evaluated based on, for example, the signal-to-noise ratio, the accuracy of spatial frequency reproduction, and the temporal stability (the low level of artifacts and flickering when multiple frames are displayed consecutively), either individually or as a combination of these factors, compared to a reference frame.

[0024] (3) Obtaining the estimated frame The system related to the reference technology inputs the input frame 22_n into the machine learning model 200 and obtains the estimated frame 24_n. The estimated frame 24_n is an image with the same number of pixels as the input number (estimated number of pixels) and an image quality that is equal to or better than the initial image quality (estimated image quality) (see Figure 3).

[0025] Here, the machine learning model 200 receives the input frame 22_n in addition to the (n-1)th auxiliary information 28_n-1 (see Figures 2 and 3). The auxiliary information 28_n-1 is based on the (n-1)th cumulative feature information 26_n-1, which describes the features of the first to (n-1)th input frames 22. Details of the cumulative feature information 26 and the auxiliary information 28 will be described later.

[0026] The machine learning model 200 is a model trained using multiple training data sets, each containing a training input frame with the input pixel count and a training estimated frame with the estimated pixel count and estimated image quality.

[0027] (4) Acquisition of cumulative feature information The machine learning model 200 has a cumulative feature information output layer 202 that receives input frames 22_n and auxiliary information 28_n-1 as inputs and outputs the nth cumulative feature information 26_n which represents the features of the first to nth input frames 22 (see Figure 2). The system related to the reference technology acquires the nth cumulative feature information 26_n.

[0028] The acquired nth cumulative feature information 26_n is input to the estimation frame output layer 204, and the estimation frame output layer 204 outputs the nth estimated frame 24_n (see Figure 2). The acquired nth cumulative feature information 26_n is also stored in the memory unit 12 and used to estimate the next frame to be processed (the n+1th frame to be processed) 20_n+1, which corresponds to the estimated frame 24_n+1.

[0029] (5) Acquisition of supplementary information As mentioned above, the n-1st cumulative feature information 26_n-1 represents the characteristics of the 1st to n-1st input frames 22 (and consequently, the 1st to n-1st processing target frames 20). By using this cumulative feature information 26_n-1, which has accumulated information from past processing target frames 20, to estimate the nth estimated frame 24_n, the amount of information available for estimation increases, thus enabling the acquisition of a high-quality estimated frame 24_n.

[0030] However, if there is movement or other action in the displayed game object between the (n-1)th processing frame 20_n-1 and the nth processing frame 20_n, and the nth input frame 22_n and the cumulative feature information 26_n-1 are directly input to the machine learning model 200, a phenomenon may occur where a ghost image of the game object displayed in the (n-1)th processing frame 20_n-1 is displayed (the so-called ghosting phenomenon).

[0031] Therefore, the system relating to the reference technology obtains the (n-1)th auxiliary information 28_n-1 by applying various corrections, described later, to the cumulative feature information 26_n-1, based on information obtained during rendering (such as motion vectors and depth buffers) (see Figures 2 and 3). The obtained (n-1)th auxiliary information 28_n-1 is input to the machine learning model 200 along with the nth input frame 22_n, as described above, and is used to estimate the nth estimated frame 24_n.

[0032] As described above, according to the reference technology of this embodiment, the estimated frame 24 is estimated using auxiliary information 28, which contains accumulated past information, in addition to the input frame 22 corresponding to the current frame to be processed 20. This increases the amount of information available for estimation, making it possible to obtain a high-quality estimated frame 24_n.

[0033] [3. Overview of the Image Processing System] Next, an overview of the image processing system 1 will be described using Figures 4 and 5. Figure 4 is a diagram illustrating the overview of the image processing system 1. Figure 5 is a diagram schematically showing the processing in the image processing system 1. In particular, in the image processing system 1, in order to enable the estimation of a high-resolution still image with fewer artifacts based on a low-resolution still image, the auxiliary information generation unit 716 is configured to include a pseudo-random number addition unit 7163. Note that in the following, the explanation of configurations similar to those in the reference technology will be omitted.

[0034] Figure 6 illustrates the process of adding pseudo-motion to cumulative feature information. For example, as shown in Figure 6, if there is no movement in the displayed objects in input frames 42_n, 42_n+1, and 42_n+2, i.e., if input frames 42_n, 42_n+1, and 42_n+2 are frames related to static images, artifacts may occur in the resulting estimated frame 24.

[0035] This is likely due in part to the lack of training of machine learning models on situations where parts of a video remain in the same position for extended periods, often with little or no movement. In other words, there are limitations on the length of video footage that can be used to train machine learning models, considering the time and cost involved. Furthermore, since video footage inherently represents scenes with movement, still images like those described above tend to be scarce in the training data used to train machine learning models. For these reasons, it is difficult to adequately train machine learning models on still images.

[0036] Furthermore, it is generally known that if the same information is repeatedly input to a machine learning model with a recursive structure, artifacts in the output are amplified. In the machine learning model 500 according to this embodiment, a recursive structure is also employed, and when a static screen is used as input, the same information is repeatedly input, resulting in amplified artifacts being observed in the output estimated frame 44.

[0037] Therefore, in the image processing system 1 of this embodiment, based on the (n-1)th motion information, the pixel values ​​of pixels in the (n-1)th cumulative feature information 46_n-1 whose amount of movement is below a predetermined threshold are set to the pixels at the position moved according to a pseudo-random number. As a result, as shown in Figure 6, the features shown by each of the cumulative feature information 46_n, 46_n+1, and 46_n+2 will be slightly different. As described above, according to the image processing system 1 of this embodiment, even when a still image is the target of estimation, each of the cumulative feature information 46 will show different features, so the occurrence of artifacts in the obtained estimated frame 44 can be suppressed. The details of the image processing system 1 will be described below.

[0038] [4. Functions implemented by the image processing system] Figure 7 is a functional block diagram showing an example of the functions implemented in the image processing system 1. As shown in Figure 7, the image processing system 1 implements a game processing unit 700, a rendering unit 702, a rendering information storage unit 704, a processing target frame acquisition unit 706, a variation information acquisition unit 708, an input frame acquisition unit 710, a machine learning model storage unit 712, an estimated frame acquisition unit 714, and an auxiliary information generation unit 716. The auxiliary information generation unit 716 includes a motion information acquisition unit 7160, a pseudo-random number acquisition unit 7162, a pseudo-random number addition unit 7163, a depth information acquisition unit 7164, an occurrence pixel identification unit 7165, and an auxiliary information acquisition unit 7166. The game processing unit 700, rendering unit 702, processing target frame acquisition unit 706, variation information acquisition unit 708, input frame acquisition unit 710, estimated frame acquisition unit 714, motion information acquisition unit 7160, pseudo-random number acquisition unit 7162, pseudo-random number addition unit 7163, depth information acquisition unit 7164, appearance pixel identification unit 7165, and auxiliary information acquisition unit 7166 are mainly implemented by the control unit 10. The rendering information storage unit 704 and machine learning model storage unit 712 are mainly implemented by the storage unit 12. Note that the game processing unit 700, rendering unit 702, and rendering information storage unit 704 are functions provided by the game software.

[0039] [Game Processing Department] The game processing unit 700 performs various processes related to the game. For example, the game processing unit 700 performs processes such as placing game objects O in the virtual 3D space VS, operating or moving game objects O, and changing the viewpoint C from which the virtual 3D space VS is viewed, in response to the game program executed by the control unit 10 or user input received by the operation unit 16 (see Figure 8). Game objects O are composed of primitives such as polygons represented by 3D data. The 3D data includes geometric information indicating the positions of vertices, topological information indicating how the vertices are connected, and attribute information such as color.

[0040] [Rendering section] Figure 8 is a diagram illustrating the processing of the rendering unit 702. The rendering unit 702 generates the first to Nth (N is a natural number greater than or equal to 2) frames 40 to be processed by rendering (drawing) 3D data representing one or more game objects O as seen from a predetermined viewpoint C. The rendering unit 702 performs rendering based on the results of various processes performed by the game processing unit 700. Specifically, the rendering unit 702 performs vertex processing (vertex shading) and pixel processing (pixel shading) based on 3D data representing game objects O placed in a virtual 3D space VS. The vertex processing includes coordinate transformation processing from the view coordinate system to the screen coordinate system (perspective projection), and the perspective projection matrix (camera matrix) used in the coordinate transformation processing has numerical values ​​related to the change in viewpoint C added to it, as will be described later. The rendering unit 702 may also perform rendering based on light source information, depth information (depth buffer), texture information, normal information, etc. In addition to the above processing, the rendering unit 702 may also perform processes such as applying effects like depth of field (DoF) or motion blur. The processing of the rendering unit 702 may be set as appropriate by the game software developer. Here, the game software developer may adjust the texture's MIP according to the estimated number of pixels in the estimated frame 44. This makes it possible to suppress the generation of noise such as moiré in the estimated frame 44.

[0041] Here, the rendering unit 702 generates each processing target frame 40 by performing rendering such that the viewpoint C changes with each processing target frame 40. Even if the game processing unit 700 fixes the viewpoint C in a predetermined position, the rendering unit 702 adds a change to the viewpoint C with each processing target frame 40. As a result, as shown in Figure 8, the position of the displayed game object O changes in each of the processing target frames 40_n, 40_n+1, and 40_n+2. In other words, the rendering unit 702 applies jitter when generating each processing target frame 40. Specifically, the rendering unit 702 changes the viewpoint C with each processing target frame 40 by adding a numerical value corresponding to a size less than one pixel, which differs for each processing target frame 40, to the perspective projection matrix. The rendering unit 702 changes the viewpoint C with each processing target frame 40 according to a predetermined rule. For example, the Halton column can be used as such a rule.

[0042] [Rendering Information Storage Unit] The rendering information storage unit 704 stores information necessary for the rendering process in the rendering unit 702 and information obtained as a result of the rendering process. For example, the rendering information storage unit 704 stores the frame 40 to be processed. The rendering information storage unit 704 also stores variation information, motion information, and depth information. Details of variation information, motion information, and depth information will be described later. In addition, the rendering information storage unit 704 may store parameters used for coordinate transformation, light source information, texture information, normal information, etc.

[0043] [Processing target frame acquisition unit] The frame acquisition unit 706 acquires the first to the Nth frame 40. Specifically, the frame acquisition unit 706 acquires the first to the Nth frame 40 stored in the rendering information storage unit 704.

[0044] [Variable Information Acquisition Unit] The variation information acquisition unit 708 acquires variation information. The variation information acquisition unit 708 acquires variation information stored in the rendering information storage unit 704. Specifically, the variation information is information indicating the amount of variation of the viewpoint C between before and after the variation. It can also be said that the information indicating the amount of variation is a variation vector indicating the direction and distance of the variation. For example, since the Halton sequence described above includes information indicating the amount of variation of the viewpoint C, this information may be used as the variation information.

[0045] [Input Frame Acquisition Unit] The input frame acquisition unit 710 acquires each of the first to N-th input frames 42 by generating, based on each processing target frame 40, an input frame 42 corresponding to the processing target frame 40 and having an input pixel count equal to or greater than the initial pixel count. In the present embodiment, each input frame 42 has an input pixel count larger than the initial pixel count. That is, in the present embodiment, each input frame 42 is an image obtained by enlarging the processing target frame 40 corresponding to the input frame 42.

[0046] Specifically, based on the variation information and each pixel of each processing target frame 40, the input frame acquisition unit 710 obtains, by interpolation, the pixel value at the position corresponding to each pixel before variation in the processing target frame 40, and generates each input frame 42. FIG. 9 is a diagram illustrating processing in the input frame acquisition unit 710. In FIG. 9, a case of obtaining the n-th input frame 42_n is illustrated as an example. For example, as shown in FIG. 9, the pixel center of a certain pixel in the input frame 42_n to be acquired is P 1,0 , then the input frame acquisition unit 710 obtains P in the processing target frame 40_n 1,0 the pixel center P' of each of the four pixels closest to 0,0 , P' 1,0 , P' 0,1 , P' 1,1 , the pixel value of P 1,0 is obtained by bilinear interpolation based on the coordinates and pixel values of . Here, P' 1,0 is P by the amount of variation indicated by the variation information 1,0It is located at a position shifted from the original. The pixel values ​​of newly generated pixels through the scaling process can be determined in the same way. In addition to bilinear interpolation, various other known methods such as bicubic interpolation and Lanczos interpolation can be used as interpolation methods.

[0047] When rendering is performed so that the viewpoint C changes with each frame 40 to be processed, the amount of information in the time series increases. If each of the frames 40 to be processed obtained in this way (hereinafter referred to as "changing frame") is used for estimation, a higher quality estimated frame 44 can be obtained.

[0048] On the other hand, if the frame to be processed for variation (or an enlarged image thereof) is directly input into the machine learning model 500, the accuracy of the estimation may decrease due to the effects of the variation in viewpoint C mentioned above.

[0049] Therefore, as described above, the image processing system 1 interpolates the pixel values ​​at the positions corresponding to each pixel before the change in the processing frame 40, based on the change information and each pixel of each processing frame 40, generates each input frame 42, and inputs this to the machine learning model 500. This corrects for the effect of changes in viewpoint C, thereby suppressing a decrease in estimation accuracy.

[0050] [Machine learning models] The machine learning model 500 is a model that estimates the nth estimated frame 44_n based on the nth input frame 42_n. Specifically, the machine learning model 500 is a model that estimates the nth estimated frame 44_n based on the nth input frame 42_n and the (n-1) auxiliary information 48_n-1. Specifically, the machine learning model 500 is a Convolutional Neural Network (CNN). As the machine learning model 500, known models such as the multilayer structure ResNet with a residual connection mechanism and the so-called encoder-decoder type U-Net can be used. The model described in Non-Patent Document 1 may also be used as the machine learning model 500.

[0051] Machine learning model 500 is a model trained using multiple training datasets, each containing a training input frame with input pixel counts and a training estimated frame with estimated pixel counts. Various known methods, such as backpropagation, can be used to train machine learning model 500.

[0052] The machine learning model 500 specifically includes a cumulative feature information output layer 502, an estimated frame output layer 504, and a convolutional layer 506 (see Figure 4).

[0053] The cumulative feature information output layer 502 receives the nth input frame 42_n and the nth-1st auxiliary information 48_n-1 based on the nth-1st cumulative feature information 46_n-1 that represents the features of the first to nth input frames 42, and outputs the nth cumulative feature information 46_n that represents the features of the first to nth input frames 42_n. The cumulative feature information output layer 502 may be composed of, for example, one or more convolutional layers. The cumulative feature information 46_n-1 is image information (information in bitmap format) having the same number of pixels as the number of input pixels. The cumulative feature information 46_n-1 can also be described as a feature map that represents the features of the first to nth-1st input frames 42.

[0054] The cumulative feature information output layer 502 receives the first input frame 42_1 and the given auxiliary information, and outputs the first cumulative feature information 46_1. When n=1, since there is no previous cumulative feature information 46 and auxiliary information 48, the pre-prepared given auxiliary information is input to the cumulative feature information output layer 502 along with the first input frame 42_1.

[0055] The estimated frame output layer 504 receives the nth cumulative feature information 46_n as input and outputs the nth estimated frame 44_n. The estimated frame output layer 504 may be composed of, for example, one or more convolutional layers, similar to the cumulative feature information output layer 502. Alternatively, the estimated frame output layer 504 may be composed of one or more transposed convolutional layers (inverse convolutional layers).

[0056] The convolutional layer 506 is a layer that reduces the number of channels in the cumulative feature information 46 while maintaining the number of pixels. The cumulative feature information 46 output from the convolutional layer 506 is used for processing by the auxiliary information acquisition unit 7166. Because the dimension of the cumulative feature information 46 is reduced by the convolutional layer 506, computational costs can be reduced. The convolutional layer 506 is, for example, a convolutional layer with a kernel size of 1x1, but is not limited to this.

[0057] [Machine Learning Model Memory Unit] The machine learning model storage unit 712 stores the machine learning model 500. Specifically, the machine learning model storage unit 712 stores the parameters of the machine learning model 500 (such as the number of convolutional layers, the number of nodes used in each convolutional layer, and the weights of each node).

[0058] [Estimated Frame Acquisition Unit] The estimated frame acquisition unit 714 inputs each input frame 42 to the machine learning model 500 and acquires the first to nth estimated frames 44, each having an estimated number of pixels greater than or equal to the input number of pixels and greater than or equal to the initial number of pixels. In this embodiment, the estimated frame 44 has the same number of estimated pixels as the input number of pixels. More specifically, the estimated frame acquisition unit 714 inputs the nth input frame 42_n and the n-1st auxiliary information 48_n-1 to the machine learning model 500 and acquires the nth estimated frame 44_n.

[0059] [Auxiliary information generation section] The auxiliary information generation unit 716 generates the (n-1)th auxiliary information 48_n-1 based on the (n-1)th cumulative feature information 46_n-1. The auxiliary information generation unit 716 includes a motion information acquisition unit 7160, a pseudo-random number sequence storage unit 7161, a pseudo-random number acquisition unit 7162, a pseudo-random number addition unit 7163, a depth information acquisition unit 7164, an occurrence pixel identification unit 7165, and an auxiliary information acquisition unit 7166.

[0060] [Motion Information Acquisition Unit] The motion information acquisition unit 7160 acquires the n-th motion information, which is information indicating the amount and direction of movement from the n-th processing target frame 40_n-1 to the nth processing target frame 40_n. Specifically, the n-th motion information is image information (information in bitmap format) that has the same number of pixels as the number of input pixels and indicates the amount and direction of movement of each pixel between the n-th processing target frame 40_n-1 and the nth processing target frame 40_n. In other words, the pixel value of each pixel in the n-th motion information indicates the amount and direction of movement of each pixel between the n-th processing target frame 40_n-1 and the nth processing target frame 40_n. That is, the pixel value of each pixel in the n-th motion information is a two-dimensional vector indicating the amount and direction of movement of each pixel between the n-th processing target frame 40_n-1 and the nth processing target frame 40_n. Motion information is also called a motion vector. Specifically, the motion information acquisition unit 7160 acquires original motion information having the same number of pixels as the initial number of pixels, and performs scaling and interpolation processing on the original motion information to acquire motion information having the same number of pixels as the input number of pixels.

[0061] [Pseudo-random number generation unit] The pseudo-random number acquisition unit 7162 acquires a pseudo-random number for every 46 cumulative feature information points. Specifically, the pseudo-random number acquisition unit 7162 acquires a pseudo-random number for every 46 cumulative feature information points by generating a pseudo-random number according to a pseudo-random number generator. The pseudo-random number is either a positive or negative value. Various known pseudo-random number generators can be used as the pseudo-random number generator. The pseudo-random number acquisition unit 7162 may also acquire a pseudo-random number for every 46 cumulative feature information points from a random number table pre-stored in the memory unit 12. However, if the period of the pseudo-random number is short, such as tens to hundreds of tens

[0062] Specifically, the pseudo-random number acquisition unit 7162 acquires two pseudo-random numbers (a first pseudo-random number and a second pseudo-random number) for each cumulative feature information 46. The pseudo-random number acquisition unit 7162 can also acquire a two-dimensional pseudo-random number vector for each cumulative feature information 46. Here, it is preferable that the pseudo-random number acquisition unit 7162 acquires two pseudo-random numbers for each cumulative feature information 46 such that the values ​​of the two pseudo-random numbers for each cumulative feature information 46 are different from each other. For example, it is preferable that the pseudo-random number acquisition unit 7162 acquires the two pseudo-random numbers for each cumulative feature information 46 by generating each of the two pseudo-random numbers based on each of two different random number seeds.

[0063] More specifically, the pseudo-random number acquisition unit 7162 acquires pseudo-random numbers for each of the first to Nth cumulative feature information 26 such that the average value of the pseudo-random numbers related to each of the first to Nth cumulative feature information 26 is 0. In detail, the pseudo-random number acquisition unit 7162 acquires pseudo-random numbers for each of the first to Nth cumulative feature information 26 such that the average value of the first pseudo-random numbers related to each of the first to Nth cumulative feature information 26 is 0, and the average value of the second pseudo-random numbers related to each of the first to Nth cumulative feature information 26 is 0. As a result, the amount of movement in the pseudo-random number addition unit 7163, described later, for each cumulative feature information 26 becomes so small that it can be evaluated as not moving when averaged over time, thus suppressing the impact on estimation and preventing the occurrence of artifacts.

[0064] Furthermore, the pseudo-random number acquisition unit 7162 acquires pseudo-random numbers for each of the first to Nth cumulative feature information 26 such that the pseudo-random numbers related to each of the first to Nth cumulative feature information 26 follow a uniform distribution. Specifically, the pseudo-random number acquisition unit 7162 acquires pseudo-random numbers for each of the first to Nth cumulative feature information 26 such that the first pseudo-random number related to each of the first to Nth cumulative feature information 26 follows a uniform distribution, and the second pseudo-random number related to each of the first to Nth cumulative feature information 26 follows a uniform distribution. This makes it possible to suppress the occurrence of artifacts while more favorably reducing the impact on estimation. Note that the pseudo-random numbers related to each of the first to Nth cumulative feature information 26 may, for example, follow a normal distribution.

[0065] Furthermore, the pseudo-random number acquisition unit 7162 acquires pseudo-random numbers for each of the first to Nth cumulative feature information 26 such that the pseudo-random numbers related to each of the first to Nth cumulative feature information 26 are within a predetermined range of magnitude. Specifically, it is preferable that the pseudo-random numbers are 0.1 or less in magnitude. More specifically, the pseudo-random number acquisition unit 7162 acquires pseudo-random numbers for each of the first to Nth cumulative feature information 26 such that the first pseudo-random number related to each of the first to Nth cumulative feature information 26 is within a predetermined range of magnitude, and the second pseudo-random number related to each of the first to Nth cumulative feature information 26 is within a predetermined range of magnitude. This makes it possible to suppress the occurrence of artifacts while further optimizing the impact on estimation.

[0066] [Pseudo-random number generation section] The pseudo-random number generator 7163 adds pseudo-random numbers related to the n-1 cumulative feature information to the pixel values ​​of one or more pixels in the n-1 motion information. Here, since the pixel values ​​of each pixel included in the n-1 motion information are two-dimensional vectors having two elements, the pseudo-random number generator 7163, more specifically, adds each of the two pseudo-random numbers related to the n-1 cumulative feature information to each of these two elements.

[0067] Specifically, the pseudo-random number generator 7163 adds pseudo-random numbers related to the n-1st cumulative feature information to the pixel values ​​of pixels in the n-1st motion information whose amount of motion is below a predetermined threshold. Alternatively, the pseudo-random number generator 7163 may add pseudo-random numbers related to the n-1st cumulative feature information to the pixel values ​​of all pixels in the n-1st motion information.

[0068] According to the pseudo-random number addition unit 7163 described above, even when a static screen is the target of estimation, each accumulated feature information 46 will exhibit different features, thereby suppressing the occurrence of artifacts in the resulting estimated frame 44.

[0069] [Depth information acquisition section] The depth information acquisition unit 7164 acquires the (n-1)th depth information, which indicates the depth of each pixel in the (n-1)th processing target frame 40_n-1, and the nth depth information, which indicates the depth of each pixel in the nth processing target frame 40_n. Specifically, the depth information is image information (information in bitmap format) having the same number of pixels as the input number of pixels. The depth information is also called a depth buffer, Z buffer, or depth buffer. Specifically, the depth information acquisition unit 7164 acquires original depth information having the same number of pixels as the initial number of pixels, and performs scaling and interpolation processing on the original depth information to acquire depth information having the same number of pixels as the input number of pixels.

[0070] [Pixel Identification Section] The pixel appearance identification unit 7165 identifies the nth appearing pixel 422_n, which is a pixel in the nth input frame 42_n that displays all or part of the game object O that was not displayed in the nth input frame 42_n-1, based on the (n-1)th depth information and the nth depth information (see Figure 5). Specifically, the pixel appearance identification unit 7165 identifies the nth appearing pixel 422_n based on the difference between the (n-1)th depth information and the nth depth information. Alternatively, the pixel appearance identification unit 7165 may identify the nth appearing pixel 422_n based on the (n-1)th perspective projection matrix related to the nth input frame 42_n-1 and the nth perspective projection matrix related to the nth input frame 42_n. Furthermore, the pixel appearance identification unit 7165 may identify the nth appearing pixel 422_n using the (n-1)th motion information. More specifically, the occurrence pixel identification unit 7165 identifies the nth occurrence pixel 422_n and generates the nth occurrence pixel information, which is image information indicating the position of the nth occurrence pixel 422_n.

[0071] [Auxiliary information acquisition department] The auxiliary information acquisition unit 7166 acquires the n-th auxiliary information 48_n-1 by applying motion compensation to the n-th cumulative feature information 46_n-1 based on the n-th motion information. In this embodiment, the auxiliary information acquisition unit 7166 acquires the n-th auxiliary information 48_n-1 by applying motion compensation to the n-th cumulative feature information 46_n-1 based on the n-th motion information to which a pseudo-random number related to the n-th cumulative feature information 26_n-1 has been added. Motion compensation refers to the process of moving the pixel at position x in the n-th cumulative feature information 46_n to position x' when, for example, a pixel at position x in the n-th input frame 42_n-1 has moved to position x' in the nth input frame 42_n (see Figure 5). In other words, the auxiliary information acquisition unit 7166 acquires the n-th auxiliary information 48_n-1 based on the n-th motion information to which a pseudo-random number related to the n-th cumulative feature information 26_n-1 has been added, such that the pixel value of one or more pixels of the n-th cumulative feature information 46_n-1 is set to the pixel at the position where the pixel has moved according to the amount and direction of the pixel's movement.

[0072] If there is movement of the game object O between the nth processing frame 40_n and the (n-1)th processing frame 40_n-1, and the nth estimated frame 44_n is obtained by directly inputting the nth input frame 42_n and the (n-1)th cumulative feature information 46_n-1 into the machine learning model 500, a ghosting phenomenon may occur in the output nth estimated frame 44_n, where the afterimage of the game object O that was displayed in the nth input frame 42_n is displayed.

[0073] Therefore, in the image processing system 1, motion compensation is applied to the (n-1)th cumulative feature information 46_n-1 based on the (n-1)th motion information, and the (n-1)th auxiliary information 48_n-1 is obtained. When obtaining the nth estimated frame 44_n, this (n-1)th auxiliary information 48_n-1 is input to the machine learning model 500. This makes it possible to suppress the ghosting phenomenon described above.

[0074] Furthermore, in this embodiment, motion compensation is applied to the n-1st cumulative feature information 46_n-1 based on the n-1st motion information to which a pseudo-random number related to the n-1st cumulative feature information 26_n-1 has been added, thereby obtaining the n-1st auxiliary information 48_n-1. As a result, even when a static screen is the target of estimation, each cumulative feature information 46 will exhibit different features, thereby suppressing the occurrence of artifacts in the obtained estimated frame 44.

[0075] Furthermore, the auxiliary information acquisition unit 7166 acquires the (n-th)th auxiliary information 48_n-1 by replacing the pixel value of the nth occurrence pixel 422_n in the (n-th)th cumulative feature information 46_n-1 with a predetermined value. Specifically, the auxiliary information acquisition unit 7166 acquires the (n-th)th auxiliary information 48_n-1 by replacing the pixel value of the nth occurrence pixel 422_n in the (n-th)th cumulative feature information 46_n-1 with a predetermined value based on the nth occurrence pixel information. The predetermined value may be a constant value such as 0 (black), or it may be the pixel value of the nth occurrence pixel 422_n in the nth input frame 42_n.

[0076] If, in the nth processing frame 40_n, all or part of a game object O that was not displayed in the (n-th)th processing frame 40_n-1 is displayed, and the nth estimated frame 44_n is obtained by directly inputting the nth input frame 42_n and the (n-th)th cumulative feature information 46_n-1 into the machine learning model 500, the aforementioned ghosting phenomenon may occur in the output nth estimated frame 44_n.

[0077] Therefore, the image processing system 1 identifies the nth occurrence pixel 422_n, which is a pixel in the nth input frame 42_n that displays all or part of the game object O that was not displayed in the (n-th)th input frame 42_n-1. The (n-th)th auxiliary information 48_n-1 is obtained by replacing the pixel value of the nth occurrence pixel 422_n in the (n-th)th cumulative feature information 46_n-1 with a predetermined value. This makes it possible to suppress the ghosting phenomenon described above.

[0078] [4. Processing performed by the image processing system] Figure 10 is a flowchart showing an example of the processing flow performed by the image processing system 1. The processing shown in Figure 10 is performed by the control unit 10 operating according to the program stored in the storage unit 12.

[0079] (1) Processing when n=1 First, the control unit 10 acquires the first frame to be processed 40_1 (S1000). Based on the first frame to be processed 40_1, the control unit 10 acquires the first input frame 42_1 (S1002). Then, the control unit 10 inputs the first input frame 42_1 and the given auxiliary information to the machine learning model 500 and acquires the first estimated frame 44_1 and the first cumulative feature information 46_1 (S1004).

[0080] (2) Processing when n≧2 The control unit 10 obtains the nth frame to be processed 40_n (S1006). Based on the nth frame to be processed 40_n, the control unit 10 obtains the nth input frame 42_n (S1008).

[0081] Next, the control unit 10 acquires the (n-1)th motion information (S1010). The control unit 10 also acquires the (n-1)th depth information and the nth depth information (S1012), and identifies the nth occurrence pixel 422_n based on the (n-1)th depth information and the nth depth information (S1014). The control unit 10 adds pseudo-random numbers related to the (n-1)th cumulative feature information to the pixel values ​​of one or more pixels of the (n-1)th motion information (S1015). The control unit 10 acquires the (n-1)th auxiliary information 48_n-1 based on the (n-1)th cumulative feature information 46_n-1, the (n-1)th motion information, and the nth occurrence pixel 422_n (S1016). The control unit 10 then inputs the nth input frame 42_n and the (n-1)th auxiliary information 48_n-1 to the machine learning model 500 and obtains the nth estimated frame 44_n and the nth cumulative feature information 46_n (S1018). The control unit 10 determines whether or not the next frame exists (S1020), and if it determines that the next frame exists (S1020; Y), it increments n = n+1 and repeats the process from S1006 to S1018. If the control unit 10 determines that the next frame does not exist (S1020; N), it terminates this process. Alternatively, if the control unit 10 determines that the next frame does not exist (S1020; N), it may display the first to the nth estimated frames 44 on the display unit 18.

[0082] [5. Summary] According to the image processing system 1 of this embodiment described above, the nth estimated frame 44_n is estimated using the nth cumulative feature information 46_n-1, which represents the features of the first to nth-first input frames 42. In other words, in addition to the information of the nth processing target frame 40_n, the information of the first to nth-first processing target frames 40 can be used for estimation, so the amount of information available for estimation increases, and a high-quality estimated frame 44_n can be obtained.

[0083] Furthermore, according to the image processing system 1 of this embodiment, even when a still image is the target of estimation, each cumulative feature information 46 will exhibit different features, thus suppressing the occurrence of artifacts in the resulting estimated frame 44.

[0084] Furthermore, the present invention is not limited to the embodiments described above. Also, the specific strings and numbers shown above and in the drawings are illustrative examples and the invention is not limited to these.

[0085] For example, in this embodiment, we have illustrated the case where the number of input pixels is greater than the initial number of pixels, and the number of input pixels and the estimated number of pixels are the same. However, the number of input pixels and the initial number of pixels may be the same, and the estimated number of pixels may be greater than the number of input pixels. In other words, the input frame 42 does not necessarily have to be an enlarged version of the frame 40 to be processed.

[0086] Furthermore, although this embodiment describes a case where the processing of the auxiliary information acquisition unit 7166 is performed after the processing of the pseudo-random number addition unit 7163, it is also possible to add the pseudo-random number vector acquired by the pseudo-random number acquisition unit 7162 to the pixel value of one or more pixels of the auxiliary information 28 after the processing of the auxiliary information acquisition unit 7166. In short, it is sufficient for the auxiliary information acquisition unit 7166 to acquire the (n-1) auxiliary information such that, based on the (n-1) motion information, the pixel value of one or more pixels of the (n-1) cumulative feature information is set to the pixel at the position moved according to the pseudo-random number.

[0087] Furthermore, in this embodiment, we have illustrated the case where a common pseudorandom number is added to the pixel values ​​of one or more pixels of the (n-1)th motion information. However, it is also possible to add different pseudorandom numbers to each of the pixel values ​​of one or more pixels of the (n-1)th motion information.

[0088] Alternatively, the frame 40 to be processed may be directly input into the machine learning model 500.

[0089] [6. Addendum] (1) An image processing system comprising at least one processor, The at least one processor is Having a predetermined number of input pixels, the system acquires the first to Nth input frames corresponding to the first to Nth frames to be processed (where N is a natural number greater than or equal to 2). Each of the aforementioned input frames is input to a machine learning model to obtain the first to Nth estimated frames, each having an estimated number of pixels equal to or greater than the number of input pixels. The aforementioned machine learning model, A cumulative feature information output layer receives the nth input frame (n=2,3,...,N), the nth auxiliary information based on the nth cumulative feature information which represents the characteristics of the first to nth input frames and has the same number of pixels as the input pixels, and outputs the nth cumulative feature information which represents the characteristics of the first to nth input frames. An estimated frame output layer receives the nth cumulative feature information as input and outputs the nth estimated frame, Includes, This is trained using multiple training data sets, each including a training input frame having the aforementioned input pixel count and a training estimation frame having the aforementioned estimated pixel count. The aforementioned at least one processor further, The (n-1) motion information is obtained, which is information indicating the amount and direction of movement of each pixel between the (n-1) frame to be processed and the nth frame to be processed. Based on the (n-1) motion information, the (n-1) auxiliary information is obtained such that the pixel values ​​of one or more pixels of the (n-1) cumulative feature information are set to the pixels at the positions moved according to pseudo-random numbers. Image processing system. (2) The at least one processor is Based on the (n-1) motion information, the (n-1) auxiliary information is acquired such that the pixel value of the pixel among the (n-1) cumulative feature information whose amount of motion is less than or equal to a predetermined threshold is set to the pixel at the position moved according to the pseudo-random number. (1) The image processing system described above. (3) The at least one processor is The pseudorandom numbers are acquired for each of the first to Nth cumulative feature information such that the average value of the pseudorandom numbers related to each of the first to Nth cumulative feature information is 0. The image processing system described in (1) or (2). (4) The pseudorandom numbers are acquired for each of the first to Nth cumulative feature information such that the pseudorandom numbers related to each of the first to Nth cumulative feature information follow a uniform distribution. An image processing system as described in any of (1) to (3). (5) The pseudorandom numbers for each of the first to Nth cumulative feature information are acquired such that the pseudorandom numbers for each of the first to Nth cumulative feature information are within a predetermined range. An image processing system as described in any of (1) to (4). (6) Each of the aforementioned processing frames is an image obtained by rendering 3D data showing one or more objects viewed from a predetermined viewpoint. An image processing system as described in any of (1) to (5). (7) The at least one processor is Based on the (n-1) motion information, motion compensation is applied to the (n-1) cumulative feature information to acquire the (n-1) auxiliary information. An image processing system as described in any of (1) to (6). (8) The cumulative feature information output layer receives the first input frame and given auxiliary information as input and outputs the first cumulative feature information. An image processing system as described in any of (1) to (7). [Explanation of Symbols]

[0090] 1 Image processing system, 10 Control unit, 12 Memory unit, 14 Communication unit, 16 Operation unit, 18 Display unit, 19 Audio output unit, 40 Frame to be processed, 42 Input frame, 44 Estimated frame, 46 Cumulative feature information, 48 Auxiliary information, 500 Machine learning model, 502 Cumulative feature information output layer, 504 Estimated frame output layer, 506 Convolutional layer, 422 Occurring pixels, 700 Game processing unit, 702 Rendering unit, 704 Rendering information storage unit, 706 Frame to be processed acquisition unit, 708 Fluctuation information acquisition unit, 710 Input frame acquisition unit, 712 Machine learning model storage unit, 714 Estimated frame acquisition unit, 716 Auxiliary information generation unit, 7160 Motion information acquisition unit, 7162 Pseudo-random number acquisition unit, 7163 Pseudo-random number addition unit, 7164 Depth information acquisition unit, 7165 Pixel identification unit, 7166 auxiliary information acquisition unit, C viewpoint, O game object, VS virtual 3D space.

Claims

1. An image processing system comprising at least one processor, The at least one processor is Having a predetermined number of input pixels, the system acquires the first to Nth input frames corresponding to the first to Nth processing target frames (where N is a natural number greater than or equal to 2). Each of the aforementioned input frames is input to a machine learning model to obtain the first to Nth estimated frames, each having an estimated number of pixels equal to or greater than the number of input pixels. The aforementioned machine learning model, A cumulative feature information output layer receives the nth input frame (n = 2, 3, ..., N) and the nth-1st auxiliary information based on the nth-1st cumulative feature information, which is image information that shows the characteristics of the first to the nth input frames and has the same number of pixels as the number of input pixels, and outputs the nth cumulative feature information that shows the characteristics of the first to the nth input frames. An estimated frame output layer receives the nth cumulative feature information and outputs the nth estimated frame, Includes, This is trained using multiple training data sets, each including a training input frame having the aforementioned input pixel count and a training estimation frame having the aforementioned estimated pixel count. The aforementioned at least one processor further, The (n-1) motion information is obtained, which is information indicating the amount and direction of movement of each pixel between the (n-1) frame to be processed and the nth frame to be processed. Based on the n-1 motion information, the n-1 auxiliary information is acquired such that the pixel values ​​of one or more pixels in the n-1 cumulative feature information are set to the pixels at the positions moved according to pseudorandom numbers. Image processing system.

2. The at least one processor is Based on the n-1 motion information, the n-1 auxiliary information is acquired such that the pixel values ​​of the pixels in the n-1 cumulative feature information whose amount of motion is less than or equal to a predetermined threshold are set to the pixels at the positions moved according to the pseudo-random numbers. The image processing system according to claim 1.

3. The at least one processor is The pseudorandom numbers are acquired for each of the first to Nth cumulative feature information such that the average value of the pseudorandom numbers related to each of the first to Nth cumulative feature information is 0. The image processing system according to claim 1 or 2.

4. The pseudorandom numbers are acquired for each of the first to Nth cumulative feature information such that the pseudorandom numbers related to each of the first to Nth cumulative feature information follow a uniform distribution. The image processing system according to claim 1 or 2.

5. The pseudorandom numbers for each of the first to N cumulative feature information are acquired such that the pseudorandom numbers for each of the first to N cumulative feature information are within a predetermined range. The image processing system according to claim 1 or 2.

6. Each of the aforementioned processing frames is an image obtained by rendering three-dimensional data showing one or more objects viewed from a predetermined viewpoint. The image processing system according to claim 1 or 2.

7. The at least one processor is Based on the n-1 motion information, motion compensation is applied to the n-1 cumulative feature information to acquire the n-1 auxiliary information. The image processing system according to claim 1 or 2.

8. The cumulative feature information output layer receives the first input frame and given auxiliary information as input and outputs the first cumulative feature information. The image processing system according to claim 1 or 2.

9. The processor Having a predetermined number of input pixels, the system acquires the first to Nth input frames corresponding to the first to Nth processing target frames (where N is a natural number greater than or equal to 2). Each of the aforementioned input frames is input to a machine learning model to obtain the first to Nth estimated frames, each having an estimated number of pixels equal to or greater than the number of input pixels. The aforementioned machine learning model, A cumulative feature information output layer receives the nth input frame (n = 2, 3, ..., N) and the nth-1st auxiliary information based on the nth-1st cumulative feature information, which is image information that shows the characteristics of the first to the nth input frames and has the same number of pixels as the number of input pixels, and outputs the nth cumulative feature information that shows the characteristics of the first to the nth input frames. An estimated frame output layer receives the nth cumulative feature information and outputs the nth estimated frame, Includes, This is trained using multiple training data sets, each including a training input frame having the aforementioned input pixel count and a training estimation frame having the aforementioned estimated pixel count. The aforementioned processor further, The (n-1)th motion information is obtained, which is information indicating the amount and direction of movement of each pixel between the (n-1)th input frame and the nth input frame. Based on the (n-1) motion information, the pixel values ​​of one or more pixels in the (n-1) cumulative feature information are set to the pixels at the positions moved according to pseudo-random numbers. Image processing methods.

10. Means for acquiring the first to Nth input frames, each corresponding to a first to Nth (where N is a natural number greater than or equal to 2) frame to be processed, having a predetermined number of input pixels. Means for inputting each of the aforementioned input frames into a machine learning model to obtain first to Nth estimated frames, each having an estimated number of pixels equal to or greater than the number of input pixels, It is a program that makes a computer function as such. The aforementioned machine learning model, A cumulative feature information output layer receives the nth input frame (n = 2, 3, ..., N) and the nth-1st auxiliary information based on the nth-1st cumulative feature information, which is image information that shows the characteristics of the first to the nth input frames and has the same number of pixels as the number of input pixels, and outputs the nth cumulative feature information that shows the characteristics of the first to the nth input frames. An estimated frame output layer receives the nth cumulative feature information and outputs the nth estimated frame, Includes, This is trained using multiple training data sets, each including a training input frame having the aforementioned input pixel count and a training estimation frame having the aforementioned estimated pixel count. The aforementioned program further, Means for acquiring n-1 motion information, which is information indicating the amount and direction of movement of each pixel between the n-1 input frame and the nth input frame. Means for setting the pixel value of one or more pixels of the n-1 cumulative feature information to the pixel at the position moved according to a pseudo-random number, based on the n-1 motion information, A program that makes a computer function.

Images