Inspection system, inspection method, and inspection program

Abstract

Provided are an inspection system, an inspection method, and an inspection program capable of easily determining whether information is correct, the information being included in each frame of a low-image-quality moving image serving as a source. This game program inspection system includes: a processor; a storage unit for storing commands to be executed by the processor and a machine learning model trained with a plurality of pieces of training data; and a display unit. The storage unit stores first to N-th (where N is a natural number equal to or greater than 2) input frames having a predetermined number of input pixels, and attached information for each input frame. The processor edits the attached information of the n-th (n = 2, 3, ... N) stored in the storage unit according to a user instruction. The machine learning model outputs the n-th estimated frame having an estimated number of pixels equal to or greater than the number of input pixels, on the basis of the first to the (n-1)th input frames and the attached information, and also the n-th input frame and the edited attached information. The display unit displays the n-th estimated frame.

Description

Inspection system, inspection method, and inspection program

[0001] The present disclosure relates to an inspection system, an inspection method, and an inspection program.

[0002] Conventionally, a technique (super-resolution) for estimating a high-quality still image based on a low-quality still image using a machine learning model is known (see Non-Patent Document 1 below).

[0003] Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang. Learning a Deep Convolutional Network for Image Super-Resolution, in Proceedings of European Conference on Computer Vision (ECCV), 2014

[0004] In super-resolution of moving images, it is believed that higher quality moving images can be estimated by taking into account not only information about each frame to be processed but also information about past frames of the frame. However, in an image processing system that estimates high quality moving images by taking into account information about past frames, it is difficult to determine whether the information contained in each frame input to the system is correct.

[0005] The present disclosure aims to provide an inspection system, inspection method, and inspection program that can easily determine whether the information contained in each frame of the original low-quality video is correct in an image processing technique that uses information from past frames to estimate a high-quality video based on a low-quality video.

[0006] The game program inspection system of the present disclosure is a game program inspection system including a processor, a memory unit that stores instructions to be executed by the processor and a machine learning model learned using multiple training data, and a display unit, wherein the memory unit stores first to Nth (N is a natural number greater than or equal to 2) input frames each having a predetermined number of input pixels, along with attached information for each of the input frames, the processor edits the nth (n = 2, 3, ..., N) attached information stored in the memory unit in response to a user instruction, the machine learning model outputs an nth estimated frame having an estimated number of pixels greater than or equal to the number of input pixels, based on the first to n-1th input frames and the attached information, and the nth input frame and the edited attached information, and the display unit displays the nth estimated frame.

[0007] 1 is a diagram illustrating an example of the hardware configuration of an image processing system and an inspection system; FIG. 2 is a diagram illustrating an overview of the image processing system; FIG. 3 is a diagram illustrating the processing of the image processing system; FIG. 4 is a functional block diagram illustrating an example of functions realized by the image processing system; FIG. 5 is a diagram illustrating the processing of a rendering unit; FIG. 6 is a diagram illustrating the processing of an intermediate frame acquisition unit; FIG. 7 is a flow diagram illustrating an example of the flow of processing executed in the image processing system; FIG. 8 is a diagram illustrating the processing executed in the inspection system; FIG. 9 is a diagram illustrating the processing executed in the inspection system; and FIG. 10 is a diagram illustrating information stored in a rendering information storage unit for each frame.

[0008] Hereinafter, an example of an embodiment of an image processing system according to the present disclosure will be described with reference to the drawings.

[0009] 1 is a diagram showing an example of the hardware configuration of an image processing system 1 and an inspection system 2. The image processing system 1 is a computer such as a game console. The inspection system 2 is a computer on which a program for creating a game program is installed. As shown in FIG. 1, the image processing system 1 and the inspection system 2 include a control unit 10, a storage unit 12, a communication unit 14, an operation unit 16, a display unit 18, and an audio output unit 19.

[0010] The control unit 10 includes a program control device such as a CPU that operates according to programs installed in the image processing system 1 and the inspection system 2. The control unit 10 also includes a GPU (Graphics Processing Unit) that draws images in a frame buffer based on graphics commands and data supplied from the CPU.

[0011] The storage unit 12 includes, for example, a main storage device such as a ROM or RAM, and an auxiliary storage device such as a HDD or SSD. The storage unit 12 stores programs executed by the control unit 10, etc. The storage unit 12 stores instructions executed by the processor and a machine learning model trained using multiple training data. For example, the storage unit 12 stores, in addition to programs for implementing various functions of the image processing system 1 described below, a game program (game software), for example. The storage unit 12 also reserves a frame buffer area in which images are drawn by the GPU.

[0012] The communication unit 14 is a communication interface such as an Ethernet (registered trademark) module or a wireless LAN module.

[0013] The operation unit 16 is a user interface such as a keyboard, mouse, or game console controller, and receives operation inputs from the user and outputs signals indicating the contents of the inputs to the control unit 10 .

[0014] The display unit 18 is a display device such as a liquid crystal display or an organic EL display, and displays various images according to instructions from the control unit 10 .

[0015] The audio output unit 19 is, for example, a speaker, and outputs audio represented by audio data generated by the image processing system 1 .

[0016] In addition to the devices described above, the image processing system 1 may also include an optical disc drive that reads optical discs such as DVD-ROMs and Blu-ray (registered trademark) discs, a USB (Universal Serial Bus) port, and the like.

[0017] 2. Overview of the Image Processing System Fig. 2 is a diagram showing an overview of the image processing system 1. Fig. 3 is a diagram showing the processing of the image processing system 1. In this embodiment, an example is shown in which the image processing system 1 is used to improve the image quality of gameplay video in a game. The gameplay video is a video generated in response to a game program executed by the control unit 10 or user input accepted by the operation unit 16, and is composed of a plurality of still images (frames) that are time-series data. The processing performed by the image processing system 1 is mainly as follows:

[0018] (1) Generation of Input Frames First, the image processing system 1 generates an image (input frame) depicting one or more game objects by rendering three-dimensional data representing the game objects as viewed from a predetermined viewpoint. This input frame is an image having a predetermined number of pixels (input pixel count) and a predetermined image quality (input image quality) (see FIG. 3). The input frame 20 is composed of color information. The input frame is generated at predetermined intervals. The number of pixels of the input frame is, for example, 1920 x 1080 (1080p). Each generated input frame is not displayed directly on the display unit 18, but is temporarily stored in the storage unit 12 for subsequent processing. Note that the following description mainly illustrates processing for the nth (nth) input frame 20_n, but similar processing is also performed for other input frames (i.e., n = 2, 3, ..., N).

[0019] (2) Acquisition of Intermediate Frame Based on the acquired input frame 20_n, the image processing system 1 acquires a frame (intermediate frame) 22_n having a number of pixels (intermediate pixel number) greater than the number of input pixels. The number of intermediate pixels is, for example, 3840 x 2160 (4K). Specifically, the input frame 20_n is subjected to enlargement and interpolation processing to generate the intermediate frame 22_n (see FIG. 3 ).

[0020] It should be noted that although the intermediate frame 22_n has a larger number of pixels than the input frame 20_n, its image quality is not necessarily improved sufficiently. In other words, the image quality of a frame does not simply refer to the number of pixels (high resolution). The image quality of a frame may be evaluated based on, for example, a higher signal-to-noise ratio, a higher reproducibility of spatial frequencies, or a higher temporal stability (fewer artifacts or flicker when multiple frames are displayed consecutively) compared to a reference frame, or a combination of these.

[0021] (3) Obtaining an Estimated Frame The image processing system 1 inputs the intermediate frame 22_n to the machine learning model 200 and obtains an estimated frame 24_n. The estimated frame 24_n is an image having the same number of pixels (estimated pixel number) as the number of intermediate pixels and image quality (estimated image quality) that is equal to or higher than the input image quality (see FIG. 3 ).

[0022] Here, in addition to the intermediate frame 22_n, the (n-1)th auxiliary information 28_n-1 is input to the machine learning model 200 (see FIGS. 2 and 3). The auxiliary information 28_n-1 is information based on the (n-1)th cumulative feature information 26_n-1 that indicates the features of the first to (n-1)th intermediate frames 22. Details of the cumulative feature information 26 and the auxiliary information 28 will be described later.

[0023] The machine learning model 200 is a model trained using a plurality of training data sets, each of which includes a training intermediate frame having an intermediate number of pixels generated based on a training input frame having an input pixel number and input image quality, and a training estimated frame having an estimated pixel number and estimated image quality. Details of the machine learning model 200 will be described later.

[0024] (4) Acquisition of Cumulative Feature Information The machine learning model 200 has a cumulative feature information output layer 202 that receives the intermediate frame 22_n and auxiliary information 28_n-1 and outputs n-th cumulative feature information 26_n that indicates the features of the first to n-th intermediate frames 22 (see FIG. 2). The image processing system 1 acquires the n-th cumulative feature information 26_n.

[0025] The acquired n-th cumulative feature information 26_n is input to the estimated frame output layer 204, which outputs the n-th estimated frame 24_n (see FIG. 2 ). The acquired n-th cumulative feature information 26_n is also stored in the storage unit 12 and is used to estimate an estimated frame 24_n+1 corresponding to the next input frame (the (n+1)th input frame) 20_n+1.

[0026] (5) Acquisition of Auxiliary Information As described above, the (n-1)th accumulated feature information 26_n-1 is information that indicates the features of the first to (n-1)th intermediate frames 22 (and thus the first to (n-1)th input frames 20). If the accumulated feature information 26_n-1, which accumulates information on past input frames 20, is used to estimate the nth estimated frame 24_n, the amount of information available for estimation increases, making it possible to obtain a high-quality estimated frame 24_n.

[0027] However, if there is movement or the like in the displayed game object between the (n-1)th input frame 20_n-1 and the nth input frame 20_n, when the nth intermediate frame 22_n and the cumulative feature information 26_n-1 are input as is to the machine learning model 200, a phenomenon (the so-called ghost phenomenon) may occur in which an afterimage of the game object that was displayed in the (n-1)th input frame 20_n-1 is displayed.

[0028] Therefore, the image processing system 1 acquires the (n-1)th auxiliary information 28_n-1 by applying various corrections (described later) to the cumulative feature information 26_n-1 based on auxiliary information (motion vectors, depth buffers, etc.) obtained during rendering (see FIGS. 2 and 3). As described above, the acquired (n-1)th auxiliary information 28_n-1 is input to the machine learning model 200 together with the nth intermediate frame 22_n, and is used to estimate the nth estimated frame 24_n.

[0029] As described above, the image processing system 1 according to this embodiment estimates the estimated frame 24 using auxiliary information 28, which stores past information, in addition to the intermediate frame 22 corresponding to the current input frame 20. This increases the amount of information available for estimation, making it possible to obtain a high-quality estimated frame 24_n. The image processing system 1 will now be described in detail.

[0030] 4 is a functional block diagram showing an example of functions realized by the image processing system 1 and the inspection system 2. As shown in FIG. 4, the image processing system 1 realizes a game processing unit 300, a rendering unit 302, a rendering information storage unit 304, an input frame acquisition unit 306, a variation information acquisition unit 308, an intermediate frame acquisition unit 310, a machine learning model storage unit 312, an estimated frame acquisition unit 314, a motion information acquisition unit 316, a depth information acquisition unit 318, an appearance pixel identification unit 320, and an auxiliary information acquisition unit 322. The inspection system 2 realizes, in addition to the units of the image processing system 1, an editing unit 324, an auxiliary information processing unit 326, and a display unit 328. The game processing unit 300, rendering unit 302, input frame acquisition unit 306, variation information acquisition unit 308, intermediate frame acquisition unit 310, estimated frame acquisition unit 314, motion information acquisition unit 316, depth information acquisition unit 318, appearance pixel identification unit 320, auxiliary information acquisition unit 322, editing unit 324, and auxiliary information processing unit 326 are mainly realized by the control unit 10. The rendering information storage unit 304 and machine learning model storage unit 312 are mainly realized by the storage unit 12. The function of the display unit 328 is mainly realized by the display unit 18, which is a device. Note that the game processing unit 300, rendering unit 302, and rendering information storage unit 304 are functions provided by game software.

[0031] [Game Processing Unit] The game processing unit 300 executes various processes related to the game. For example, in response to a game program executed by the control unit 10 or user input received by the operation unit 16, the game processing unit 300 executes processes such as placing a game object O in a virtual three-dimensional space VS, operating or moving the game object O, and changing the viewpoint C from which the virtual three-dimensional space VS is viewed (see FIG. 5 ). The game object O is composed of primitives such as polygons represented by three-dimensional data. The three-dimensional data includes geometric information indicating the positions of vertices, topological information indicating how the vertices are connected, and attribute information such as color.

[0032] [Rendering Unit] FIG. 5 is a diagram illustrating the processing of the rendering unit 302. The rendering unit 302 generates first to Nth (N is a natural number equal to or greater than 2) input frames 20 by rendering (drawing) three-dimensional data representing one or more game objects O viewed from a predetermined viewpoint C. The rendering unit 302 performs rendering based on the results of various processes executed by the game processing unit 300. Specifically, the rendering unit 302 performs vertex processing (vertex shading) and pixel processing (pixel shading) based on three-dimensional data representing game objects O placed in the virtual three-dimensional space VS. The vertex processing includes a coordinate transformation process (perspective projection) from the view coordinate system to the screen coordinate system, and a numerical value related to a change in the viewpoint C is added to the perspective projection matrix (camera matrix) used in the coordinate transformation process, as described below. The rendering unit 302 may perform rendering based on light source information, depth information (depth buffer), texture information, normal information, and the like. In addition to the above processes, the rendering unit 302 may also perform processes that apply effects such as depth of field (DoF) and motion blur. The processes performed by the rendering unit 302 may be set as appropriate by game software developers, etc. Here, the game software developers, etc. may adjust the MIP of the texture according to the estimated number of pixels of the estimated frame 24, etc. This makes it possible to suppress the occurrence of noise such as moiré in the estimated frame 24.

[0033] Here, the rendering unit 302 generates each input frame 20 by performing rendering such that the viewpoint C varies for each input frame 20. Here, even if the game processing unit 300 fixes the viewpoint C at a predetermined position, the rendering unit 302 varies the viewpoint C for each input frame 20. As a result, as shown in FIG. 5 , the position of the displayed game object O varies in each of input frames 20_n, 20_n+1, and 20_n+2. In other words, the rendering unit 302 applies jitter when generating each input frame 20. Specifically, the rendering unit 302 varies the viewpoint C for each input frame 20 by adding a numerical value corresponding to a sub-pixel size that differs for each input frame 20 to the perspective projection matrix. The rendering unit 302 varies the viewpoint C for each input frame 20 according to a predetermined rule. For example, a Halton sequence can be used as such a rule.

[0034] [Rendering Information Storage Unit] The rendering information storage unit 304 stores information necessary for the rendering process in the rendering unit 302 and information obtained as a result of the rendering process. For example, the rendering information storage unit 304 stores first to Nth (N is a natural number equal to or greater than 2) input frames 20 each having a predetermined number of input pixels. The rendering information storage unit 304 also stores auxiliary information for each input frame 20 along with the input frames 20. The auxiliary information includes variation information, motion information, and depth information. Details of the auxiliary information will be described later. The rendering information storage unit 304 also stores information (appearance pixel information and auxiliary information 28) generated based on the auxiliary information. The rendering information storage unit 304 may also store parameters used for coordinate transformation, light source information, texture information, normal information, etc.

[0035] [Input Frame Acquisition Unit] The input frame acquisition unit 306 acquires the first to Nth input frames 20. Specifically, the input frame acquisition unit 306 acquires the first to Nth input frames 20 stored in the rendering information storage unit 304.

[0036] [Variation Information Acquisition Unit] The variation information acquisition unit 308 acquires variation information. The variation information acquisition unit 308 acquires variation information stored in the rendering information storage unit 304. Specifically, the variation information is information indicating the amount of variation of the viewpoint C between before and after the variation. The information indicating the amount of variation can also be said to be a variation vector indicating the direction and distance of the variation. For example, since the above-mentioned Halton sequence includes information indicating the amount of variation of the viewpoint C, this information may be used as the variation information.

[0037] [Intermediate Frame Acquisition Unit] The intermediate frame acquisition unit 310 acquires first to Nth intermediate frames 22, respectively, based on each input frame 20 by generating an intermediate frame 22 that corresponds to the input frame 20 and has a number of intermediate pixels equal to or greater than the number of input pixels. In this embodiment, each intermediate frame 22 has a number of intermediate pixels greater than the number of input pixels. That is, in this embodiment, each intermediate frame 22 is an enlarged image of the input frame 20 corresponding to that intermediate frame 22.

[0038] Specifically, the intermediate frame acquisition unit 310 interpolates pixel values ​​at positions in each input frame 20 corresponding to the pixel values ​​before the fluctuation based on the fluctuation information and each pixel in the input frame 20, thereby generating each intermediate frame 22. FIG. 6 is a diagram illustrating the processing performed by the intermediate frame acquisition unit 310. FIG. 6 illustrates an example of obtaining the nth intermediate frame 22_n. For example, as shown in FIG. 6 , if the pixel center of a pixel in the intermediate frame 22_n to be acquired is P1,0, the intermediate frame acquisition unit 310 uses bilinear interpolation to obtain the pixel value of P1,0 based on the coordinates and pixel values ​​of the pixel centers P'0,0, P'1,0, P'0,1, and P'1,1 of the four pixels closest to P1,0 in the input frame 20_n. Here, P'1,0 is shifted from P1,0 by the amount of fluctuation indicated by the fluctuation information. The pixel values ​​of pixels newly generated by the enlargement process are similarly obtained. As the interpolation method, various known methods such as bicubic interpolation and Lanczos interpolation can be used in addition to bilinear interpolation.

[0039] When rendering is performed so that the viewpoint C varies for each input frame 20, the amount of time-series information increases. However, by using each input frame 20 obtained in this way (hereinafter referred to as a "variable input frame") for estimation, an estimated frame 24 with higher image quality can be obtained.

[0040] On the other hand, if the varying input frame (or an enlarged image thereof) is input directly to the machine learning model 200, the influence of the variation in viewpoint C described above may result in a decrease in the accuracy of estimation.

[0041] Therefore, as described above, the image processing system 1 interpolates pixel values ​​at positions in the input frame 20 corresponding to the pixels before the fluctuation based on the fluctuation information and each pixel of the input frame 20, generates each intermediate frame 22, and inputs this to the machine learning model 200. This corrects the influence of fluctuations in the viewpoint C, thereby preventing a decrease in the accuracy of estimation.

[0042] [Machine Learning Model] The machine learning model 200 is a model that estimates the nth estimated frame 24_n based on the nth intermediate frame 22_n. Specifically, the machine learning model 200 is a model that estimates the nth estimated frame 24_n based on the nth intermediate frame 22_n and the n-1th auxiliary information 28_n-1. The machine learning model 200 functions as part of the image processing system 1 inspection and also as part of the inspection system 2. In either case, the machine learning model 200 is the same, but when the machine learning model 200 functions as part of the inspection system 2, it outputs an nth estimated frame having a number of estimated pixels equal to or greater than the number of input pixels based on the first to (n-1)th input frames 20 and auxiliary information, and the nth input frame 20_n and edited auxiliary information. Specifically, the machine learning model 200 is a convolutional neural network (CNN). As the machine learning model 200, for example, a well-known model such as a multi-layered ResNet with a residual connection mechanism or a so-called encoder-decoder U-Net can be used. The model described in Non-Patent Document 1 may also be used as the machine learning model 200.

[0043] The machine learning model 200 is a model trained using a plurality of training data sets, each of which includes a training intermediate frame having an intermediate number of pixels generated based on a training input frame having an input pixel number, and a training estimation frame having an estimated pixel number. Various known techniques, such as backpropagation, can be used to train the machine learning model 200.

[0044] Specifically, the machine learning model 200 includes an accumulated feature information output layer 202, an estimated frame output layer 204, and a convolution layer 206 (see FIG. 2).

[0045] The cumulative feature information output layer 202 receives the nth intermediate frame 22_n and the n-1th auxiliary information 28_n-1 based on the n-1th cumulative feature information 26_n-1 indicating the features of the first to n-1th intermediate frames 22, and outputs the nth cumulative feature information 26_n indicating the features of the first to n-th intermediate frames 22_n. The cumulative feature information output layer 202 may be composed of, for example, one or more convolutional layers. The cumulative feature information 26_n-1 is image information (bitmap information) having the same number of pixels as the number of intermediate pixels. The cumulative feature information 26_n-1 can also be considered a feature map indicating the features of the first to n-1th intermediate frames 22.

[0046] The cumulative feature information output layer 202 receives the first intermediate frame 22_1 and given auxiliary information, and outputs first cumulative feature information 26_1. When n=1, there is no previous cumulative feature information 26 or auxiliary information 28, so the given auxiliary information prepared in advance is input to the cumulative feature information output layer 202 together with the first intermediate frame 22_1.

[0047] The estimated frame output layer 204 receives the n-th accumulated feature information 26_n and outputs the n-th estimated frame 24_n. The estimated frame output layer 204 may be configured, for example, by one or more convolutional layers, similar to the accumulated feature information output layer 202. Alternatively, the estimated frame output layer 204 may be configured by one or more transposed convolutional layers (deconvolutional layers).

[0048] The convolution layer 206 is a layer that reduces the number of channels of the accumulated feature information 26 while maintaining the number of pixels of the accumulated feature information 26. The accumulated feature information 26 output from the convolution layer 206 is subjected to processing in the auxiliary information acquisition unit 322. The convolution layer 206 reduces the dimensions of the accumulated feature information 26, thereby reducing calculation costs. The convolution layer 206 is, for example, a convolution layer with a kernel size of 1×1, but is not limited to this.

[0049] [Machine Learning Model Storage Unit] The machine learning model storage unit 312 stores the machine learning model 200. Specifically, the machine learning model storage unit 312 stores parameters of the machine learning model 200 (such as the number of convolutional layers, the number of nodes used in each convolutional layer, and the weight of each node).

[0050] [Estimated Frame Acquisition Unit] The estimated frame acquisition unit 314 inputs each intermediate frame 22 into the machine learning model 200 to acquire first to Nth estimated frames 24, each having an estimated pixel count greater than the input pixel count and equal to or greater than the intermediate pixel count. In this embodiment, the estimated frame 24 has an estimated pixel count equal to the intermediate pixel count. More specifically, the estimated frame acquisition unit 314 inputs the nth intermediate frame 22_n and the (n-1)th auxiliary information 28_n-1 into the machine learning model 200 to acquire the nth estimated frame 24_n.

[0051] [Motion Information Acquisition Unit] The motion information acquisition unit 316 acquires nth motion information, which is information indicating the amount and direction of motion from the n-1th input frame 20_n-1 to the nth input frame 20_n. Specifically, for example, the nth motion information is motion information included in the associated information attached to the nth input frame. Furthermore, the nth motion information is information having the same number of pixels as the number of intermediate pixels. The motion information is image information (bitmap format information) indicating the amount and direction of motion of each pixel between the n-1th input frame 20_n-1 and the nth input frame 20_n. The motion information is also called a motion vector. Specifically, the motion information acquisition unit 316 acquires motion information having the same number of pixels as the number of input pixels, and performs enlargement and interpolation processing on the motion information to acquire motion information having the same number of pixels as the number of intermediate pixels. Hereinafter, motion information having the same number of pixels as the number of input pixels will also be referred to as original motion information.

[0052] [Depth Information Acquisition Unit] The depth information acquisition unit 318 acquires n-1th depth information indicating the depth of each pixel in the n-1th input frame 20_n-1 and nth depth information indicating the depth of each pixel in the nth input frame 20_n. Specifically, the depth information is image information having the same number of pixels as the intermediate pixel count (bitmap format information). The depth information is also called a depth buffer, Z buffer, or depth buffer. Specifically, the depth information acquisition unit 318 acquires depth information having the same number of pixels as the input pixel count and performs enlargement and interpolation processing on the depth information to acquire depth information having the same number of pixels as the intermediate pixel count. Hereinafter, the depth information having the same number of pixels as the input pixel count will also be referred to as original depth information.

[0053] [Appearance Pixel Identification Unit] Based on the (n-1)th depth information and the (n)th depth information, the appearance pixel identification unit 320 identifies the nth appearance pixel 222_n, which is a pixel in the nth intermediate frame 22_n where all or part of the game object O, which is not displayed in the (n-1)th intermediate frame 22_n-1, is displayed (see FIG. 3 ). Specifically, the appearance pixel identification unit 320 identifies the nth appearance pixel 222_n based on the difference between the (n-1)th depth information and the nth depth information. Note that the appearance pixel identification unit 320 may also identify the nth appearance pixel 222_n based on the (n-1)th perspective projection matrix associated with the (n-1)th intermediate frame 22_n-1 and the nth perspective projection matrix associated with the nth intermediate frame 22_n. Alternatively, the appearance pixel identification unit 320 may identify the nth appearance pixel 222_n using the (n-1)th motion information. More specifically, the appearance pixel specifying section 320 specifies the nth appearance pixel 222_n and generates nth appearance pixel information, which is image information indicating the position of the nth appearance pixel 222_n.

[0054] [Auxiliary Information Acquisition Unit] The auxiliary information acquisition unit 322 acquires the (n-1)th auxiliary information 28_n-1 by applying motion compensation to the (n-1)th cumulative feature information 26_n-1 based on the (n-1)th motion information (excluding the original motion information). Motion compensation refers to, for example, a process of moving a pixel at position x in the (n-1)th cumulative feature information 26_n to position x' in the (n-1)th intermediate frame 22_n-1, assuming that the pixel at position x in the (n-1)th intermediate frame 22_n has moved to position x' (see FIG. 3 ). That is, the auxiliary information acquisition unit 322 acquires the (n-1)th auxiliary information 28_n-1 based on the (n-1)th motion information by setting the pixel values ​​of one or more pixels in the (n-1)th cumulative feature information 26_n-1 to pixels at positions moved in accordance with the amount and direction of the pixel's movement.

[0055] If there is movement of the game object O between the nth input frame 20_n and the n-1th input frame 20_n-1, if the nth intermediate frame 22_n and the n-1th cumulative feature information 26_n-1 are input directly into the machine learning model 200 when obtaining the nth estimated frame 24_n, a ghost phenomenon may occur in which an afterimage of the game object O that was displayed in the nth intermediate frame 22_n is displayed in the output nth estimated frame 24_n.

[0056] Therefore, as described above, the image processing system 1 applies motion compensation to the (n-1)th accumulated feature information 26_n-1 based on the (n-1)th motion information to acquire the (n-1)th auxiliary information 28_n-1, and when acquiring the nth estimated frame 24_n, this (n-1)th auxiliary information 28_n-1 is input to the machine learning model 200. This makes it possible to suppress the ghost phenomenon described above.

[0057] Furthermore, the auxiliary information acquisition unit 322 acquires the (n-1)th auxiliary information 28_n-1 by replacing the pixel value of the nth appearance pixel 222_n in the (n-1)th cumulative feature information 26_n-1 with a predetermined value. Specifically, the auxiliary information acquisition unit 322 acquires the (n-1)th auxiliary information 28_n-1 by replacing the pixel value of the nth appearance pixel 222_n in the (n-1)th cumulative feature information 26_n-1 with a predetermined value based on the nth appearance pixel information. The predetermined value may be a constant value such as 0 (black), or may be the pixel value of the nth appearance pixel 222_n in the nth intermediate frame 22_n.

[0058] If all or part of a game object O that is not displayed in the n-1th input frame 20_n-1 is displayed in the nth input frame 20_n, and the nth intermediate frame 22_n and the n-1th cumulative feature information 26_n-1 are input directly into the machine learning model 200 when obtaining the nth estimated frame 24_n, the aforementioned ghost phenomenon may occur in the output nth estimated frame 24_n.

[0059] Therefore, as described above, the image processing system 1 identifies the nth appearance pixel 222_n, which is a pixel in the nth intermediate frame 22_n where all or part of the game object O that is not displayed in the (n-1)th intermediate frame 22_n-1 is displayed, and then replaces the pixel value of the nth appearance pixel 222_n in the (n-1)th cumulative feature information 26_n-1 with a predetermined value to obtain the (n-1)th auxiliary information 28_n-1. This makes it possible to suppress the ghost phenomenon described above.

[0060] [Editing Unit] The editing unit 324 edits the nth ancillary information stored in the rendering information storage unit 304 in response to a user instruction. Specifically, for example, the editing unit 324 changes the value of motion information associated with the nth input frame 20_n selected by the user to a value specified by the user. Here, the motion information to be changed is motion information having the same number of pixels as the number of input pixels (i.e., original motion information). The value of the changed motion information is stored in the rendering information storage unit 304 in association with other ancillary information for the nth input frame 20_n. Note that the ancillary information edited in response to a user instruction is not limited to motion information, and may also be variation information or depth information.

[0061] [Ancillary Information Processing Unit] The auxiliary information processing unit 326 generates a difference image between an image obtained by applying auxiliary information to the (n-1)th input frame 20_n-1 based on the nth auxiliary information and an image represented by the nth input frame 20_n. Specifically, for example, the auxiliary information processing unit 326 generates a difference image between an image obtained by applying motion compensation to the (n-1)th input frame 20_n-1 based on the nth motion information and an image represented by the nth input frame 20_n.

[0062] [Display Unit] The display unit 328 displays the nth estimated frame 24_n. Specifically, for example, the display unit 328 displays the nth estimated frame 24_n acquired by the estimated frame acquisition unit 314 by inputting the nth intermediate frame 22_n and the (n-1)th auxiliary information 28_n-1 into the machine learning model 200. The display unit 328 also displays the difference image generated by the auxiliary information processing unit 326. In addition, the display unit 328 displays the input frame 20 and each piece of auxiliary information stored in the rendering information storage unit 304. The display unit 328 may display some or all of the estimated frame 24, each piece of auxiliary information, and difference image selected by the user side by side.

[0063] 7 is a flow diagram showing an example of the flow of processing executed in the image processing system 1. The processing shown in FIG. 7 is executed by the control unit 10 operating in accordance with a program stored in the storage unit 12.

[0064] (1) Processing for n=1 First, the control unit 10 acquires a first input frame 20_1 (S700). The control unit 10 acquires a first intermediate frame 22_1 based on the first input frame 20_1 (S702). Then, the control unit 10 inputs the first intermediate frame 22_1 and given auxiliary information to the machine learning model 200, and acquires a first estimated frame 24_1 and first cumulative feature information 26_1 (S704).

[0065] (2) Processing when n≧2 The control unit 10 acquires the n-th input frame 20_n (S706).The control unit 10 acquires the n-th intermediate frame 22_n based on the n-th input frame 20_n (S708).

[0066] Next, the control unit 10 acquires the (n-1)th motion information (S710). The control unit 10 also acquires the (n-1)th depth information and the nth depth information (S712) and identifies the nth appearance pixel 222_n based on the (n-1)th depth information and the nth depth information (S714). The control unit 10 acquires the (n-1)th auxiliary information 28_n-1 based on the (n-1)th cumulative feature information 26_n-1, the (n-1)th motion information, and the nth appearance pixel 222_n (S716). The control unit 10 then inputs the nth intermediate frame 22_n and the (n-1)th auxiliary information 28_n-1 into the machine learning model 200, and acquires the nth estimated frame 24_n and the nth cumulative feature information 26_n (S718). The control unit 10 determines whether or not a next frame exists (S720), and if it determines that a next frame exists (S720; Y), it increments n to n+1 and repeats the processes of S706 to S718. If the control unit 10 determines that a next frame does not exist (S720; N), it ends this process. Note that if it determines that a next frame does not exist (S720; N), the control unit 10 may cause the display unit 18 to display the first to Nth estimated frames 24 as they are.

[0067] 5. Processing Executed in the Inspection System With reference to FIGS. 8 to 10 , the processing and inspection method executed by the inspection system 2 when motion information is edited by a user's instruction will be described as an example. The auxiliary information edited in this processing is original motion information among the motion information. FIGS. 8 and 9 show the input frame 20 and the difference image that change as a result of this processing. The processing shown in FIGS. 8 and 9 is executed by the control unit 10 operating in accordance with a program stored in the storage unit 12. FIG. 10 shows the information stored in the rendering information storage unit 304 for each frame. In FIGS. 8 to 10 , the input frame is represented as "Color," the motion information (original motion information) is represented as "MV," the depth information (original depth information) is represented as "Depth" and "Previous Depth," the pixel information that appears is represented as "Exposure," and the auxiliary information is represented as "RFM." Note that "Previous Depth" refers to the depth information of the frame immediately preceding "Depth." Moreover, each piece of information is denoted by adding (n) to indicate that it belongs to the nth frame.

[0068] FIG. 8A shows the nth motion information, i.e., a motion vector representing the amount and direction of movement of game object Q from the (n-1)th input frame to the nth input frame. For simplicity's sake, the explanations of FIGS. 8 to 10 assume that the information for each frame includes only game object Q and no other game objects when viewed from a specific viewpoint. As shown in FIG. 8A, the nth motion information, MV(n), represents information about the movement of spherical game object Q from the (n-1)th input frame to the nth input frame, from the bottom left to the top right. The arrow points from the game object in the nth input frame 20_n toward the game object in the (n-1)th input frame 20_n-1.

[0069] FIG. 8(b) is a diagram showing the (n-1)th input frame 20_n-1, i.e., color information Color(n-1). FIG. 8(c) is a diagram showing the nth input frame 20_n, i.e., color information Color(n). Both FIG. 8(b) and FIG. 8(c) show game object Q, as seen from a certain viewpoint, displayed at a predetermined position in the background. FIG. 8(b) is the input frame 20 one frame before FIG. 8(c), and the game object Q shown in FIG. 8(b) is located to the lower left of the game object Q shown in FIG. 8(c). All of FIG. 8(a) to FIG. 8(c) are information generated by the rendering unit 302 and stored in the rendering information storage unit 304.

[0070] The auxiliary information processing unit 326 generates an image by applying motion compensation to the (n-1)th input frame 20_n based on the nth motion information. Specifically, for example, the auxiliary information processing unit 326 generates an image by applying motion compensation (MC(MV(n), Color(n-1))) to the input frame (Color(n-1)) shown in FIG. 8B using the motion information MV(n) shown in FIG. 8A. Here, motion compensation refers to a process of moving a pixel at position x in the (n-1)th input frame 20_n-1 to position x' when a pixel at position x in the (n-1)th input frame 20_n-1 has moved to position x' in the nth input frame 20_n. The nth motion information is image information that indicates the amount and direction of movement of each pixel between the (n-1)th input frame 20_n-1 and the nth input frame 20_n. Therefore, the auxiliary information processing unit 326 generates color information (Color(n)') shown in Figure 8(d) by setting each pixel value of the game object Q shown in Figure 8(b) to a pixel at a position moved in accordance with the amount and direction of movement of the pixel, based on the motion information MV(n) in Figure 8(a). Note that the auxiliary information processing unit 326 also processes the pixels representing the background of input frame 20_n-1 shown in Figure 8(b), but since the pixels representing the background have no motion information value (value 0), they do not change as a result of the above processing.

[0071] Furthermore, the auxiliary information processing unit 326 generates a difference image between the image to which motion compensation has been applied and the image represented by the nth input frame 20_n. Specifically, for example, a difference image (Diff.Color=Color(n)′−Color(n)) between the color information to which motion compensation has been applied (Color(n)′) shown in FIG. 8D and the input frame (Color(n)) shown in FIG. 8C is generated. FIG. 8E is a diagram showing the generated difference image. As shown in FIG. 8E, a crescent-shaped pattern is shown in the area showing game project Q in the nth input frame. In other words, it can be seen that there is a discrepancy between the nth input frame 20_n estimated from the nth motion information and the n−1th input frame 20_n−1 and the nth input frame 20_n obtained from the rendering unit. The user can easily determine whether the attached information is correct by having each piece of information shown in (a) to (e) of Figures 8 displayed on the display unit 328 as a visible image.

[0072] Next, editing of auxiliary information will be described. (a) to (c) of FIG. 9 are all the same as (a) to (c) of FIG. 8. The editing unit 324 edits the nth motion information MV(n) stored in the rendering information storage unit 304 in response to a user instruction. Specifically, as information for editing the nth motion information shown in (a) of FIG. 9, the user operates the operation unit 16 to input the shift amount (MV.Offset) of the starting point position in the x and y directions and the magnification ratio (MV.Scale) in the x and y directions. The editing unit 324 acquires the edited nth motion information MV'(n) based on the input shift amount and magnification ratio. For example, the motion information MV'(n) is calculated using the formula MV'(n) = MV.Offset + MV.Scale * MV(n).

[0073] The auxiliary information processing unit 326 generates an image by applying motion compensation to the (n-1)th input frame based on the nth motion information. Here, as described above, the nth motion information has been changed from MV(n) to MV'(n). Therefore, the auxiliary information processing unit 326 generates an image by applying motion compensation (MC(MV'(n), Color(n-1)))) to the input frame (Color(n-1)) shown in FIG. 9B using the edited motion information MV'(n). FIG. 9D shows the generated color information (Color(n)'').

[0074] Furthermore, the auxiliary information processing unit 326 generates a difference image between the image to which motion compensation has been applied and the image represented by the nth input frame 20_n. Specifically, for example, the auxiliary information processing unit 326 generates a difference image (Diff.Color=Color(n)″−Color(n)) between the color information to which motion compensation has been applied (Color(n)″) shown in FIG. 9D and the input frame (Color(n)) shown in FIG. 9C. FIG. 9E is a diagram showing the generated difference image. As shown in FIG. 9E, the crescent-shaped pattern is not shown. In other words, it can be seen that the nth input frame 20_n estimated from the edited motion information and the n-1th input frame 20_n−1 is approximately consistent with the nth input frame 20_n obtained from the rendering unit 302. The user can easily obtain MV′(n), which is the motion information that should originally be generated by the rendering unit 302.

[0075] As described above, the auxiliary information is information that reflects multiple past input frames 20 and auxiliary information. However, since images in games can change significantly depending on the scene, the auxiliary information 28 across scene changes only contains a small amount of information. Therefore, it is desirable to perform the inspection method according to the present disclosure within the same scene.

[0076] FIG. 10 shows the information stored in the rendering information storage unit 304 for each frame. The symbols added above the information for each frame indicate the frame order. Assume that a scene change occurs between the -k-1th (k is a constant) input frame 20 and the -kth input frame 20. In this case, the above inspection method is preferably performed after a certain number of frames have passed since the scene change, so that the auxiliary information contains sufficient information about past frames. In FIG. 10, the 0th auxiliary information (RFM) 28 is stored in the rendering information storage unit 304. The 0th auxiliary information 28 is information that reflects the -kth to -1st input frames 20 and auxiliary information.

[0077] Since the 0th auxiliary information 28 is stored in the rendering information storage unit 304, the machine learning model 200 can output high-quality images that contain sufficient information from past frames for each frame unless there is a subsequent scene change.

[0078] The user performs the above inspection method on the first frame and thereafter. For example, as shown in Fig. 10, the user changes MV, which is the n-th auxiliary information, to MV'. This causes a difference image to be displayed, as shown in Fig. 9.

[0079] In the embodiment shown in FIG. 10 , the input frame, motion information, depth information, pixel appearance information, and original depth information are stored for every frame. Storing this information for every frame allows the user to identify frames for which incorrect auxiliary information was generated by the rendering unit. Meanwhile, the auxiliary information 28 only needs to contain sufficient information from past frames, and is rarely subject to editing. Therefore, the auxiliary information 28 is stored only for predetermined frames. The auxiliary information 28 may be stored every certain number of frames (e.g., every 32 frames). Furthermore, there is no problem even if the auxiliary information 28 is stored for every frame.

[0080] In the above description, the case where motion information out of the auxiliary information is edited has been described, but other information may be edited as long as it is information stored in the rendering information storage unit 304 .

[0081] [6. Summary] According to the image processing system 1 of this embodiment described above, the nth estimated frame 24_n is estimated using the (n-1)th cumulative feature information 26_n-1 that indicates the features of the first to (n-1)th intermediate frames 22. Then, according to the inspection system 2 of this embodiment, when estimating a high-quality moving image using the image processing system 1, it becomes possible to easily determine whether the information included in each frame of the original low-quality moving image is correct.

[0082] The invention according to the present disclosure is not limited to the above-described embodiments. Furthermore, the specific character strings and numerical values ​​described above and in the drawings are merely examples, and the invention is not limited to these character strings and numerical values.

[0083] For example, in the present embodiment, the number of intermediate pixels is greater than the number of input pixels, and the number of intermediate pixels is equal to the number of estimated pixels. However, the number of intermediate pixels may be equal to the number of input pixels, and the number of estimated pixels may be greater than the number of intermediate pixels. In other words, the intermediate frame 22 does not necessarily have to be an enlarged version of the input frame 20.

[0084] [7. Supplementary Notes] (1) A game program inspection system including: a processor; a storage unit that stores instructions to be executed by the processor and a machine learning model trained using a plurality of training data; and a display unit, wherein the storage unit stores first to Nth (N is a natural number equal to or greater than two) input frames each having a predetermined number of input pixels, along with attached information for each of the input frames; the processor edits the nth (n=2, 3, ..., N) attached information stored in the storage unit in response to a user instruction; the machine learning model outputs an nth estimated frame having an estimated number of pixels equal to or greater than the number of input pixels, based on the first to n-1th input frames and the attached information, and the nth input frame and the edited attached information; and the display unit displays the nth estimated frame. (2) The inspection system according to (1), wherein the processor: based on each of the input frames, generates intermediate frames corresponding to the input frames and having an intermediate pixel number equal to or greater than the input pixel number, thereby obtaining first to Nth intermediate frames, respectively; and inputs each of the intermediate frames to a machine learning model to obtain first to Nth estimated frames having an estimated pixel number greater than the input pixel number and equal to or greater than the intermediate pixel number, respectively; the machine learning model includes: an accumulated feature information output layer that receives the nth intermediate frame and n-1th auxiliary information based on n-1th accumulated feature information indicating features of the first to n-1th intermediate frames, and outputs the nth accumulated feature information indicating features of the first to nth intermediate frames; and an estimated frame output layer that receives the nth accumulated feature information, and outputs the nth estimated frame, and is trained using a plurality of training data, each of which includes a training intermediate frame having the intermediate pixel number generated based on a training input frame having the input pixel number, and a training estimated frame having the estimated pixel number.(3) The inspection system according to (2), wherein the auxiliary information includes nth motion information indicating an amount and direction of motion from the (n-1)th input frame to the nth input frame, and the processor acquires the nth auxiliary information by applying motion compensation to the (n-1)th accumulated feature information based on the nth motion information. (4) The inspection system according to (3), wherein the processor generates a difference image between an image obtained by applying motion compensation to the (n-1)th input frame based on the nth motion information and an image represented by the nth input frame, and the display unit displays the difference image. (5) The inspection system according to (2), wherein each of the input frames is an image obtained by rendering three-dimensional data showing one or more objects seen from a predetermined viewpoint. (6) The inspection system according to (5), wherein the auxiliary information includes n-th depth information indicating a depth of each pixel of the n-th input frame, and the processor identifies an n-th appearance pixel, which is a pixel of the n-th intermediate frame in which all or part of the object not displayed in the n-1th intermediate frame is displayed, based on the n-1th depth information and the n-th depth information, among the pixels of the n-th intermediate frame, and acquires the n-1th auxiliary information by replacing the pixel value of the n-th appearance pixel in the n-1th cumulative feature information with a predetermined value. (7) The inspection system according to (6), wherein each of the input frames is an image obtained by performing the rendering so that the viewpoint varies for each of the input frames, and the processor acquires variation information which is information related to the variation of the viewpoint for each of the input frames in the rendering, and generates each of the intermediate frames by interpolating pixel values ​​in the input frames at positions corresponding to each pixel before the variation, based on the variation information and each pixel of each of the input frames. (8) The inspection system for a game program according to (2), wherein the cumulative feature information output layer receives the first intermediate frame and given auxiliary information and outputs the first cumulative feature information. (9) The inspection system according to (2), wherein the cumulative feature information is image information having the same number of pixels as the number of intermediate pixels.(10) An inspection method, in which a memory unit stores first to Nth (N is a natural number equal to or greater than 2) input frames each having a predetermined number of input pixels, along with attached information for each of the input frames; a processor edits the nth (n=2, 3, ..., N) attached information stored in the memory unit in response to a user instruction; a machine learning model trained using a plurality of training data outputs an nth estimated frame having an estimated number of pixels equal to or greater than the number of input pixels, based on the first to n-1th input frames and the attached information, and the nth input frame and the edited attached information; and a display unit displays the nth estimated frame. (11) A test program for causing a computer to function as: a storage means for storing first to Nth (N is a natural number equal to or greater than 2) input frames each having a predetermined number of input pixels, along with attached information for each of the input frames; an editing means for editing the nth (n=2, 3, ..., N) attached information stored in the storage unit in response to a user instruction; an output means for outputting an nth estimated frame having an estimated number of pixels equal to or greater than the number of input pixels, based on the first to n-1th input frames and the attached information, and the nth input frame and the edited attached information, using a machine learning model trained using a plurality of training data; and a display means for displaying the nth estimated frame.

Claims

1. A game program inspection system comprising: a processor; a memory unit storing instructions executed by the processor and a machine learning model trained using a plurality of training data; and a display unit, wherein the memory unit stores 1st to Nth (N is a natural number equal to or greater than 2) input frames having a predetermined number of input pixels along with associated information for each of the input frames; the processor edits the nth (n=2, 3, ..., N) associated information stored in the memory unit in response to a user instruction; the machine learning model outputs an nth estimated frame having an estimated number of pixels equal to or greater than the number of input pixels based on the 1st to n-1th input frames and the associated information, and the nth input frame and the edited associated information; and the display unit displays the nth estimated frame.

2. The inspection system of claim 1, wherein the processor: based on each of the input frames, generates an intermediate frame that corresponds to the input frame and has an intermediate number of pixels equal to or greater than the input pixel number, to obtain first to Nth intermediate frames, respectively; and inputs each of the intermediate frames to a machine learning model to obtain first to Nth estimated frames, respectively, having an estimated number of pixels greater than the input pixel number and equal to or greater than the intermediate pixel number, wherein the machine learning model includes: an cumulative feature information output layer that receives as input the nth intermediate frame and n-1th auxiliary information based on n-1th cumulative feature information indicating features of the 1st to n-1th intermediate frames, and outputs the nth cumulative feature information indicating features of the 1st to nth intermediate frames; and an estimated frame output layer that receives as input the nth cumulative feature information, and outputs the nth estimated frame, and is trained using a plurality of training data each including a learning intermediate frame having the intermediate number of pixels generated based on a learning input frame having the input pixel number, and a learning estimated frame having the estimated number of pixels.

3. The inspection system of claim 2, wherein the auxiliary information includes nth motion information which is information indicating the amount and direction of motion from the n-1th input frame to the nth input frame, and the processor obtains the nth auxiliary information by applying motion compensation to the n-1th accumulated feature information based on the nth motion information.

4. The inspection system of claim 3, wherein the processor generates a difference image between an image obtained by applying motion compensation to the (n-1)th input frame based on the nth motion information and an image represented by the nth input frame, and the display unit displays the difference image.

5. The inspection system of claim 2, wherein each of the input frames is an image obtained by performing a rendering of three-dimensional data showing one or more objects as viewed from a predetermined viewpoint.

6. The inspection system of claim 5, wherein the auxiliary information includes nth depth information indicating a depth of each pixel of the nth input frame, and the processor identifies an nth appearance pixel, which is a pixel of the nth intermediate frame in which all or part of the object not displayed in the n-1th intermediate frame is displayed, based on the n-1th depth information and the nth depth information, and obtains the n-1th auxiliary information by replacing the pixel value of the nth appearance pixel in the n-1th accumulated feature information with a predetermined value.

7. The inspection system described in claim 6, wherein each of the input frames is an image obtained by performing the rendering so that the viewpoint varies for each of the input frames, and the processor acquires variation information which is information relating to the variation of the viewpoint for each of the input frames in the rendering, and based on the variation information and each pixel of each of the input frames, determines by interpolation pixel values ​​at positions in the input frame which correspond to each pixel before the variation, thereby generating each of the intermediate frames.

8. The game program inspection system according to claim 2, wherein the cumulative feature information output layer receives the first intermediate frame and given auxiliary information and outputs the first cumulative feature information.

9. The inspection system according to claim 2, wherein the cumulative feature information is image information having the same number of pixels as the intermediate number of pixels.

10. An inspection method, in which a memory unit stores 1st to Nth (N is a natural number equal to or greater than 2) input frames having a predetermined number of input pixels along with auxiliary information for each of the input frames, a processor edits the nth (n=2, 3, ..., N) auxiliary information stored in the memory unit in response to a user's instruction, a machine learning model trained using a plurality of training data outputs an nth estimated frame having an estimated number of pixels equal to or greater than the number of input pixels based on the 1st to N-1th input frames and the auxiliary information, and the nth input frame and the edited auxiliary information, and a display unit displays the nth estimated frame.

11. A testing program for causing a computer to function as: a storage means for storing 1st to Nth (N is a natural number equal to or greater than 2) input frames having a predetermined number of input pixels together with auxiliary information for each of the input frames; an editing means for editing the nth (n=2, 3, ..., N) auxiliary information stored in the storage unit in response to a user's instruction; an output means for outputting an nth estimated frame having an estimated number of pixels equal to or greater than the number of input pixels, based on the 1st to N-1th input frames and the auxiliary information, and the nth input frame and the edited auxiliary information, using a machine learning model trained using a plurality of training data; and a display means for displaying the nth estimated frame.