Image processing system, image processing method, and program

Abstract

The present invention makes it possible to apply high-precision super-resolution processing to a moving image generated from a texture-changed object without relying on motion information. In an image processing system according to the present invention, a processor acquires first to Nth input frames having a plurality of pixel inputs, acquires first to Nth intermediate frames from each of the input frames, acquires first to Nth estimated frames from each of the intermediate frames, identifies an nth color change pixel based on texture information of an object, and outputs the identified nth color change pixel. The n-th color changing pixel includes color information that changes irrespective of the motion of the object in the n-th intermediate frame, and the n-th auxiliary information is acquired by replacing a pixel value of the color changing pixel in the n-th cumulative feature information with a predetermined value. Further, in the image processing system, the machine learning model includes an output layer that outputs the n-th cumulative feature information and an output layer that outputs the n-th estimated frame, and the machine learning model is trained with a plurality of items of training data including a training intermediate frame, auxiliary information in which the color change pixel has been replaced by a predetermined value, and the training estimated frame.

Description

TECHNICAL FIELD

[0001] The present application relates to an image processing system, an image processing method, and a program. BACKGROUND

[0002] Conventionally, a technique of estimating a high-quality still image based on a low-quality still image using a machine learning model (super-resolution) is known (see Non-Patent Literature 1 below). PRIOR ART

[0003] [Non-Patent Literature]

[0004] [Non-Patent Literature 1] Chao Dong, Chen Change Loy, Kaiming He, Xiao ou Tang. Learning a Deep Convolutional Network for Image Super-Resolution. In Proceedings of the European Conference on Computer Vision (ECCV), 2014 SUMMARY

[0005] [Problem to be Solved by the Invention]

[0006] The inventors of the present application are considering applying the above-described super-resolution to a moving image such as a game screen. In super-resolution of a moving image, it is thought that a moving image of higher image quality can be estimated by considering not only information about each frame to be processed but also information about past frames of these frames. In particular, deterioration of image quality due to ghosting can be avoided by considering information indicating motion of an object such as a motion vector. However, there are cases where the texture of an object changes independently of the motion information of the object, such as when the object is a mirror or when the object has an animation texture. When super-resolution processing that takes motion information into account is performed on a moving image generated from such an object, it can actually result in a decrease in image quality.

[0007] An object of the present disclosure is to provide an image processing system, an image processing method, and a program that can perform high-precision super-resolution processing on a moving image generated from an object having a changing texture without relying on motion information in an image processing device that estimates a high-quality moving image based on a low-quality moving image using motion information and information from past frames.

[0008] [Means for Solving the Problem]

[0009] The image processing system according to the present invention is an image processing system comprising at least one processor, wherein: the at least one processor acquires each of the first to Nth input frames (N is a natural number of 2 or greater) having a predetermined number of input pixels; acquires the first to Nth intermediate frames by generating intermediate frames based on the input frames, the intermediate frames having an intermediate pixel number equal to or greater than the number of input pixels and corresponding to the input frames; and acquires the first to Nth estimated frames having an estimated pixel number equal to or greater than the intermediate pixel number and greater than the number of input pixels by inputting each intermediate frame into a machine learning model; the machine learning model includes a cumulative feature information output layer having the nth intermediate frame (n=2,3,…). N), and inputting (n-1)th auxiliary information based on (n-1)th cumulative feature information indicating features from the first to the (n-1)th intermediate frames, and wherein outputting (n)th cumulative feature information indicating features from the first to the nth intermediate frames; and an estimated frame output layer, wherein inputting (n)th cumulative feature information, and wherein outputting (n)th estimated frames; at least one processor identifies (n)th color-changing pixels based on texture information of an object, the (n)th color-changing pixels including color information that changes independently of the motion of the object in the (n)th intermediate frame, and obtains (n)th auxiliary information by replacing the pixel values ​​of the color-changing pixels in the (n)th cumulative feature information with predetermined values; and a machine learning model learns using multiple training data, the multiple training data respectively including learning intermediate frames with a number of intermediate pixels, learning intermediate frames generated based on learning intermediate frames with a number of intermediate pixels, learning intermediate frames generated based on learning input frames with a number of input pixels, wherein auxiliary information in which color-changing pixels are replaced with predetermined values, and learning estimated frames with a number of estimated pixels. Attached Figure Description

[0010] Figure 1 This is a diagram illustrating an example of the hardware configuration of an image processing system.

[0011] Figure 2 This is a diagram showing an overview of an image processing system.

[0012] Figure 3 It is a schematic diagram illustrating the processing in an image processing system.

[0013] Figure 4 This is a function block diagram illustrating an example of the functionality implemented by an image processing system.

[0014] Figure 5 This is a diagram illustrating the processing of the rendering unit.

[0015] Figure 6 This is a diagram illustrating the processing in the intermediate frame acquisition unit.

[0016] Figure 7This is a schematic diagram illustrating the processing used to define the color-changing pixels.

[0017] Figure 8 This is a flowchart illustrating an example of the processing flow performed in an image processing system. Detailed Implementation

[0018] An example of an embodiment of the image processing system according to the present disclosure will now be described with reference to the accompanying drawings.

[0019] [1. Hardware configuration of the image processing system]

[0020] Figure 1 This diagram illustrates an example of the hardware configuration of an image processing system 1. The image processing system 1 is, for example, a computer such as a game console (gaming device). Figure 1 As shown, the image processing system 1 includes a control unit 10, a storage unit 12, a communication unit 14, an operation unit 16, a display unit 18, and an audio output unit 19.

[0021] The control unit 10 includes, for example, a program control device such as a CPU that operates according to a program installed in the image processing system 1. The control unit 10 also includes a GPU (graphics processing unit) that renders an image in a frame buffer based on graphics commands or data provided from the CPU.

[0022] Storage unit 12 includes, for example, a main storage device such as ROM or RAM, and an auxiliary storage device such as HDD or SSD. Storage unit 12 stores programs executed by control unit 10. In addition to programs for implementing various functions of image processing system 1, as described later, storage unit 12 also stores, for example, game programs (game software). Furthermore, storage unit 12 also has a frame buffer area reserved for images rendered by the GPU.

[0023] Communication unit 14 is a communication interface such as an Ethernet (registered trademark) module or a wireless LAN module.

[0024] The operation unit 16 is a user interface such as a keyboard, mouse, or game console controller, and receives operation input from the user and outputs signals indicating input details to the control unit 10.

[0025] The display unit 18 is a display device such as a liquid crystal display or an organic EL display, and displays various images according to instructions from the control unit 10.

[0026] The audio output unit 19 is, for example, a speaker, and outputs audio represented by audio data generated by the image processing system 1.

[0027] Note that, in addition to the devices mentioned above, the image processing system 1 may also include an optical disc drive for reading optical discs such as DVD-ROM and Blu-ray (registered trademark) discs, a USB (Universal Serial Bus) port, etc.

[0028] [2. Overview of Image Processing Systems]

[0029] Figure 2 This is a diagram showing an outline of the image processing system 1. Figure 3 This diagram schematically illustrates the processing in the image processing system 1. In this embodiment, an example of using the image processing system 1 to improve the image quality of gameplay motion images in a game will be described. Gameplay motion images are motion images generated in response to a game program executed by the control unit 10, user input received by the operation unit 16, etc., and are composed of multiple still images (frames) as timing data. The processing performed in the image processing system 1 is mainly as follows.

[0030] (1) Generation of input frames

[0031] First, the image processing system 1 generates an image (input frame) depicting one or more game objects by rendering 3D data showing the game objects as seen from a predetermined viewpoint. This input frame is an image with a predetermined number of pixels (input pixel count) and a predetermined image quality (input image quality) (see [link to image processing system]). Figure 3 Input frames are generated at predetermined time intervals. The number of pixels in an input frame is, for example, 1920 × 1080 (1080p). The generated input frames are not displayed as is in display unit 18, but are temporarily stored in storage unit 12 for use in subsequent processing. In the following description, the processing of the nth input frame 20_n will be given primarily as an example, but similar processing is performed on other input frames (i.e., n = 2, 3, ..., N).

[0032] (2) Acquisition of intermediate frames

[0033] Image processing system 1 acquires a frame (intermediate frame) 22_n with a pixel count (intermediate pixel count) greater than the input pixel count based on the acquired input frame 20_n. The intermediate pixel count is, for example, 3840 × 2160 (4K). Specifically, intermediate frame 22_n is generated by performing magnification and interpolation processing on the input frame 20_n (see [link to documentation]). Figure 3 ).

[0034] It should be noted that although the intermediate frame 22_n has more pixels than the input frame 20_n, its image quality is not necessarily adequately improved. That is, the image quality of a frame is not simply a matter of the number of pixels (high resolution). When compared with a reference frame, the image quality of a frame can be evaluated based on factors such as a high SN ratio, high spatial frequency reproducibility, high temporal stability (less artifacts and flicker when displaying multiple frames consecutively), or a combination thereof.

[0035] (3) Estimation of frame acquisition

[0036] Image processing system 1 inputs intermediate frame 22_n into machine learning model 200 to obtain estimated frame 24_n. Estimated frame 24_n is an image with the same number of pixels as the intermediate frame (estimated pixel count) and an image quality equal to or higher than the input image quality (estimated image quality) (see [link to image processing system]). Figure 3 ).

[0037] Here, in addition to the intermediate frame 22_n, the (n-1)th auxiliary information 28_n-1 is input into the machine learning model 200 (see...). Figure 2 and Figure 3 The auxiliary information 28_n-1 is the (n-1)th cumulative feature information 26_n-1 based on the features indicating the characteristics of the first to the (n-1)th intermediate frames 22. The cumulative feature information 26 and the auxiliary information 28 will be explained in detail later.

[0038] Machine learning model 200 is a model trained using multiple training datasets. Each training dataset includes intermediate training frames with the number of intermediate pixels generated based on the number of input pixels and the quality of the input image, and estimated training frames with the estimated number of pixels and the estimated image quality. Details of machine learning model 200 will be described in more detail later.

[0039] (4) Acquisition of cumulative feature information

[0040] The machine learning model 200 has a cumulative feature information output layer 202, which receives intermediate frames 22_n and auxiliary information 28_n-1, and outputs the nth cumulative feature information 26_n indicating the features of the first to nth input frames 22 (see [link]). Figure 2 Image processing system 1 acquires the nth cumulative feature information 26_n.

[0041] The acquired nth cumulative feature information 26_n is input to the estimation frame output layer 204, and the nth estimated frame 24_n is output from the estimation frame output layer 204 (see Figure 2The acquired nth cumulative feature information 26_n is also stored in storage unit 12 and used to estimate the estimated frame 24_n+1 corresponding to the next input frame (n+1th input frame) 20_n+1.

[0042] (5) Acquisition of auxiliary information

[0043] As described above, the (n-1)th cumulative feature information 26_n-1 is information indicating the features of the first to (n-1)th intermediate frames 22 (and therefore the first to (n-1)th input frames 20). In this way, by using the cumulative feature information 26_n-1, which is the cumulative information of the past input frames 20, to estimate the nth estimated frame 24_n, the amount of information available for estimation is increased, making it possible to obtain a high-quality estimated frame 24_n.

[0044] However, when there is motion or other issues in the game objects displayed between the (n-1)th input frame 20_n-1 and the nth input frame 20_n, when the nth intermediate frame 22_n and the accumulated feature information 26_n-1 are directly input into the machine learning model 200, a phenomenon of afterimages of the game objects displayed in the (n-1)th input frame 20_n-1 (so-called ghosting) may occur.

[0045] Therefore, the image processing system 1 obtains the (n-1)th auxiliary information 28_n-1 by applying various corrections described below to the accumulated feature information 26_n-1 based on information acquired during rendering (information indicating motion vectors, depth buffers, texture types, etc.). Figure 2 and Figure 3 As described above, the acquired (n-1)th auxiliary information 28_n-1, together with the nth intermediate frame 22_n, is input into the machine learning model 200 and used to estimate the nth estimated frame 24_n.

[0046] As described above, the image processing system 1 according to this embodiment estimates the estimation frame 24 using auxiliary information 28 that accumulates past information, in addition to the intermediate frame 22 corresponding to the current input frame 20. This increases the amount of information available for estimation, enabling the acquisition of a high-quality estimation frame 24_n. The image processing system 1 will now be described in detail.

[0047] [3. Functions implemented by the image processing system]

[0048] Figure 4 This is a functional block diagram illustrating an example of the functions implemented by image processing system 1. For example... Figure 4As shown, in the image processing system 1, the following components are implemented: game processing unit 300, rendering unit 302, rendering information storage unit 304, input frame acquisition unit 306, change information acquisition unit 308, intermediate frame acquisition unit 310, machine learning model storage unit 312, estimation frame acquisition unit 314, motion information acquisition unit 316, depth information acquisition unit 318, emergent pixel recognition unit 320, auxiliary information acquisition unit 322, and color change pixel information acquisition unit 324. The game processing unit 300, rendering unit 302, input frame acquisition unit 306, change information acquisition unit 308, intermediate frame acquisition unit 310, estimation frame acquisition unit 314, motion information acquisition unit 316, depth information acquisition unit 318, emergent pixel recognition unit 320, auxiliary information acquisition unit 322, and color change pixel information acquisition unit 324 are mainly implemented by the control unit 10. The rendering information storage unit 304 and the machine learning model storage unit 312 are mainly implemented by the storage unit 12. The game processing unit 300, the rendering unit 302, and the rendering information storage unit 304 are functions provided by the game software.

[0049] [Game Processing Unit]

[0050] The game processing unit 300 performs various game-related processes. For example, according to the game program executed by the control unit 10 and the user input received by the operation unit 16, the game processing unit 300 performs processes such as configuring the game object O in the virtual three-dimensional space VS, manipulating or moving the game object O, and changing the viewpoint C of observing the virtual three-dimensional space VS (see reference). Figure 5 The game object O is composed of primitives such as polygons represented by three-dimensional data. The three-dimensional data includes geometric information such as the position of vertices, topological information indicating how the vertices are connected, and attribute information such as color.

[0051] [Rendering Unit]

[0052] Figure 5This is a diagram illustrating the processing of the rendering unit 302. The rendering unit 302 generates first to Nth (N is a natural number greater than 2) input frames 20 by performing rendering (depicting processing) of 3D data representing one or more game objects O viewed from a predetermined viewpoint C. The rendering unit 302 performs rendering based on the results of various processes performed by the game processing unit 300. Specifically, the rendering unit 302 performs vertex processing (vertex shading) and pixel processing (pixel shading) based on 3D data representing game objects O configured in the virtual 3D space VS. Vertex processing includes coordinate transformation processing (perspective projection) from the view coordinate system to the screen coordinate system, and adds values ​​related to the change in viewpoint C to the perspective projection matrix (camera matrix) used in the coordinate transformation processing, as described later. The rendering unit 302 can perform rendering based on light source information, depth information (depth buffer), texture information, normal information, etc. The texture information of the game objects can be animated textures (animation information) as moving images, or color information (mirror mapping information) incident on the viewpoint C of the game objects, which act as mirrors. Furthermore, the texture information of the game object can be color information (ray tracing information) calculated by extending a straight line connecting the viewpoint C and each pixel on the rendered surface in space, and taking into account transmission and refraction to calculate the light intensity at the first point on the surface of the hit object. In addition to the above processing, the rendering unit 302 can also perform processing such as applying effects like depth of field (DoF) and motion blur. The processing of the rendering unit 302 can be appropriately set by game software developers, etc. Here, game software developers, etc., can adjust the texture MIP based on the estimated number of pixels in the estimated frame 24, etc. This allows noise such as moiré fringes generated in the estimated frame 24 to be suppressed.

[0053] Here, the rendering unit 302 generates each input frame 20 by performing rendering, causing the viewpoint C to change for each input frame 20. Even when the game processing unit 300 fixes the viewpoint C in a predetermined position, the rendering unit 302 makes the viewpoint C different for each input frame 20. As a result, as... Figure 5 As shown, the position of the displayed game object O varies in each input frame 20_n, 20_n+1, and 20_n+2. In other words, the rendering unit 302 applies jitter when generating each input frame 20. Specifically, the rendering unit 302 changes the viewpoint C for each input frame 20 by adding a value corresponding to a size less than one pixel that differs for each input frame 20 to the perspective projection matrix. The rendering unit 302 changes the viewpoint C of each input frame 20 according to a predetermined rule. For example, a Halton sequence can be used as such a rule.

[0054] [Rendering Information Storage Unit]

[0055] The rendering information storage unit 304 stores information required for rendering processing in the rendering unit 302 and information obtained as a result of the rendering processing. For example, the rendering information storage unit 304 stores the input frame 20. The rendering information storage unit 304 also stores change information, motion information, and depth information. Change information, motion information, and depth information will be described in detail later. In addition, when the texture information of the game object is animation information, mirror mapping information, or ray tracing information, the rendering information storage unit 304 can store color-changing pixel information representing the distribution of pixels generated based on that information among the pixels of the input frame 20. Furthermore, the rendering information storage unit 304 can store parameters used in coordinate transformations, light source information, texture information, normal information, etc.

[0056] [Input Frame Acquisition Unit]

[0057] The input frame acquisition unit 306 acquires each of the first to Nth input frames 20. Specifically, the input frame acquisition unit 306 acquires the first to Nth input frames 20 stored in the rendering information storage unit 304.

[0058] [Change Information Acquisition Unit]

[0059] The change information acquisition unit 308 acquires change information. The change information acquisition unit 308 acquires the change information stored in the rendering information storage unit 304. Specifically, the change information is information indicating the amount of change in viewpoint C before and after the change. The information indicating the amount of change can also be referred to as a change vector indicating the direction and distance of the change. For example, the Halton sequence described above contains information indicating the amount of change in viewpoint C, and therefore this information can be used as change information.

[0060] [Intermediate Frame Acquisition Unit]

[0061] The intermediate frame acquisition unit 310 generates intermediate frames 22 corresponding to the input frame 20, with the number of intermediate pixels equal to or greater than the number of input pixels, and acquires the first to the Nth intermediate frames 22 based on each input frame 20. In this embodiment, each intermediate frame 22 has a number of intermediate pixels greater than the number of input pixels. That is, in this embodiment, each intermediate frame 22 is a magnified image of the input frame 20 corresponding to that intermediate frame 22.

[0062] Specifically, the intermediate frame acquisition unit 310, based on the change information and each pixel of each input frame 20, finds the pixel value at the position in the input frame 20 corresponding to each pixel before the change by interpolation, and generates each intermediate frame 22. Figure 6 This is a diagram illustrating the processing in the intermediate frame acquisition unit 310. Figure 6 An example of finding the nth intermediate frame 22_n is shown. For example, as... Figure 6As shown, when the pixel center of the pixel in the intermediate frame 22_n to be acquired is defined as P1,0, the intermediate frame acquisition unit 310 finds the pixel value of P1,0 by bilinear interpolation based on the coordinates and pixel values ​​of the pixel centers P'0,0, P'1,0, P'0,1, and P'1,1 of the four pixels closest to P1,0 in the input frame 20_n. Here, P'1,0 is located at a position offset from P1,0 by the amount of change indicated by the change information. The pixel value of the newly generated pixel through the magnification process is found in the same way. In addition to bilinear interpolation, various known techniques such as bicubic interpolation and Lanczos interpolation can be used as interpolation methods.

[0063] When rendering is performed so that the viewpoint C changes for each input frame 20, the amount of time-series information increases, but a higher quality estimated frame 24 can be obtained by estimating each input frame 20 obtained in this way (hereinafter referred to as the "varied input frame").

[0064] Conversely, when a changed input frame (or a magnified image thereof) is directly fed into the machine learning model 200, the aforementioned effect of viewpoint C variation may lead to a decrease in estimation accuracy.

[0065] Therefore, as described above, in the image processing system 1, based on the change information and each pixel of each input frame 20, the pixel value at the position in the input frame 20 corresponding to each pixel before the change is found through interpolation, generating each intermediate frame 22 and inputting it into the machine learning model 200. This corrects for the impact of changes in viewpoint C, thereby preventing a decrease in estimation accuracy.

[0066] [Machine Learning Model]

[0067] Machine learning model 200 is a model that estimates the nth estimated frame 24_n based on the nth intermediate frame 22_n. Specifically, machine learning model 200 is a model that estimates the nth estimated frame 24_n based on the nth intermediate frame 22_n and the (n-1)th auxiliary information 28_n-1. Specifically, machine learning model 200 is a convolutional neural network (CNN). Known models such as multilayer ResNet with residual connection mechanisms, so-called encoder-decoder type U-Net, etc., can be used as machine learning model 200. The model described in Non-Patent Document 1 can be used as machine learning model 200.

[0068] Machine learning model 200 is a model taught using multiple training datasets. Each training dataset includes intermediate training frames with a number of intermediate pixels generated from training input frames with a number of input pixels, auxiliary information 28 replacing color-changing pixels with predetermined values, and training estimation frames with an estimated number of pixels. Various known techniques, such as backpropagation, can be used to teach machine learning model 200. In the auxiliary information 28 included in the training data, color-changing pixels are replaced with predetermined values. That is, machine learning model 200 has been learned using auxiliary information 28 without applying motion compensation (described later) to the color-changing pixels.

[0069] Specifically, the machine learning model 200 includes a cumulative feature information output layer 202, an estimated frame output layer 204, and a convolutional layer 206 (see [link to model 200]). Figure 2 ).

[0070] The cumulative feature information output layer 202 receives the nth intermediate frame 22_n and the (n-1)th auxiliary information 28_n_n, based on the (n-1)th cumulative feature information 26_n_1 indicating the features of the first to nth intermediate frames 22, and outputs the nth cumulative feature information 26_n indicating the features of the first to nth intermediate frames 22_n. The cumulative feature information output layer 202 may be composed of, for example, one or more convolutional layers. The cumulative feature information 26_n_1 is image information (bitmap format information) with the same number of pixels as the number of intermediate pixels. The cumulative feature information 26_n_1 can also be called a feature map, which indicates the features of the first to nth intermediate frames 22.

[0071] The cumulative feature information output layer 202 receives the first intermediate frame 22_1 and the assigned auxiliary information, and outputs the first cumulative feature information 26_1. When n=1, there is no previous cumulative feature information 26 or auxiliary information 28, so the pre-prepared assigned auxiliary information is input into the cumulative feature information output layer 202 together with the first intermediate frame 22_1.

[0072] The estimated frame output layer 204 receives the nth accumulated feature information 26_n and outputs the nth estimated frame 24_n. Similar to the accumulated feature information output layer 202, the estimated frame output layer 204 may be composed of one or more convolutional layers, for example. Alternatively, the estimated frame output layer 204 may be composed of one or more transposed convolutional layers (deconvolutional layers).

[0073] Convolutional layer 206 is a layer that reduces the number of channels in the accumulated feature information 26 while maintaining the number of pixels. The accumulated feature information 26 output from convolutional layer 206 is used for processing in auxiliary information acquisition unit 322. Convolutional layer 206 can reduce the dimension of accumulated feature information 26, thereby reducing computational cost. Convolutional layer 206 is, for example, a convolutional layer with a kernel size of 1×1, but is not limited to this.

[0074] [Machine Learning Model Storage Unit]

[0075] Machine learning model storage unit 312 stores machine learning model 200. Specifically, machine learning model storage unit 312 stores the parameters of machine learning model 200 (such as the number of convolutional layers, the number of nodes used in each convolutional layer, and the weight of each node).

[0076] [Estimated Frame Acquisition Unit]

[0077] The estimation frame acquisition unit 314 inputs each intermediate frame 22 into the machine learning model 200 and acquires the first to the Nth estimation frames 24. Each estimation frame 24 has an estimated number of pixels greater than the number of input pixels and equal to or greater than the number of intermediate pixels. In this embodiment, the estimation frame 24 has the same number of estimated pixels as the number of intermediate pixels. More specifically, the estimation frame acquisition unit 314 inputs the nth intermediate frame 22_n and the (n-1)th auxiliary information 28_n-1 into the machine learning model 200 to acquire the nth estimation frame 24_n.

[0078] [Motion Information Acquisition Unit]

[0079] The motion information acquisition unit 316 acquires the (n-1)th motion information, which is information indicating the amount and direction of motion from the (n-1)th input frame 20_n-1 to the nth input frame 20_n. Specifically, the (n-1)th motion information is image information (bitmap format information) with the same number of pixels as the intermediate pixels and indicating the amount and direction of motion of each pixel between the (n-1)th input frame 20_n-1 and the nth input frame 20_n. Motion information is also referred to as a motion vector. Specifically, the motion information acquisition unit 316 acquires motion information with the same number of pixels as the input pixels and performs amplification and interpolation processing on the motion information to obtain motion information with the same number of pixels as the intermediate pixels. The motion information acquisition unit 316 acquires information indicating no motion (e.g., value 0) as motion information for pixels generated from a game object without motion by rendering. Here, when the texture information of the game object is animation information, mirror mapping information, or ray tracing information, the motion information acquisition unit 316 may acquire information indicating that the pixel is a color-changing pixel instead of information indicating no motion. Color changes in pixels are represented by information such as infinity (INF) or non-numeric (NaN) (see [link to relevant documentation]). Figure 7 ).

[0080] [Depth Information Acquisition Unit]

[0081] The depth information acquisition unit 318 acquires the (n-1)th depth information indicating the depth of each pixel in the (n-1)th input frame 20_n-1, and the nth depth information indicating the depth of each pixel in the nth input frame 20_n. Specifically, the depth information is image information (bitmap format information) with the same number of pixels as the intermediate pixels. The depth information is also referred to as a depth buffer or Z-buffer. Specifically, the depth information acquisition unit 318 acquires depth information with the same number of pixels as the input pixels, and then performs magnification and interpolation processing on the depth information to obtain depth information with the same number of pixels as the intermediate pixels.

[0082] Emergent pixel recognition unit

[0083] Emerging pixel determination unit 320, based on the (n-1)th depth information and the nth depth information, determines the pixels in the nth intermediate frame 22_n that are displayed in the (n-1)th intermediate frame 22_n-1 (refer to...). Figure 3 The nth emerging pixel 222_n is a portion of the game object O not shown in the image. Specifically, the emerging pixel recognition unit 320 identifies the nth emerging pixel 222_n based on the difference between the (n-1)th depth information and the nth depth information. Furthermore, the emerging pixel recognition unit 320 can identify the nth emerging pixel 222_n based on the (n-1)th perspective projection matrix associated with the (n-1)th intermediate frame 22_n and the nth perspective projection matrix associated with the nth intermediate frame 22_n. Additionally, the emerging pixel recognition unit 320 can identify the nth emerging pixel 222_n by using the (n-1)th motion information. More specifically, the emerging pixel recognition unit 320 identifies the nth emerging pixel 222_n and generates nth emerging pixel information, which is image information indicating the position of the nth emerging pixel 222_n.

[0084] [Color Change Pixel Information Acquisition Unit]

[0085] The color-change pixel information acquisition unit 324 identifies the nth color-change pixel based on the object's texture information. This nth color-change pixel includes color information that changes independently of the object's motion in the nth intermediate frame 22, and acquires the color-change pixel information. Specifically, the illustrated... Figure 7 The diagram shown illustrates the process of recognizing color-changing pixels. Figure 7 The upper part shows a schematic diagram illustrating how the scenery reflected on the windshield in racing games and the like is rendered from the driver's seat viewpoint. Figure 7 A row of trees facing the sky is shown, with the rearview mirror 700 positioned near the center. Each tree is a game object composed of leaves and a trunk, and color information such as green or brown is added to the tree game object as a texture.

[0086] On the other hand, the rearview mirror 700 is a game object consisting of a mirror and parts other than the mirror (such as a frame). The scenery seen from viewpoint C as reflected by the rearview mirror 700 is added to a part of the mirror as a texture. For example, an image generated by rendering in a direction symmetrical to viewpoint C (the direction of mirror reflection) using the position of the rearview mirror 700 as a new viewpoint is attached as a texture to the game object called the rearview mirror.

[0087] Motion information acquisition unit 316 acquires Figure 7 The motion information shown in the lower left corner is referred to as the nth motion information, which indicates the amount and direction of motion from the (n-1)th input frame 20_n-1 to the nth input frame 20_n. Specifically, since viewpoint C is located inside the car, the scenery reflected on the windshield changes according to the movement of the car, which is the game object. For example, the motion information acquisition unit 316 acquires motion information indicating that the pixel representing the tree has moved 0.0f in the x direction and 0.1f in the y direction. The motion information acquisition unit 316 also acquires motion information about the pixels representing the rearview mirror. Here, the rearview mirror 700 is fixed inside the vehicle and is stationary when viewed from viewpoint C. However, the motion information acquisition unit 316 does not acquire motion information indicating that the rearview mirror 700 is in its original stationary state (i.e., motion information indicating movement of 0 in the x direction and 0 in the y direction), but instead acquires information about pixels that change color (e.g., NaN in the x direction and NaN in the y direction).

[0088] The color-change pixel information acquisition unit 324 identifies the nth color-change pixel, which includes color information that changes independently of the motion of the object in the nth intermediate frame 22 of the motion information, and acquires the color-change pixel information. That is, the color-change pixel information acquisition unit 324 identifies pixels in the motion information that contain information indicating they are color-change pixels (here, NaN), and generates... Figure 7 The color-change pixel information shown in the lower right corner is image information (bitmap format information), where the identified color-change pixels are 0, and all pixels other than the color-change pixels are reference RFM. Note that Refer RFM indicates that no calculations are performed on the pixels during the processing of the color-change pixel information by the auxiliary information acquisition unit. Specifically, the color-change pixel information acquisition unit 324 acquires color-change pixel information with the same number of pixels as the input pixels, and then performs amplification and interpolation processing on the color-change information to obtain color-change pixel information with the same number of pixels as the intermediate pixels.

[0089] Note that when the rendering information storage unit 304 stores color-changing pixel information representing the distribution of pixels generated based on animation information, mirror mapping information, or ray tracing information, the color-changing pixel information acquisition unit 324 can acquire the color-changing pixel information from the rendering information storage unit 304 without using motion information.

[0090] [Auxiliary Information Acquisition Unit]

[0091] The auxiliary information acquisition unit 322 acquires the (n-1)th auxiliary information 28_n-1 by applying motion compensation to the (n-1)th accumulated feature information 26_n-1 based on the (n-1)th motion information. Motion compensation refers to the process of moving the pixel at position x in the (n-1)th accumulated feature information 26_n to position x'. For example, when the pixel at position x in the (n-1)th intermediate frame 22_n-1 has already moved to position x' in the nth intermediate frame 22_n (see...). Figure 3 That is, the auxiliary information acquisition unit 322 acquires the (n-1)th auxiliary information 28_n-1 based on the (n-1)th motion information by setting the pixel value of one or more pixels of the (n-1)th accumulated feature information 26_n-1 to the pixel of the position moved according to the amount and direction of the pixel's motion.

[0092] When there is motion of game object O between the nth input frame 20_n and the (n-1)th input frame 20_n-1, when obtaining the nth estimated frame 24_n, when the nth intermediate frame 22_n and the (n-1)th accumulated feature information 26_n-1 are directly input into the machine learning model 200, a ghosting phenomenon may occur, in which the afterimage of game object O displayed in the nth intermediate frame 22_n is displayed in the output nth estimated frame 24_n.

[0093] Therefore, in the image processing system 1, as described above, motion compensation is applied to the (n-1)th accumulated feature information 26_n-1 based on the (n-1)th motion information to obtain the (n-1)th auxiliary information 28_n-1, and when the nth estimated frame 24_n is obtained, the (n-1)th auxiliary information 28_n-1 is input into the machine learning model 200. This suppresses the aforementioned ghosting phenomenon.

[0094] Furthermore, the auxiliary information acquisition unit 322 acquires the (n-1)th auxiliary information 28_n-1 by replacing the pixel value of the nth emerging pixel 222_n in the (n-1)th accumulated feature information 26_n-1 with a predetermined value. Specifically, the auxiliary information acquisition unit 322 acquires the (n-1)th auxiliary information 28_n-1 based on the nth emerging pixel information by replacing the pixel value of the nth emerging pixel 222_n in the (n-1)th accumulated feature information 26_n-1 with a predetermined value. The predetermined value can be a constant value such as 0 (black), or it can be the pixel value of the nth emerging pixel 222_n in the nth intermediate frame 22_n.

[0095] When all or part of the game object O that was not displayed in the (n-1)th input frame 20_n is displayed, and when the nth intermediate frame 22_n and the (n-1)th cumulative feature information 26_n-1 are directly input into the machine learning model 200 to obtain the nth estimated frame 24_n, the above-mentioned ghosting phenomenon may occur in the output nth estimated frame 24_n.

[0096] Therefore, as described above, the image processing system 1 determines the nth emerging pixel 222_n in the pixels of the nth intermediate frame 22_n that displays all or part of the game object O that is not displayed in the (n-1)th intermediate frame 22_n-1, and obtains the (n-1)th auxiliary information 28_n-1 by replacing the pixel value of the nth emerging pixel 222_n in the (n-1)th accumulated feature information 26_n-1 with a predetermined value. This suppresses the aforementioned ghosting phenomenon.

[0097] Furthermore, the auxiliary information acquisition unit 322 acquires the nth auxiliary information by replacing the pixel value of the color-changing pixel at the nth accumulated feature information with a predetermined value. Specifically, the auxiliary information acquisition unit 322 acquires the nth auxiliary information 28_n by replacing the pixel value of the nth color-changing pixel in the nth accumulated feature information 26_n with a predetermined value based on the nth color-changing pixel information. The predetermined value can be a constant value such as 0 (black), or it can be the pixel value of the nth color-changing pixel n in the nth intermediate frame 22_n. The auxiliary information acquisition unit 322 does not perform the above replacement for pixels other than the color-changing pixels.

[0098] As described above, the color-changing pixels identified are those whose color information has already been acquired based on the texture changes of the game object, regardless of the game object's motion information. Even if the game object is moving, the motion information representing that movement is unrelated to the game object's texture. When the aforementioned motion compensation is applied to such color-changing pixels, the image quality will degrade.

[0099] In image processing system 1, as described above, based on the texture information of the object, the nth color-changing pixel includes color information that changes independently of the motion of the object in the nth intermediate frame, and the nth auxiliary information is obtained by replacing the pixel value of the color-changing pixel at the nth accumulated feature information with a predetermined value. Thus, the aforementioned image quality degradation can be suppressed.

[0100] Note that the motion compensation, replacement processing based on appearance pixel information, and replacement processing based on color change pixel information performed by the auxiliary information acquisition unit 322 can be performed entirely or partially on a cumulative feature information 26.

[0101] [4. Processing performed in the image processing system]

[0102] Figure 8 This is a flowchart illustrating an example of the processing flow performed in the image processing system 1. The processing is performed by the control unit 10 according to a program stored in the storage unit 12. Figure 7 The processing is shown.

[0103] (1) Handling when n=1

[0104] First, the control unit 10 acquires a first input frame 20_1 (S700). The control unit 10 acquires a first intermediate frame 22_1 based on the first input frame 20_1 (S702). Then, the control unit 10 inputs the first intermediate frame 22_1 and the assigned auxiliary information into the machine learning model 200, and acquires a first estimated frame 24_1 and first cumulative feature information 26_1 (S704).

[0105] (2) Handling when n≥2

[0106] Control unit 10 acquires the nth input frame 20_n (S706). Control unit 10 acquires the nth intermediate frame 22_n based on the nth input frame 20_n (S708).

[0107] Next, the control unit 10 acquires the (n-1)th motion information (S710). Additionally, the control unit 10 acquires the (n-1)th depth information and the nth depth information (S712) and identifies the nth emerging pixel 222_n based on the (n-1)th depth information and the nth depth information (S714). The control unit 10 acquires the (n-1)th auxiliary information 28_n-1 based on the (n-1)th accumulated feature information 26_n-1, the (n-1)th motion information, and the nth emerging pixel 222_n (S716). The control unit 10 then inputs the nth intermediate frame 22_n and the (n-1)th auxiliary information 28_n-1 into the machine learning model 200 to acquire the nth estimated frame 24_n and the nth accumulated feature information 26_n (S718). The control unit 10 determines whether a next frame exists (S720), and if it determines that a next frame exists (S720: yes), it increments n to n+1 and repeats the processing from S706 to S718. If the control unit 10 determines that there is no next frame (S720: N), it ends the process. If the control unit 10 determines that there is no next frame (S720: No), it can make the display unit 18 directly display the first to the Nth estimated frames 24.

[0108] [5. Summary]

[0109] According to the image processing system 1 of the above embodiment, based on the texture information of the object, an nth color-changing pixel is obtained. This nth color-changing pixel includes color information that changes independently of the motion of the object in the nth intermediate frame. Nth auxiliary information is obtained by replacing the pixel value of the color-changing pixel in the nth accumulated feature information with a predetermined value. In other words, in an image processing apparatus that estimates a high-quality motion image based on a low-quality motion image using motion information and information from past frames, high-precision super-resolution processing can be performed on a motion image generated from an object with varying texture, without relying on motion information.

[0110] The invention disclosed herein is not limited to the embodiments described above. Furthermore, the specific strings or values ​​described above and those in the accompanying drawings are examples, and the invention is not limited to these strings or values.

[0111] For example, in this embodiment, an example has been given where the number of intermediate pixels is greater than the number of input pixels and the number of intermediate pixels is the same as the number of estimated pixels. However, the number of intermediate pixels can be the same as the number of input pixels and the number of estimated pixels can be greater than the number of intermediate pixels. That is to say, intermediate frame 22 does not necessarily need to be a magnified version of input frame 20.

[0112] [6. Postscript] (1)

[0114] An image processing system includes at least one processor, wherein:

[0115] The at least one processor

[0116] Acquire each input frame from the first input frame to the Nth input frame (N is a natural number of 2 or greater) with a predetermined number of input pixels;

[0117] The first to Nth intermediate frames are obtained by generating intermediate frames based on the input frame. The intermediate frames have an intermediate pixel count that is equal to or greater than the number of input pixels and correspond to the input frame.

[0118] as well as

[0119] The first to Nth estimated frames are obtained by inputting each intermediate frame into a machine learning model. The estimated frames have an estimated number of pixels that is equal to or greater than the number of intermediate pixels and greater than the number of input pixels.

[0120] The machine learning model

[0121] It includes a cumulative feature information output layer, which has an nth intermediate frame (n=2, 3, ..., N) and inputs n-1 auxiliary information based on the n-1th cumulative feature information indicating the features from the first intermediate frame to the (n-1th)th intermediate frame, and outputs the nth cumulative feature information indicating the features from the first intermediate frame to the nth intermediate frame;

[0122] And an estimated frame output layer, wherein the nth accumulated feature information is input, and wherein the nth estimated frame is output;

[0123] The at least one processor

[0124] The nth color-changing pixel is identified based on the texture information of the object. The nth color-changing pixel includes color information that changes independently of the object's motion in the nth intermediate frame. The nth auxiliary information is obtained by replacing the pixel value of the color-changing pixel in the nth accumulated feature information with a predetermined value.

[0125] The machine learning model

[0126] Learning is performed using multiple training data sets, each including a learning intermediate frame with the specified number of intermediate pixels. These learning intermediate frames are generated based on the learning intermediate frames with the specified number of intermediate pixels, and are also generated based on the learning input frames with the specified number of input pixels.

[0127] The color-changing pixels are replaced with the auxiliary information of a predetermined value, and the learned estimation frame has the estimated number of pixels. (2)

[0129] According to the image processing system described in (1), wherein

[0130] Each input frame is an image obtained by rendering three-dimensional data depicting one or more objects as seen from a predetermined viewpoint. (3)

[0132] According to the image processing system described in (2), wherein

[0133] Each input frame is an image obtained by rendering the viewpoint so that it changes for each input frame.

[0134] The at least one processor acquires change information, which is information related to the change of viewpoint in each input frame during the rendering, and

[0135] Based on the change information and each pixel of each input frame, each intermediate frame is generated by interpolating the pixel values ​​at positions corresponding to each pixel before the change in the input frame. (4)

[0137] According to the image processing system described in (2) or (3), wherein,

[0138] The at least one processor

[0139] Obtain the (n-1)th motion information, which is information indicating the amount and direction of motion from the (n-1)th input frame to the nth input frame, and

[0140] The (n-1)th auxiliary information is obtained by applying motion compensation to the (n-1)th cumulative feature information based on the (n-1)th motion information. (5)

[0142] According to the image processing system described in (4), wherein

[0143] The at least one processor

[0144] Obtain the (n-1)th depth information indicating the depth of each pixel in the (n-1)th input frame and the nth depth information indicating the depth of each pixel in the nth input frame.

[0145] The nth emerging pixel is identified based on the (n-1)th depth information and the nth depth information. The nth emerging pixel is a pixel in the nth intermediate frame that displays all or part of the object not displayed in the (n-1)th intermediate frame.

[0146] The (n-1)th auxiliary information is obtained by replacing the pixel value of the nth emerging pixel in the (n-1)th accumulated feature information with a predetermined value. (6)

[0148] According to the image processing system described in (1) or (2), wherein,

[0149] The cumulative feature information output layer is input with a first intermediate frame and assigned auxiliary information, and outputs the first cumulative feature information. (7)

[0151] According to the image processing system described in (1) or (2), wherein,

[0152] The accumulated feature information is image information with the same number of pixels as the intermediate pixels. (8)

[0154] According to the image processing system described in (1) or (2),

[0155] The color-changing pixels are represented by information indicating infinity or not being a number. (9)

[0157] An image processing method, wherein: processor

[0158] Acquire each input frame from the first input frame to the Nth input frame (N is a natural number of 2 or greater) with a predetermined number of input pixels;

[0159] The first to Nth intermediate frames are obtained by generating intermediate frames based on the input frame. The intermediate frames have an intermediate pixel count that is equal to or greater than the number of input pixels and correspond to the input frame.

[0160] as well as

[0161] The first to Nth estimated frames are obtained by inputting each intermediate frame into a machine learning model. The estimated frames have an estimated number of pixels that is equal to or greater than the number of intermediate pixels and greater than the number of input pixels.

[0162] The machine learning model

[0163] It includes a cumulative feature information output layer, which has an nth intermediate frame (n=2, 3, ..., N) and inputs n-1 auxiliary information based on the n-1th cumulative feature information indicating the features from the first intermediate frame to the (n-1th)th intermediate frame, and outputs the nth cumulative feature information indicating the features from the first intermediate frame to the nth intermediate frame;

[0164] And an estimated frame output layer, wherein the nth accumulated feature information is input, and wherein the nth estimated frame is output;

[0165] The processor

[0166] The nth color-changing pixel is identified based on the texture information of the object. The nth color-changing pixel includes color information that changes independently of the object's motion in the nth intermediate frame. The nth auxiliary information is obtained by replacing the pixel value of the color-changing pixel in the nth accumulated feature information with a predetermined value.

[0167] The machine learning model

[0168] Learning is performed using multiple training data sets, each including a learning intermediate frame with the specified number of intermediate pixels. These learning intermediate frames are generated based on the learning intermediate frames with the specified number of intermediate pixels, and are also generated based on the learning input frames with the specified number of input pixels.

[0169] The color-changing pixels are replaced with the auxiliary information of a predetermined value, and the learned estimation frame has the estimated number of pixels. (10)

[0171] A program for making:

[0172] Input frame acquisition component

[0173] Acquire each input frame from the first input frame to the Nth input frame (N is a natural number of 2 or greater) with a predetermined number of input pixels;

[0174] The intermediate frame acquisition unit acquires the first to Nth intermediate frames by generating intermediate frames based on the input frame. The intermediate frames have an intermediate pixel count that is equal to or greater than the number of input pixels and correspond to the input frame.

[0175] And an estimated frame acquisition component, which acquires the first to the Nth estimated frames by inputting each intermediate frame into a machine learning model function in a computer, wherein the estimated frame has an estimated number of pixels that is equal to or greater than the number of intermediate pixels and greater than the number of input pixels;

[0176] in

[0177] The machine learning model

[0178] It includes a cumulative feature information output layer, which has an nth intermediate frame (n=2, 3, ..., N) and inputs n-1 auxiliary information based on the n-1th cumulative feature information indicating the features from the first intermediate frame to the (n-1th)th intermediate frame, and outputs the nth cumulative feature information indicating the features from the first intermediate frame to the nth intermediate frame;

[0179] And an estimated frame output layer, wherein the nth accumulated feature information is input, and wherein the nth estimated frame is output;

[0180] The program also enables

[0181] The recognition component identifies the nth color-changing pixel based on the object's texture information. The nth color-changing pixel includes color information that changes independently of the object's motion in the nth intermediate frame. The nth auxiliary information is obtained by replacing the pixel value of the color-changing pixel in the nth accumulated feature information with a predetermined value.

[0182] The machine learning model

[0183] Learning is performed using multiple training data sets, each including a learning intermediate frame with the specified number of intermediate pixels. These learning intermediate frames are generated based on the learning intermediate frames with the specified number of intermediate pixels, and are also generated based on the learning input frames with the specified number of input pixels.

[0184] The color-changing pixels are replaced with the auxiliary information of a predetermined value, and the learned estimation frame has the estimated number of pixels.

Claims

1. An image processing system, comprising at least one processor, wherein: The at least one processor Acquire each input frame from the first input frame to the Nth input frame (N is a natural number of 2 or greater) with a predetermined number of input pixels; The first to Nth intermediate frames are obtained by generating intermediate frames based on the input frame. The intermediate frames have an intermediate pixel count that is equal to or greater than the number of input pixels and correspond to the input frame. as well as The first to Nth estimated frames are obtained by inputting each intermediate frame into a machine learning model. The estimated frames have an estimated number of pixels that is equal to or greater than the number of intermediate pixels and greater than the number of input pixels. The machine learning model It includes a cumulative feature information output layer, which has an nth intermediate frame (n=2, 3, ..., N) and inputs n-1 auxiliary information based on the n-1th cumulative feature information indicating the features from the first intermediate frame to the (n-1th)th intermediate frame, and outputs the nth cumulative feature information indicating the features from the first intermediate frame to the nth intermediate frame; And an estimated frame output layer, wherein the nth accumulated feature information is input, and wherein the nth estimated frame is output; The at least one processor The nth color-changing pixel is identified based on the texture information of the object. The nth color-changing pixel includes color information that changes independently of the object's motion in the nth intermediate frame. The nth auxiliary information is obtained by replacing the pixel value of the color-changing pixel in the nth accumulated feature information with a predetermined value. The machine learning model Learning is performed using multiple training data sets, each including a learning intermediate frame with the specified number of intermediate pixels. These learning intermediate frames are generated based on the learning intermediate frames with the specified number of intermediate pixels, and are also generated based on the learning input frames with the specified number of input pixels. The color-changing pixels are replaced with the auxiliary information of a predetermined value, and the learned estimation frame has the estimated number of pixels.

2. The image processing system according to claim 1, wherein, Each input frame is an image obtained by rendering three-dimensional data depicting one or more objects as seen from a predetermined viewpoint.

3. The image processing system according to claim 2, wherein, Each input frame is an image obtained by rendering the viewpoint so that it changes for each input frame. The at least one processor acquires change information, which is information related to the change of viewpoint in each input frame during the rendering, and Based on the change information and each pixel of each input frame, each intermediate frame is generated by interpolating the pixel values ​​at positions corresponding to each pixel before the change in the input frame.

4. The image processing system according to claim 2 or 3, wherein, The at least one processor Obtain the (n-1)th motion information, which is information indicating the amount and direction of motion from the (n-1)th input frame to the nth input frame, and The (n-1)th auxiliary information is obtained by applying motion compensation to the (n-1)th cumulative feature information based on the (n-1)th motion information.

5. The image processing system according to claim 4, wherein, The at least one processor Obtain the (n-1)th depth information indicating the depth of each pixel in the (n-1)th input frame and the nth depth information indicating the depth of each pixel in the nth input frame. The nth emerging pixel is identified based on the (n-1)th depth information and the nth depth information. The nth emerging pixel is a pixel in the nth intermediate frame that displays all or part of the object not displayed in the (n-1)th intermediate frame. The (n-1)th auxiliary information is obtained by replacing the pixel value of the nth emerging pixel in the (n-1)th accumulated feature information with a predetermined value.

6. The image processing system according to claim 1 or 2, wherein, The cumulative feature information output layer is input with a first intermediate frame and assigned auxiliary information, and outputs the first cumulative feature information.

7. The image processing system according to claim 1 or 2, wherein, The accumulated feature information is image information with the same number of pixels as the intermediate pixels.

8. The image processing system according to claim 1 or 2, in, The color-changing pixels are represented by information indicating infinity or not a number.

9. An image processing method, wherein: Acquire each input frame from the first input frame to the Nth input frame (N is a natural number of 2 or greater) with a predetermined number of input pixels; The first to Nth intermediate frames are obtained by generating intermediate frames based on the input frame. The intermediate frames have an intermediate pixel count that is equal to or greater than the number of input pixels and correspond to the input frame. as well as The first to Nth estimated frames are obtained by inputting each intermediate frame into a machine learning model. The estimated frames have an estimated number of pixels that is equal to or greater than the number of intermediate pixels and greater than the number of input pixels. The machine learning model It includes a cumulative feature information output layer, which has an nth intermediate frame (n=2, 3, ..., N) and inputs n-1 auxiliary information based on the n-1th cumulative feature information indicating the features from the first intermediate frame to the (n-1th)th intermediate frame, and outputs the nth cumulative feature information indicating the features from the first intermediate frame to the nth intermediate frame; And an estimated frame output layer, wherein the nth accumulated feature information is input, and wherein the nth estimated frame is output; The nth color-changing pixel is identified based on the texture information of the object. The nth color-changing pixel includes color information that changes independently of the object's motion in the nth intermediate frame. The nth auxiliary information is obtained by replacing the pixel value of the color-changing pixel in the nth accumulated feature information with a predetermined value. The machine learning model Learning is performed using multiple training data sets, each including a learning intermediate frame with the specified number of intermediate pixels. These learning intermediate frames are generated based on the learning intermediate frames with the specified number of intermediate pixels, and are also generated based on the learning input frames with the specified number of input pixels. The color-changing pixels are replaced with the auxiliary information of a predetermined value, and the learned estimation frame has the estimated number of pixels.

10. A program for making: Input frame acquisition component Acquire each input frame from the first input frame to the Nth input frame (N is a natural number of 2 or greater) with a predetermined number of input pixels; The intermediate frame acquisition unit acquires the first to Nth intermediate frames by generating intermediate frames based on the input frame. The intermediate frames have an intermediate pixel count that is equal to or greater than the number of input pixels and correspond to the input frame. And an estimated frame acquisition component, which acquires the first to the Nth estimated frames by inputting each intermediate frame into a machine learning model function in a computer, wherein the estimated frame has an estimated number of pixels that is equal to or greater than the number of intermediate pixels and greater than the number of input pixels; in The machine learning model It includes a cumulative feature information output layer, which has an nth intermediate frame (n=2, 3, ..., N) and inputs n-1 auxiliary information based on the n-1th cumulative feature information indicating the features from the first intermediate frame to the (n-1th)th intermediate frame, and outputs the nth cumulative feature information indicating the features from the first intermediate frame to the nth intermediate frame; And an estimated frame output layer, wherein the nth accumulated feature information is input, and wherein the nth estimated frame is output; The program also enables The recognition component identifies the nth color-changing pixel based on the object's texture information. The nth color-changing pixel includes color information that changes independently of the object's motion in the nth intermediate frame. The nth auxiliary information is obtained by replacing the pixel value of the color-changing pixel in the nth accumulated feature information with a predetermined value. The machine learning model Learning is performed using multiple training data sets, each including a learning intermediate frame with the specified number of intermediate pixels. These learning intermediate frames are generated based on the learning intermediate frames with the specified number of intermediate pixels, and are also generated based on the learning input frames with the specified number of input pixels. The color-changing pixels are replaced with the auxiliary information of a predetermined value, and the learned estimation frame has the estimated number of pixels.

Images