Image processing system, image processing method, and program

Abstract

An estimated frame acquisition unit (314) inputs each intermediate frame (22) to a machine learning model (200), and acquires each of first to Nth estimated frames (24) having an estimated number of pixels equal to or greater than an intermediate number of pixels greater than the input number of pixels. A machine learning model (200) includes a cumulative feature information output layer (202) to which an n-th intermediate frame (22n) (n = 2, 3,..., N) and (n-1) th auxiliary information (28n-1) based on (n-1) th cumulative feature information (26n-1) indicating features of first to (n-1) th intermediate frames (22) are input, wherein the accumulated feature information output layer (202) outputs n-th accumulated feature information (26n) indicating features of the first to n-th intermediate frames (22). The machine learning model 200 further includes an estimated frame output layer 204 that inputs the nth cumulative feature information (26n), wherein the estimated frame output layer 204 outputs the nth estimated frame (24n). A machine learning model (200) has been learned by a plurality of pieces of training data each including a learning intermediate frame having an intermediate pixel number generated based on a learning input frame having an input pixel number and a learning estimation frame having an estimated pixel number.

Description

Technical Field

[0001] This invention relates to image processing systems, image processing methods, and programs. Background Technology

[0002] The technique of using conventional machine learning models to estimate high-resolution single images based on low-resolution single images (super-resolution) is conventionally known (see Non-Patent Literature 1 below).

[0003] Reference List

[0004] Non-patent literature

[0005] Non-patent literature 1: Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang. Learning a Deep Convolutional Network for Image Super Resolution, Proceedings of the European Conference on Computer Vision (ECCV), 2014. Summary of the Invention

[0006] Technical issues

[0007] The inventors have investigated applying the aforementioned super-resolution to moving images, such as game graphics. Here, in super-resolution of moving images, it is believed that a higher quality moving image can be estimated not only by using information from each frame as the processing target, but also by considering information from past frames. However, because in conventional super-resolution, as mentioned above, a single image (still image) is considered as the target, even when this technique is directly applied to moving images, information from past frames in the estimation of high-quality moving images is not adequately considered.

[0008] The purpose of this invention is to provide an image processing system, image processing method, and program that utilizes information from past frames to estimate high-quality dynamic images based on low-quality dynamic images.

[0009] Solution to the problem

[0010] The image processing system according to the present invention includes at least one processor, wherein the at least one processor: acquires each of the first to Nth input frames (N is a natural number equal to or greater than 2) having a predetermined number of input pixels; acquires each of the first to Nth intermediate frames by generating an intermediate frame corresponding to the input frame and having an intermediate number of pixels equal to or greater than the number of input pixels, based on each of the input frames; and inputs each of the intermediate frames into a machine learning model, and acquires each of the first to Nth estimated frames having an estimated number of pixels greater than the number of input pixels and equal to or greater than the number of intermediate pixels. The machine learning model includes: a cumulative feature information output layer, which is input to the nth intermediate frame (n=2, 3, …, N) and (n-1)th auxiliary information based on the (n-1)th cumulative feature information indicating the features of the first to (n-1)th intermediate frames, wherein the cumulative feature information output layer outputs the nth cumulative feature information indicating the features of the first to nth intermediate frames; and an estimated frame output layer, which is input to the nth cumulative feature information, wherein the estimated frame output layer outputs the nth estimated frame. The machine learning model has been trained using multiple training datasets, which include intermediate training frames with intermediate pixel counts generated from the input frames with the input pixel count, and estimated training frames with the estimated pixel count. Attached Figure Description

[0011] [ Figure 1 [ ] is a diagram illustrating an example of the hardware configuration of an image processing system.

[0012] [ Figure 2 [ ] is a diagram illustrating an overview of an image processing system.

[0013] [ Figure 3 [Illustrated diagram illustrating the processing of an image processing system]

[0014] [ Figure 4 [ ] is a function block diagram illustrating an example of the functions implemented by an image processing system.

[0015] [ Figure 5 [] is a diagram illustrating the processing within the rendering unit.

[0016] [ Figure 6 [ ] is a diagram illustrating the processing in the intermediate frame acquisition unit.

[0017] [ Figure 7 [ ] is a flowchart illustrating an example of the process performed by an image processing system. Detailed Implementation

[0018] Examples of embodiments of the image processing system according to the present invention will now be described with reference to the accompanying drawings.

[0019] [1. Hardware configuration of the image processing system]

[0020] Figure 1 This is a diagram illustrating an example of the hardware configuration of an image processing system 1. Image processing system 1 is, for example, a computer such as a game console (game machine). Figure 1 As shown, the image processing system 1 includes a control unit 10, a storage unit 12, a communication unit 14, an operation unit 16, a display unit 18, and an audio output unit 19.

[0021] The control unit 10 includes, for example, a program control device (such as a CPU that operates according to a program installed in the image processing system 1). Furthermore, the control unit 10 also includes a GPU (graphics processing unit), which draws images in a frame buffer based on graphics commands and data provided from the CPU.

[0022] Storage unit 12 includes, for example, main storage devices (such as ROM, RAM, etc.) and auxiliary storage devices (such as HDD, SSD, etc.). Programs executed by control unit 10 are stored in storage unit 12. In addition to programs used to implement all the functions of the image processing system 1 mentioned below, storage unit 12 also stores, for example, game programs (game software). Furthermore, storage unit 12 secures a frame buffer area where images are rendered by the GPU.

[0023] Communication unit 14 is a communication interface (such as an Ethernet (registered trademark) module or a wireless LAN module).

[0024] The operation unit 16 is a user interface (such as a keyboard or mouse, a controller for a game console, etc.) that receives user operation input and outputs signals indicating its content to the control unit 10.

[0025] Display unit 18 is a display device (such as a liquid crystal display, organic EL display, etc.) that displays various images according to the instructions of control unit 10.

[0026] The audio output unit 19 is, for example, a speaker, which outputs audio as indicated by the audio data generated by the image processing system 1.

[0027] In addition to the devices mentioned above, the image processing system 1 may also include an optical disc drive for reading optical discs (such as DVD-ROM or Blu-ray discs) or a USB (Universal Serial Bus) port.

[0028] [2. Overview of Image Processing Systems]

[0029] Figure 2 This is a diagram illustrating the general structure of image processing system 1. Figure 3This diagram schematically illustrates the processing of the image processing system 1. In this embodiment, the image processing system 1 is used to improve the image quality of playing dynamic images in a game. Playing dynamic images is a dynamic image generated based on a game program executed by the control unit 10 or user input received by the operation unit 16, and consists of multiple still images (frames) as time-series data. The processing that occurs in the image processing system 1 is mainly as follows.

[0030] (1) Generation of input frames

[0031] First, the image processing system 1 generates an image (input frame) depicting one or more of these game objects by rendering 3D data indicating the view from a specified viewpoint. This input frame is an image with a specified number of pixels (input pixel count) (see reference). Figure 3 Input frames are generated at predetermined intervals. The number of pixels in the input frame is, for example, 1920×1080 (1080p). Each generated input frame is not displayed as is on the display unit 18, but is stored in the storage unit 12 at once, and subsequent processing is applied to the generated input frames. In the following description, the processing targeting the nth input frame 20_n is mainly illustrated. At the same time, the same processing is performed on other input frames (i.e., n=2, 3, ..., N).

[0032] (2) Acquisition of intermediate frames

[0033] Image processing system 1 acquires a frame (intermediate frame) 22_n with a number of pixels greater than the number of input pixels (intermediate pixel count) based on the acquired input frame 20_n. The number of intermediate pixels is, for example, 3840×2160 (4K). Specifically, intermediate frame 22_n is generated by performing magnification and interpolation processing on input frame 20_n (see [link to documentation]). Figure 3 ).

[0034] Here, although the intermediate frame 22_n has a larger number of pixels than the input frame 20_n, it should be noted that its image quality is not necessarily sufficiently improved. That is, the image quality of a frame does not simply mean a large number of pixels (high level of image quality). When compared with a frame used as a standard, the image quality of a frame can be evaluated based on each or a combination of factors such as, for example, a high SN ratio, high reproducibility of spatial frequency, and high temporal stability (few artifacts or flicker when displaying multiple frames consecutively).

[0035] (3) Estimation of frame acquisition

[0036] Image processing system 1 inputs intermediate frame 22_n into machine learning model 200 and obtains estimated frame 24_n. Estimated frame 24_n is an image with the same number of pixels as the intermediate frame (estimated number of pixels) and an image quality (estimated image quality) equal to or greater than the input image quality (see [link to image processing system]). Figure 3 ).

[0037] Here, in addition to the intermediate frame 22_n, the (n-1)th auxiliary information 28_n-1 is input into the machine learning model 200 (see...). Figure 2 and Figure 3 Auxiliary information 28_n-1 is the (n-1)th cumulative feature information 26_n-1 based on the features indicating the features of the first to (n-1)th intermediate frames 22. Details of cumulative feature information 26 and auxiliary information 28 are described below.

[0038] Machine learning model 200 is a model that has been learned using multiple training datasets. These datasets include intermediate training frames with the number of intermediate pixels generated from the learning input frames, which have the input pixel count and input image quality, and estimated training frames with the estimated pixel count and estimated image quality. Detailed information about machine learning model 200 is provided below.

[0039] (4) Acquisition of cumulative feature information

[0040] The machine learning model 200 has intermediate frames 22_n and auxiliary information 28_n-1 as input, and has a cumulative feature information output layer 202 that outputs indications of the first to nth intermediate frames 22 (see...). Figure 2 The image processing system 1 acquires the nth cumulative feature information 26_n.

[0041] The acquired nth cumulative feature information 26_n is input to the estimation frame output layer 204, and the nth estimated frame 24_n is output from the estimation frame output layer 204 (see [link]). Figure 2 ).

[0042] The acquired nth cumulative feature information 26_n is also stored in storage unit 12 and is provided to the estimation of the estimation frame 24_n+1 corresponding to the next input frame (the (n+1)th input frame) 20_n+1.

[0043] (5) Acquisition of auxiliary information

[0044] As described above, the (n-1)th cumulative feature information 26_n-1 is information indicating the features of the first to (n-1)th intermediate frames 22 (which in the long term are the first to (n-1)th input frames 20). If the cumulative feature information 26_n1, which indicates the information of the past input frames 20, is used for the estimation of the nth estimation frame 24_n, the information available for estimation increases, and thus a high-quality estimation frame 24_n can be obtained.

[0045] However, when, for example, the game object shown moves between the (n-1)th input frame 20_n-1 and the nth input frame 20_n, when the nth intermediate frame 22_n and the accumulated feature information 26_n-1 are input into the machine learning model 200 as is, a phenomenon may occur where the residual image of the game object that was already displayed in the (n-1)th input frame 20_n-1 is eventually displayed (the so-called ghosting phenomenon).

[0046] Therefore, the image processing system 1 obtains the (n-1)th auxiliary information 28_n-1 by applying various corrections mentioned below to the accumulated feature information 26_n-1 based on information available at rendering time (motion vectors, depth buffers, etc.) (see below). Figure 2 and Figure 3 As mentioned above, the acquired (n-1)th auxiliary information 28_n-1, together with the nth intermediate frame 22_n, is input into the machine learning model 200, and the estimate of the nth estimated frame 24_n is applied to information 28_n-1.

[0047] As explained above, the image processing system 1 according to this embodiment uses, in addition to the intermediate frame 22 corresponding to the current input frame 20, auxiliary information 28 indicating the accumulation of past information to estimate the estimation frame 24. Therefore, the information available for estimation is increased, and thus a high-quality estimation frame 24_n can be obtained. Details of the image processing system 1 will now be described.

[0048] [3. Functions implemented by the image processing system]

[0049] Figure 4 This is a functional block diagram illustrating an example of the functions implemented by the image processing system 1. For example... Figure 4As shown, in the image processing system 1, a game processing unit 300, a rendering unit 302, a rendering information storage unit 304, an input frame acquisition unit 306, a change information acquisition unit 308, an intermediate frame acquisition unit 310, a machine learning model storage unit 312, an estimation frame acquisition unit 314, a motion information acquisition unit 316, a depth information acquisition unit 318, an appearance pixel recognition unit 320, and an auxiliary information acquisition unit 322 are implemented primarily by the control unit 10. The rendering information storage unit 304 and the machine learning model storage unit 312 are primarily implemented by the storage unit 12. The game processing unit 300, the rendering unit 302, and the rendering information storage unit 304 have functions provided by the game software.

[0050] [Game Processing Unit]

[0051] The game processing unit 300 performs various game-related processes. For example, based on the game program executed by the control unit 10 or user input received by the operation unit 16, the game processing unit 300 performs processes such as: arranging the game object O in the virtual three-dimensional space VS, manipulating or moving the game object O, and changing the viewpoint C of the virtual three-dimensional space VS (see...). Figure 5 The game object O is configured with primitives such as polygons indicated by 3D data. The 3D data includes geometric information such as the position of vertices, phase information indicating how vertices are bound, and attribute information such as color.

[0052] [Rendering Unit]

[0053] Figure 5This diagram illustrates the processing in rendering unit 302. Rendering unit 302 generates first to Nth (N is a natural number greater than 2) input frames 20 by performing rendering (depicting processing) of the 3D data of one or more game objects O as observed from a specified viewpoint C. Rendering unit 302 performs rendering based on the results of various processing performed by game processing unit 300. Specifically, rendering unit 302 performs vertex processing (vertex shading) and pixel processing (pixel shading) based on the 3D data of game objects O arranged in virtual 3D space VS. Vertex processing includes coordinate transformation processing (perspective projection) from view coordinate system to screen coordinate system, and adds values ​​related to the change in viewpoint C to the perspective projection matrix (camera matrix) used for coordinate transformation processing, as mentioned below. Rendering unit 302 can also perform rendering based on light source information or depth information (depth buffer), texture information, and normal information, etc. In addition to the aforementioned processing, rendering unit 302 can also perform processing such as applying effects like depth of field (DoF) or motion blur. Game software developers and others can appropriately configure the processing in rendering unit 302. Here, game software developers and others can adjust the texture MIP based on the estimated number of pixels in the estimated frame 24. This allows them to suppress noise such as moiré fringes in the estimated frame 24.

[0054] Here, the rendering unit 302 generates each input frame 20 by performing rendering, causing the viewpoint C to change for each input frame 20. Even if the game processing unit 300 has fixed the viewpoint C at a predetermined position, the rendering unit 302 still changes the viewpoint C for each input frame 20. The result is as follows: Figure 5 As shown, in each of the input frames 20_n, 20_n+1, and 20_n+2, the position of the displayed game object O changes. In other words, the rendering unit 302 applies dithering when generating each input frame 20. Specifically, the rendering unit 302 changes the viewpoint C for each input frame 20 by adding a value corresponding to a size less than one pixel to the perspective projection matrix; this value is different for each input frame 20. The rendering unit 302 changes the viewpoint C for each input frame 20 according to a predetermined rule. For example, a Halton sequence can be used as such a rule.

[0055] [Rendering Information Storage Unit]

[0056] The rendering information storage unit 304 stores information required for the rendering process of the rendering unit 302, as well as information obtainable as a result of the rendering process. For example, the rendering information storage unit 304 stores the input frame 20. Furthermore, the rendering information storage unit 304 stores change information, motion information, and depth information. Detailed information on change information, motion information, and depth information is described below. In addition, the rendering information storage unit 304 may store parameters used for coordinate transformation, light source information, texture information, and normal information, etc.

[0057] [Input Frame Acquisition Unit]

[0058] The input frame acquisition unit 306 acquires each of the first to Nth input frames 20. Specifically, the input frame acquisition unit 306 acquires each of the first to Nth input frames 20 stored in the rendering information storage unit 304.

[0059] [Change information acquisition unit]

[0060] The change information acquisition unit 308 acquires change information. The change information acquisition unit 308 acquires change information stored in the rendering information storage unit 304. The change information is information related to changes in viewpoint C for each input frame 20 during rendering. Specifically, the change information is information indicating the amount of change in viewpoint C before and after the change. The information indicating the amount of change can also be a change vector indicating the direction and distance of the change. For example, since information indicating the amount of change in viewpoint C is included in the aforementioned Halton sequence, such information can be used as change information.

[0061] [Intermediate Frame Acquisition Unit]

[0062] The intermediate frame acquisition unit 310 acquires each of the first to Nth intermediate frames 22 by generating an intermediate frame 22 corresponding to the input frame 20 and having an intermediate pixel number equal to or greater than the input pixel number based on each input frame 20. In this embodiment, each intermediate frame 22 has an intermediate pixel number greater than the input pixel number. That is, in this embodiment, each intermediate frame 22 is an image obtained by magnifying the input frame 20 corresponding to that intermediate frame 22.

[0063] Specifically, the intermediate frame acquisition unit 310 obtains the pixel value corresponding to the position of each pixel in the input frame 20 before the change by interpolation based on the change information and each pixel of each input frame 20, thereby generating each intermediate frame 22. Figure 6 This diagram illustrates the processing in the intermediate frame acquisition unit 310. Figure 6 This illustrates the case of obtaining the nth intermediate frame 22_n. For example, as... Figure 6As shown, if the pixel center of the pixel to be acquired in intermediate frame 22_n is P1,0, then the intermediate frame acquisition unit 310 obtains the pixel value of P1,0 through bilinear interpolation based on the coordinates and pixel values ​​of the pixel centers P'0,0, P'1,0, P'0,1, and P'1,1 of the four corresponding pixels closest to P1,0 in the input frame 20_n. Here, P'1,0 is located at the position indicated by the amount of change from the offset change information of P1,0. The pixel value of the newly generated pixel is obtained similarly through magnification processing. In addition to bilinear interpolation, various known methods such as bicubic interpolation and Lanczos interpolation can be used as interpolation methods.

[0064] When rendering is performed so that the viewpoint C changes for each input frame 20, each input frame 20 obtained in this way (hereinafter referred to as the "changed input frame") is used for estimation, thereby obtaining an estimated frame 24 with higher image quality.

[0065] On the other hand, if the changed input frame (or its magnified image) is input into the machine learning model 200 as is, the estimation accuracy may eventually decrease due to the change in viewpoint C mentioned above.

[0066] Therefore, as described above, in the image processing system 1, based on the change information and each pixel of each input frame 20, pixel values ​​corresponding to the positions of each pixel before the change are obtained in the input frame 20 through interpolation, generating each intermediate frame 22 and inputting it into the machine learning model 200. Thus, the effect of the change in viewpoint C is corrected, thereby preventing a decrease in estimation accuracy.

[0067] [Machine Learning Model]

[0068] Machine learning model 200 is a model that estimates the nth estimated frame 24_n based on the nth intermediate frame 22_n. Specifically, machine learning model 200 is a model that estimates the nth estimated frame 24_n based on the nth intermediate frame 22_n and (n-1)th auxiliary information 28_n-1. Specifically, machine learning model 200 is a convolutional neural network (CNN). Known models such as multi-layer ResNet with residual connection mechanism and so-called encoder-decoder type U-Net can be used as machine learning model 200. The model described in Non-Patent Document 1 can also be used as machine learning model 200.

[0069] Machine learning model 200 is a model that has been learned using multiple training datasets. These datasets include intermediate training frames with intermediate pixel counts generated from the input frames (which have input pixel counts) and estimated training frames with estimated pixel counts. Various well-known methods, such as backpropagation, can be used to learn machine learning model 200.

[0070] Specifically, the machine learning model 200 includes a cumulative feature information output layer 202, an estimated frame output layer 204, and a convolutional layer 206 (see [link to model 200]). Figure 2 ).

[0071] The cumulative feature information output layer 202 receives an nth intermediate frame 22_n as input and (n-1)th auxiliary information 28_n-1 based on the (n-1)th cumulative feature information 26_n-1 indicating the features of the first to (n-1)th intermediate frames 22, and outputs the nth cumulative feature information 26_n indicating the features of the first to (n-1)th intermediate frames 22_n. The cumulative feature information output layer 202 can be configured by, for example, one or more convolutional layers. The cumulative feature information 26_n-1 is image information (bitmap format information) with the same number of pixels as the intermediate pixels. The cumulative feature information 26_n-1 can also be a feature map indicating the features of the first to (n-1)th intermediate frames 22.

[0072] The cumulative feature information output layer 202 receives a first intermediate frame 22_1 and given auxiliary information as input, and outputs first cumulative feature information 26_1. When n=1, since there is no cumulative feature information 26 and auxiliary information 28 before, the pre-prepared given auxiliary information is input to the cumulative feature information output layer 202 together with the first intermediate frame 22_1.

[0073] The estimated frame output layer 204 receives nth accumulated feature information 26_n as input and outputs nth estimated frame 24_n. The estimated frame output layer 204 can be configured with, for example, one or more convolutional layers, similar to the accumulated feature information output layer 202. Alternatively, the estimated frame output layer 204 can also be configured with one or more transposed convolutional layers (inverse convolutional layers).

[0074] Convolutional layer 206 is a layer that maintains the number of pixels in the accumulated feature information 26 while reducing its channel number. The accumulated feature information 26 output from convolutional layer 206 is processed by auxiliary information acquisition unit 322 to which it is applied. Since the dimension of the accumulated feature information 26 is reduced according to convolutional layer 206, computational cost can be suppressed. Convolutional layer 206 is, for example, a convolutional layer with a kernel size of 1×1, but is not limited to this.

[0075] [Machine Learning Model Storage Unit]

[0076] Machine learning model storage unit 312 stores machine learning model 200. Specifically, machine learning model storage unit 312 stores the parameters of machine learning model 200 (the number of convolutional layers, the number of nodes used in each convolutional layer, and the weight of each node, etc.).

[0077] [Estimated Frame Acquisition Unit]

[0078] The estimation frame acquisition unit 314 inputs each intermediate frame 22 into the machine learning model 200 and acquires each of the first to Nth estimation frames 24 having an estimated number of pixels equal to or greater than the number of intermediate pixels, where the number of intermediate pixels is greater than the number of input pixels. In this embodiment, the estimation frame 24 has the same estimated number of pixels as the number of intermediate pixels. More specifically, the estimation frame acquisition unit 314 inputs the nth intermediate frame 22_n and the (n-1)th auxiliary information 28_n-1 into the machine learning model 200 and acquires the nth estimation frame 24_n.

[0079] [Motion Information Acquisition Unit]

[0080] The motion information acquisition unit 316 acquires the (n-1)th motion information, which is information indicating the amount and direction of motion from the (n-1)th input frame 20_n-1 to the nth input frame 20_n. Specifically, the (n-1)th motion information is image information (bitmap format information) having the same number of pixels as the intermediate pixels and indicating the amount and direction of motion of each pixel between the (n-1)th input frame 20_n-1 and the nth input frame 20_n. Motion information is also referred to as motion vector. Specifically, the motion information acquisition unit 316 acquires the original motion information having the same number of pixels as the input pixels, and acquires motion information having the same number of pixels as the intermediate pixels by performing amplification and interpolation processing on the original motion information.

[0081] [Depth Information Acquisition Unit]

[0082] The depth information acquisition unit 318 acquires the (n-1)th depth information indicating the depth of each pixel in the (n-1)th input frame 20_n-1, and the nth depth information indicating the depth of each pixel in the nth input frame 20_n. Specifically, the depth information is image information (bitmap format information) with the same number of pixels as the intermediate pixels. The depth information is also referred to as a depth buffer or Z buffer. Specifically, the depth information acquisition unit 318 acquires the original depth information with the same number of pixels as the input pixels, and acquires depth information with the same number of pixels as the intermediate pixels by performing magnification and interpolation processing on the original depth information.

[0083] [Pixel recognition unit appears]

[0084] The pixel recognition unit 320, based on the (n-1)th depth information and the nth depth information, identifies the nth appearing pixel 222_n as appearing in the (n-1)th intermediate frame 22_n (see [link to relevant documentation]). Figure 3The pixel appearance is the pixel that is fully or partially displayed in the game object O that is not shown in the image. Specifically, the pixel appearance recognition unit 320 identifies the nth appearing pixel 222_n based on the difference between the (n-1)th depth information and the nth depth information. The pixel appearance recognition unit 320 can also specify the nth appearing pixel 222_n based on the (n-1)th perspective projection matrix associated with the (n-1)th intermediate frame 22_n and the nth perspective projection matrix associated with the nth intermediate frame 22_n. In addition, the pixel appearance recognition unit 320 can also specify the nth appearing pixel 222_n by utilizing the (n-1)th motion information. More specifically, the pixel appearance recognition unit 320 specifies the nth appearing pixel 222_n and generates nth appearing pixel information, which is image information indicating the position of the nth appearing pixel 222_n.

[0085] [Auxiliary Information Acquisition Unit]

[0086] The auxiliary information acquisition unit 322 acquires the (n-1)th auxiliary information 28_n-1 by applying motion compensation to the (n-1)th cumulative feature information 26_n-1 based on the (n-1)th motion information. Motion compensation is the process of moving the pixel at position x of the (n-1)th cumulative feature information 26_n to position x'. For example, if the pixel at position x of the (n-1)th intermediate frame 22_n-1 has already moved to position x' of the nth intermediate frame 22_n (see...). Figure 3 That is, the auxiliary information acquisition unit 322 acquires the (n-1)th auxiliary information 28_n-1 by setting the pixel value of one or more pixels of the (n-1)th accumulated feature information 26_n-1 to the pixel at the moving position according to the amount and direction of the pixel's motion information.

[0087] When the game object O moves between the nth input frame 20_n and the (n-1)th input frame 20_n-1, if the nth intermediate frame 22_n and the (n-1)th accumulated feature information 26_n-1 are input into the machine learning model 200 as is when the nth estimated frame 24_n is obtained, a ghosting phenomenon may occur in the nth estimated frame 24_n to be output. In this ghosting phenomenon, the residual image of the game object O displayed in the nth intermediate frame 22_n is finally displayed.

[0088] Therefore, as described above, the image processing system 1 is configured such that it obtains (n-1) auxiliary information 28_n-1 by applying motion compensation to the (n-1)th accumulated feature information 26_n-1 based on the (n-1)th motion information, and inputs the (n-1)th auxiliary information 28_n-1 into the machine learning model 200 when acquiring the nth estimated frame 24_n. This suppresses the aforementioned ghosting phenomenon.

[0089] Furthermore, the auxiliary information acquisition unit 322 acquires the (n-1)th auxiliary information 28_n-1 by converting the pixel value of the nth occurrence pixel 222_n in the (n-1)th accumulated feature information 26_n-1 into a predetermined value. Specifically, the auxiliary information acquisition unit 322 acquires the (n-1)th auxiliary information 28_n-1 based on the nth occurrence pixel information by converting the pixel value of the nth occurrence pixel 222_n in the (n-1)th accumulated feature information 26_n-1 into a predetermined value. The predetermined value may be a fixed value such as 0 (black), or it may be the pixel value of the nth occurrence pixel 222_n in the nth intermediate frame 22_n.

[0090] In the nth input frame 20_n, if the game object O that was not displayed in the (n-1)th input frame 20_n-1 is fully or partially displayed, when obtaining the nth estimated frame 24_n, if the nth intermediate frame 22_n and the (n-1)th cumulative feature information 26_n-1 are input into the machine learning model 200 as is, the above-mentioned ghosting phenomenon may occur in the nth estimated frame 24_n to be output.

[0091] Therefore, as described above, the image processing system 1 is configured to specify the nth occurrence pixel 222_n in the pixels of the nth intermediate frame 22_n, which is the game object O that is not displayed in the (n-1)th intermediate frame 22_n-1, and to obtain the (n-1)th auxiliary information 28_n-1 by converting the pixel value of the nth occurrence pixel 222_n in the (n-1)th accumulated feature information 26_n-1 into a predetermined value. This allows the aforementioned ghosting phenomenon to be suppressed.

[0092] [4. Processing performed by the image processing system]

[0093] Figure 7 This is a flowchart illustrating an example of the processing flow performed by the image processing system 1. The control unit 10 executes the process according to the program stored in the storage unit 12. Figure 7 The processing is shown.

[0094] (1) Processing when n=1

[0095] First, the control unit 10 acquires a first input frame 20_1 (S700). Based on the first input frame 20_1, the control unit 10 acquires a first intermediate frame 22_1 (S702). Then, the control unit 10 inputs the first intermediate frame 22_1 and the given auxiliary information into the machine learning model 200 to acquire a first estimated frame 24_1 and first cumulative feature information 26_1 (S704).

[0096] (2) Treatment of n≥2

[0097] Control unit 10 acquires the nth input frame 20_n (S706). Control unit 10 acquires the nth intermediate frame 22_n based on the nth input frame 20_n (S708).

[0098] Next, the control unit 10 acquires the (n-1)th motion information (S710). Furthermore, the control unit 10 acquires the (n-1)th depth information and the nth depth information (S712), and based on the (n-1)th depth information and the nth depth information, designates the nth appearing pixel 222_n (S714). The control unit 10 acquires the (n-1)th auxiliary information 28_n-1 based on the (n-1)th accumulated feature information 26_n-1, the (n-1)th motion information, and the nth appearing pixel 222_n (S716). Then, the control unit 10 inputs the nth intermediate frame 22_n and the (n-1)th auxiliary information 28_n-1 into the machine learning model 200, and acquires the nth estimated frame 24_n and the nth accumulated feature information 26_n (S718). Control unit 10 determines whether a next frame exists (S720), and if it does, it increments the frame count to n=n+1 and repeats the processing from S706 to S718. If control unit 10 has determined that no next frame exists (S720; N), the processing terminates. If control unit 10 determines that no next frame exists (S720; N), the first to Nth estimated frames 24 can be displayed on display unit 18 as is.

[0099] [5. Summary]

[0100] According to the image processing system 1 described above in relation to this embodiment, the (n-1)th cumulative feature information 26_n-1, which indicates the features of the first to (n-1)th intermediate frames 22, is used to estimate the nth estimated frame 24_n. That is, in addition to the information of the nth input frame 20_n, since the information of the first to (n-1)th input frames 20 can be used for estimation, the information available for estimation increases, and a high-resolution estimated frame 24_n can be obtained.

[0101] This invention is not limited to the foregoing embodiments. Furthermore, the specific strings or numbers mentioned above and those in the accompanying drawings are examples, and this invention is not limited to these strings or numbers.

[0102] For example, in this embodiment, the case where the number of intermediate pixels is greater than the number of input pixels and the number of intermediate pixels and the number of estimated pixels are the same is illustrated. However, the number of intermediate pixels and the number of input pixels can also be the same, and the number of estimated pixels can also be greater than the number of intermediate pixels. That is, intermediate frame 22 is not necessarily an enlarged input frame 20.

[0103] [6. Appendix] (1)

[0105] An image processing system includes at least one processor, wherein

[0106] The at least one processor:

[0107] Obtain each of the first to Nth input frames (N is a natural number equal to or greater than 2) having a specified number of input pixels;

[0108] Based on each of the input frames, each of the first to Nth intermediate frames is obtained by generating an intermediate frame corresponding to the input frame and having an intermediate number of pixels equal to or greater than the number of input pixels; and

[0109] Each of the intermediate frames is input into the machine learning model, and each of the first to Nth estimated frames is obtained with an estimated number of pixels equal to or greater than the number of intermediate pixels, wherein the number of intermediate pixels is greater than the number of input pixels; wherein

[0110] The machine learning model includes:

[0111] A cumulative feature information output layer is input to the nth intermediate frame (n=2, 3, ..., N) and (n-1)th auxiliary information based on the (n-1)th cumulative feature information indicating the features of the first to (n-1)th intermediate frames, wherein the cumulative feature information output layer outputs the nth cumulative feature information indicating the features of the first to nth intermediate frames; and

[0112] An estimation frame output layer is input with the nth accumulated feature information, and outputs the nth estimated frame; wherein...

[0113] The machine learning model has been trained using multiple training datasets, which include:

[0114] A learning intermediate frame with the specified number of intermediate pixels is generated based on a learning input frame having the specified number of input pixels, and...

[0115] A learned estimated frame with an estimated number of pixels. (2)

[0117] According to the image processing system of (1), where

[0118] Each of the input frames is an image that can be obtained by performing the rendering of three-dimensional data of one or more objects as seen from a specified viewpoint. (3)

[0120] According to the image processing system in (2), where

[0121] Each of the input frames is an image that can be obtained by performing rendering so that the viewpoint changes for each of the input frames, and

[0122] The at least one processor:

[0123] Acquire change information, which is information relating to changes in the viewpoint of each of the input frames in the rendering; and

[0124] Based on the change information and each pixel in each of the input frames, pixel values ​​corresponding to the positions of each pixel before the change are obtained through interpolation in the input frames, and each of the intermediate frames is generated. (4)

[0126] According to the image processing system of (2) or (3), wherein,

[0127] The at least one processor

[0128] Obtain the (n-1)th motion information, which is information indicating the amount and direction of motion from the (n-1)th input frame to the nth input frame, and

[0129] Based on the (n-1)th motion information, the (n-1)th auxiliary information is obtained by applying motion compensation to the (n-1)th cumulative feature information. (5)

[0131] According to the image processing system in (4), where

[0132] The at least one processor,

[0133] A depth information acquisition device acquires (n-1)th depth information indicating the depth of each pixel in the (n-1)th input frame and nth depth information indicating the depth of each pixel in the nth input frame.

[0134] Based on the (n-1)th depth information and the nth depth information, the nth appearing pixel in the nth intermediate frame is designated as the pixel for the complete or partial display of the object that was not displayed in the (n-1)th intermediate frame, and

[0135] The (n-1)th auxiliary information is obtained by converting the pixel value of the nth occurrence pixel in the (n-1)th cumulative feature information into a specified value. (6)

[0137] According to the image processing system from (1) to (5), where

[0138] The first intermediate frame and given auxiliary information are input to the cumulative feature information output layer, and the cumulative feature information output layer outputs the first cumulative feature information. (7)

[0140] According to the image processing system from (1) to (6), where

[0141] The accumulated feature information is image information with the same number of pixels as the intermediate pixel number.

Claims

1. An image processing system, comprising at least one processor, wherein, The at least one processor: Obtain each of the first to Nth input frames (N is a natural number equal to or greater than 2) having a specified number of input pixels; Based on each of the input frames, each of the first to Nth intermediate frames is obtained by generating an intermediate frame corresponding to the input frame and having an intermediate number of intermediate pixels equal to or greater than the number of input pixels; as well as Each of the intermediate frames is input into the machine learning model, and each of the first to Nth estimated frames is obtained having an estimated number of pixels equal to or greater than the intermediate pixel count, wherein the intermediate pixel count is greater than the input pixel count. The machine learning model includes: A cumulative feature information output layer, which is input to the nth intermediate frame (n=2, 3, ..., N) and (n-1)th auxiliary information based on the (n-1)th cumulative feature information indicating the features of the first to (n-1)th intermediate frames, outputs the nth cumulative feature information indicating the features of the first to nth intermediate frames; and An estimation frame output layer is input with the nth accumulated feature information and outputs the nth estimated frame. The machine learning model has been trained using multiple training datasets, which include: A learning intermediate frame with the specified number of intermediate pixels is generated based on a learning input frame having the specified number of input pixels, and... A learned estimated frame with an estimated number of pixels.

2. The image processing system according to claim 1, wherein, Each of the input frames is an image that can be obtained by performing the rendering of three-dimensional data of one or more objects as seen from a specified viewpoint.

3. The image processing system according to claim 2, wherein, Each of the input frames is an image that can be obtained by performing rendering so that the viewpoint changes for each of the input frames, and The at least one processor: Acquire change information, which is information relating to changes in the viewpoint of each of the input frames in the rendering; and Based on the change information and each pixel in each of the input frames, pixel values ​​corresponding to the positions of each pixel before the change are obtained by interpolation in the input frames, and each of the intermediate frames is generated.

4. The image processing system according to claim 2 or 3, wherein, The at least one processor Obtain the (n-1)th motion information, which is information indicating the amount and direction of motion from the (n-1)th input frame to the nth input frame, and The (n-1)th auxiliary information is obtained by applying motion compensation to the (n-1)th cumulative feature information based on the (n-1)th motion information.

5. The image processing system according to claim 4, wherein, The at least one processor, It also includes a depth information acquisition device, which acquires (n-1)th depth information indicating the depth of each pixel in the (n-1)th input frame and nth depth information indicating the depth of each pixel in the nth input frame. Based on the (n-1)th depth information and the nth depth information, the nth appearing pixel in the nth intermediate frame is designated as the pixel for the full or partial display of an object not displayed in the (n-1)th intermediate frame, and The (n-1)th auxiliary information is obtained by converting the pixel value of the nth occurrence pixel in the (n-1)th cumulative feature information into a specified value.

6. The image processing system according to claim 1 or 2, wherein The first intermediate frame and given auxiliary information are input to the cumulative feature information output layer, and the cumulative feature information output layer outputs the first cumulative feature information.

7. The image processing system according to claim 1 or 2, in, Cumulative feature information is image information with the same number of pixels as the intermediate pixel number.

8. An image processing method, wherein, processor: Obtain each of the first to Nth input frames (N is a natural number equal to or greater than 2) having a specified number of input pixels; Based on each of the input frames, each of the first to Nth intermediate frames is obtained by generating an intermediate frame corresponding to the input frame and having an intermediate number of intermediate pixels equal to or greater than the number of input pixels; as well as Each of the intermediate frames is input into the machine learning model, and each of the first to Nth estimated frames is obtained having an estimated pixel count equal to or greater than the number of intermediate pixels, wherein the number of intermediate pixels is greater than the number of input pixels. The machine learning model includes: The cumulative feature information output layer has an input of an nth intermediate frame (n=2, 3, ..., N) and (n-1)th auxiliary information based on the (n-1)th cumulative feature information indicating the features of the first to (n-1)th intermediate frames, and outputs the nth cumulative feature information indicating the features of the first to nth intermediate frames; as well as An estimation frame output layer, which receives nth accumulated feature information as input and outputs the nth estimated frame, wherein... The machine learning model has been trained using multiple training datasets, which respectively include... A learning intermediate frame with the specified number of intermediate pixels is generated based on a learning input frame having the specified number of input pixels, and... A learned estimated frame with an estimated number of pixels.

9. A program for running on a computer, The procedure includes: An input frame acquisition device acquires each of the first to Nth input frames (N is a natural number equal to or greater than 2) having a specified number of input pixels; An intermediate frame acquisition device, which acquires each of the first to Nth intermediate frames based on each of the input frames by generating an intermediate frame corresponding to the input frame and having an intermediate number of pixels equal to or greater than the number of input pixels; as well as An estimation frame acquisition device inputs each of the intermediate frames into a machine learning model and acquires each of the first to Nth estimation frames having an estimated number of pixels equal to or greater than the number of intermediate pixels, wherein the number of intermediate pixels is greater than the number of input pixels. The machine learning model includes: The cumulative feature information output layer has an input of an nth intermediate frame (n=2, 3, ..., N) and (n-1)th auxiliary information based on the (n-1)th cumulative feature information indicating the features of the first to (n-1)th intermediate frames, and outputs the nth cumulative feature information indicating the features of the first to nth intermediate frames; as well as An estimation frame output layer, which receives nth accumulated feature information as input and outputs the nth estimated frame, wherein... The machine learning model has been trained using multiple training datasets, which respectively include... A learning intermediate frame with the specified number of intermediate pixels is generated based on a learning input frame having the specified number of input pixels, and... A learned estimated frame with an estimated number of pixels.

Images