Image processing system, image processing method, and program

Abstract

The present invention improves estimation performance and reduces processing load. In an image processing system (1), at least one processor acquires each of first to n-th processing target frames (20), acquires each of first to n-th input frames (24), acquires each of first to n-th estimation frames (26) by inputting each input frame (24) to a machine learning model (200). The machine learning model (200) includes a cumulative feature information output layer (202) that receives input of the n-th input frame (24) and (n-1)-th auxiliary information (30) to output n-th cumulative feature information (28) representing the features of the first to n-th input frames (24), and the at least one processor adjusts the resolution of (n-1)-th cumulative feature information (28) so that the resolution is less than an estimated resolution, and acquires the (n-1)-th auxiliary information (30) on the basis of (n-1)-th cumulative feature information (29) with the adjusted resolution.

Description

Image processing system, image processing method and program

[0001] The present disclosure relates to an image processing system, an image processing method, and a program.

[0002] Conventionally, a technique (super-resolution) for estimating a high-quality still image based on a low-quality still image using a machine learning model is known (see Non-Patent Document 1 below).

[0003] Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang. Learning a Deep Convolutional Network for Image Super-Resolution, in Proceedings of European Conference on Computer Vision (ECCV), 2014

[0004] To achieve super-resolution of moving images such as game screens, the inventors of the present application are studying a system having a recursive configuration in which information indicating the characteristics of the current frame (the nth frame) and past frames (the 1st to n-1th frames) is input into a machine learning model to improve the image quality of the nth frame. By using information on past frames in addition to the current frame in this way, it is expected that the estimation performance of the machine learning model will improve.

[0005] However, in a system having the above-described recursive configuration, if information on past frames with improved image quality is used as is, the processing load will increase.

[0006] An object of the present disclosure is to provide an image processing system, an image processing method, and a program that improve estimation performance and reduce processing load.

[0007] An image processing system according to the present disclosure is an image processing system including at least one processor, wherein the at least one processor acquires first to nth (n is a natural number greater than or equal to 2) frames to be processed, each having a predetermined initial number of pixels, and, based on each frame to be processed, generates an input frame that corresponds to the frame to be processed and has a number of input pixels greater than the initial number of pixels, thereby acquiring first to nth input frames, and inputs each input frame to a machine learning model to acquire first to nth estimated frames, each having an estimated number of pixels, the machine learning model including an accumulated feature information output layer that receives as input the nth input frame and n-1th auxiliary information based on n-1th cumulative feature information that indicates features of the first to nth input frames, and outputs the nth cumulative feature information that indicates the features of the first to nth input frames, and the at least one processor adjusts the number of pixels in the n-1th cumulative feature information so that it is less than the estimated number of pixels, and acquires the n-1th auxiliary information based on the n-1st cumulative feature information with the adjusted number of pixels.

[0008] 1 is a diagram illustrating an example of a hardware configuration of an image processing system. FIG. 2 is a diagram illustrating an overview of the image processing system. FIG. 3 is a diagram illustrating a schematic view of processing of the image processing system. FIG. 4 is a functional block diagram illustrating an example of functions implemented in the image processing system. FIG. 5 is a diagram illustrating processing in a rendering unit. FIG. 6 is a diagram illustrating processing in an intermediate frame acquisition unit. FIG. 7 is a diagram illustrating a schematic view of an intermediate frame. FIG. 8 is a diagram illustrating another example of an intermediate frame. FIG. 9 is a flow diagram illustrating an example of a flow of processing executed in the image processing system. FIG. 10 is a flow diagram illustrating an example of a flow of processing executed in the image processing system. FIG. 11 is a diagram illustrating an overview of an image processing system 1 according to a modified example of this embodiment. FIG. 12 is a functional block diagram illustrating an example of functions implemented in an image processing system according to a modified example of this embodiment.

[0009] Hereinafter, an example of an embodiment of an image processing system according to the present disclosure will be described with reference to the drawings.

[0010] 1 is a diagram showing an example of the hardware configuration of an image processing system 1. The image processing system 1 is a computer such as a game console. As shown in FIG. 1, the image processing system 1 includes a control unit 10, a storage unit 12, a communication unit 14, an operation unit 16, a display unit 18, and an audio output unit 19.

[0011] The control unit 10 includes at least one processor. The control unit 10 may be, for example, a program-controlled device such as a CPU that operates according to a program installed in the image processing system 1. The control unit 10 also includes a GPU (Graphics Processing Unit) that draws images in a frame buffer based on graphics commands and data supplied from the CPU.

[0012] The storage unit 12 includes, for example, a main storage device such as a ROM or a RAM, and an auxiliary storage device such as a HDD or an SSD. The storage unit 12 stores programs executed by the control unit 10. The storage unit 12 stores, for example, a game program (game software) in addition to programs for realizing various functions of the image processing system 1, which will be described later. The storage unit 12 also reserves an area for a frame buffer in which images are drawn by the GPU.

[0013] The communication unit 14 is a communication interface such as an Ethernet (registered trademark) module or a wireless LAN module.

[0014] The operation unit 16 is a user interface such as a keyboard, mouse, or game console controller, and receives operation inputs from the user and outputs signals indicating the contents of the inputs to the control unit 10 .

[0015] The display unit 18 is a display device such as a liquid crystal display or an organic EL display, and displays various images according to instructions from the control unit 10 .

[0016] The audio output unit 19 is, for example, a speaker, and outputs audio represented by audio data generated by the image processing system 1 .

[0017] In addition to the devices described above, the image processing system 1 may also include an optical disc drive that reads optical discs such as DVD-ROMs and Blu-ray (registered trademark) discs, a USB (Universal Serial Bus) port, and the like.

[0018] 2. Overview of the Image Processing System Fig. 2 is a diagram showing an overview of the image processing system 1. Fig. 3 is a diagram showing the processing of the image processing system 1. In this embodiment, an example is shown in which the image processing system 1 is used to improve the image quality of gameplay video in a game. The gameplay video is a video generated in response to a game program executed by the control unit 10 or user input accepted by the operation unit 16, and is composed of a plurality of still images (frames) that are time-series data. The processing performed by the image processing system 1 is mainly as follows:

[0019] (1) Generation of Processing Frame The image processing system 1 generates an image (processing frame 20) in which one or more game objects are drawn by rendering three-dimensional data representing the game objects as seen from a predetermined viewpoint. This processing frame 20 is an image having a predetermined number of pixels (initial pixel number) and a predetermined image quality (initial image quality). The processing frame 20 is generated at predetermined time intervals. The initial pixel number is, for example, 3840 x 2160 (4K).

[0020] Each generated frame to be processed is not displayed directly on the display unit 18, but is temporarily stored in the storage unit 12 and then subjected to subsequent processing. Note that in the following explanation, processing for the nth (2≦n≦N, n and N are natural numbers equal to or greater than 2) frame to be processed 20_n will be mainly exemplified, but similar processing is also performed on other frames to be processed (i.e., n=2, 3, ..., N).

[0021] (2) Acquisition of Intermediate Frame Based on the acquired processing target frame 20_n, the image processing system 1 acquires an intermediate frame 22_n having a number of intermediate pixels greater than the initial number of pixels. The intermediate frame 22_n is generated by performing enlargement and interpolation processing on the processing target frame 20_n. The number of intermediate pixels (indicated as "8K-" in FIG. 2 and other figures) is preferably greater than 3840 x 2160 (4K) and less than 7680 x 4320 (8K). For example, the number of intermediate pixels is the number of pixels obtained by thinning out a predetermined pixel row from 8K, and may be 7680 x 2160, 3840 x 4320, or the like.

[0022] (3) Acquisition of Input Frame Based on the acquired intermediate frame 22_n, the image processing system 1 acquires an input frame 24_n having a number of input pixels greater than the number of intermediate pixels. The input frame 24_n is generated by performing enlargement and interpolation processing on the processing target frame 20_n. The number of input pixels is, for example, 7680 x 4320 (8K).

[0023] It should be noted that although the input frame 24_n has a larger number of pixels than the target frame 20_n, its image quality is not necessarily sufficiently improved. In other words, the image quality of a frame does not simply refer to the number of pixels (high resolution). The image quality of a frame may be evaluated based on, for example, a high signal-to-noise ratio, high spatial frequency reproducibility, high temporal stability (low artifacts and flicker when multiple frames are displayed consecutively), etc., when compared to a reference frame, or a combination of these.

[0024] (4) Obtaining an Estimated Frame The image processing system 1 inputs the input frame 24_n to the machine learning model 200 and obtains an estimated frame 26_n. The estimated frame 26_n is an image having the same number of estimated pixels as the number of input pixels and image quality (estimated image quality) that is equal to or higher than the initial image quality. Therefore, if the number of input pixels is 7680 x 4320 (8K), the estimated number of pixels will also be 7680 x 4320 (8K).

[0025] Here, in addition to the input frame 24_n, the (n-1)th auxiliary information 30_n-1 is input to the machine learning model 200 (see FIGS. 2 and 3). The auxiliary information 30_n-1 is information based on the (n-1)th cumulative feature information 28_n-1 that indicates the features of the first to (n-1)th input frames 24. Details of the cumulative feature information 28 and the auxiliary information 30 will be described later.

[0026] (5) Acquisition of Cumulative Feature Information The machine learning model 200 has an cumulative feature information output layer 202 that receives input frame 24_n and auxiliary information 30_n-1 and outputs n-th cumulative feature information 28_n that indicates the features of the first to n-th input frames 24 (see FIG. 2). The image processing system 1 acquires the n-th cumulative feature information 28_n.

[0027] The acquired n-th cumulative feature information 28_n is input to the estimated frame output layer 204, which outputs the n-th estimated frame 26_n (see FIG. 2 ). The acquired n-th cumulative feature information 28_n is also stored in the storage unit 12 and is used to estimate an estimated frame 26_n+1 corresponding to the next processing target frame (the (n+1)th processing target frame) 20_n+1.

[0028] (6) Acquisition of Auxiliary Information As described above, the (n-1)th accumulated feature information 28_n-1 is information that indicates the features of the first to (n-1)th input frames 24 (and thus the first to (n-1)th processing target frames 20). If the accumulated feature information 28_n-1, which accumulates information on past processing target frames 20, is used to estimate the nth estimated frame 26_n, the amount of information available for estimation increases, making it possible to obtain a high-quality estimated frame 26_n.

[0029] However, if there is movement or the like in the displayed game object between the (n-1)th processing frame 20_n-1 and the nth processing frame 20_n, when the nth input frame 24_n and the cumulative feature information 28_n-1 are input as is to the machine learning model 200, a phenomenon (the so-called ghost phenomenon) may occur in which an afterimage of the game object that was displayed in the (n-1)th processing frame 20_n-1 is displayed.

[0030] Therefore, the image processing system 1 acquires the (n-1)th auxiliary information 30_n-1 by applying various corrections (described later) to the cumulative feature information 28_n-1 based on information obtained during rendering (motion vectors, depth buffers, etc.) (see FIGS. 2 and 3). As described above, the acquired (n-1)th auxiliary information 30_n-1 is input to the machine learning model 200 together with the nth input frame 24_n, and is used to estimate the nth estimated frame 26_n.

[0031] According to the image processing system 1 of this embodiment, an estimated frame 26 is estimated using auxiliary information 30, which stores past information, in addition to the input frame 24 corresponding to the current processing target frame 20. This increases the amount of information available for estimation, making it possible to obtain a high-quality estimated frame 26_n.

[0032] (7) Increasing or Decreasing the Number of Pixels Here, the cumulative feature information 28 output from the machine learning model 200 is information having the same number of pixels as the estimated frame 26. That is, the cumulative feature information 28 is information having 8K pixels. In this embodiment, the number of pixels in the cumulative feature information 28 is adjusted to be the same as the number of intermediate pixels in the intermediate frame 22. Then, based on the cumulative feature information 28 having the same number of pixels as the number of intermediate pixels, auxiliary information 30 having the same number of pixels as the number of intermediate pixels is obtained. Specifically, the number of pixels in the auxiliary information 30 is the number of pixels obtained by thinning out a predetermined pixel row from 8K, and may be 7680 x 2160, 3840 x 4320, or the like.

[0033] Furthermore, in this embodiment, the number of pixels in the auxiliary information 30 is increased to be the same as the number of input pixels in the input frame 24. Specifically, the number of pixels in the auxiliary information 30 is increased to 7680 x 4320 (8K). Then, the input frame 24 and the auxiliary information 30 having the same number of pixels as the number of input pixels are input to the machine learning model 200.

[0034] In this way, by adopting a configuration in which the auxiliary information 30 is acquired using cumulative feature information 28 having a smaller number of pixels than the estimated number of pixels of the estimated frame 26, it is possible to reduce memory usage and processing time when acquiring the auxiliary information 30. Such a reduction in processing load is particularly effective when the image processing system 1 is applied to a game console with limited processing performance. Furthermore, while reducing the number of pixels (amount of information) of the auxiliary information 30 too much may result in insufficient estimation accuracy in the machine learning model 200, in this embodiment, by increasing the number of pixels contained in the auxiliary information 30 compared to the initial number of pixels of the frame 20 to be processed, information from past frames can be effectively used and estimation accuracy can be maintained.

[0035] 4 is a functional block diagram showing an example of functions realized in the image processing system 1. As shown in Fig. 4, the image processing system 1 realizes a game processing unit 400, a rendering unit 402, a rendering information storage unit 404, a processing target frame acquisition unit 406, a variation information acquisition unit 408, an intermediate frame acquisition unit 410, an input frame acquisition unit 414, a machine learning model storage unit 417, an estimated frame acquisition unit 418, a cumulative feature information acquisition unit 422, an auxiliary information acquisition unit 424, a motion information acquisition unit 426, a depth information acquisition unit 428, an appearance pixel identification unit 430, and a pixel number increase unit 432.

[0036] The game processing unit 400, rendering unit 402, processing target frame acquisition unit 406, variation information acquisition unit 408, intermediate frame acquisition unit 410, input frame acquisition unit 414, estimated frame acquisition unit 418, cumulative feature information acquisition unit 422, auxiliary information acquisition unit 424, motion information acquisition unit 426, depth information acquisition unit 428, appearance pixel identification unit 430, and pixel number increase unit 432 are mainly realized by the control unit 10. The rendering information storage unit 404 and machine learning model storage unit 417 are mainly realized by the storage unit 12. Note that the game processing unit 400, rendering unit 402, and rendering information storage unit 404 are functions provided by the game software.

[0037] [Game Processing Unit] The game processing unit 400 executes various processes related to the game. For example, in response to a game program executed by the control unit 10 or user input received by the operation unit 16, the game processing unit 400 executes processes such as placing a game object O in a virtual three-dimensional space VS, operating or moving the game object O, and changing the viewpoint C from which the virtual three-dimensional space VS is viewed (see FIG. 5 ). The game object O is composed of primitives such as polygons represented by three-dimensional data. The three-dimensional data includes geometric information indicating the positions of vertices, topological information indicating how the vertices are connected, and attribute information such as color.

[0038] [Rendering Unit] FIG. 5 is a diagram illustrating the processing of the rendering unit 402. The rendering unit 402 generates first to Nth (N is a natural number equal to or greater than 2) processing frames 20 by rendering (drawing) three-dimensional data representing one or more game objects O viewed from a predetermined viewpoint C. The rendering unit 402 performs rendering based on the results of various processes executed by the game processing unit 400. Specifically, the rendering unit 402 performs vertex processing (vertex shading) and pixel processing (pixel shading) based on three-dimensional data representing game objects O placed in the virtual three-dimensional space VS. The vertex processing includes a coordinate transformation process (perspective projection) from the view coordinate system to the screen coordinate system, and a numerical value related to a change in the viewpoint C is added to the perspective projection matrix (camera matrix) used in the coordinate transformation process, as described below. The rendering unit 402 may perform rendering based on light source information, depth information (depth buffer), texture information, normal information, etc.

[0039] Here, the rendering unit 402 generates each processing frame 20 by performing rendering such that the viewpoint C varies for each processing frame 20. Here, even if the game processing unit 400 fixes the viewpoint C at a predetermined position, the rendering unit 402 varies the viewpoint C for each processing frame 20. As a result, as shown in FIG. 5 , the position of the displayed game object O varies in each processing frame 20_n, 20_n+1, and 20_n+2. In other words, the rendering unit 402 applies jitter when generating each processing frame 20. Specifically, the rendering unit 402 varies the viewpoint C for each processing frame 20 by adding a numerical value corresponding to a sub-pixel size that differs for each processing frame 20 to the perspective projection matrix. The rendering unit 402 varies the viewpoint C for each processing frame 20 according to a predetermined rule. For example, a Halton sequence can be used as such a rule.

[0040] [Rendering Information Storage Unit] The rendering information storage unit 404 stores information necessary for the rendering process in the rendering unit 402 and information obtained as a result of the rendering process. For example, the rendering information storage unit 404 stores the processing target frame 20. The rendering information storage unit 404 also stores variation information, motion information, and depth information. Details of the variation information, motion information, and depth information will be described later. In addition, the rendering information storage unit 404 may store parameters used for coordinate transformation, light source information, texture information, normal information, etc.

[0041] [Processing Frame Acquisition Unit] The processing frame acquisition unit 406 acquires the first to Nth processing frames 20. Specifically, the processing frame acquisition unit 406 acquires the first to Nth processing frames 20 stored in the rendering information storage unit 404.

[0042] [Variation Information Acquisition Unit] The variation information acquisition unit 408 acquires variation information. The variation information acquisition unit 408 acquires variation information stored in the rendering information storage unit 404. Specifically, the variation information is information indicating the amount of variation of the viewpoint C between before and after the variation. The information indicating the amount of variation can also be said to be a variation vector indicating the direction and distance of the variation. For example, since the above-mentioned Halton sequence includes information indicating the amount of variation of the viewpoint C, this information may be used as the variation information.

[0043] [Intermediate Frame Acquisition Unit] The intermediate frame acquisition unit 410 includes a pixel number increase unit 410a that enlarges the processing target frame 20. The intermediate frame acquisition unit 410 increases the number of pixels in the first to Nth processing target frames 20 using the pixel number increase unit 410a, and acquires the first to Nth intermediate frames 22, respectively. Specifically, the intermediate frame acquisition unit 410 interpolates pixel values ​​at positions in the processing target frame 20 that correspond to each pixel before the fluctuation, based on the fluctuation information and each pixel of each processing target frame 20, and generates each intermediate frame 22.

[0044] FIG. 6 is a diagram illustrating the processing performed by the intermediate frame acquisition unit 410. FIG. 6 illustrates an example of obtaining the nth intermediate frame 22_n. For example, as shown in FIG. 6, if the pixel center of a pixel in the intermediate frame 22_n to be acquired is P1,0, the intermediate frame acquisition unit 410 uses bilinear interpolation to obtain the pixel value of P1,0 based on the coordinates and pixel values ​​of the pixel centers P'0,0, P'1,0, P'0,1, and P'1,1 of the four pixels closest to P1,0 in the processing target frame 20_n. Here, P'1,0 is shifted from P1,0 by the amount of variation indicated by the variation information. The pixel values ​​of pixels newly generated by the enlargement process are similarly obtained. In addition to bilinear interpolation, various other well-known interpolation methods, such as bicubic interpolation and Lanczos interpolation, can also be used.

[0045] When rendering is performed so that the viewpoint C varies for each processing frame 20, the amount of time-series information increases. However, by using each processing frame 20 obtained in this way (hereinafter referred to as a "variable processing frame") for estimation, an estimated frame 26 with higher image quality can be obtained.

[0046] On the other hand, if the frame to be subjected to variation processing (or an enlarged image thereof) is input directly into the machine learning model 200, the influence of the variation in viewpoint C described above may result in a decrease in the accuracy of estimation.

[0047] Therefore, as described above, the image processing system 1 interpolates pixel values ​​at positions in the processing frame 20 corresponding to the pixels before the fluctuation based on the fluctuation information and each pixel of each processing frame 20, thereby generating each intermediate frame 22. Then, the number of pixels in each intermediate frame 22 is further increased to generate an input frame 24, which is input to the machine learning model 200. This corrects for the influence of fluctuations in the viewpoint C, thereby preventing a decrease in the accuracy of estimation.

[0048] [Input Frame Acquisition Unit] The input frame acquisition unit 414 includes a pixel number increase unit 414a that enlarges the intermediate frame 22. The input frame acquisition unit 414 increases the number of pixels in the first to Nth intermediate frames 22 using the pixel number increase unit 414a, and acquires the first to Nth input frames 24, respectively. The pixel number may be increased by a method such as bilinear interpolation.

[0049] [Machine Learning Model] The machine learning model 200 is a model that estimates the nth estimated frame 26_n based on the nth input frame 24_n. Specifically, the machine learning model 200 is a model that estimates the nth estimated frame 26_n based on the nth input frame 24_n and the n-1th auxiliary information 30_n-1. Specifically, the machine learning model 200 is a convolutional neural network (CNN). As the machine learning model 200, for example, a well-known model such as a multi-layered ResNet having a residual connection mechanism or a so-called encoder-decoder U-Net can be used. The model described in Non-Patent Document 1 may also be used as the machine learning model 200.

[0050] The machine learning model 200 is a model trained using a plurality of training data sets, each of which includes a training input frame having an input pixel count and a training estimation frame having an estimated pixel count. Various known techniques, such as backpropagation, can be used to train the machine learning model 200.

[0051] Specifically, the machine learning model 200 includes an accumulated feature information output layer 202, an estimated frame output layer 204, and a convolution layer 206 (see FIG. 2).

[0052] The cumulative feature information output layer 202 receives the n-th input frame 24_n and the (n-1)th auxiliary information 30_n-1 based on the (n-1)th cumulative feature information 28_n-1 indicating the features of the first to n-1th input frames 24, and outputs the n-th cumulative feature information 28_n indicating the features of the first to n-th input frames 24_n. The cumulative feature information output layer 202 may be composed of, for example, one or more convolutional layers.

[0053] The cumulative feature information 28_n-1 is image information (bitmap information) having the same number of pixels as the number of input pixels. The cumulative feature information 28_n-1 can also be considered a feature map that indicates the features of the first to (n-1)th input frames 24.

[0054] The cumulative feature information output layer 202 receives the first input frame 24_1 and given auxiliary information, and outputs first cumulative feature information 28_1. When n=1, there is no previous cumulative feature information 28 or auxiliary information 30, so the given auxiliary information prepared in advance is input to the cumulative feature information output layer 202 together with the first input frame 24_1.

[0055] The estimated frame output layer 204 receives the n-th accumulated feature information 28_n and outputs the n-th estimated frame 26_n. The estimated frame output layer 204 may be composed of, for example, one or more convolutional layers, similar to the accumulated feature information output layer 202. Alternatively, the estimated frame output layer 204 may be composed of one or more transposed convolutional layers (deconvolutional layers).

[0056] The convolutional layer 206 is a layer that reduces the number of channels of the accumulated feature information 28 while maintaining the number of pixels of the accumulated feature information 28. The accumulated feature information 28 output from the convolutional layer 206 is subjected to processing by the auxiliary information acquisition unit 424. The convolutional layer 206 reduces the dimensions of the accumulated feature information 28, thereby reducing calculation costs. The convolutional layer 206 is, for example, a convolutional layer with a kernel size of 1×1, but is not limited to this.

[0057] [Machine Learning Model Storage Unit] The machine learning model storage unit 417 stores the machine learning model 200. Specifically, the machine learning model storage unit 417 stores parameters of the machine learning model 200 (such as the number of convolutional layers, the number of nodes used in each convolutional layer, and the weight of each node).

[0058] [Estimated Frame Acquisition Unit] The estimated frame acquisition unit 418 inputs each input frame 24 to the machine learning model 200 to acquire first to Nth estimated frames 26, each having an estimated number of pixels. In this embodiment, the estimated frame 26 has the same number of estimated pixels as the number of input pixels. More specifically, the estimated frame acquisition unit 418 inputs the nth input frame 24_n and the (n-1)th auxiliary information 30_n-1 to the machine learning model 200 to acquire the nth estimated frame 26_n.

[0059] [Motion Information Acquisition Unit] The motion information acquisition unit 426 acquires n-1th motion information, which is information indicating the amount and direction of motion from the n-1th processing frame 20_n-1 to the nth processing frame 20_n. Specifically, the n-1th motion information has the same number of pixels as the number of intermediate pixels, and is image information (bitmap format information) indicating the amount and direction of motion of each pixel between the n-1th processing frame 20_n-1 and the nth processing frame 20_n. The motion information is also called a motion vector. Specifically, the motion information acquisition unit 426 acquires original motion information having the same number of pixels as the number of initial pixels, and performs enlargement and interpolation processing on the original motion information to acquire motion information having the same number of pixels as the number of intermediate pixels.

[0060] [Depth Information Acquisition Unit] The depth information acquisition unit 428 acquires (n-1)th depth information indicating the depth of each pixel in the (n-1)th processing target frame 20_n-1 and (n)th depth information indicating the depth of each pixel in the nth processing target frame 20_n. Specifically, the depth information is image information having the same number of pixels as the intermediate pixel count (bitmap format information). The depth information is also called a depth buffer, Z buffer, or depth buffer. Specifically, the depth information acquisition unit 428 acquires original depth information having the same number of pixels as the initial pixel count, and performs enlargement and interpolation processing on the original depth information to acquire depth information having the same number of pixels as the intermediate pixel count.

[0061] [Appearance Pixel Identification Unit] Based on the (n-1)th depth information and the (n)th depth information, the appearance pixel identification unit 430 identifies the (n)th appearance pixel 222_n, which is a pixel of the nth input frame 24_n where all or part of the game object O, which is not displayed in the (n-1)th intermediate frame 22_n-1, is displayed (see FIG. 3 ). Specifically, the appearance pixel identification unit 430 identifies the (n)th appearance pixel 222_n based on the difference between the (n-1)th depth information and the (n)th depth information. Note that the appearance pixel identification unit 430 may also identify the (n)th appearance pixel 222_n based on the (n-1)th perspective projection matrix associated with the (n-1)th intermediate frame 22_n-1 and the (n)th perspective projection matrix associated with the (n)th intermediate frame 22_n. Alternatively, the appearance pixel identification unit 430 may identify the (n)th appearance pixel 222_n using the (n-1)th motion information. More specifically, the appearance pixel specifying section 430 specifies the nth appearance pixel and generates nth appearance pixel information, which is image information indicating the position of the nth appearance pixel 222_n.

[0062] [Cumulative Feature Information Acquisition Unit] The cumulative feature information acquisition unit 422 includes a pixel number adjustment unit 422a. The pixel number adjustment unit 422a adjusts the number of pixels contained in the cumulative feature information 28 to the same number of pixels as the intermediate pixel number. The cumulative feature information acquisition unit 422 acquires the cumulative feature information 28 with the adjusted pixel number.

[0063] [Auxiliary Information Acquisition Unit] The auxiliary information acquisition unit 424 acquires the (n-1)th auxiliary information 30_n-1 by applying motion compensation to the (n-1)th cumulative feature information 28_n-1 based on the (n-1)th motion information. Motion compensation refers to, for example, a process of moving a pixel at position x in the (n-1)th cumulative feature information 28_n to position x' when a pixel at position x in the (n-1)th intermediate frame 22_n-1 has moved to position x' in the nth intermediate frame 22_n (see FIG. 5 ). That is, the auxiliary information acquisition unit 424 acquires the (n-1)th auxiliary information 30_n-1 based on the (n-1)th motion information by setting the pixel values ​​of one or more pixels in the (n-1)th cumulative feature information 28_n-1 to pixels at positions moved in accordance with the amount and direction of the pixel's movement.

[0064] If there is movement of the game object O between the nth processing frame 20_n and the n-1th processing frame 20_n-1, when obtaining the nth estimated frame 26_n, if the nth input frame 24_n and the n-1th accumulated feature information 28_n-1 are input directly into the machine learning model 200, a ghost phenomenon may occur in which an afterimage of the game object O that was displayed in the nth input frame 24_n is displayed in the output nth estimated frame 26_n.

[0065] Therefore, in the image processing system 1, as described above, motion compensation is applied to the (n-1)th accumulated feature information 28_n-1 based on the (n-1)th motion information to obtain the (n-1)th auxiliary information 30_n-1, and when obtaining the nth estimated frame 26_n, this (n-1)th auxiliary information 30_n-1 is input to the machine learning model 200. This makes it possible to suppress the ghost phenomenon described above.

[0066] 7A and 7B, the intermediate frame of this embodiment will be described. As described above, the intermediate frame 22 may be a frame formed by thinning out vertical or horizontal pixel rows from a plurality of pixels arranged in a grid that constitutes 8K, and interpolating information from pixel rows adjacent to the thinned pixel rows.

[0067] 7A is a schematic diagram showing an example in which a horizontal row of pixels is thinned out and information from pixels adjacent to the thinned out pixels is interpolated. In FIG. 7A, the thinned out pixels are indicated by dotted lines, and the direction of the interpolated information is indicated by arrows. The same applies to FIG. 7B.

[0068] Alternatively, the intermediate frame 22 may be a frame obtained by thinning out one pixel from a plurality of pixels arranged in a grid pattern that constitutes 8K, and interpolating information about each pixel adjacent to the thinned pixel. Fig. 7B schematically shows an example in which information about pixels adjacent to each thinned pixel is interpolated. In the example shown in Fig. 7B, the number of pixels adjacent to the thinned pixel is greater than in the example shown in Fig. 7A, and therefore information can be interpolated with higher accuracy.

[0069] The number of intermediate pixels is not limited to 7680×2160 or 3840×4320, and may be at least greater than the number of initial pixels and less than the number of estimated pixels.

[0070] 8A and 8B are flow diagrams showing an example of the flow of processing executed in the image processing system 1. The processing shown in Fig. 8A and Fig. 7B is executed by the control unit 10 operating in accordance with a program stored in the storage unit 12. Note that Fig. 8A shows processing when n=1, and Fig. 8B shows processing when n=2 and thereafter.

[0071] (1) Processing for n=1 First, the control unit 10 acquires a first processing target frame 20_1 (S100). The control unit 10 acquires a first intermediate frame 22_1 based on the first processing target frame 20_1 (S102). At this time, the number of pixels in the frame is increased.

[0072] The control unit 10 also acquires a first input frame 24_1 based on the first intermediate frame 22_1 (S104), increasing the number of pixels in the frame.

[0073] Furthermore, the control unit 10 inputs the first input frame 24_1 and given auxiliary information to the machine learning model 200, and obtains the first estimated frame 26_1 and first cumulative feature information 28_1 (S106).

[0074] (2) Processing for n≧2 The control unit 10 acquires the nth processing target frame 20_n (S108). The control unit 10 acquires the nth intermediate frame 22_n based on the nth processing target frame 20_n (S110). At this time, the number of pixels in the frame is increased.

[0075] Next, the control unit 10 acquires the n-th motion information (S112). The control unit 10 also acquires the (n-1)th depth information and the n-th depth information (S114), and identifies the n-th appearance pixel 222_n based on the (n-1)th depth information and the n-th depth information (S116).

[0076] The control unit 10 also acquires the (n-1)th cumulative feature information acquired previously (S118) and adjusts the number of pixels thereof (S120).The control unit 10 then acquires the (n-1)th auxiliary information 30_n-1 based on the (n-1)th cumulative feature information 28_n-1, the nth motion information, and the nth appearance pixel 222_n (S122).

[0077] The control unit 10 also acquires the n-th input frame 24_n based on the n-th intermediate frame 22_n (S124), increasing the number of pixels in the frame.

[0078] Then, the control unit 10 inputs the nth input frame 24_n and the n-1th auxiliary information 30_n-1 into the machine learning model 200, and obtains the nth estimated frame 26_n and the nth cumulative feature information 28_n (S126).

[0079] The control unit 10 then determines whether or not the next frame exists (S128), and if it determines that the next frame exists (S128; Y), it increments n to n+1 and repeats the processes of S108 to S126. If it determines that the next frame does not exist (S128; N), the control unit 10 ends this process.

[0080] [6. Modification] Fig. 9 is a diagram showing an overview of an image processing system 1 according to a modification of this embodiment. Fig. 10 is a functional block diagram showing an example of functions realized by an image processing system according to a modification of this embodiment. Note that the same reference numerals are used for configurations similar to those of this embodiment described with reference to Figs. 2, 4, etc., and descriptions thereof will be omitted.

[0081] In the above embodiment, an example was described in which the number of pixels in the processing target frame 20 is increased in two stages before inputting it to the machine learning model 400, but in a modified example, an example is described in which the number of pixels in the processing target frame 20 is increased in one stage before inputting it to the machine learning model 200.

[0082] Specifically, the input frame acquisition unit 414 acquires an input frame 24 having a number of input pixels greater than the initial number of pixels of the processing target frame 20. The initial number of pixels may be, for example, 4K. In this case, the number of input pixels may be greater than 4K and less than 8K.

[0083] An input frame 24 is then input to the machine learning model 400, and an estimated frame 26 having a greater number of estimated pixels than the number of input pixels is output. The number of pixels contained in the auxiliary information 30 input to the machine learning model 200 may be the same as the number of input pixels. Therefore, the image processing system 1 of the modified example may not include the pixel number increasing unit 432 shown in FIG. 4 . The subsequent processing is similar to that of the present embodiment described with reference to FIG. 4 , etc. Note that the machine learning model 200 of the modified example may be trained in advance so that the number of pixels in the output frame is greater than the number of pixels in the input frame. Specifically, the machine learning model 200 may be a model trained using a plurality of training data sets, each of which includes a training input frame having an input pixel count greater than 4K and less than 8K (8K-) and a training estimation frame having an estimated pixel count of 8K.

[0084] In the modified example, as in the above-described embodiment, it is possible to maintain the estimation accuracy and reduce the processing load.

Claims

1. An image processing system including at least one processor, wherein the at least one processor: acquires first to nth (n is a natural number equal to or greater than 2) processing frames each having a predetermined initial number of pixels, generates an input frame corresponding to the processing frame and having a number of input pixels greater than the initial number of pixels based on each processing frame, thereby acquiring the first to nth input frames, respectively, and inputs each of the input frames to a machine learning model to acquire first to nth estimated frames each having an estimated number of pixels, the machine learning model including an accumulated feature information output layer that receives the nth input frame and n-1th auxiliary information based on n-1th cumulative feature information indicating features of the first to nth input frames, and outputs the nth cumulative feature information indicating features of the first to nth input frames, the at least one processor: adjusts the number of pixels contained in the n-1st cumulative feature information so that it is less than the estimated number of pixels, and acquires the n-1st auxiliary information based on the n-1st cumulative feature information with the adjusted number of pixels.

2. The image processing system of claim 1, wherein the at least one processor obtains the first to nth intermediate frames by generating, based on each of the frames to be processed, an intermediate frame that corresponds to the frame to be processed and has a number of intermediate pixels greater than the initial number of pixels, and obtains the first to nth input frames by generating, based on each of the intermediate frames, an input frame that corresponds to the intermediate frame and has a number of input pixels greater than the number of intermediate pixels.

3. The image processing system according to claim 2, wherein the number of input pixels and the number of estimated pixels are the same.

4. The image processing system of claim 1, wherein the machine learning model is trained to output the estimated frame having a greater number of estimated pixels than a number of input pixels that the input frame has.

5. An image processing method, comprising: a processor acquiring first to nth (n is a natural number greater than or equal to 2) processing frames each having a predetermined initial number of pixels; generating, based on each processing frame, an input frame corresponding to the processing frame and having a number of input pixels greater than the initial number of pixels, thereby acquiring first to nth input frames each; inputting each input frame to a machine learning model to acquire first to nth estimated frames each having an estimated number of pixels, the machine learning model including an accumulated feature information output layer that receives the nth input frame and n-1th auxiliary information based on n-1th cumulative feature information indicating features of the first to n-1th input frames, and outputs the nth cumulative feature information indicating the features of the first to nth input frames; the processor adjusting the number of pixels contained in the n-1st cumulative feature information so that it is less than the estimated number of pixels; and acquiring the n-1st auxiliary information based on the n-1st cumulative feature information with the adjusted number of pixels.

6. A program causing a computer to function as: means for acquiring first to nth (n is a natural number of 2 or more) processing frames each having a predetermined initial number of pixels; means for acquiring first to nth input frames by generating, based on each processing frame, an input frame that corresponds to the processing frame and has a number of input pixels greater than the initial number of pixels, to acquire first to nth estimated frames each having an estimated number of pixels; wherein the machine learning model includes a cumulative feature information output layer that receives the nth input frame and n-1th auxiliary information based on n-1th cumulative feature information indicating features of the 1st to n-1th input frames, and outputs the nth cumulative feature information indicating features of the 1st to nth input frames; means for adjusting the number of pixels in the n-1th cumulative feature information so that it is less than the estimated number of pixels; and means for acquiring the n-1st auxiliary information based on the n-1st cumulative feature information with the adjusted number of pixels.

Images