Image processing system, image processing method, and program

Abstract

The present invention improves image quality according to human visual characteristics. Provided is an image processing system (1) including at least one processor, wherein the at least one processor acquires an input frame (22) having a linear relationship between a pixel value and a luminance value, performs nonlinear conversion on the input frame (22) to generate a converted input frame (22C) having a non-linear relationship between a pixel value and a luminance value according to human visual characteristics, and inputs the converted input frame (22C) into a machine learning model (200) to acquire an estimation frame (24).

Description

Image processing system, image processing method and program

[0001] The present disclosure relates to an image processing system, an image processing method, and a program.

[0002] Conventionally, a technique (super-resolution) for estimating a high-quality still image based on a low-quality still image using a machine learning model is known (see Non-Patent Document 1 below).

[0003] Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang. Learning a Deep Convolutional Network for Image Super-Resolution, in Proceedings of European Conference on Computer Vision (ECCV), 2014

[0004] In the above technology, when an input frame in which pixel values ​​and luminance values ​​have a linear relationship is input to a machine learning model, image quality is improved equally in all luminance regions. Here, human visual characteristics have different sensitivity to distinguishing luminance differences for each luminance region. Specifically, human visual characteristics are highly sensitive to luminance differences in low-luminance regions and less sensitive to luminance differences in high-luminance regions. Therefore, even if image quality is improved in high-luminance regions, it is difficult to recognize that image quality has improved. On the other hand, it is preferable to improve image quality even more in low-luminance regions.

[0005] An object of the present disclosure is to provide an image processing system, an image processing method, and a program that improve image quality in accordance with human visual characteristics.

[0006] The image processing system according to the present disclosure is an image processing system including at least one processor, which acquires an input frame in which pixel values ​​and luminance values ​​have a linear relationship, performs a nonlinear transformation on the input frame to generate a transformed input frame in which pixel values ​​and luminance values ​​have a nonlinear relationship according to human visual characteristics, and inputs the transformed input frame into a machine learning model to acquire an estimated frame.

[0007] 1 is a diagram illustrating an example of a hardware configuration of an image processing system. FIG. 2 is a diagram illustrating an overview of the image processing system. FIG. 3 is a diagram illustrating the processing of the image processing system. FIG. 4 is a functional block diagram illustrating an example of functions implemented in the image processing system. FIG. 5 is a diagram illustrating the processing of a rendering unit. FIG. 6 is a diagram illustrating the processing of an input frame acquisition unit. FIG. 7 is a diagram illustrating the frame conversion processing in this embodiment. FIG. 8 is a graph illustrating an example of the relationship between pixel values ​​and luminance values ​​in an input frame. FIG. 9 is a graph illustrating an example of the relationship between pixel values ​​and luminance values ​​in a converted input frame. FIG. 10 is a graph illustrating an example of a conversion function. FIG. 11 is a flow diagram illustrating an example of the flow of processing executed in the image processing system. FIG. 12 is a diagram illustrating the frame conversion processing in a first modified example of this embodiment. FIG. 13 is a diagram illustrating the frame conversion processing in a second modified example of this embodiment. FIG. 14 is a diagram illustrating the frame conversion processing in a third modified example of this embodiment.

[0008] Hereinafter, an example of an embodiment of an image processing system according to the present disclosure will be described with reference to the drawings.

[0009] 1 is a diagram showing an example of the hardware configuration of an image processing system 1. The image processing system 1 is a computer such as a game console. As shown in FIG. 1, the image processing system 1 includes a control unit 10, a storage unit 12, a communication unit 14, an operation unit 16, a display unit 18, and an audio output unit 19.

[0010] The control unit 10 includes a program control device such as a CPU that operates according to a program installed in the image processing system 1. The control unit 10 also includes a GPU (Graphics Processing Unit) that draws an image in a frame buffer based on graphics commands and data supplied from the CPU.

[0011] The storage unit 12 includes, for example, a main storage device such as a ROM or a RAM, and an auxiliary storage device such as a HDD or an SSD. The storage unit 12 stores programs executed by the control unit 10. The storage unit 12 stores, for example, a game program (game software) in addition to programs for realizing various functions of the image processing system 1, which will be described later. The storage unit 12 also reserves an area for a frame buffer in which images are drawn by the GPU.

[0012] The communication unit 14 is a communication interface such as an Ethernet (registered trademark) module or a wireless LAN module.

[0013] The operation unit 16 is a user interface such as a keyboard, mouse, or game console controller, and receives operation inputs from the user and outputs signals indicating the contents of the inputs to the control unit 10 .

[0014] The display unit 18 is a display device such as a liquid crystal display or an organic EL display, and displays various images according to instructions from the control unit 10 .

[0015] The audio output unit 19 is, for example, a speaker, and outputs audio represented by audio data generated by the image processing system 1 .

[0016] In addition to the devices described above, the image processing system 1 may also include an optical disc drive that reads optical discs such as DVD-ROMs and Blu-ray (registered trademark) discs, a USB (Universal Serial Bus) port, and the like.

[0017] 2. Overview of the Image Processing System Fig. 2 is a diagram showing an overview of the image processing system 1. Fig. 3 is a diagram showing the processing of the image processing system 1. In this embodiment, an example is shown in which the image processing system 1 is used to improve the image quality of gameplay video in a game. The gameplay video is a video generated in response to a game program executed by the control unit 10 or user input accepted by the operation unit 16, and is composed of a plurality of still images (frames) that are time-series data. The processing performed by the image processing system 1 is mainly as follows:

[0018] (1) Generation of Processing Frames The image processing system 1 generates an image (processing frame) depicting one or more game objects by rendering three-dimensional data representing the game objects as viewed from a predetermined viewpoint. This processing frame is an image having a predetermined number of pixels (initial pixel count) and a predetermined image quality (initial image quality) (see FIG. 3 ). Processing frames are generated at predetermined time intervals. The number of pixels of the processing frame is, for example, 1920 x 1080 (1080p). Each generated processing frame is not displayed directly on the display unit 18 but is temporarily stored in the storage unit 12 for subsequent processing. Note that the following description mainly illustrates processing for the nth processing frame 20_n (2≦n≦N, where n and N are natural numbers equal to or greater than 2), but similar processing is also performed for other processing frames (i.e., n=2, 3, ..., N).

[0019] (2) Obtaining an Input Frame Based on the obtained processing target frame 20_n, the image processing system 1 obtains a frame (input frame) 22_n having a number of pixels (number of input pixels) greater than the initial number of pixels. The number of input pixels is, for example, 3840 x 2160 (4K). Specifically, the processing target frame 20_n is subjected to enlargement and interpolation processing to generate the input frame 22_n (see FIG. 3 ).

[0020] It should be noted that although the input frame 22_n has a larger number of pixels than the target frame 20_n, its image quality is not necessarily sufficiently improved. In other words, the image quality of a frame does not simply refer to the number of pixels (high resolution). The image quality of a frame may be evaluated based on, for example, a high signal-to-noise ratio, high spatial frequency reproducibility, high temporal stability (low artifacts and flicker when multiple frames are displayed consecutively), etc., when compared to a reference frame, or a combination of these.

[0021] (3) Obtaining an Estimated Frame The image processing system 1 inputs the input frame 22_n to the machine learning model 200 and obtains an estimated frame 24_n. The estimated frame 24_n is an image having the same number of pixels (estimated number of pixels) as the number of input pixels and an image quality (estimated image quality) that is equal to or higher than the initial image quality (see FIG. 3 ).

[0022] Here, in addition to the input frame 22_n, the (n-1)th auxiliary information 28_n-1 is input to the machine learning model 200 (see FIGS. 2 and 3). The auxiliary information 28_n-1 is information based on the (n-1)th cumulative feature information 26_n-1 that indicates the features of the first to (n-1)th input frames 22. Details of the cumulative feature information 26 and the auxiliary information 28 will be described later.

[0023] (4) Acquisition of Cumulative Feature Information The machine learning model 200 has an cumulative feature information output layer 202 that receives input frame 22_n and auxiliary information 28_n-1 and outputs n-th cumulative feature information 26_n that indicates the features of the first to n-th input frames 22 (see FIG. 2). The image processing system 1 acquires the n-th cumulative feature information 26_n.

[0024] The acquired n-th cumulative feature information 26_n is input to the estimated frame output layer 204, which outputs the n-th estimated frame 24_n (see FIG. 2 ). The acquired n-th cumulative feature information 26_n is also stored in the storage unit 12 and is used to estimate an estimated frame 24_n+1 corresponding to the next processing target frame (the (n+1)th processing target frame) 20_n+1.

[0025] (5) Acquisition of Auxiliary Information As described above, the (n-1)th accumulated feature information 26_n-1 is information that indicates the features of the first to (n-1)th input frames 22 (and thus the first to (n-1)th processing target frames 20). If the accumulated feature information 26_n-1, which accumulates information on past processing target frames 20, is used to estimate the nth estimated frame 24_n, the amount of information available for estimation increases, making it possible to obtain a high-quality estimated frame 24_n.

[0026] However, if there is movement or the like in the displayed game object between the (n-1)th processing frame 20_n-1 and the nth processing frame 20_n, when the nth input frame 22_n and the cumulative feature information 26_n-1 are input as is to the machine learning model 200, a phenomenon (the so-called ghost phenomenon) may occur in which an afterimage of the game object that was displayed in the (n-1)th processing frame 20_n-1 is displayed.

[0027] Therefore, the image processing system 1 acquires the (n-1)th auxiliary information 28_n-1 by applying various corrections (described later) to the cumulative feature information 26_n-1 based on information obtained during rendering (motion vectors, depth buffers, etc.) (see FIGS. 2 and 3). As described above, the acquired (n-1)th auxiliary information 28_n-1 is input to the machine learning model 200 together with the nth input frame 22_n, and is used to estimate the nth estimated frame 24_n.

[0028] As described above, the image processing system 1 according to this embodiment estimates the estimated frame 24 using auxiliary information 28 that stores past information in addition to the input frame 22 that corresponds to the current processing target frame 20. This increases the amount of information available for estimation, making it possible to obtain a high-quality estimated frame 24_n.

[0029] 4 is a functional block diagram showing an example of functions realized in the image processing system 1. As shown in Fig. 4, the image processing system 1 realizes a game processing unit 400, a rendering unit 402, a rendering information storage unit 404, a processing target frame acquisition unit 406, a variation information acquisition unit 408, an input frame acquisition unit 410, a machine learning model storage unit 412, an estimated frame acquisition unit 414, an auxiliary information acquisition unit 416, a motion information acquisition unit 418, a depth information acquisition unit 420, an appearance pixel identification unit 422, a nonlinear conversion unit 501, and a linear conversion unit 502.

[0030] The game processing unit 400, rendering unit 402, processing target frame acquisition unit 406, variation information acquisition unit 408, input frame acquisition unit 410, estimated frame acquisition unit 414, auxiliary information acquisition unit 416, motion information acquisition unit 418, depth information acquisition unit 420, appearance pixel identification unit 422, nonlinear conversion unit 501, and linear conversion unit 502 are mainly realized by the control unit 10. The rendering information storage unit 404 and machine learning model storage unit 412 are mainly realized by the storage unit 12. Note that the game processing unit 400, rendering unit 402, and rendering information storage unit 404 are functions provided by the game software.

[0031] [Game Processing Unit] The game processing unit 400 executes various processes related to the game. For example, in response to a game program executed by the control unit 10 or user input received by the operation unit 16, the game processing unit 400 executes processes such as placing a game object O in a virtual three-dimensional space VS, operating or moving the game object O, and changing the viewpoint C from which the virtual three-dimensional space VS is viewed (see FIG. 5 ). The game object O is composed of primitives such as polygons represented by three-dimensional data. The three-dimensional data includes geometric information indicating the positions of vertices, topological information indicating how the vertices are connected, and attribute information such as color.

[0032] [Rendering Unit] FIG. 5 is a diagram illustrating the processing of the rendering unit 402. The rendering unit 402 generates first to Nth (N is a natural number equal to or greater than 2) processing frames 20 by rendering (drawing) three-dimensional data representing one or more game objects O viewed from a predetermined viewpoint C. The rendering unit 402 performs rendering based on the results of various processes executed by the game processing unit 400. Specifically, the rendering unit 402 performs vertex processing (vertex shading) and pixel processing (pixel shading) based on three-dimensional data representing game objects O placed in the virtual three-dimensional space VS. The vertex processing includes a coordinate transformation process (perspective projection) from the view coordinate system to the screen coordinate system, and a numerical value related to a change in the viewpoint C is added to the perspective projection matrix (camera matrix) used in the coordinate transformation process, as described below. The rendering unit 402 may perform rendering based on light source information, depth information (depth buffer), texture information, normal information, etc.

[0033] Here, the rendering unit 402 generates each processing frame 20 by performing rendering such that the viewpoint C varies for each processing frame 20. Here, even if the game processing unit 400 fixes the viewpoint C at a predetermined position, the rendering unit 402 varies the viewpoint C for each processing frame 20. As a result, as shown in FIG. 5 , the position of the displayed game object O varies in each processing frame 20_n, 20_n+1, and 20_n+2. In other words, the rendering unit 402 applies jitter when generating each processing frame 20. Specifically, the rendering unit 402 varies the viewpoint C for each processing frame 20 by adding a numerical value corresponding to a sub-pixel size that differs for each processing frame 20 to the perspective projection matrix. The rendering unit 402 varies the viewpoint C for each processing frame 20 according to a predetermined rule. For example, a Halton sequence can be used as such a rule.

[0034] [Rendering Information Storage Unit] The rendering information storage unit 404 stores information necessary for the rendering process in the rendering unit 402 and information obtained as a result of the rendering process. For example, the rendering information storage unit 404 stores the processing target frame 20. The rendering information storage unit 404 also stores variation information, motion information, and depth information. Details of the variation information, motion information, and depth information will be described later. In addition, the rendering information storage unit 404 may store parameters used for coordinate transformation, light source information, texture information, normal information, etc.

[0035] [Processing Frame Acquisition Unit] The processing frame acquisition unit 406 acquires the first to Nth processing frames 20. Specifically, the processing frame acquisition unit 406 acquires the first to Nth processing frames 20 stored in the rendering information storage unit 404.

[0036] [Variation Information Acquisition Unit] The variation information acquisition unit 408 acquires variation information. The variation information acquisition unit 408 acquires variation information stored in the rendering information storage unit 404. Specifically, the variation information is information indicating the amount of variation of the viewpoint C between before and after the variation. The information indicating the amount of variation can also be said to be a variation vector indicating the direction and distance of the variation. For example, since the above-mentioned Halton sequence includes information indicating the amount of variation of the viewpoint C, this information may be used as the variation information.

[0037] [Input Frame Acquisition Unit] The input frame acquisition unit 410 acquires the first to Nth input frames 22 by generating, based on each processing target frame 20, an input frame 22 that corresponds to the processing target frame 20 and has a number of input pixels equal to or greater than the initial number of pixels. In this embodiment, each input frame 22 has a number of input pixels greater than the initial number of pixels. That is, in this embodiment, each input frame 22 is an enlarged image of the processing target frame 20 that corresponds to the input frame 22.

[0038] Specifically, the input frame acquisition unit 410 generates each input frame 22 by interpolating pixel values ​​at positions in the processing frame 20 corresponding to each pixel before the fluctuation based on the fluctuation information and each pixel in the processing frame 20. FIG. 6 is a diagram illustrating the processing performed by the input frame acquisition unit 410. FIG. 6 illustrates an example of obtaining the nth input frame 22_n. For example, as shown in FIG. 6, if the pixel center of a pixel in the input frame 22_n to be acquired is P1,0, the input frame acquisition unit 410 uses bilinear interpolation to obtain the pixel value of P1,0 based on the coordinates and pixel values ​​of the pixel centers P'0,0, P'1,0, P'0,1, and P'1,1 of the four pixels closest to P1,0 in the processing frame 20_n. Here, P'1,0 is shifted from P1,0 by the amount of fluctuation indicated by the fluctuation information. The pixel values ​​of pixels newly generated by the enlargement process are similarly obtained. As the interpolation method, various known methods such as bicubic interpolation and Lanczos interpolation can be used in addition to bilinear interpolation.

[0039] When rendering is performed so that the viewpoint C varies for each processing frame 20, the amount of time-series information increases. However, by using each processing frame 20 obtained in this way (hereinafter referred to as a "variable processing frame") for estimation, an estimated frame 24 with higher image quality can be obtained.

[0040] On the other hand, if the frame to be subjected to variation processing (or an enlarged image thereof) is input directly into the machine learning model 200, the influence of the variation in viewpoint C described above may result in a decrease in the accuracy of estimation.

[0041] Therefore, as described above, in the image processing system 1, pixel values ​​at positions in the processing frame 20 corresponding to each pixel before the fluctuation are obtained by interpolation based on the fluctuation information and each pixel of each processing frame 20, and each input frame 22 is generated and input to the machine learning model 200. This corrects for the influence of fluctuations in the viewpoint C, making it possible to prevent a decrease in the accuracy of estimation.

[0042] [Machine Learning Model] The machine learning model 200 is a model that estimates the nth estimated frame 24_n based on the nth input frame 22_n. Specifically, the machine learning model 200 is a model that estimates the nth estimated frame 24_n based on the nth input frame 22_n and the n-1th auxiliary information 28_n-1. Specifically, the machine learning model 200 is a convolutional neural network (CNN). As the machine learning model 200, for example, a well-known model such as a multi-layered ResNet having a residual connection mechanism or a so-called encoder-decoder U-Net can be used. The model described in Non-Patent Document 1 may also be used as the machine learning model 200.

[0043] The machine learning model 200 is a model trained using a plurality of training data sets, each of which includes a training input frame having an input pixel count and a training estimation frame having an estimated pixel count. Various known techniques, such as backpropagation, can be used to train the machine learning model 200.

[0044] Specifically, the machine learning model 200 includes an accumulated feature information output layer 202, an estimated frame output layer 204, and a convolution layer 206 (see FIG. 2).

[0045] The cumulative feature information output layer 202 receives the nth input frame 22_n and the n-1th auxiliary information 28_n-1 based on the n-1th cumulative feature information 26_n-1 indicating the features of the first to n-1th input frames 22, and outputs the nth cumulative feature information 26_n indicating the features of the first to n-th input frames 22_n. The cumulative feature information output layer 202 may be configured, for example, with one or more convolutional layers. The cumulative feature information 26_n-1 is image information (bitmap information) having the same number of pixels as the number of input pixels. The cumulative feature information 26_n-1 can also be considered a feature map indicating the features of the first to n-1th input frames 22.

[0046] The cumulative feature information output layer 202 receives the first input frame 22_1 and given auxiliary information, and outputs first cumulative feature information 26_1. When n=1, there is no previous cumulative feature information 26 or auxiliary information 28, so the given auxiliary information prepared in advance is input to the cumulative feature information output layer 202 together with the first input frame 22_1.

[0047] The estimated frame output layer 204 receives the n-th accumulated feature information 26_n and outputs the n-th estimated frame 24_n. The estimated frame output layer 204 may be configured, for example, by one or more convolutional layers, similar to the accumulated feature information output layer 202. Alternatively, the estimated frame output layer 204 may be configured by one or more transposed convolutional layers (deconvolutional layers).

[0048] The convolutional layer 206 is a layer that reduces the number of channels of the accumulated feature information 26 while maintaining the number of pixels of the accumulated feature information 26. The accumulated feature information 26 output from the convolutional layer 206 is subjected to processing by the auxiliary information acquisition unit 416. The convolutional layer 206 reduces the dimensions of the accumulated feature information 26, thereby reducing calculation costs. The convolutional layer 206 is, for example, a convolutional layer with a kernel size of 1×1, but is not limited to this.

[0049] [Machine Learning Model Storage Unit] The machine learning model storage unit 412 stores the machine learning model 200. Specifically, the machine learning model storage unit 412 stores parameters of the machine learning model 200 (such as the number of convolutional layers, the number of nodes used in each convolutional layer, and the weight of each node).

[0050] [Estimated Frame Acquisition Unit] The estimated frame acquisition unit 414 inputs each input frame 22 to the machine learning model 200 and acquires first to Nth estimated frames 24, each having a number of estimated pixels greater than the initial number of pixels and equal to or greater than the number of input pixels. In this embodiment, the estimated frame 24 has the same number of estimated pixels as the number of input pixels. More specifically, the estimated frame acquisition unit 414 inputs the nth input frame 22_n and the (n-1)th auxiliary information 28_n-1 to the machine learning model 200 and acquires the nth estimated frame 24_n.

[0051] [Motion Information Acquisition Unit] The motion information acquisition unit 418 acquires n-1th motion information, which is information indicating the amount and direction of motion from the n-1th processing frame 20_n-1 to the nth processing frame 20_n. Specifically, the n-1th motion information has the same number of pixels as the number of input pixels, and is image information (bitmap format information) indicating the amount and direction of motion of each pixel between the n-1th processing frame 20_n-1 and the nth processing frame 20_n. The motion information is also called a motion vector. Specifically, the motion information acquisition unit 418 acquires original motion information having the same number of pixels as the initial number of pixels, and performs enlargement and interpolation processing on the original motion information to acquire motion information having the same number of pixels as the input pixel number.

[0052] [Depth Information Acquisition Unit] The depth information acquisition unit 420 acquires n-1th depth information indicating the depth of each pixel in the n-1th processing target frame 20_n-1 and nth depth information indicating the depth of each pixel in the nth processing target frame 20_n. Specifically, the depth information is image information having the same number of pixels as the number of input pixels (information in bitmap format). The depth information is also called a depth buffer, Z buffer, or depth buffer. Specifically, the depth information acquisition unit 422 acquires original depth information having the same number of pixels as the initial number of pixels, and performs enlargement and interpolation processing on the original depth information to acquire depth information having the same number of pixels as the number of input pixels.

[0053] [Appearance Pixel Identification Unit] Based on the (n-1)th depth information and the (n)th depth information, the appearance pixel identification unit 422 identifies an nth appearance pixel 422_n, which is a pixel in the nth input frame 22_n where all or part of a game object O that is not displayed in the (n-1)th input frame 22_n-1 is displayed (see FIG. 3 ). Specifically, the appearance pixel identification unit 422 identifies the nth appearance pixel 422_n based on the difference between the (n-1)th depth information and the nth depth information. Note that the appearance pixel identification unit 422 may also identify the nth appearance pixel 422_n based on the (n-1)th perspective projection matrix associated with the (n-1)th input frame 22_n-1 and the nth perspective projection matrix associated with the nth input frame 22_n. Alternatively, the appearance pixel identification unit 422 may identify the nth appearance pixel 422_n using the (n-1)th motion information. More specifically, the appearance pixel specifying unit 422 specifies the nth appearance pixel 422_n, and generates nth appearance pixel information, which is image information indicating the position of the nth appearance pixel 422_n.

[0054] [Auxiliary Information Acquisition Unit] The auxiliary information acquisition unit 416 acquires the (n-1)th auxiliary information 28_n-1 by applying motion compensation to the (n-1)th cumulative feature information 26_n-1 based on the (n-1)th motion information. Motion compensation refers to, for example, a process of moving a pixel at position x in the (n-1)th cumulative feature information 26_n to position x' when a pixel at position x in the (n-1)th input frame 22_n-1 has moved to position x' in the nth input frame 22_n (see FIG. 5 ). That is, the auxiliary information acquisition unit 416 acquires the (n-1)th auxiliary information 28_n-1 based on the (n-1)th motion information by setting the pixel values ​​of one or more pixels in the (n-1)th cumulative feature information 26_n-1 to pixels at positions moved in accordance with the amount and direction of the pixel's movement.

[0055] If there is movement of the game object O between the nth processing frame 20_n and the n-1th processing frame 20_n-1, when obtaining the nth estimated frame 24_n, if the nth input frame 22_n and the n-1th accumulated feature information 26_n-1 are input directly into the machine learning model 200, a ghost phenomenon may occur in which an afterimage of the game object O that was displayed in the nth input frame 22_n is displayed in the output nth estimated frame 24_n.

[0056] Therefore, in the image processing system 1, as described above, motion compensation is applied to the (n-1)th accumulated feature information 26_n-1 based on the (n-1)th motion information to obtain the (n-1)th auxiliary information 28_n-1, and when obtaining the nth estimated frame 24_n, this (n-1)th auxiliary information 28_n-1 is input to the machine learning model 200. This makes it possible to suppress the ghost phenomenon described above.

[0057] The nonlinear conversion unit 501 performs nonlinear conversion on the input frame 22, in which pixel values ​​and luminance values ​​have a linear relationship, to generate a frame in which pixel values ​​and luminance values ​​have a nonlinear relationship. Details of the nonlinear conversion performed by the nonlinear conversion unit 501 will be described later.

[0058] The linear conversion unit 502 generates a frame in which the pixel values ​​and the luminance values ​​have a linear relationship by performing a nonlinear conversion on the estimated frame 24 in which the pixel values ​​and the luminance values ​​have a nonlinear relationship. Details of the nonlinear conversion performed by the linear conversion unit 502 will be described later.

[0059] [4. Frame Conversion Processing] Fig. 7 is a diagram illustrating frame conversion processing in this embodiment. Fig. 8A is a graph showing an example of the relationship between pixel values ​​and luminance values ​​in an input frame. Fig. 8B is a graph showing an example of the relationship between pixel values ​​and luminance values ​​in a converted input frame. Fig. 8C is a graph showing an example of a conversion function.

[0060] In this embodiment, the processing target frame 20 and the input frame 22 generated based on the processing target frame 20 are images in which the relationship between pixel values ​​and luminance values ​​is linear. Each pixel included in the image is displayed by the display unit 18 at a brightness according to the luminance value corresponding to the pixel value.

[0061] Humans have a visual characteristic that they are highly sensitive to luminance differences in low-luminance areas and less sensitive to luminance differences in high-luminance areas. Therefore, even if the image quality of a frame is improved in a high-luminance area, it is difficult for them to recognize that the image quality has improved. On the other hand, if the image quality of a frame is improved in a low-luminance area, they can easily recognize that the image quality has improved.

[0062] Therefore, in this embodiment, a configuration is adopted in which more gradations are assigned to low-luminance areas than to high-luminance areas in accordance with human visual characteristics, and the input frame 22 is then input to the machine learning model 200.

[0063] As shown in FIG. 7 , before the input frame 22 is input to the machine learning model 200, a nonlinear transformation unit 501 performs a nonlinear transformation on the input frame 22. The nonlinear transformation may be performed, for example, based on the transformation function shown in FIG. 8C . Hereinafter, the input frame 22 that has undergone nonlinear transformation will be referred to as the "transformed input frame 22C." As shown in FIG. 8B , the pixel values ​​and luminance values ​​of the transformed input frame 22C have a nonlinear relationship. FIG. 7 shows an example in which nonlinear transformation is performed based on a transformation function (PQ EOTF-1) based on the so-called PQ (Perceptual Quantizer) method. The PQ method is a type of standard for realizing video representation that is in line with human visual characteristics.

[0064] FIG. 8A shows an example in which luminance values ​​L1 to L4 correspond to pixel values ​​P1 to P4 in the input frame 22, respectively. FIG. 8B shows an example in which luminance values ​​L1 to L4 correspond to pixel values ​​P1' to P4' in the converted input frame 22C, respectively. As shown in FIGS. 8A and 8B, in low-luminance regions (luminance values ​​L1, L2), the converted input frame 22C has a wider range of corresponding pixel values ​​than the input frame 22. On the other hand, in high-luminance regions (luminance values ​​L3, L4), the range of corresponding pixel values ​​in the input frame 22 and the converted input frame 22C is approximately the same. In other words, the converted input frame 22C is an image in which more pixel values ​​(gradations) are assigned to low-luminance regions than to high-luminance regions.

[0065] The transformed input frame 22C is input to the machine learning model 200. Then, the machine learning model 200 outputs an estimated frame 24. As with the transformed input frame 22C, the estimated frame 24 has a nonlinear relationship between its pixel values ​​and luminance values.

[0066] The linear transformation unit 502 performs nonlinear transformation on the estimated frame 24 to generate a frame in which the relationship between pixel values ​​and luminance values ​​is linear. This nonlinear transformation may be an inverse transformation of the nonlinear transformation performed by the nonlinear transformation unit 501. FIG. 7 illustrates an example in which nonlinear transformation is performed based on a PQ transformation function (PQ EOTF). Hereinafter, the estimated frame 24 that has undergone nonlinear transformation is also referred to as the "transformed estimated frame 24C." The transformed estimated frame 24C is output to the display unit 18 after various visual effects are applied by the game engine. The reason for converting the estimated frame 24 into a frame in which the relationship between pixel values ​​and luminance values ​​is linear is that a frame in which the relationship between pixel values ​​and luminance values ​​is linear is that it is more suitable for various processes leading up to display on the display unit 18. However, as described in the second modified example below, nonlinear transformation of the estimated frame 24 is not essential.

[0067] In this embodiment, the training data used to learn the machine learning model 200 may be data including a learning input frame in which pixel values ​​and brightness values ​​have a nonlinear relationship, and a learning estimation frame in which pixel values ​​and brightness values ​​have a nonlinear relationship.

[0068] 9 is a flow diagram showing an example of the flow of processing executed in the image processing system 1. The processing shown in FIG. 9 is executed by the control unit 10 operating in accordance with a program stored in the storage unit 12.

[0069] (1) Processing for n=1 First, the control unit 10 acquires a first processing frame 20_1 (S100). The control unit 10 acquires a first input frame 22_1 based on the first processing frame 20_1 (S102). Then, the control unit 10 acquires a first converted input frame 22C_1, which has a nonlinear relationship between pixel values ​​and luminance values ​​generated by performing a nonlinear conversion on the first input frame 22_1 (S104). Furthermore, the control unit 10 inputs the first converted input frame 22C_1 and given auxiliary information to the machine learning model 200, and acquires a first estimated frame 24_1 and first cumulative feature information 26_1 (S106).

[0070] (2) Processing when n≧2 The control unit 10 acquires the nth processing target frame 20_n (S108). The control unit 10 acquires the nth input frame 22_n based on the nth processing target frame 20_n (S110).

[0071] Next, the control unit 10 acquires the (n-1)th motion information (S112). The control unit 10 also acquires the (n-1)th depth information and the nth depth information (S114), and identifies the nth appearance pixel 422_n based on the (n-1)th depth information and the nth depth information (S116). The control unit 10 acquires the (n-1)th auxiliary information 28_n-1 based on the (n-1)th cumulative feature information 26_n-1, the (n-1)th motion information, and the nth appearance pixel 422_n (S118).

[0072] The control unit 10 also acquires a first converted input frame 22C_n-1 in which the relationship between pixel values ​​and luminance values ​​is nonlinear, generated by performing nonlinear conversion on the (n-1)th input frame 22_n-1 (S120).

[0073] Then, the control unit 10 inputs the nth converted input frame 22C_n and the n-1th auxiliary information 28_n-1 into the machine learning model 200, and obtains the nth estimated frame 24_n and the nth cumulative feature information 26_n (S122).

[0074] The control unit 10 also obtains the nth transformed estimated frame 24C_n, which has a linear relationship between pixel values ​​and luminance values ​​generated by performing a nonlinear transformation on the nth estimated frame 24_n (S124).

[0075] The control unit 10 then determines whether or not the next frame exists (S126), and if it determines that the next frame exists (S126; Y), it increments n to n+1 and repeats the processes of S108 to S124. If it determines that the next frame does not exist (S126; N), the control unit 10 ends this process.

[0076] [6. Summary] According to the image processing system 1 according to the present embodiment described above, the nth estimated frame 24_n is estimated using the (n-1)th accumulated feature information 26_n-1 indicating the features of the first to (n-1)th input frames 22. That is, in addition to the information of the nth input frame 22_n, the information of the first to (n-1)th input frames 22 can be used for estimation, so that the amount of information available for estimation increases, and a high-quality estimated frame 24_n can be obtained.

[0077] Furthermore, according to the image processing system 1 of this embodiment, the input frame 22 is input to the machine learning model 200 after reducing the number of gradations allocated to high-brightness regions in the input frame 22, thereby reducing the processing load on the machine learning model 200. Such a reduction in processing load is particularly effective when the image processing system 1 is applied to a game console with limited processing performance. Furthermore, the input frame 22 is input to the machine learning model 200 after increasing the number of gradations allocated to low-brightness regions in the input frame 22, thereby making it possible to obtain an estimated frame 24 that is suited to human visual characteristics.

[0078] 7. Frame Conversion Processing in Modifications of the Present Embodiment Next, frame conversion processing in each modification of the present embodiment will be described with reference to FIGS.

[0079] Fig. 10 is a diagram illustrating frame transformation processing in a first modified example of this embodiment. While Fig. 7 illustrates an example of nonlinear transformation using the PQ method, the nonlinear transformation method is not limited to this. Fig. 10 illustrates an example in which a transformed estimated frame 24C is generated by performing nonlinear transformation using the so-called gamma method on the estimated frame 24.

[0080] While the PQ method described above provides higher image quality and more accurate color reproduction, it uses complex algorithms for data processing and encoding, resulting in a heavy processing load. Therefore, as shown in Figure 10, by applying the conversion function of the PQ method to the nonlinear conversion of the input frame 22, it is possible to achieve high image quality in low-luminance areas, while applying the gamma method to the nonlinear conversion of the estimated frame 24, thereby reducing the processing load.

[0081] The conversion method is not limited to the above, and any method that converts the relationship between pixel values ​​and luminance values ​​linearly and nonlinearly may be used, for example, the HLG (Hybrid Log-Gamma) method.

[0082] FIG. 11 is a diagram illustrating the frame conversion process in a second modified example of this embodiment. While FIGS. 7 and 9 show examples in which nonlinear conversion is performed on the estimated frame 24, this nonlinear conversion does not have to be performed. This is because, depending on the processing method of the game engine and the specifications of the display, it may be possible to display a frame in which the relationship between pixel values ​​and luminance values ​​is nonlinear. Therefore, as shown in FIG. 11, an estimated frame 24 in which the relationship between pixel values ​​and luminance values ​​is nonlinear may be output to the display unit 18.

[0083] FIG. 12 is a diagram illustrating frame conversion processing in a third modified example of this embodiment. In the above-described FIGS. 7 , 10 , and 11 , an example is shown in which the machine learning model 200 outputs an estimated frame 24 in which the pixel values ​​and luminance values ​​have a nonlinear relationship based on a transformed input frame 22C in which the pixel values ​​and luminance values ​​have a nonlinear relationship. However, this is not limiting. That is, as shown in FIG. 12 , the machine learning model 200 may output an estimated frame 24 in which the pixel values ​​and luminance values ​​have a linear relationship based on the input of a transformed input frame 22C in which the pixel values ​​and luminance values ​​have a nonlinear relationship. In this case, the machine learning model 200 may be trained using multiple training data sets including a training input frame in which the pixel values ​​and luminance values ​​have a nonlinear relationship and a training estimated frame in which the pixel values ​​and luminance values ​​have a linear relationship. In the third modified example, nonlinear conversion of the estimated frame 24 output by the machine learning model 200 is not required, thereby reducing the processing load on the control unit 10.

[0084] In the above embodiment and each modification, examples have been described in which standardized existing methods such as the PQ method and the gamma method are used, but the present invention is not limited to these. For example, the conversion function according to the PQ method may be replaced with an approximate expression. This reduces the processing load required for nonlinear conversion by the control unit 10.

[0085] Furthermore, the image generated by the nonlinear conversion of the nonlinear conversion unit 501 may be in a YUV space, which represents color using a combination of a luminance signal (Y), a difference between the luminance signal and a blue component (U), and a difference between the luminance signal and a red component (V). However, the color space is not limited to such an existing color space, and a unique color space may also be used. For example, a loss function representing the difference between the estimated frame 24 output by the machine learning model 200 and the training estimated frame may be obtained, and a color space suitable for the performance of the machine learning model 200 may be searched for based on the loss function. Then, the frame converted into the searched color space may be input to the machine learning model 200.

[0086] Furthermore, the invention according to the present disclosure is not limited to the above-described embodiment and each modified example. For example, in the present embodiment, the case where the number of input pixels is greater than the initial number of pixels and the number of input pixels is the same as the estimated number of pixels is illustrated. However, the number of input pixels may be the same as the initial number of pixels and the estimated number of pixels may be greater than the number of input pixels. In other words, the input frame 22 does not necessarily have to be an enlarged version of the processing target frame 20.

[0087] Furthermore, in the present embodiment and each of the modified examples, a moving image in which input frames 22 are input sequentially has been described as an example, but the present invention is not limited to this, and still images may also be input to the image processing system 1. In other words, N may be 1.

[0088] [8. Supplementary Notes] (1) An image processing system including at least one processor, wherein the at least one processor acquires an input frame in which pixel values ​​and luminance values ​​have a linear relationship, performs a nonlinear transformation on the input frame to generate a transformed input frame in which pixel values ​​and luminance values ​​have a nonlinear relationship according to human visual characteristics, and inputs the transformed input frame to a machine learning model to obtain an estimated frame. (2) The image processing system described in (1), wherein the at least one processor acquires first to Nth input frames (N is a natural number greater than or equal to 2), respectively, performs a nonlinear transformation on the first to Nth input frames to generate the first to Nth transformed input frames, respectively, and inputs the first to Nth transformed input frames to the machine learning model to obtain the first to Nth estimated frames, respectively. (3) The image processing system according to (2), wherein the machine learning model includes: an accumulated feature information output layer that receives the nth (2≦n≦N) transformed input frame and n-1th auxiliary information based on n-1th accumulated feature information, which is image information indicating features of the 1st to n-1th transformed input frames, and outputs the nth accumulated feature information indicating features of the 1st to n-th transformed input frames; and an estimated frame output layer that receives the nth accumulated feature information and outputs the nth estimated frame. (4) The image processing system according to any of (1) to (3), wherein the at least one processor performs a nonlinear transformation on the estimated frame to generate a transformed estimated frame in which pixel values ​​and luminance values ​​have a linear relationship. (5) The image processing system according to (4), wherein the machine learning model has been trained using a plurality of training data including a training input frame in which pixel values ​​and luminance values ​​have a nonlinear relationship, and a training estimated frame in which pixel values ​​and luminance values ​​have a nonlinear relationship. (6) The image processing system according to any one of (1) to (5), wherein the machine learning model is learned using a plurality of training data including a training input frame in which pixel values ​​and luminance values ​​have a nonlinear relationship and a training estimation frame in which pixel values ​​and luminance values ​​have a linear relationship.(7) The image processing system according to any one of (1) to (5), wherein each of the input frames is an image obtained by rendering three-dimensional data showing one or more objects viewed from a predetermined viewpoint. (8) An image processing method, wherein a processor acquires an input frame in which pixel values ​​and luminance values ​​have a linear relationship, performs a nonlinear transformation on the input frame to generate a transformed input frame in which pixel values ​​and luminance values ​​have a nonlinear relationship according to human visual characteristics, and inputs the transformed input frame to a machine learning model to obtain an estimated frame. (9) A program for causing a computer to function as: input frame acquisition means for acquiring an input frame in which pixel values ​​and luminance values ​​have a linear relationship, conversion means for performing a nonlinear transformation on the input frame to generate a transformed input frame in which pixel values ​​and luminance values ​​have a nonlinear relationship according to human visual characteristics, and estimated frame acquisition means for inputting the transformed input frame to a machine learning model to obtain an estimated frame.

Claims

1. An image processing system including at least one processor, wherein the at least one processor acquires an input frame in which pixel values ​​and luminance values ​​have a linear relationship; performs a nonlinear transformation on the input frame to generate a transformed input frame in which pixel values ​​and luminance values ​​have a nonlinear relationship according to human visual characteristics; and inputs the transformed input frame to a machine learning model to acquire an estimated frame.

2. The image processing system of claim 1, wherein the at least one processor: acquires the first to Nth input frames (N is a natural number greater than or equal to 2), respectively; generates the first to Nth transformed input frames, respectively, by performing a nonlinear transformation on the first to Nth input frames, respectively; and inputs the first to Nth transformed input frames to the machine learning model to acquire the first to Nth estimated frames, respectively.

3. The image processing system of claim 2, wherein the machine learning model includes: an accumulated feature information output layer that receives the nth (2≦n≦N) transformed input frame and n-1th auxiliary information based on n-1th accumulated feature information which is image information indicating features of the 1st to n-1th transformed input frames, and outputs the nth accumulated feature information indicating features of the 1st to nth transformed input frames; and an estimated frame output layer that receives the nth accumulated feature information, and outputs the nth estimated frame.

4. The image processing system according to claim 1 or 3, wherein the at least one processor performs a nonlinear transformation on the estimated frame to generate a transformed estimated frame in which pixel values ​​and luminance values ​​have a linear relationship.

5. The image processing system of claim 4, wherein the machine learning model is trained using a plurality of training data including a training input frame in which pixel values ​​and brightness values ​​have a nonlinear relationship, and a training estimation frame in which pixel values ​​and brightness values ​​have a nonlinear relationship.

6. The image processing system of claim 1 or 3, wherein the machine learning model is trained using a plurality of training data including a training input frame in which pixel values ​​and brightness values ​​have a nonlinear relationship, and a training estimation frame in which pixel values ​​and brightness values ​​have a linear relationship.

7. An image processing system according to claim 1 or 3, wherein each of the input frames is an image obtained by performing a rendering of three-dimensional data showing one or more objects as seen from a given viewpoint.

8. An image processing method, comprising: a processor acquiring an input frame in which pixel values ​​and luminance values ​​have a linear relationship; performing a nonlinear transformation on the input frame to generate a transformed input frame in which pixel values ​​and luminance values ​​have a nonlinear relationship according to human visual characteristics; and inputting the transformed input frame into a machine learning model to acquire an estimated frame.

9. A program for causing a computer to function as: an input frame acquisition means for acquiring an input frame in which pixel values ​​and luminance values ​​have a linear relationship; a conversion means for performing a nonlinear conversion on the input frame to generate a converted input frame in which pixel values ​​and luminance values ​​have a nonlinear relationship according to human visual characteristics; and an estimated frame acquisition means for inputting the converted input frame into a machine learning model to acquire an estimated frame.

Images