Learning method, machine learning model, and program

Abstract

The present invention makes it possible to realize semi-transparent expression with high accuracy. In this learning method for a machine learning model (200), a processor acquires a plurality of n-th input frames (22) generated by a plurality of instances of rendering, acquires an n-th learning input frame (122) including an object (O) displayed at an intermediate transparency acquired on the basis of the transparencies of each of objects (O) displayed in the plurality of n-th input frames (22), acquires a plurality of n-th estimation frames (24) generated by a plurality of instances of rendering, acquires an n-th learning estimation frame (124) including an object (O) displayed at an intermediate transparency acquired on the basis of the transparencies of each of objects (O) displayed in the plurality of n-th estimation frames (24), and learns the machine learning model (200) using a plurality of pieces of training data including the n-th learning input frame (122) and the n-th learning estimation frame (124).

Description

Learning methods, machine learning models, and programs

[0001] The present disclosure relates to a learning method, a machine learning model, and a program.

[0002] Conventionally, a technique (super-resolution) for estimating a high-quality still image based on a low-quality still image using a machine learning model is known (see Non-Patent Document 1 below).

[0003] Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang. Learning a Deep Convolutional Network for Image Super-Resolution, in Proceedings of European Conference on Computer Vision (ECCV), 2014

[0004] The inventors of the present application are investigating the use of a machine learning model with a recursive structure that utilizes information from past frames to achieve super-resolution of moving images such as game screens. One known method for achieving semi-transparent expression in moving images is alpha blending, which combines multiple frames using calculations that use alpha coefficients.

[0005] Another method for simulating translucent rendering is to use frames in which objects are displayed stochastically, which can reduce the processing load compared to alpha blending.

[0006] However, when input frames in which an object is displayed stochastically are used in the learning stage of the above-mentioned machine learning model, blinking or repeated light and dark patterns are learned. When a machine learning model that has been trained in this way is used, the learning may affect areas other than the object to be displayed translucently, resulting in blinking or repeated light and dark patterns being reflected in the estimated frames. Furthermore, if the number of input frames in which an object is displayed stochastically is insufficient, the accuracy of the learning will not improve, making it difficult to achieve the intended pseudo-transparent display.

[0007] The present disclosure aims to provide a learning method, a machine learning model, and a program that enable semi-transparent representation to be realized with high accuracy.

[0008] A learning method according to the present disclosure is a learning method for a machine learning model that receives first to Nth (N is a natural number) input frames as input and generates first to Nth estimated frames, respectively, wherein at least some of the first to Nth input frames include objects displayed with probabilistically different transparency levels, and a processor acquires a plurality of nth (1≦n≦N) input frames generated by multiple renderings, acquires an nth learning input frame including the object displayed with an intermediate transparency level obtained based on the transparency level of each of the objects displayed in the plurality of nth input frames, acquires the nth estimated frames generated by multiple renderings, acquires the nth learning estimated frame including the object displayed with an intermediate transparency level obtained based on the transparency level of each of the objects displayed in the plurality of nth estimated frames, and learns the machine learning model using a plurality of training data including the nth learning input frame and the nth learning estimated frame associated with the nth learning input frame.

[0009] 1 is a diagram illustrating an example of a hardware configuration of an image processing system. FIG. 2 is a diagram illustrating an overview of the image processing system. FIG. 3 is a diagram illustrating the processing of the image processing system. FIG. 4 is a functional block diagram illustrating an example of a function implemented in the image processing system. FIG. 5 is a diagram illustrating the processing of a rendering unit. FIG. 6 is a diagram illustrating the processing in an input frame acquisition unit. FIG. 7 is a flow diagram illustrating an example of the flow of processing executed in the image processing system. FIG. 8 is a diagram illustrating an example of an input frame for implementing pseudo-semi-transparent expression. FIG. 9 is a diagram illustrating the generation of a learning input frame. FIG. 10 is a diagram illustrating the generation of a learning estimation frame. FIG. 11 is a functional block diagram illustrating an example of a function implemented in a learning device during learning. FIG. 12 is a flow diagram illustrating an example of a training data generation process. FIG. 13 is a diagram illustrating an example of an estimation frame output when input frames in which an object with a transparency of 0% is displayed and input frames in which an object with a transparency of 100% is displayed are alternately input to the machine learning model of this embodiment.

[0010] Hereinafter, an example of an embodiment of an image processing system according to the present disclosure will be described with reference to the drawings.

[0011] 1 is a diagram showing an example of the hardware configuration of an image processing system 1. The image processing system 1 is a computer such as a game console. As shown in FIG. 1, the image processing system 1 includes a control unit 10, a storage unit 12, a communication unit 14, an operation unit 16, a display unit 18, and an audio output unit 19.

[0012] The control unit 10 includes a program control device such as a CPU that operates according to a program installed in the image processing system 1. The control unit 10 also includes a GPU (Graphics Processing Unit) that draws an image in a frame buffer based on graphics commands and data supplied from the CPU.

[0013] The storage unit 12 includes, for example, a main storage device such as a ROM or a RAM, and an auxiliary storage device such as a HDD or an SSD. The storage unit 12 stores programs executed by the control unit 10. The storage unit 12 stores, for example, a game program (game software) in addition to programs for realizing various functions of the image processing system 1, which will be described later. The storage unit 12 also reserves an area for a frame buffer in which images are drawn by the GPU.

[0014] The communication unit 14 is a communication interface such as an Ethernet (registered trademark) module or a wireless LAN module.

[0015] The operation unit 16 is a user interface such as a keyboard, mouse, or game console controller, and receives operation inputs from the user and outputs signals indicating the contents of the inputs to the control unit 10 .

[0016] The display unit 18 is a display device such as a liquid crystal display or an organic EL display, and displays various images according to instructions from the control unit 10 .

[0017] The audio output unit 19 is, for example, a speaker, and outputs audio represented by audio data generated by the image processing system 1 .

[0018] In addition to the devices described above, the image processing system 1 may also include an optical disc drive that reads optical discs such as DVD-ROMs and Blu-ray (registered trademark) discs, a USB (Universal Serial Bus) port, and the like.

[0019] 2. Overview of the Image Processing System Fig. 2 is a diagram showing an overview of the image processing system 1. Fig. 3 is a diagram showing the processing of the image processing system 1. In this embodiment, an example is shown in which the image processing system 1 is used to improve the image quality of gameplay video in a game. The gameplay video is a video generated in response to a game program executed by the control unit 10 or user input accepted by the operation unit 16, and is composed of a plurality of still images (frames) that are time-series data. The processing performed by the image processing system 1 is mainly as follows:

[0020] (1) Generation of Processing Frames The image processing system 1 generates an image (processing frame) depicting one or more game objects by rendering three-dimensional data representing the game objects as viewed from a predetermined viewpoint. This processing frame is an image having a predetermined number of pixels (initial pixel count) and a predetermined image quality (initial image quality) (see FIG. 3 ). Processing frames are generated at predetermined time intervals. The number of pixels of the processing frame is, for example, 1920 x 1080 (1080p). Each generated processing frame is not displayed directly on the display unit 18 but is temporarily stored in the storage unit 12 for subsequent processing. Note that the following description mainly illustrates processing for the nth processing frame 20_n (2≦n≦N, where n and N are natural numbers equal to or greater than 2), but similar processing is also performed for other processing frames (i.e., n=2, 3, ..., N).

[0021] (2) Obtaining an Input Frame Based on the obtained processing target frame 20_n, the image processing system 1 obtains a frame (input frame) 22_n having a number of pixels (number of input pixels) greater than the initial number of pixels. The number of input pixels is, for example, 3840 x 2160 (4K). Specifically, the processing target frame 20_n is subjected to enlargement and interpolation processing to generate the input frame 22_n (see FIG. 3 ).

[0022] It should be noted that although the input frame 22_n has a larger number of pixels than the target frame 20_n, its image quality is not necessarily sufficiently improved. In other words, the image quality of a frame does not simply refer to the number of pixels (high resolution). The image quality of a frame may be evaluated based on, for example, a high signal-to-noise ratio, high spatial frequency reproducibility, high temporal stability (low artifacts and flicker when multiple frames are displayed consecutively), etc., when compared to a reference frame, or a combination of these.

[0023] (3) Obtaining an Estimated Frame The image processing system 1 inputs the input frame 22_n to the machine learning model 200 and obtains an estimated frame 24_n. The estimated frame 24_n is an image having the same number of pixels (estimated number of pixels) as the number of input pixels and an image quality (estimated image quality) that is equal to or higher than the initial image quality (see FIG. 3 ).

[0024] Here, in addition to the input frame 22_n, the (n-1)th auxiliary information 28_n-1 is input to the machine learning model 200 (see FIGS. 2 and 3). The auxiliary information 28_n-1 is information based on the (n-1)th cumulative feature information 26_n-1 that indicates the features of the first to (n-1)th input frames 22. Details of the cumulative feature information 26 and the auxiliary information 28 will be described later.

[0025] (4) Acquisition of Cumulative Feature Information The machine learning model 200 has an cumulative feature information output layer 202 that receives input frame 22_n and auxiliary information 28_n-1 and outputs n-th cumulative feature information 26_n that indicates the features of the first to n-th input frames 22 (see FIG. 2). The image processing system 1 acquires the n-th cumulative feature information 26_n.

[0026] The acquired n-th cumulative feature information 26_n is input to the estimated frame output layer 204, which outputs the n-th estimated frame 24_n (see FIG. 2 ). The acquired n-th cumulative feature information 26_n is also stored in the storage unit 12 and is used to estimate an estimated frame 24_n+1 corresponding to the next processing target frame (the (n+1)th processing target frame) 20_n+1.

[0027] (5) Acquisition of Auxiliary Information As described above, the (n-1)th accumulated feature information 26_n-1 is information that indicates the features of the first to (n-1)th input frames 22 (and thus the first to (n-1)th processing target frames 20). If the accumulated feature information 26_n-1, which accumulates information on past processing target frames 20, is used to estimate the nth estimated frame 24_n, the amount of information available for estimation increases, making it possible to obtain a high-quality estimated frame 24_n.

[0028] However, if there is movement or the like in the displayed game object between the (n-1)th processing frame 20_n-1 and the nth processing frame 20_n, when the nth input frame 22_n and the cumulative feature information 26_n-1 are input as is to the machine learning model 200, a phenomenon (the so-called ghost phenomenon) may occur in which an afterimage of the game object that was displayed in the (n-1)th processing frame 20_n-1 is displayed.

[0029] Therefore, the image processing system 1 acquires the (n-1)th auxiliary information 28_n-1 by applying various corrections (described later) to the cumulative feature information 26_n-1 based on information obtained during rendering (motion vectors, depth buffers, etc.) (see FIGS. 2 and 3). As described above, the acquired (n-1)th auxiliary information 28_n-1 is input to the machine learning model 200 together with the nth input frame 22_n, and is used to estimate the nth estimated frame 24_n.

[0030] As described above, the image processing system 1 according to this embodiment estimates the estimated frame 24 using the auxiliary information 28, which stores past information, in addition to the input frame 22 corresponding to the current processing target frame 20. This increases the amount of information available for estimation, making it possible to obtain a high-quality estimated frame 24_n.

[0031] 4 is a functional block diagram showing an example of functions realized in the image processing system 1. As shown in Fig. 4, the image processing system 1 realizes a game processing unit 400, a rendering unit 402, a rendering information storage unit 404, a processing target frame acquisition unit 406, a variation information acquisition unit 408, an input frame acquisition unit 410, a machine learning model storage unit 412, an estimated frame acquisition unit 414, an auxiliary information acquisition unit 416, a motion information acquisition unit 418, a depth information acquisition unit 420, and an appearance pixel identification unit 422.

[0032] The game processing unit 400, rendering unit 402, processing target frame acquisition unit 406, variation information acquisition unit 408, input frame acquisition unit 410, estimated frame acquisition unit 414, auxiliary information acquisition unit 416, motion information acquisition unit 418, depth information acquisition unit 420, and appearance pixel identification unit 422 are mainly realized by the control unit 10. The rendering information storage unit 404 and machine learning model storage unit 412 are mainly realized by the storage unit 12. Note that the game processing unit 400, rendering unit 402, and rendering information storage unit 404 are functions provided by game software.

[0033] [Game Processing Unit] The game processing unit 400 executes various processes related to the game. For example, in response to a game program executed by the control unit 10 or user input received by the operation unit 16, the game processing unit 400 executes processes such as placing a game object O in a virtual three-dimensional space VS, operating or moving the game object O, and changing the viewpoint C from which the virtual three-dimensional space VS is viewed (see FIG. 5 ). The game object O is composed of primitives such as polygons represented by three-dimensional data. The three-dimensional data includes geometric information indicating the positions of vertices, topological information indicating how the vertices are connected, and attribute information such as color.

[0034] [Rendering Unit] FIG. 5 is a diagram illustrating the processing of the rendering unit 402. The rendering unit 402 generates first to Nth (N is a natural number equal to or greater than 2) processing frames 20 by rendering (drawing) three-dimensional data representing one or more game objects O viewed from a predetermined viewpoint C. The rendering unit 402 performs rendering based on the results of various processes executed by the game processing unit 400. Specifically, the rendering unit 402 performs vertex processing (vertex shading) and pixel processing (pixel shading) based on three-dimensional data representing game objects O placed in the virtual three-dimensional space VS. The vertex processing includes a coordinate transformation process (perspective projection) from the view coordinate system to the screen coordinate system, and a numerical value related to a change in the viewpoint C is added to the perspective projection matrix (camera matrix) used in the coordinate transformation process, as described below. The rendering unit 402 may perform rendering based on light source information, depth information (depth buffer), texture information, normal information, etc.

[0035] Here, the rendering unit 402 generates each processing frame 20 by performing rendering such that the viewpoint C varies for each processing frame 20. Here, even if the game processing unit 400 fixes the viewpoint C at a predetermined position, the rendering unit 402 varies the viewpoint C for each processing frame 20. As a result, as shown in FIG. 5 , the position of the displayed game object O varies in each processing frame 20_n, 20_n+1, and 20_n+2. In other words, the rendering unit 402 applies jitter when generating each processing frame 20. Specifically, the rendering unit 402 varies the viewpoint C for each processing frame 20 by adding a numerical value corresponding to a sub-pixel size that differs for each processing frame 20 to the perspective projection matrix. The rendering unit 402 varies the viewpoint C for each processing frame 20 according to a predetermined rule. For example, a Halton sequence can be used as such a rule.

[0036] [Rendering Information Storage Unit] The rendering information storage unit 404 stores information necessary for the rendering process in the rendering unit 402 and information obtained as a result of the rendering process. For example, the rendering information storage unit 404 stores the processing target frame 20. The rendering information storage unit 404 also stores variation information, motion information, and depth information. Details of the variation information, motion information, and depth information will be described later. In addition, the rendering information storage unit 404 may store parameters used for coordinate transformation, light source information, texture information, normal information, etc.

[0037] [Processing Frame Acquisition Unit] The processing frame acquisition unit 406 acquires the first to Nth processing frames 20. Specifically, the processing frame acquisition unit 406 acquires the first to Nth processing frames 20 stored in the rendering information storage unit 404.

[0038] [Variation Information Acquisition Unit] The variation information acquisition unit 408 acquires variation information. The variation information acquisition unit 408 acquires variation information stored in the rendering information storage unit 404. Specifically, the variation information is information indicating the amount of variation of the viewpoint C between before and after the variation. The information indicating the amount of variation can also be said to be a variation vector indicating the direction and distance of the variation. For example, since the above-mentioned Halton sequence includes information indicating the amount of variation of the viewpoint C, this information may be used as the variation information.

[0039] [Input Frame Acquisition Unit] The input frame acquisition unit 410 acquires the first to Nth input frames 22 by generating, based on each processing target frame 20, an input frame 22 that corresponds to the processing target frame 20 and has a number of input pixels equal to or greater than the initial number of pixels. In this embodiment, each input frame 22 has a number of input pixels greater than the initial number of pixels. That is, in this embodiment, each input frame 22 is an enlarged image of the processing target frame 20 that corresponds to the input frame 22.

[0040] Specifically, the input frame acquisition unit 410 generates each input frame 22 by interpolating pixel values ​​at positions in the processing frame 20 corresponding to each pixel before the fluctuation based on the fluctuation information and each pixel in the processing frame 20. FIG. 6 is a diagram illustrating the processing performed by the input frame acquisition unit 410. FIG. 6 illustrates an example of obtaining the nth input frame 22_n. For example, as shown in FIG. 6, if the pixel center of a pixel in the input frame 22_n to be acquired is P1,0, the input frame acquisition unit 410 uses bilinear interpolation to obtain the pixel value of P1,0 based on the coordinates and pixel values ​​of the pixel centers P'0,0, P'1,0, P'0,1, and P'1,1 of the four pixels closest to P1,0 in the processing frame 20_n. Here, P'1,0 is shifted from P1,0 by the amount of fluctuation indicated by the fluctuation information. The pixel values ​​of pixels newly generated by the enlargement process are similarly obtained. As the interpolation method, various known methods such as bicubic interpolation and Lanczos interpolation can be used in addition to bilinear interpolation.

[0041] When rendering is performed so that the viewpoint C varies for each processing frame 20, the amount of time-series information increases. However, by using each processing frame 20 obtained in this way (hereinafter referred to as a "variable processing frame") for estimation, an estimated frame 24 with higher image quality can be obtained.

[0042] On the other hand, if the frame to be subjected to variation processing (or an enlarged image thereof) is input directly into the machine learning model 200, the influence of the variation in viewpoint C described above may result in a decrease in the accuracy of estimation.

[0043] Therefore, as described above, in the image processing system 1, pixel values ​​at positions in the processing frame 20 corresponding to each pixel before the fluctuation are obtained by interpolation based on the fluctuation information and each pixel of each processing frame 20, and each input frame 22 is generated and input to the machine learning model 200. This corrects for the influence of fluctuations in the viewpoint C, thereby preventing a decrease in the accuracy of estimation.

[0044] [Machine Learning Model] The machine learning model 200 is a model that estimates the nth estimated frame 24_n based on the nth input frame 22_n. Specifically, the machine learning model 200 is a model that estimates the nth estimated frame 24_n based on the nth input frame 22_n and the n-1th auxiliary information 28_n-1. Specifically, the machine learning model 200 is a convolutional neural network (CNN). As the machine learning model 200, for example, a well-known model such as a multi-layered ResNet having a residual connection mechanism or a so-called encoder-decoder U-Net can be used. The model described in Non-Patent Document 1 may also be used as the machine learning model 200.

[0045] The machine learning model 200 is a model trained using a plurality of training data sets, each of which includes a training input frame and a training estimation frame. Various well-known techniques, such as backpropagation, can be used to train the machine learning model 200. Details of the training data will be described later.

[0046] Specifically, the machine learning model 200 includes an accumulated feature information output layer 202, an estimated frame output layer 204, and a convolution layer 206 (see FIG. 2).

[0047] The cumulative feature information output layer 202 receives the nth input frame 22_n and the n-1th auxiliary information 28_n-1 based on the n-1th cumulative feature information 26_n-1 indicating the features of the first to n-1th input frames 22, and outputs the nth cumulative feature information 26_n indicating the features of the first to n-th input frames 22_n. The cumulative feature information output layer 202 may be configured, for example, with one or more convolutional layers. The cumulative feature information 26_n-1 is image information (bitmap information) having the same number of pixels as the number of input pixels. The cumulative feature information 26_n-1 can also be considered a feature map indicating the features of the first to n-1th input frames 22.

[0048] The cumulative feature information output layer 202 receives the first input frame 22_1 and given auxiliary information, and outputs first cumulative feature information 26_1. When n=1, there is no previous cumulative feature information 26 or auxiliary information 28, so the given auxiliary information prepared in advance is input to the cumulative feature information output layer 202 together with the first input frame 22_1.

[0049] The estimated frame output layer 204 receives the n-th accumulated feature information 26_n and outputs the n-th estimated frame 24_n. The estimated frame output layer 204 may be configured, for example, by one or more convolutional layers, similar to the accumulated feature information output layer 202. Alternatively, the estimated frame output layer 204 may be configured by one or more transposed convolutional layers (deconvolutional layers).

[0050] The convolutional layer 206 is a layer that reduces the number of channels of the accumulated feature information 26 while maintaining the number of pixels of the accumulated feature information 26. The accumulated feature information 26 output from the convolutional layer 206 is subjected to processing by the auxiliary information acquisition unit 416. The convolutional layer 206 reduces the dimensions of the accumulated feature information 26, thereby reducing calculation costs. The convolutional layer 206 is, for example, a convolutional layer with a kernel size of 1×1, but is not limited to this.

[0051] [Machine Learning Model Storage Unit] The machine learning model storage unit 412 stores the machine learning model 200. Specifically, the machine learning model storage unit 412 stores parameters of the machine learning model 200 (such as the number of convolutional layers, the number of nodes used in each convolutional layer, and the weight of each node).

[0052] [Estimated Frame Acquisition Unit] The estimated frame acquisition unit 414 inputs each input frame 22 to the machine learning model 200 and acquires first to Nth estimated frames 24, each having a number of estimated pixels greater than the initial number of pixels and equal to or greater than the number of input pixels. In this embodiment, the estimated frame 24 has the same number of estimated pixels as the number of input pixels. More specifically, the estimated frame acquisition unit 414 inputs the nth input frame 22_n and the (n-1)th auxiliary information 28_n-1 to the machine learning model 200 and acquires the nth estimated frame 24_n.

[0053] [Motion Information Acquisition Unit] The motion information acquisition unit 418 acquires n-1th motion information, which is information indicating the amount and direction of motion from the n-1th processing frame 20_n-1 to the nth processing frame 20_n. Specifically, the n-1th motion information has the same number of pixels as the number of input pixels, and is image information (bitmap format information) indicating the amount and direction of motion of each pixel between the n-1th processing frame 20_n-1 and the nth processing frame 20_n. The motion information is also called a motion vector. Specifically, the motion information acquisition unit 418 acquires original motion information having the same number of pixels as the initial number of pixels, and performs enlargement and interpolation processing on the original motion information to acquire motion information having the same number of pixels as the input pixel number.

[0054] [Depth Information Acquisition Unit] The depth information acquisition unit 420 acquires n-1th depth information indicating the depth of each pixel in the n-1th processing target frame 20_n-1 and nth depth information indicating the depth of each pixel in the nth processing target frame 20_n. Specifically, the depth information is image information having the same number of pixels as the number of input pixels (information in bitmap format). The depth information is also called a depth buffer, Z buffer, or depth buffer. Specifically, the depth information acquisition unit 422 acquires original depth information having the same number of pixels as the initial number of pixels, and performs enlargement and interpolation processing on the original depth information to acquire depth information having the same number of pixels as the number of input pixels.

[0055] [Appearance Pixel Identification Unit] Based on the (n-1)th depth information and the (n)th depth information, the appearance pixel identification unit 422 identifies an nth appearance pixel 422_n, which is a pixel in the nth input frame 22_n where all or part of a game object O that is not displayed in the (n-1)th input frame 22_n-1 is displayed (see FIG. 3 ). Specifically, the appearance pixel identification unit 422 identifies the nth appearance pixel 422_n based on the difference between the (n-1)th depth information and the nth depth information. Note that the appearance pixel identification unit 422 may also identify the nth appearance pixel 422_n based on the (n-1)th perspective projection matrix associated with the (n-1)th input frame 22_n-1 and the nth perspective projection matrix associated with the nth input frame 22_n. Alternatively, the appearance pixel identification unit 422 may identify the nth appearance pixel 422_n using the (n-1)th motion information. More specifically, the appearance pixel specifying unit 422 specifies the nth appearance pixel 422_n, and generates nth appearance pixel information, which is image information indicating the position of the nth appearance pixel 422_n.

[0056] [Auxiliary Information Acquisition Unit] The auxiliary information acquisition unit 416 acquires the (n-1)th auxiliary information 28_n-1 by applying motion compensation to the (n-1)th cumulative feature information 26_n-1 based on the (n-1)th motion information. Motion compensation refers to, for example, a process of moving a pixel at position x in the (n-1)th cumulative feature information 26_n to position x' when a pixel at position x in the (n-1)th input frame 22_n-1 has moved to position x' in the nth input frame 22_n (see FIG. 5 ). That is, the auxiliary information acquisition unit 416 acquires the (n-1)th auxiliary information 28_n-1 based on the (n-1)th motion information by setting the pixel values ​​of one or more pixels in the (n-1)th cumulative feature information 26_n-1 to pixels at positions moved in accordance with the amount and direction of the pixel's movement.

[0057] If there is movement of the game object O between the nth processing frame 20_n and the n-1th processing frame 20_n-1, when obtaining the nth estimated frame 24_n, if the nth input frame 22_n and the n-1th accumulated feature information 26_n-1 are input directly into the machine learning model 200, a ghost phenomenon may occur in which an afterimage of the game object O that was displayed in the nth input frame 22_n is displayed in the output nth estimated frame 24_n.

[0058] Therefore, in the image processing system 1, as described above, motion compensation is applied to the (n-1)th accumulated feature information 26_n-1 based on the (n-1)th motion information to obtain the (n-1)th auxiliary information 28_n-1, and when obtaining the nth estimated frame 24_n, this (n-1)th auxiliary information 28_n-1 is input to the machine learning model 200. This makes it possible to suppress the ghost phenomenon described above.

[0059] 7 is a flow diagram showing an example of the flow of processing executed in the image processing system 1. The processing shown in FIG. 7 is executed by the control unit 10 operating in accordance with a program stored in the storage unit 12.

[0060] (1) Processing for n=1 First, the control unit 10 acquires a first processing frame 20_1 (S100). The control unit 10 acquires a first input frame 22_1 based on the first processing frame 20_1 (S102). Then, the control unit 10 inputs the first input frame 22_1 and given auxiliary information to the machine learning model 200, and acquires a first estimated frame 24_1 and first cumulative feature information 26_1 (S104).

[0061] (2) Processing when n≧2 The control unit 10 acquires the nth processing target frame 20_n (S106). The control unit 10 acquires the nth input frame 22_n based on the nth processing target frame 20_n (S108).

[0062] Next, the control unit 10 acquires the (n-1)th motion information (S110). The control unit 10 also acquires the (n-1)th depth information and the nth depth information (S112), and identifies the nth appearance pixel 422_n based on the (n-1)th depth information and the nth depth information (S114). The control unit 10 acquires the (n-1)th auxiliary information 28_n-1 based on the (n-1)th cumulative feature information 26_n-1, the (n-1)th motion information, and the nth appearance pixel 422_n (S116).

[0063] In addition, the control unit 10 inputs the nth input frame 22_n and the n-1th auxiliary information 28_n-1 into the machine learning model 200, and obtains the nth estimated frame 24_n and the nth cumulative feature information 26_n (S118).

[0064] The control unit 10 then determines whether or not the next frame exists (S120), and if it determines that the next frame exists (S120; Y), it increments n to n+1 and repeats the processes of S108 to S124. If it determines that the next frame does not exist (S120; N), the control unit 10 ends this process.

[0065] [5. Learning Method] Next, a learning method for a machine learning model will be described with reference to FIGS. 8 to 10. FIG. 8 is a diagram showing an example of an input frame for realizing pseudo-transparent representation. FIG. 9 is a diagram for explaining the generation of a learning input frame in this embodiment. FIG. 10 is a diagram for explaining the generation of a learning estimated frame in this embodiment. Note that in FIGS. 8 to 10, the number of pixels of the estimated frame 24 is shown to be greater than the number of pixels of the input frame 22, taking visual clarity into consideration. However, as described above, the estimated frame 24 only needs to have higher image quality than the input frame 22, and the number of pixels of the input frame 22 and the estimated frame 24 may be the same. The same applies to FIG. 13 described below.

[0066] A known technique is to alternately or randomly display frames in which an object O is displayed and frames in which the object O is not displayed, thereby making the object O appear pseudo-transparent. Fig. 8 shows an example in which the object O is displayed in a first input frame 22_1 at time T1 and a third input frame 22_3 at time T3, and is not displayed in a second input frame 22_2 at time T2 and a fourth input frame 22_4 at time T4.

[0067] In the input frame 22 used to achieve this pseudo-transparent representation, the object O is set to be displayed with different transparency levels probabilistically. Transparency is a parameter set for each pixel separately from the pixel value. For example, if the transparency is 0%, the area is displayed as non-transparent, and if the transparency is 100%, the area is displayed as transparent.

[0068] In the first input frame 22_1 and the third input frame 22_3 shown in Fig. 8, the transparency of the object O is 0%, i.e., non-transparent, and the object O is displayed visibly. In the second input frame 22_2 and the fourth input frame 22_4 shown in Fig. 8, the transparency of the object O is 100%, i.e., transparent, and the object O is displayed invisible. In this way, the object O exists in each input frame 22, but its transparency is different. Furthermore, the object O included in each input frame 22 is represented by pixels containing common depth information.

[0069] Here, when a machine learning model is trained using frames including an object O that is displayed with probabilistically different transparency levels as training data, the model learns blinking or repeated light and dark patterns. When a machine learning model trained in this way is used, the model may be affected by the learning even in areas other than the object O that is to be displayed semi-transparently, resulting in the blinking or repeated light and dark patterns being reflected in the estimated frames. Furthermore, if the number of input frames in which the object is displayed stochastically is insufficient, the accuracy of the learning does not improve, making it difficult to achieve the intended pseudo-semi-transparent pattern.

[0070] Therefore, in this embodiment, the machine learning model 200 is trained using a plurality of training data sets including a training input frame 122 and a training estimation frame 124, which will be described below.

[0071] [Learning Input Frame] The nth learning input frame 122_n is a frame including an object O displayed at an intermediate transparency obtained based on the transparency of each of the objects O displayed in the multiple nth input frames 22_n.

[0072] First, at each time (each input frame), rendering is performed multiple times to obtain multiple input frames 22. Fig. 9 shows an example in which multiple first input frames 22_1 are obtained at time T1, and a first training input frame 122_1 is generated based on the multiple first input frames 22_1.

[0073] The first training input frame 122_1 may be, for example, an average of the transparency of each pixel of a plurality of first input frames 22_1, thereby generating a first training input frame 122_1 including the object O displayed with an intermediate transparency.

[0074] For the input frames 22_2, 22_3, . . . at times T2, T3, . . . , the learning input frames 122_2, 22_3, .

[0075] It is preferable that a common jitter is applied to the multiple n-th input frames 22_n. By applying a common jitter, it is possible to average the transparency of each pixel in the multiple n-th input frames 22_n for each pixel. Furthermore, it is preferable that a developer or other person who generates training data can appropriately adjust a counter related to the transparency period, thereby making it possible to set the ratio of n-th input frames 22_n that contain non-transparent and transparent objects O among the multiple n-th input frames 22_n in which objects O with different transparency levels are displayed probabilistically.

[0076] [Learning Estimated Frame] The nth learning estimated frame 124_n is a frame including an object O displayed at an intermediate transparency obtained based on the transparency of each of the objects O displayed in the multiple nth estimated frames 24_n.

[0077] First, at each time (each estimated frame), rendering is performed multiple times to obtain multiple estimated frames 24. Fig. 10 shows an example in which multiple first estimated frames 24_1 are obtained at time T1, and a first training estimated frame 124_1 is generated based on the multiple first estimated frames 24_1.

[0078] The first training estimation frame 124_1 may be, for example, an average of the transparency of each pixel of a plurality of first estimation frames 24_1, thereby generating a first training estimation frame 124_1 including the object O displayed with an intermediate transparency.

[0079] For the estimated frames 24_2, 24_3, . . . at times T2, T3, . . . , the training estimated frames 124_2, 24_3, .

[0080] It is preferable that a common jitter is applied to the multiple n-th estimated frames 24_n. By applying a common jitter, it is possible to average the transparency of each pixel in the multiple n-th estimated frames 24_n for each pixel. Furthermore, it is preferable that a developer or other person who generates training data can appropriately adjust a counter related to the transparency period, thereby making it possible to set the ratio of n-th estimated frames 24_n containing non-transparent and transparent objects O among the multiple n-th estimated frames 24_n in which objects O with different transparency levels are displayed probabilistically.

[0081] [Functions Implemented by the Learning Device] Fig. 11 is a functional block diagram showing an example of functions implemented by an image processing system (herein referred to as the learning device) during learning. As shown in Fig. 11, the learning device 500 includes the above-described machine learning model 200, an input frame acquisition unit 510, a learning input frame acquisition unit 520, an estimated frame acquisition unit 530, a learning estimated frame acquisition unit 540, and a learning unit 550. The input frame acquisition unit 510, the learning input frame acquisition unit 520, the estimated frame acquisition unit 530, the learning estimated frame acquisition unit 540, and the learning unit 550 are mainly implemented by the control unit 10.

[0082] The input frame acquisition unit 510 acquires multiple n-th input frames 22 generated by multiple renderings. The training input frame acquisition unit 520 acquires an n-th training input frame 122_n including an object O displayed at an intermediate transparency acquired based on the transparency of each of the objects O displayed in the multiple n-th input frames 22_n.

[0083] The estimated frame acquisition unit 530 acquires multiple n-th estimated frames 24_n that are output by inputting the n-th input frame 22_n to the machine learning model 200 and generated by multiple renderings. The training estimated frame acquisition unit 540 acquires the n-th training estimated frame 124_n that includes the object O displayed at an intermediate transparency acquired based on the transparency of each of the objects O displayed in the multiple n-th estimated frames.

[0084] The learning unit 550 causes the machine learning model 200 to learn the n-th learning input frame 122_n and the n-th learning estimation frame 124_n associated with the n-th learning input frame 122_n as training data.

[0085] 12 is a flow diagram showing an example of the training data generation process. The control unit 10 acquires multiple n-th input frames 22_n generated by multiple renderings (S200). The control unit 10 acquires an n-th learning input frame 122 generated based on the multiple n-th input frames 22_n (S202).

[0086] The control unit 10 also acquires a plurality of n-th estimated frames 24_n generated by multiple renderings (S204), and an n-th training estimated frame 124_n generated based on the plurality of n-th estimated frames 24_n (S206).

[0087] The control unit 10 performs training of the machine learning model 200 using the nth training data set consisting of the nth training input frame 122_n and the nth training estimation frame 124_n acquired in S202 and S206 (S208).

[0088] If the control unit 10 has not completed learning using the first to Nth training data sets (N in S210), it increments n to n+1 and repeats the processes of S200 to S208. If the control unit 10 has completed learning using the first to Nth training data sets (Y in S210), it ends this process.

[0089] 6. Example of Estimated Frames to be Output FIG. 13 shows an example of estimated frames to be output when input frames in which an object with a transparency of 0% is displayed and input frames in which an object with a transparency of 100% is displayed are alternately input to the machine learning model of this embodiment.

[0090] As described with reference to FIG. 2 , the image processing system 1 uses a machine learning model 200 having a recursive configuration that utilizes information from past frames. Therefore, the resolution of the input frame 22 in which an opaque object O is displayed is increased by referring to past frames in which a transparent object O is displayed, and the resolution of the input frame 22 in which a transparent object O is displayed is increased by referring to past frames in which an opaque object O is displayed. Therefore, as shown in FIG. 13 , regardless of whether the input frame 22 in which an object O with 0% transparency is displayed or the input frame 22 in which an object O with 100% transparency is input, an estimated frame 24 in which an object O with an intermediate transparency is displayed is output. As a result, the object O can be visually perceived by the user as translucent.

[0091] [7. Summary] According to the image processing system 1 according to the present embodiment described above, the nth estimated frame 24_n is estimated using the (n-1)th accumulated feature information 26_n-1 indicating the features of the first to (n-1)th input frames 22. That is, in addition to the information of the nth input frame 22_n, the information of the first to (n-1)th input frames 22 can be used for estimation, increasing the amount of information available for estimation and enabling the acquisition of a high-quality estimated frame 24_n.

[0092] Furthermore, regardless of the method of semi-transparent expression in the input frame 22, it is possible to output a highly accurate estimated frame 24.

[0093] Furthermore, the invention according to the present disclosure is not limited to the above-described embodiment and each modified example. For example, in the present embodiment, the case where the number of input pixels is greater than the initial number of pixels and the number of input pixels is the same as the estimated number of pixels is illustrated. However, the number of input pixels may be the same as the initial number of pixels and the estimated number of pixels may be greater than the number of input pixels. In other words, the input frame 22 does not necessarily have to be an enlarged version of the processing target frame 20.

[0094] [8. Supplementary Note] For example, the learning method may be configured as follows: (1) A learning method for a machine learning model that receives first to Nth (N is a natural number) input frames and generates first to Nth estimated frames, respectively, wherein at least some of the first to Nth input frames include objects displayed with probabilistically different transparency levels, and a processor: acquires a plurality of nth (1≦n≦N) input frames generated by multiple renderings, acquires an nth learning input frame including the object displayed with an intermediate transparency level obtained based on the transparency level of each of the objects displayed in the plurality of nth input frames, acquires the nth estimated frames generated by multiple renderings, acquires the nth learning estimated frame including the object displayed with an intermediate transparency level obtained based on the transparency level of each of the objects displayed in the plurality of nth estimated frames, and learns the machine learning model using a plurality of training data including the nth learning input frame and the nth learning estimated frame associated with the nth learning input frame. (2) The learning method according to (1), wherein the machine learning model includes: an cumulative feature information output layer that receives the nth input frame and (n-1)th auxiliary information based on (n-1)th cumulative feature information, which is image information indicating features of the 1st to n-1th input frames, and outputs the nth cumulative feature information indicating features of the 1st to n-th input frames; and an estimated frame output layer that receives the nth cumulative feature information and outputs the nth estimated frame. (3) The learning method according to (1) or (2), wherein the number of pixels of the nth estimated frame is equal to or greater than the number of pixels of the nth input frame. (4) The learning method according to any of (1) to (3), wherein the intermediate transparency of the object displayed in the nth learning input frame is an average of the transparency of the object displayed in each of the multiple nth input frames.(5) The learning method according to any one of (1) to (4), wherein the intermediate transparency of the object displayed in the nth learning estimated frame is an average of the transparency of the object displayed in each of the plurality of nth estimated frames. (6) The learning method according to any one of (1) to (5), wherein the plurality of nth input frames include an input frame including the object with a transparency of 100% and an input frame including the object with a transparency of 0%. (7) The learning method according to any one of (1) to (6), wherein each of the input frames is an image obtained by rendering three-dimensional data showing one or more objects seen from a predetermined viewpoint.

Claims

1. A learning method for a machine learning model that generates first to Nth estimated frames by inputting first to Nth input frames (N is a natural number), wherein at least some of the first to Nth input frames include objects displayed with probabilistically different transparency, and a processor: acquires a plurality of nth (1≦n≦N) input frames generated by multiple renderings, acquires an nth learning input frame including the object displayed with an intermediate transparency obtained based on the transparency of each of the objects displayed in the plurality of nth input frames, acquires a plurality of the nth estimated frames generated by multiple renderings, acquires an nth learning estimated frame including the object displayed with an intermediate transparency obtained based on the transparency of each of the objects displayed in the plurality of nth estimated frames, and learns the machine learning model using a plurality of training data including the nth learning input frame and the nth learning estimated frame corresponding to the nth learning input frame.

2. The learning method of claim 1, wherein the machine learning model includes: an cumulative feature information output layer that receives the nth input frame and n-1th auxiliary information based on n-1th cumulative feature information which is image information indicating features of the 1st to n-1th input frames, and outputs the nth cumulative feature information indicating features of the 1st to nth input frames; and an estimated frame output layer that receives the nth cumulative feature information, and outputs the nth estimated frame.

3. The learning method according to claim 1 or 2, wherein the number of pixels of the nth estimated frame is equal to or greater than the number of pixels of the nth input frame.

4. A learning method as described in claim 1 or 2, wherein the intermediate transparency of the object displayed in the nth learning input frame is an average of the transparency of the object displayed in each of the multiple nth input frames.

5. A learning method according to claim 1 or 2, wherein the median transparency of the object displayed in the nth learning estimated frame is an average of the transparency of the object displayed in each of the multiple nth estimated frames.

6. The learning method according to claim 1 or 2, wherein the plurality of nth input frames include an input frame including the object having a transparency of 100% and an input frame including the object having a transparency of 0%.

7. A method according to claim 1 or 2, wherein each of the input frames is an image obtained by performing a rendering of three-dimensional data showing one or more objects as seen from a given viewpoint.

8. A machine learning model that generates first to Nth estimated frames, respectively, by receiving first to Nth input frames (N is a natural number), wherein at least some of the first to Nth input frames include an object displayed with probabilistically different transparency, and the machine learning model is trained with multiple training data including: an nth learning input frame including the object displayed with an intermediate transparency obtained based on the transparency of each of the objects displayed in a plurality of nth (1≦n≦N) input frames generated by multiple renderings; and an nth learning estimation frame corresponding to the nth learning input frame, including the object displayed with an intermediate transparency obtained based on the transparency of each of the objects displayed in the plurality of nth estimation frames generated by multiple renderings.

9. A program for causing a computer to function as: an input frame acquisition means for acquiring a plurality of nth (n is a natural number) input frames including an object displayed with probabilistically different transparency generated by multiple renderings; a learning input frame acquisition means for acquiring an nth learning input frame including the object displayed with an intermediate transparency acquired based on the transparency of each of the objects displayed in the plurality of nth input frames; an estimation frame acquisition means for acquiring a plurality of nth estimation frames including an object displayed with probabilistically different transparency generated by multiple renderings; a learning estimation frame acquisition means for acquiring an nth learning estimation frame including the object displayed with an intermediate transparency acquired based on the transparency of each of the objects displayed in the plurality of nth estimation frames; and a learning means for training the machine learning model using a plurality of training data including the nth learning input frame and the nth learning estimation frame corresponding to the nth learning input frame.

Images