Image processing system, image processing method, and program

Abstract

The at least one processor: acquires first to N-th input frames (42) (N is a natural number equal to or greater than 3) each having a number of input pixels equal to or greater than the prescribed number of initial pixels according to first to N-th frames-to-be-processed (40) having the number of initial pixels; acquires first to i-th estimation frames (44) (i is a natural number equal to or greater than 1 and equal to or smaller than N-2) each having a number of estimation pixels greater than the number of initial pixels on the basis of first to i-th input frames (42) and a first machine-learning model (510); and acquires (i+1)-th to j-th estimation frames (44) (j is a natural number equal to or greater than i+2 and equal to and smaller than N) on the basis of (i+1)-th to j-th input frames (42) and a second machine-learning model (520).

Description

Image processing system, image processing method and program

[0001] The present invention relates to an image processing system, an image processing method, and a program.

[0002] Conventionally, a technique (super-resolution) for estimating a high-quality image based on a low-quality image using a machine learning model has been known (see Non-Patent Document 1 below).

[0003] Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang. Learning a Deep Convolutional Network for Image Super-Resolution, in Proceedings of European Conference on Computer Vision (ECCV), 2014

[0004] The inventors of the present application have been studying a system (hereinafter sometimes referred to as the "reference technology") with the following recursive configuration in order to achieve super-resolution of moving images such as game screens. That is, this system inputs the current frame, i.e., the nth (nth) frame, and information on past frames, i.e., cumulative feature information indicating the features of frames 1 to n-1 (1st to n-1th) into a machine learning model to improve the image quality of the nth frame (see FIG. 2).

[0005] In this way, by using cumulative feature information that accumulates information on past frames in addition to the current frame for estimation, we can expect to improve the estimation accuracy of the machine learning model.

[0006] However, if one machine learning model is used to estimate both the early frames and the later frames, as in the above-mentioned reference technology, the accuracy of the estimation for the early frames will be lower than that for the later frames because not much information about past frames has been accumulated in the early frames. In particular, the decrease in estimation accuracy is more pronounced for the first frame because no information about past frames has been accumulated.

[0007] In view of the above-mentioned problems, an object of the present invention is to provide an image processing system, an image processing method, and a program that enable accurate estimation of high-quality frames even for early frames.

[0008] An image processing system according to the present invention is an image processing system including at least one processor, wherein the at least one processor acquires first to N-th input frames, each having a number of input pixels equal to or greater than a predetermined initial number of pixels, corresponding to first to N-th processing target frames (N is a natural number equal to or greater than 3), and acquires first to i-th estimated frames, each having a number of estimated pixels greater than the initial number of pixels, based on the first to i-th input frames (i is a natural number equal to or greater than 1 and N-2) and a first machine learning model, and The first machine learning model obtains the (i+1)-th to (j)-th estimated frames based on the (i+1)-th to (j)-th input frames (j is a natural number between i+2 and N) and a second machine learning model, and the first machine learning model outputs the n-th estimated frame and the n-th accumulated feature information indicating features of the 1st to n-th input frames based on the n-th input frame (n is a natural number between 1 and i), and the second machine learning model obtains the (i+1)-th to (j)-th estimated frames based on the (i+1)-th input frame and the features of the 1st to i-th input frames output from the first machine learning model. and outputs the (i+1)th estimated frame and the (i+1)th cumulative feature information indicating features of the first to (i+1)th input frames based on the i-th cumulative feature information indicating the number of input pixels, and outputs the m-th estimated frame and the m-th cumulative feature information indicating features of the first to m-th input frames based on the m-th input frame (m is a natural number between i+2 and j) and the (m-1)th cumulative feature information indicating features of the first to (m-1)th input frames, and the first machine learning model further outputs the m-th estimated frame and the m-th cumulative feature information indicating features of the first to m-th input frames based on the m-th input frame (m is a natural number between i+2 and j) and the (m-1)th cumulative feature information indicating features of the first to m-th input frames. and first to i-th training data respectively including the (i+1)th to j-th learning input frames having the input pixel count and the (i+1)th to j-th learning estimation frames having the estimated pixel count, and the second machine learning model is trained using the (i+1)th to j-th training data respectively including the (i+1)th to j-th learning input frames having the input pixel count and the (i+1)th to j-th learning estimation frames having the estimated pixel count, and is trained based on the ith learning cumulative feature information output from the first machine learning model that indicates the features of the first to i-th learning input frames.

[0009] 1 is a diagram showing an example of the hardware configuration of an image processing system; FIG. 2 is a diagram showing an overview of a reference technology; FIG. 3 is a diagram showing a schematic diagram of processing in the reference technology; FIG. 4 is a diagram showing an overview of an image processing system; FIG. 5 is a diagram showing an overview of an image processing system; FIG. 6 is a diagram showing an overview of an image processing system; FIG. 7 is a functional block diagram showing an example of functions realized in the image processing system; FIG. 8 is a diagram explaining processing in a rendering unit; FIG. 9 is a diagram explaining processing in an input frame acquisition unit; FIG. 10 is a flow diagram showing an example of the flow of processing executed in the image processing system; FIG. 11 is a flow diagram showing an example of the flow of processing executed in the image processing system;

[0010] Hereinafter, an example of an embodiment of an image processing system according to the present invention will be described with reference to the drawings.

[0011] 1 is a diagram showing an example of the hardware configuration of an image processing system 1. The image processing system 1 is a computer such as a game console. As shown in FIG. 1, the image processing system 1 includes a control unit 10, a storage unit 12, a communication unit 14, an operation unit 16, a display unit 18, and an audio output unit 19.

[0012] The control unit 10 includes a program control device such as a CPU that operates according to a program installed in the image processing system 1. The control unit 10 also includes a GPU (Graphics Processing Unit) that draws an image in a frame buffer based on graphics commands and data supplied from the CPU.

[0013] The storage unit 12 includes, for example, a main storage device such as a ROM or a RAM, and an auxiliary storage device such as a HDD or an SSD. The storage unit 12 stores programs executed by the control unit 10. The storage unit 12 stores, for example, game programs (game software) in addition to programs for realizing various functions of the image processing system 1 described below. The storage unit 12 also reserves an area for a frame buffer in which images are drawn by the GPU.

[0014] The communication unit 14 is a communication interface such as an Ethernet (registered trademark) module or a wireless LAN module.

[0015] The operation unit 16 is a user interface such as a keyboard, mouse, or game console controller, and receives operation inputs from the user and outputs signals indicating the contents of the inputs to the control unit 10 .

[0016] The display unit 18 is a display device such as a liquid crystal display or an organic EL display, and displays various images according to instructions from the control unit 10 .

[0017] The audio output unit 19 is, for example, a speaker, and outputs audio represented by audio data generated by the image processing system 1 .

[0018] In addition to the devices described above, the image processing system 1 may also include an optical disc drive that reads optical discs such as DVD-ROMs and Blu-ray (registered trademark) discs, a USB (Universal Serial Bus) port, and the like.

[0019] [2. Overview of the Reference Technology] Before describing the image processing system 1 according to this embodiment, a reference technology that is the premise of the image processing system 1 according to this embodiment will be described using Figures 2 and 3. Figure 2 is a diagram showing an overview of the reference technology. Figure 3 is a diagram showing a schematic diagram of the processing in the reference technology. Here, an example is shown in which the reference technology is used to improve the image quality of gameplay video in a game. Gameplay video is video generated in response to a game program executed by a control unit or user input accepted by an operation unit, and is composed of a plurality of still images (frames) that are time-series data. The processing performed by the reference technology is mainly as follows:

[0020] (1) Generation of Processing Frames First, the system according to the reference technology generates an image (processing frame) depicting one or more game objects by rendering three-dimensional data representing the game objects as viewed from a predetermined viewpoint. This processing frame is an image having a predetermined number of pixels (initial pixel count) and a predetermined image quality (initial image quality) (see FIG. 3 ). Processing frames are generated at predetermined time intervals. The number of pixels of the processing frame is, for example, 1920 x 1080 (1080p). Each generated processing frame is not displayed directly on the display unit 18, but is temporarily stored in the storage unit 12 for subsequent processing. Note that the following description mainly illustrates processing for the kth processing frame 20_k, but similar processing is also performed for other processing frames (i.e., k = 2, 3, ..., N).

[0021] (2) Obtaining an Input Frame The system according to the reference technology obtains a frame (input frame) 22_k having a number of pixels (number of input pixels) greater than the initial number of pixels based on the obtained processing target frame 20_k. The number of input pixels is, for example, 3840 × 2160 (4K). Specifically, the processing target frame 20_k is subjected to enlargement and interpolation processing to generate the input frame 22_k (see FIG. 3 ).

[0022] It should be noted that although input frame 22_k has a larger number of pixels than target frame 20_k, its image quality is not necessarily sufficiently improved. In other words, the image quality of a frame does not simply refer to the number of pixels (high resolution). The image quality of a frame may be evaluated based on, for example, a higher signal-to-noise ratio, better spatial frequency reproducibility, or higher temporal stability (fewer artifacts or flickering when multiple frames are displayed consecutively) compared to a reference frame, or a combination of these.

[0023] (3) Obtaining an Estimated Frame The system according to the reference technology inputs the input frame 22_k to the machine learning model 200 and obtains the estimated frame 24_k. The estimated frame 24_k is an image having the same number of pixels (estimated number of pixels) as the number of input pixels and an image quality (estimated image quality) that is equal to or higher than the initial image quality (see FIG. 3 ).

[0024] Here, in addition to the input frame 22_k, the (k-1)th auxiliary information 28_k-1 is input to the machine learning model 200 (see FIGS. 2 and 3). The auxiliary information 28_k-1 is information based on the (k-1)th cumulative feature information 26_k-1 that indicates the features of the first to (k-1)th input frames 22. Details of the cumulative feature information 26 and the auxiliary information 28 will be described later.

[0025] The machine learning model 200 is a model trained using a plurality of training data sets, each of which includes a training input frame having an input pixel count and a training estimation frame having an estimated pixel count and estimated image quality.

[0026] (4) Acquisition of Cumulative Feature Information The machine learning model 200 has an cumulative feature information output layer 202 that receives input frame 22_k and auxiliary information 28_k−1 and outputs k-th cumulative feature information 26_k indicating the features of the first to k-th input frames 22 (see FIG. 2 ). The system according to the reference technology acquires the k-th cumulative feature information 26_k.

[0027] The acquired k-th cumulative feature information 26_k is input to the estimated frame output layer 204, which outputs the k-th estimated frame 24_k (see FIG. 2 ). The acquired k-th cumulative feature information 26_k is also stored in the storage unit 12 and is used to estimate an estimated frame 24_k+1 corresponding to the next processing target frame (the (k+1)th processing target frame) 20_k+1.

[0028] (5) Acquisition of Auxiliary Information As described above, the k-1st accumulated feature information 26_k-1 is information that indicates the features of the first to k-1st input frames 22 (and thus the first to k-1st processing target frames 20). If the accumulated feature information 26_k-1, which accumulates information on past processing target frames 20, is used to estimate the kth estimated frame 24_k, the amount of information available for estimation increases, and therefore an estimated frame 24_k with high image quality can be obtained.

[0029] However, if there is movement or the like in the displayed game object between the k-1st processing frame 20_k-1 and the kth processing frame 20_k, when the kth input frame 22_k and the cumulative feature information 26_k-1 are input as is to the machine learning model 200, a phenomenon (the so-called ghost phenomenon) may occur in which an afterimage of the game object that was displayed in the k-1st processing frame 20_k-1 is displayed.

[0030] Therefore, the system according to the reference technology acquires the (k-1)th auxiliary information 28_k-1 by applying various corrections (described later) to the cumulative feature information 26_k-1 based on information obtained during rendering (such as a motion vector or a depth buffer) (see "auxiliary information generation unit 316" in FIG. 2 and FIG. 3). As described above, the acquired (k-1)th auxiliary information 28_k-1 is input to the machine learning model 200 together with the kth input frame 22_k, and is used to estimate the kth estimated frame 24_k.

[0031] As described above, according to the reference technique of this embodiment, the estimated frame 24 is estimated using auxiliary information 28, which stores past information, in addition to the input frame 22 corresponding to the current processing target frame 20. This increases the amount of information available for estimation, making it possible to obtain a high-quality estimated frame 24_.

[0032] 4A to 4C, an overview of the image processing system 1 will be described. Figures 4A to 4C are diagrams showing an overview of the image processing system 1. Note that, in the following, descriptions of configurations similar to those of the reference technology may be omitted.

[0033] According to the above-mentioned reference technology, cumulative feature information (auxiliary information) in which information on past frames is accumulated in addition to the current frame is used for estimation, thereby improving the estimation accuracy using a machine learning model.

[0034] However, if one machine learning model is used to estimate both the early frames and the later frames, as in the reference technology, the accuracy of the estimation for the early frames will be lower than that for the later frames because not much information about past frames has been accumulated in the early frames. In particular, the decrease in estimation accuracy is more pronounced for the first frame because no information about past frames has been accumulated.

[0035] Therefore, in the image processing system 1 according to this embodiment, a machine learning model (first machine learning model 510) that performs estimation for early frames and a machine learning model (second machine learning model 520) that performs estimation for frames after the early frames are separately prepared. This will be specifically described below.

[0036] (1) Processing of the First Input Frame First, the first machine learning model 510 outputs a first estimated frame 44_1 and first cumulative feature information 46_1 indicating features of the first input frame 42_1 based on the first input frame 42_1 (see FIG. 4A ). Here, given auxiliary information 48_0 (given feature information) is input to the first machine learning model 510 along with the first input frame 42_1. Note that in the image processing system 1, as in the reference technology, the first auxiliary information 48_1 is generated based on the first cumulative feature information 46_1.

[0037] The first machine learning model 510 is trained using first training data, which includes a first training input frame having an input pixel count and a first training estimation frame having an estimated pixel count.

[0038] (2) Processing of the Second Input Frame The second machine learning model 520 outputs the second estimated frame 44_2 and second cumulative feature information 46_2 indicating the features of the first and second input frames based on the second input frame 42_2 and the first cumulative feature information (first auxiliary information 48_1 in this embodiment) (see FIG. 4B ). Here, the first auxiliary information 48_1 input to the second machine learning model 520 is output from the first machine learning model 510, as described above.

[0039] As a result, information indicating the characteristics of the input frame 42 extracted by the first machine learning model 510 is passed on to the second machine learning model 520, so that the second machine learning model 520 can also use information indicating the characteristics of input frames 42 prior to the second input frame 42_2 for estimation.

[0040] (3) Processing of the nth input frame Thereafter, the second machine learning model 520 outputs the nth estimated frame 44_n and the nth cumulative feature information 46_n indicating the features of the first to mth input frames based on the nth input frame 42_n (n is a natural number greater than or equal to 3 and less than or equal to N) and the n-1th cumulative feature information indicating the features of the first to n-1th input frames (in this embodiment, the n-1th auxiliary information 48_n-1) (see Figure 4C).

[0041] The second machine learning model 520 is trained using second to Nth training data, each of which includes second to Nth training input frames having an input pixel count and second to Nth training estimation frames having an estimated pixel count. In this embodiment, the training of the second machine learning model 520 and the training of the first machine learning model 510 are performed independently of each other. The second machine learning model 520 is trained based on first cumulative learning feature information that indicates the features of the first training input frame and is output from the first machine learning model 510. That is, when the second machine learning model 520 is trained, a process similar to that described in (2) above is performed.

[0042] According to the above configuration, since estimation for the early frames and estimation for the later frames are performed using separate machine learning models, it is possible to perform estimation with high accuracy even for the early frames. Details of the image processing system 1 are described below.

[0043] 5 is a functional block diagram showing an example of functions realized by the image processing system 1. As shown in Fig. 5, the image processing system 1 realizes a game processing unit 600, a rendering unit 602, a rendering information storage unit 604, a processing target frame acquisition unit 606, a variation information acquisition unit 608, an input frame acquisition unit 610, a machine learning model storage unit 612, an estimated frame acquisition unit 614, and an auxiliary information generation unit 616. The auxiliary information generation unit 616 includes a motion information acquisition unit 6160, a depth information acquisition unit 6162, an appearance pixel identification unit 6164, and an auxiliary information acquisition unit 6166. The game processing unit 600, rendering unit 602, processing target frame acquisition unit 606, variation information acquisition unit 608, input frame acquisition unit 610, estimated frame acquisition unit 614, motion information acquisition unit 6160, depth information acquisition unit 6162, appearance pixel identification unit 6164, and auxiliary information acquisition unit 6166 are mainly realized by the control unit 10. The rendering information storage unit 604 and machine learning model storage unit 612 are mainly realized by the storage unit 12. Note that the game processing unit 600, rendering unit 602, and rendering information storage unit 604 are functions provided by game software.

[0044] [Game Processing Unit] The game processing unit 600 executes various processes related to the game. For example, in response to a game program executed by the control unit 10 or user input received by the operation unit 16, the game processing unit 600 executes processes such as placing a game object O in a virtual three-dimensional space VS, operating or moving the game object O, and changing the viewpoint C from which the virtual three-dimensional space VS is viewed (see FIG. 6 ). The game object O is composed of primitives such as polygons represented by three-dimensional data. The three-dimensional data includes geometric information indicating the positions of vertices, topological information indicating how the vertices are connected, and attribute information such as color.

[0045] [Rendering Unit] FIG. 6 is a diagram illustrating the processing of the rendering unit 602. The rendering unit 602 generates first to Nth (N is a natural number equal to or greater than 2) processing frames 40 by rendering (drawing) three-dimensional data representing one or more game objects O viewed from a predetermined viewpoint C. The rendering unit 602 performs rendering based on the results of various processes executed by the game processing unit 600. Specifically, the rendering unit 602 performs vertex processing (vertex shading) and pixel processing (pixel shading) based on three-dimensional data representing game objects O placed in the virtual three-dimensional space VS. The vertex processing includes a coordinate transformation process (perspective projection) from the view coordinate system to the screen coordinate system, and a numerical value related to a change in the viewpoint C is added to the perspective projection matrix (camera matrix) used in the coordinate transformation process, as described below. The rendering unit 602 may perform rendering based on light source information, depth information (depth buffer), texture information, normal information, and the like. In addition to the above processing, the rendering unit 602 may also execute processing to apply effects such as depth of field (DoF) and motion blur. The processing of the rendering unit 602 may be set as appropriate by a game software developer or the like. Here, the game software developer or the like may adjust the MIP of the texture according to the estimated number of pixels of the estimated frame 44 or the like. This makes it possible to suppress the occurrence of noise such as moiré in the estimated frame 44.

[0046] Here, the rendering unit 602 generates each processing frame 40 by performing rendering such that the viewpoint C varies for each processing frame 40. Here, even if the game processing unit 600 fixes the viewpoint C at a predetermined position, the rendering unit 602 varies the viewpoint C for each processing frame 40. As a result, as shown in FIG. 6 , the position of the displayed game object O varies in each processing frame 40_n, 40_n+1, and 40_n+2. In other words, the rendering unit 602 applies jitter when generating each processing frame 40. Specifically, the rendering unit 602 varies the viewpoint C for each processing frame 40 by adding a numerical value corresponding to a fractional pixel size that differs for each processing frame 40 to the perspective projection matrix. The rendering unit 602 varies the viewpoint C for each processing frame 40 according to a predetermined rule. For example, a Halton sequence can be used as such a rule.

[0047] [Rendering Information Storage Unit] The rendering information storage unit 604 stores information necessary for the rendering process in the rendering unit 602 and information obtained as a result of the rendering process. For example, the rendering information storage unit 604 stores the processing target frame 40. The rendering information storage unit 604 also stores variation information, motion information, and depth information. Details of the variation information, motion information, and depth information will be described later. In addition, the rendering information storage unit 604 may store parameters used for coordinate transformation, light source information, texture information, normal information, etc.

[0048] [Processing Frame Acquisition Unit] The processing frame acquisition unit 606 acquires each of the first to Nth processing frames 40. Specifically, the processing frame acquisition unit 606 acquires each of the first to Nth processing frames 40 stored in the rendering information storage unit 604.

[0049] [Variation Information Acquisition Unit] The variation information acquisition unit 608 acquires variation information. The variation information acquisition unit 608 acquires variation information stored in the rendering information storage unit 604. Specifically, the variation information is information indicating the amount of variation of the viewpoint C between before and after the variation. The information indicating the amount of variation can also be said to be a variation vector indicating the direction and distance of the variation. For example, since the above-mentioned Halton sequence includes information indicating the amount of variation of the viewpoint C, this information may be used as the variation information.

[0050] [Input Frame Acquisition Unit] The input frame acquisition unit 610 acquires first to Nth input frames 42, each having a number of input pixels equal to or greater than the initial number, corresponding to first to Nth (N is a natural number equal to or greater than 3) processing target frames 40 each having a predetermined initial number of pixels. In this embodiment, each input frame 42 has a number of input pixels greater than the initial number of pixels. That is, in this embodiment, each input frame 42 is an enlarged image of the processing target frame 40 corresponding to that input frame 42.

[0051] Specifically, the input frame acquisition unit 610 interpolates pixel values ​​at positions in the processing frame 40 that correspond to the pixels before the fluctuation, based on the fluctuation information and each pixel in each processing frame 40, to generate each input frame 42. Fig. 7 is a diagram illustrating the processing in the input frame acquisition unit 610. Fig. 7 illustrates an example of obtaining the n-th input frame 42_n. For example, as shown in Fig. 7, if the pixel center of a pixel in the input frame 42_n to be obtained is P １，０ In this case, the input frame acquisition unit 610 calculates P １，０ The pixel centers P' of the four pixels closest to ０，０ , P' １，０ , P' ０，１ , P' １，１ Based on the coordinates and pixel values ​​of P １，０ The pixel value of P' is calculated by bilinear interpolation. １，０ is the amount of fluctuation indicated by the fluctuation information. １，０The pixel values ​​of the pixels newly generated by the enlargement process are calculated in the same manner. Note that, in addition to bilinear interpolation, various other well-known interpolation methods such as bicubic interpolation and Lanczos interpolation can also be used as the interpolation method.

[0052] When rendering is performed so that the viewpoint C varies for each processing frame 40, the amount of time-series information increases. However, by using each processing frame 40 obtained in this way (hereinafter referred to as a "variable processing frame") for estimation, an estimated frame 44 with higher image quality can be obtained.

[0053] On the other hand, if the frame to be subjected to variation processing (or an enlarged image thereof) is input directly into the first machine learning model 510 or the second machine learning model 520, the accuracy of the estimation may be reduced due to the influence of the variation in viewpoint C described above.

[0054] Therefore, as described above, in the image processing system 1, pixel values ​​at positions in each processing frame 40 corresponding to each pixel before the variation are obtained by interpolation based on the variation information and each pixel in the processing frame 40, and each input frame 42 is generated and input to the first machine learning model 510 or the second machine learning model 520. This corrects for the influence of variation in the viewpoint C, thereby preventing a decrease in estimation accuracy.

[0055] [First Machine Learning Model] The first machine learning model 510 is a model that estimates a first estimated frame 44_1 based on a first input frame 42_1. Specifically, the first machine learning model 510 outputs a first estimated frame 44_1 based on the first input frame 42_1 and given auxiliary information 48_0. Note that the given auxiliary information 48_0 is data in the same format as the auxiliary information 48_1 and 48_n-1 described below. Specifically, the first machine learning model 510 is a convolutional neural network (CNN). As the first machine learning model 510, for example, a multi-layered ResNet with a residual connection mechanism or a so-called encoder-decoder U-Net can be used. The model described in Non-Patent Document 1 may be used as the first machine learning model 510.

[0056] The first machine learning model 510 is a model trained using first training data including a first training input frame having an input pixel count and a first training estimation frame having an estimated pixel count. More specifically, the first machine learning model 510 is trained using first training data including the first training input frame, given training auxiliary information, and the first training estimation frame having the estimated pixel count. Specifically, the first machine learning model 510 is trained based on the loss between the output when the first training input frame and the given training auxiliary information are input and the first training estimation frame. The first machine learning model 510 is trained so as to reduce the loss. Various known techniques, such as backpropagation, can be used to train the first machine learning model 510.

[0057] Specifically, the first machine learning model 510 includes an accumulated feature information output layer 512, an estimated frame output layer 514, and a convolutional layer 516 (see FIG. 4A).

[0058] The cumulative feature information output layer 512 receives the first input frame 42_1 and given auxiliary information 48_0, and outputs first cumulative feature information 46_1 indicating the features of the first input frame 42_1. The cumulative feature information output layer 512 may be composed of, for example, one or more convolutional layers. The cumulative feature information 46 is image information (bitmap information) having the same number of pixels as the number of input pixels. The cumulative feature information 46_1 can also be considered a feature map indicating the features of the first input frame 42_1.

[0059] The estimated frame output layer 514 receives the first accumulated feature information 46_1 and outputs the first estimated frame 44_1. Similar to the accumulated feature information output layer 512, the estimated frame output layer 514 may be composed of, for example, one or more convolutional layers. Alternatively, the estimated frame output layer 514 may be composed of one or more transposed convolutional layers (deconvolutional layers).

[0060] The convolution layer 516 is a layer that reduces the number of channels of the accumulated feature information 46 while maintaining the number of pixels of the accumulated feature information 46. The accumulated feature information 46 output from the convolution layer 516 is subjected to processing by the auxiliary information acquisition unit 6166. The convolution layer 516 reduces the dimensions of the accumulated feature information 46, thereby reducing calculation costs. The convolution layer 516 is, for example, a convolution layer with a kernel size of 1×1, but is not limited to this.

[0061] [Second Machine Learning Model] The second machine learning model 520 is a model that estimates second to Nth estimated frames 44 based on the second to Nth input frames 42. Specifically, the second machine learning model 520 outputs the second estimated frame 44_2 based on the second input frame 42_2 and the first cumulative feature information (in this embodiment, the first auxiliary information 48_1) output from the first machine learning model 510. The second machine learning model 520 also outputs the nth estimated frame 44_n based on the nth input frame 42_n (n is a natural number greater than or equal to 3 and less than or equal to N) and the n-1th cumulative feature information (in this embodiment, the n-1th auxiliary information 48_n-1) indicating the features of the first to n-1th input frames 42. The second machine learning model 520 is a convolutional neural network (CNN) similar to the first machine learning model 510. Known models such as a multi-layered ResNet with a residual connection mechanism or a so-called encoder-decoder U-Net can be used as the second machine learning model 520. The model described in Non-Patent Document 1 may also be used as the second machine learning model 520.

[0062] Specifically, the second machine learning model 520 includes an accumulated feature information output layer 522, an estimated frame output layer 524, and a convolutional layer 526 (see FIGS. 4B and 4C ). Note that the convolutional layer 526 has the same configuration as the convolutional layer 516, and therefore its description will be omitted.

[0063] The cumulative feature information output layer 522 receives the second input frame 42_2 and the first cumulative feature information (first auxiliary information 48_1) output from the first machine learning model 510, and outputs second cumulative feature information 46_2 indicating the features of the first to second input frames 42. The cumulative feature information output layer 522 also receives the nth input frame 42_n and the (n-1)th auxiliary information 48_n-1, and outputs nth cumulative feature information 46_n indicating the features of the first to nth input frames 42. The cumulative feature information output layer 522 may be composed of, for example, one or more convolutional layers. The cumulative feature information 46 is image information (bitmap information) having the same number of pixels as the number of input pixels. The nth cumulative feature information 46_n can also be considered a feature map indicating the features of the first to nth input frames 42.

[0064] The estimated frame output layer 524 receives the second accumulated feature information 46_2 and outputs the second estimated frame 44_2. The estimated frame output layer 524 also receives the nth accumulated feature information 46_n and outputs the nth estimated frame 44_n. Like the accumulated feature information output layer 522, the estimated frame output layer 524 may be composed of, for example, one or more convolutional layers. Alternatively, the estimated frame output layer 524 may be composed of one or more transposed convolutional layers (deconvolutional layers).

[0065] The second machine learning model 520 is trained based on first cumulative learning feature information indicating features of the first learning input frame output from the first machine learning model 510. Specifically, the second machine learning model 520 is trained based on the loss between the output when the second learning input frame and first learning auxiliary information based on the first cumulative learning feature information output from the first machine learning model 510 are input, and the second learning estimation frame. The second machine learning model 520 is trained so as to reduce the loss. Various known techniques, such as backpropagation, can be used to train the second machine learning model 520.

[0066] The second machine learning model 520 is trained using second to Nth training data, each of which includes second to Nth training input frames having an input pixel count and second to Nth training estimation frames having an estimated pixel count. Specifically, the second machine learning model 520 is trained based on the loss between the output when the nth training input frame and the (n-1)th learning auxiliary information based on the (n-1)th learning cumulative feature information indicating the features of the first to (n-1)th learning input frames are input, and the nth learning estimation frame.

[0067] In this embodiment, we will explain a case where the learning of the first machine learning model 510 and the learning of the second machine learning model 520 are performed independently of each other, but the first machine learning model 510 and the second machine learning model 520 may also be trained together.

[0068] [Machine Learning Model Storage Unit] The machine learning model storage unit 612 stores the first machine learning model 510 and the second machine learning model 520. Specifically, the machine learning model storage unit 612 stores parameters (such as the number of convolutional layers, the number of nodes used in each convolutional layer, and the weight of each node) of the first machine learning model 510 and the second machine learning model 520. Note that the first machine learning model 510 and the second machine learning model 520 have different parameters.

[0069] [Estimated Frame Acquisition Unit] The estimated frame acquisition unit 614 acquires the first estimated frame 44_1 based on the first input frame 42_1 and the first machine learning model 510. Specifically, the estimated frame acquisition unit 614 inputs the first input frame 42_1 and given auxiliary information 48_0 to the first machine learning model 510 and acquires the first estimated frame 44_1.

[0070] The estimated frame acquisition unit 614 also acquires the second to Nth estimated frames 44 based on the second to Nth input frames 42 and the second machine learning model 520. Specifically, the estimated frame acquisition unit 614 inputs the second input frame 42_2 and the first auxiliary information 48_1 to the second machine learning model 520 to acquire the second estimated frame 44_2. The estimated frame acquisition unit 614 also inputs the nth input frame 42_n and the (n-1)th auxiliary information 48_n-1 to the second machine learning model 520 to acquire the nth estimated frame 44_n. Note that in this embodiment, the estimated frame 44 has the same number of estimated pixels as the number of input pixels.

[0071] [Auxiliary Information Generation Unit] The auxiliary information generation unit 616 generates the (n-1)th auxiliary information 48_n-1 based on the (n-1)th cumulative feature information 46_n-1. The auxiliary information generation unit 616 also generates the first auxiliary information 48_1 based on the first cumulative feature information 46_1. The auxiliary information generation unit 616 includes a motion information acquisition unit 6160, a depth information acquisition unit 6162, an appearance pixel identification unit 6164, and an auxiliary information acquisition unit 6166.

[0072] [Motion Information Acquisition Unit] The motion information acquisition unit 6160 acquires the (n-1)th motion information, which is information indicating the amount and direction of motion from the (n-1)th processing frame 40_n-1 to the nth processing frame 40_n. Specifically, the (n-1)th motion information has the same number of pixels as the number of input pixels and is image information (bitmap format information) indicating the amount and direction of motion of each pixel between the (n-1)th processing frame 40_n-1 and the nth processing frame 40_n. In other words, the pixel value of each pixel in the (n-1)th motion information indicates the amount and direction of motion of each pixel between the (n-1)th processing frame 40_n-1 and the nth processing frame 40_n. That is, the pixel value of each pixel in the (n-1)th motion information is a two-dimensional vector indicating the amount and direction of motion of each pixel between the (n-1)th processing frame 40_n-1 and the nth processing frame 40_n. Motion information is also called a motion vector. Specifically, the motion information acquisition unit 6160 acquires original motion information having the same number of pixels as the initial number of pixels, and performs enlargement and interpolation processing on the original motion information to acquire motion information having the same number of pixels as the input pixel number.

[0073] Furthermore, the motion information acquisition unit 6160 acquires first motion information, which is information indicating the amount and direction of motion from the first processing target frame 40_1 to the second processing target frame 40_2.

[0074] [Depth Information Acquisition Unit] The depth information acquisition unit 6162 acquires n-1th depth information indicating the depth of each pixel in the n-1th processing target frame 40_n-1 and nth depth information indicating the depth of each pixel in the nth processing target frame 40_n. Specifically, the depth information is image information having the same number of pixels as the number of input pixels (information in bitmap format). The depth information is also called a depth buffer, Z buffer, or depth buffer. Specifically, the depth information acquisition unit 6162 acquires original depth information having the same number of pixels as the initial number of pixels, and performs enlargement and interpolation processing on the original depth information to acquire depth information having the same number of pixels as the number of input pixels.

[0075] Furthermore, the depth information acquisition unit 6162 acquires first depth information indicating the depth of each pixel of the first processing target frame 40_1.

[0076] [Appearance Pixel Identification Unit] The appearance pixel identification unit 6164 identifies the nth appearance pixel 422_n, which is a pixel of the nth input frame 42_n where all or part of the game object O, which is not displayed in the n-1th input frame 42_n-1, is displayed, based on the (n-1)th depth information and the nth depth information (see FIG. 5 ). Specifically, the appearance pixel identification unit 6164 identifies the nth appearance pixel 422_n based on the difference between the (n-1)th depth information and the nth depth information. Note that the appearance pixel identification unit 6164 may also identify the nth appearance pixel 422_n based on the (n-1)th perspective projection matrix associated with the (n-1)th input frame 42_n-1 and the nth perspective projection matrix associated with the nth input frame 42_n. Alternatively, the appearance pixel identification unit 6164 may identify the nth appearance pixel 422_n using the (n-1)th motion information. More specifically, the appearance pixel specifying unit 6164 specifies the nth appearance pixel 422_n and generates nth appearance pixel information, which is image information indicating the position of the nth appearance pixel 422_n.

[0077] In addition, based on the first depth information and the second depth information, the appearance pixel identification unit 6164 identifies a second appearance pixel 422_2 among the pixels of the second input frame 42_2, which is a pixel in which all or part of a game object O that is not displayed in the first input frame 42_1 is displayed.

[0078] [Auxiliary Information Acquisition Unit] The auxiliary information acquisition unit 6166 acquires the (n-1)th auxiliary information 48_n-1 by applying motion compensation to the (n-1)th cumulative feature information 46_n-1 based on the (n-1)th motion information. Motion compensation refers to, for example, a process of moving a pixel at position x in the (n-1)th cumulative feature information 46_n to position x' when a pixel at position x in the (n-1)th input frame 42_n-1 has moved to position x' in the nth input frame 42_n (see FIG. 3 ). That is, the auxiliary information acquisition unit 6166 acquires the (n-1)th auxiliary information 48_n-1 based on the (n-1)th motion information to which a pseudo-random number associated with the (n-1)th cumulative feature information 26_n-1 has been added, by setting the pixel values ​​of each of one or more pixels in the (n-1)th cumulative feature information 46_n-1 to pixels at positions moved in accordance with the amount and direction of the pixel's movement.

[0079] The auxiliary information acquisition unit 6166 acquires the first auxiliary information 48_1 by applying motion compensation to the first accumulated feature information 46_1 based on the first motion information.

[0080] If there is movement of the game object O between the nth processing frame 40_n and the n-1th processing frame 40_n-1, when obtaining the nth estimated frame 44_n, if the nth input frame 42_n and the n-1th accumulated feature information 46_n-1 are input directly into the machine learning model 500, a ghost phenomenon may occur in which an afterimage of the game object O that was displayed in the nth input frame 42_n is displayed in the output nth estimated frame 44_n.

[0081] Therefore, in the image processing system 1, as described above, motion compensation is applied to the (n-1)th accumulated feature information 46_n-1 based on the (n-1)th motion information to obtain the (n-1)th auxiliary information 48_n-1, and when obtaining the nth estimated frame 44_n, this (n-1)th auxiliary information 48_n-1 is input to the machine learning model 500. This makes it possible to suppress the ghost phenomenon described above.

[0082] Furthermore, the auxiliary information acquisition unit 6166 acquires the (n-1)th auxiliary information 48_n-1 by replacing the pixel value of the nth appearance pixel 422_n in the (n-1)th cumulative feature information 46_n-1 with a predetermined value. Specifically, the auxiliary information acquisition unit 6166 acquires the (n-1)th auxiliary information 48_n-1 by replacing the pixel value of the nth appearance pixel 422_n in the (n-1)th cumulative feature information 46_n-1 with a predetermined value based on the nth appearance pixel information. The predetermined value may be a constant value such as 0 (black), or may be the pixel value of the nth appearance pixel 422_n in the nth input frame 42_n.

[0083] The auxiliary information acquisition unit 6166 acquires the first auxiliary information 48_1 in such a way that the pixel value of the second appearance pixel 422_2 in the first cumulative feature information 46_1 is replaced with a predetermined value.

[0084] If all or part of a game object O that is not displayed in the n-1st processing frame 40_n-1 is displayed in the nth processing frame 40_n, and the nth input frame 42_n and the n-1st cumulative feature information 46_n-1 are input directly into the machine learning model 500 when obtaining the nth estimated frame 44_n, the aforementioned ghost phenomenon may occur in the output nth estimated frame 44_n.

[0085] Therefore, as described above, the image processing system 1 identifies the nth appearance pixel 422_n, which is a pixel in the nth input frame 42_n where all or part of the game object O that is not displayed in the (n-1)th input frame 42_n-1 is displayed, and then acquires the (n-1)th auxiliary information 48_n-1 by replacing the pixel value of the nth appearance pixel 422_n in the (n-1)th cumulative feature information 46_n-1 with a predetermined value. This makes it possible to suppress the ghost phenomenon described above.

[0086] 8A to 8C are flow diagrams showing an example of the flow of processing executed in the image processing system 1. The processing shown in Fig. 8A to 8C is executed by the control unit 10 operating in accordance with a program stored in the storage unit 12.

[0087] 8A , first, the control unit 10 acquires a first processing frame 40_1 (S800). The control unit 10 acquires a first input frame 42_1 based on the first processing frame 40_1 (S802). Then, the control unit 10 inputs the first input frame 42_1 and given auxiliary information 48_0 to the first machine learning model 510, and acquires a first estimated frame 44_1 and first cumulative feature information 46_1 (S804).

[0088] 8B, the control unit 10 acquires the second processing frame 40_2 (S806).The control unit 10 acquires the second input frame 42_2 based on the second processing frame 40_2 (S808).

[0089] The control unit 10 acquires first motion information (S810). The control unit 10 also acquires first depth information and second depth information (S812) and identifies the second appearance pixel 422_2 based on the first depth information and the second depth information (S814). The control unit 10 acquires first auxiliary information 48_1 based on the first cumulative feature information 46_1, the first motion information, and the second appearance pixel 422_2 (S816). The control unit 10 then inputs the second input frame 42_2 and the first auxiliary information 48_1 to the second machine learning model 520, and acquires the second estimated frame 44_2 and the second cumulative feature information 46_2 (S818).

[0090] Next, the control unit 10 acquires the n-th processing target frame 40_n (S820).The control unit 10 acquires the n-th input frame 42_n based on the n-th processing target frame 40_n (S822).

[0091] The control unit 10 acquires the (n-1)th motion information (S824). The control unit 10 also acquires the (n-1)th depth information and the nth depth information (S826), and identifies the nth appearance pixel 422_n based on the (n-1)th depth information and the nth depth information (S828). The control unit 10 acquires the (n-1)th auxiliary information 48_n-1 based on the (n-1)th cumulative feature information 46_n-1, the (n-1)th motion information, and the nth appearance pixel 422_n (S830). The control unit 10 then inputs the nth input frame 42_n and the (n-1)th auxiliary information 48_n-1 to the second machine learning model 520, and acquires the nth estimated frame 44_n and the nth cumulative feature information 46_n (S832). The control unit 10 determines whether or not a next frame exists (S834), and if it determines that a next frame exists (S834; Y), it increments n to n+1 and repeats the processes of S820 to S832. If the control unit 10 determines that a next frame does not exist (S834; N), it ends this process. Note that if it determines that a next frame does not exist (S834; N), the control unit 10 may cause the display unit 18 to display the first to Nth estimated frames 44 as they are.

[0092] [5. Summary] According to the image processing system 1 according to the present embodiment described above, the k-1st accumulated feature information 46_k-1 indicating the features of the first to k-1st input frames 42 (k = 2, 3, ..., N) is used to estimate the k-th estimated frame 44_k. That is, in addition to the information of the k-th processing target frame 40_k, the information of the first to k-1st processing target frames 40 can be used for estimation, increasing the amount of information available for estimation and enabling the acquisition of a high-quality estimated frame 44_k.

[0093] Furthermore, according to the image processing system 1 of this embodiment, estimations for the early frames and estimations for the frames after the early frames are performed using separate machine learning models, so that accurate estimations can be made even for the early frames.

[0094] It should be noted that the present invention is not limited to the above-described embodiment. Furthermore, the specific character strings and numerical values ​​described above and the specific character strings and numerical values ​​in the drawings are merely examples, and the present invention is not limited to these character strings and numerical values.

[0095] For example, in the present embodiment, the number of input pixels is greater than the initial number of pixels, and the number of input pixels is equal to the estimated number of pixels, but the number of input pixels may be equal to the initial number of pixels, and the estimated number of pixels may be greater than the number of input pixels. In other words, the input frame 42 does not necessarily have to be an enlarged version of the processing target frame 40.

[0096] Furthermore, the processing target frame 40 may be input directly to the machine learning model 500 .

[0097] In addition, in this embodiment, the cumulative feature information 46 is processed into auxiliary information 48 and then input to the first machine learning model 510 or the second machine learning model 520, but the cumulative feature information 46 may be input directly to the first machine learning model 510 or the second machine learning model 520.

[0098] Furthermore, in the present embodiment, a case has been described in which only the first input frame 42_1 is input to the first machine learning model 510, and the second to Nth input frames 42 are input to the second machine learning model 520. However, this is not limiting. For example, the first to third input frames 42 may be input to the first machine learning model 510, and the fourth to Nth input frames 42 may be input to the second machine learning model 520. In short, the first machine learning model 510 may estimate the first to ith estimated frames 44 based on the first to ith input frames 42 (i is a natural number between 1 and N-2). Furthermore, the second machine learning model 520 may estimate the i+1th to jth estimated frames 44 based on the i+1th to jth input frames 42 (j is a natural number between i+2 and N).

[0099] In addition, in the present embodiment, the image processing system 1 includes the first machine learning model 510 and the second machine learning model 520. However, the image processing system 1 may include more machine learning models. For example, the image processing system 1 may include a third machine learning model.

[0100] Furthermore, in this embodiment, the image processing system 1 is applied to game moving images, but the image processing system 1 is not limited to game moving images and can be applied to general moving images.

Claims

1. An image processing system including at least one processor, wherein the at least one processor: acquires first to Nth input frames having a number of input pixels equal to or greater than a predetermined initial number of pixels according to first to Nth processing target frames (N is a natural number equal to or greater than 3), the number of input pixels being equal to or greater than the initial number of pixels; acquires first to ith estimated frames having a number of estimated pixels greater than the initial number of pixels based on the first to ith input frames (i is a natural number equal to or greater than 1 and equal to or less than N-2) and a first machine learning model; acquires the i+1th to jth estimated frames based on the i+1th to jth input frames (j is a natural number equal to or greater than i+2 and equal to or less than N), the first machine learning model outputs the nth estimated frame and the nth accumulated feature information indicating features of the first to nth input frames based on the nth input frame (n is a natural number equal to or greater than 1 and equal to or less than i); and the second machine learning model: based on the (i+1)th input frame and the i-th cumulative feature information output from the first machine learning model and indicating features of the first to i-th input frames, outputting the (i+1)th estimated frame and the (i+1)th cumulative feature information indicating features of the first to i+1th input frames; and1. An image processing system comprising: a computer-readable storage device for storing image data of a first machine learning model having a first pixel count and a first cumulative feature information indicating features of the first to m-1th input frames, the computer outputting the m-th estimated frame and the m-th cumulative feature information indicating features of the first to m-1th input frames based on a first cumulative feature information indicating features of the first to m-1th input frames; the first machine learning model is further trained with first to i-th training data including first to i-th learning input frames having the input pixel count and first to i-th learning estimation frames having the estimated pixel count, respectively; and the second machine learning model is further trained with i+1-th to j-th training data including i+1-th to j-th learning input frames having the input pixel count and i+1-th to j-th learning estimation frames having the estimated pixel count, respectively, and the second machine learning model is trained based on the i-th learned cumulative feature information output from the first machine learning model and indicating features of the first to i-th learning input frames.

2. The image processing system of claim 1, wherein the first machine learning model receives the first input frame and given feature information, and outputs the first estimated frame and the first accumulated feature information.

3. A processor acquires first to Nth input frames, each having a number of input pixels equal to or greater than a predetermined initial number of pixels, corresponding to first to Nth processing frames (N is a natural number equal to or greater than 3), each having a number of input pixels equal to or greater than an initial number of pixels; acquires first to ith estimated frames, each having a number of estimated pixels greater than the initial number of pixels, based on the first to ith input frames (i is a natural number equal to or greater than 1 and equal to or less than N-2) and a first machine learning model; acquires i+1th to jth estimated frames, each based on the i+1th to jth input frames (j is a natural number equal to or greater than i+2 and equal to or less than N) and a second machine learning model; the first machine learning model outputs the nth estimated frame and the nth accumulated feature information indicating features of the first to nth input frames based on the nth input frame (n is a natural number equal to or greater than 1 and equal to or less than i); and the second machine learning model: outputting the (i+1)th estimated frame and the (i+1)th cumulative feature information indicating features of the first to (i+1)th input frames based on the (i+1)th input frame and the i'th cumulative feature information output from the first machine learning model and indicating features of the first to (i+1)th input frames; outputting the m'th estimated frame and the m'th cumulative feature information indicating features of the first to (m)th input frames based on the m'th input frame (m is a natural number equal to or greater than i+2 and equal to or less than j) and the (m-1)th cumulative feature information indicating features of the first to (m-1)th input frames; the first machine learning model is further trained using first to i'th training data, the first to i'th training data respectively including first to i'th learning input frames having the input pixel count and first to i'th learning estimation frames having the estimated pixel count; and the second machine learning model is further the image processing method being learned using the (i+1)-th to (j)-th training data, each of which includes the (i+1)-th to (j)-th learning input frames having the input pixel count and the (i+1)-th to (j)-th learning estimated frames having the estimated pixel count, and being learned based on the i-th learning accumulated feature information, which is output from the first machine learning model and indicates features of the 1st to i-th learning input frames.

4. The first to Nth processing frames (N is a natural number of 3 or more) each having a predetermined initial pixel number are a first to i-th estimated frame acquisition means for acquiring first to i-th estimated frames each having a number of input pixels greater than the initial number of pixels based on the first to i-th input frames (i is a natural number greater than or equal to 1 and less than N-2) and a first machine learning model, and a second machine learning model for acquiring the i+1 to j-th estimated frames based on the i+1 to j-th input frames (j is a natural number greater than or equal to i+2 and less than N), wherein the first machine learning model outputs the n-th estimated frame and the n-th accumulated feature information indicating features of the first to n-th input frames based on the n-th input frame (n is a natural number greater than or equal to 1 and less than i), and the second machine learning model outputs the n-th estimated frame and the n-th accumulated feature information indicating features of the first to n-th input frames based on the n-th input frame (n is a natural number greater than or equal to 1 and less than i), based on the (i+1)th input frame and the i-th cumulative feature information output from the first machine learning model and indicating features of the first to i-th input frames, outputting the (i+1)th estimated frame and the (i+1)th cumulative feature information indicating features of the first to i+1th input frames; and) and the m-1st cumulative feature information indicating features of the 1st to m-1th input frames, wherein the first machine learning model is further trained with first to i-th training data including first to i-th learning input frames having the input pixel count and first to i-th learning estimation frames having the estimated pixel count, respectively; and the second machine learning model is further trained with the i+1-th to j-th training data including the i+1-th to j-th learning input frames having the input pixel count and the i+1-th to j-th learning estimation frames having the estimated pixel count, respectively, and trained based on the i-th learning cumulative feature information output from the first machine learning model indicating features of the 1st to i-th learning input frames.

Images