Image processing method, image processing apparatus, and imaging apparatus

Abstract

To perform highly accurate machine learning tasks using a motion vector regardless of image size that a machine learning model for generating motion vectors has learned.SOLUTION: An image processing method includes: reducing a first image and a second image including at least a part of the same object in different positions, to generate a third image corresponding to the first image and a fourth image corresponding to the second image; generating a first motion vector based on the third and fourth images, using a first machine learning model; enlarging the first motion vector to generate a second motion vector; and generating a fifth image based on the first and second images and the second motion vector, using a second machine learning model.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technique for performing machine learning tasks using motion vectors generated using a machine learning model. [Background technology]

[0002] As an image processing method using motion vectors, Non-Patent Document 1 discloses a method in which a reference frame and an adjacent frame included in a video are input into a first machine learning model to generate a motion vector between the reference frame and the adjacent frame, and the motion vector is enlarged by bilinear interpolation. In this method, the enlarged motion vector, the reference frame, and the adjacent frame upscaled by a second machine learning model are input into a second machine learning model to upscale the reference frame. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Mehdi SM Sajjadi, Raviteja Vemulapalli, and Matthew Brown. Frame-recurrent video super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6626-6634, 2018. Summary of the Invention [Problem to be solved by the invention]

[0004] In the method disclosed in Non-Patent Document 1, the size of a reference frame that can achieve highly accurate upscaling is limited based on the image size trained by a first machine learning model. Specifically, the size of an image input to a first machine learning model that generates a motion vector is limited based on the image size trained by the first machine learning model. Therefore, the size of an image input to a second machine learning model that uses the motion vector generated using the first machine learning model is also limited based on the image size trained by the first machine learning model.

[0005] The present invention provides an image processing method and an image processing device that are capable of performing high-precision machine learning tasks using motion vectors without being limited by the image size on which the machine learning model that generates the motion vectors is trained. [Means for solving the problem]

[0006] An image processing method according to one aspect of the present invention includes the steps of: reducing a first image and a second image, each image including at least a portion of the same object at a different position, to generate a third image corresponding to the first image and a fourth image corresponding to the second image; generating a first motion vector based on the third image and the fourth image using a first machine learning model; enlarging the first motion vector to generate a second motion vector; and generating a fifth image based on the first image, the second image, and the second motion vector using a second machine learning model. Note that an image processing device that performs processing according to the image processing method, an imaging device including the image processing device, and a program that causes a computer to execute the processing also constitute other aspects of the present invention. [Effects of the Invention]

[0007] According to the present invention, a machine learning model that generates motion vectors can be used to perform highly accurate machine learning tasks using motion vectors without being limited by the image size on which the machine learning model has been trained. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 4 is a diagram showing the flow of an estimation phase in the first embodiment. [Figure 2] FIG. 1 is a diagram showing a conventional flow of executing a machine learning task using motion vectors generated by a machine learning model. [Figure 3] FIG. 1 is a block diagram showing the configuration of an image processing system according to a first embodiment. [Figure 4] FIG. 1 is an external view of an image processing system according to a first embodiment. [Figure 5] 10 is a flowchart showing a process for generating learning data for a first machine learning model in the first embodiment. [Figure 6] 1 is a flowchart showing the process of learning weights of a first machine learning model (first learning phase) in the first embodiment. [Figure 7] 10 is a flowchart showing a process for generating learning data for a second machine learning model in the first embodiment. [Figure 8] 10 is a flowchart showing the process of learning weights of a second machine learning model (second learning phase) in the first embodiment. [Figure 9] 10 is a flowchart relating to an estimation phase in the first embodiment. [Figure 10] FIG. 10 is a block diagram showing the configuration of an image processing system according to a second embodiment. [Figure 11] FIG. 10 is an external view of an image processing system according to a second embodiment. [Figure 12] 10 is a flowchart showing a process for generating learning data for a second machine learning model in the second embodiment. [Figure 13] 10 is a flowchart showing the process of learning weights of a second machine learning model (second learning phase) in the second embodiment. [Figure 14] FIG. 10 is a diagram showing the flow of an estimation phase in the second embodiment. [Figure 15] 10 is a flowchart showing processing in an estimation phase in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0010] Here, before specifically describing Examples 1 to 3, matters common to each of the Examples will be described.

[0011] In each embodiment, "size" refers to the number of pixels in the width and height of an image or map, and "motion vector" refers to a vector representing the movement of corresponding pixels between an image pair (two frame images).

[0012] First, the problems to be solved by each embodiment will be described. In a machine learning model that generates a motion vector within an image pair based on an image pair, the size of the image pair is limited based on the image size trained by the machine learning model. Specifically, when a motion vector is generated based on an image pair whose size is larger than a threshold multiple of the trained image size, the accuracy of the motion vector (optical flow) is significantly impaired. The threshold value is a value in the range of 1 to 2, although it varies depending on the model structure of the machine learning model and the amount of movement of objects contained in the image pair. For example, when an image pair of 256×256 pixels is input to a machine learning model that has been trained to generate a motion vector by inputting an image pair of 128×128 pixels, the accuracy of the generated motion vector is significantly impaired.

[0013] The reason for this is explained below. A machine learning model that performs the task of generating motion vectors is trained to generate motion vectors for each pixel in an image pair by taking into account a wide range of surrounding pixels. At this time, the contribution of each surrounding pixel included in the receptive field of the machine learning model is optimized according to the size of the image pair used for training. In other words, the contribution of each surrounding pixel in the receptive field is automatically adjusted based on the size of the image pair given during training, and training is performed so that a wide range of surrounding pixels in the receptive field have a high contribution in the task of generating motion vectors.

[0014] On the other hand, when generating motion vectors based on image pairs larger than the image size used for training, the receptive field is expanded compared to training, and the generated motion vectors are influenced by an unnecessarily wide range of neighboring pixels, resulting in a significant loss of accuracy.

[0015] In this way, in a machine learning model that generates motion vectors based on image pairs, the size of the image pairs is limited based on the image size learned by the machine learning model. Accordingly, the image size in a machine learning task that uses the motion vectors generated by the machine learning model is also limited based on the image size learned by the machine learning model.

[0016] FIG. 2 shows a conventional process for executing a machine learning task using motion vectors generated by a machine learning model. FIG. 2 shows the simplest conventional process for generating an output image 104 from an image pair, images 101 and 102. In this conventional process, images 101 and 102 are first input to machine learning model 111, which generates a motion vector 103 between images 101 and 102. Next, images 101 and 102 and the motion vector 103 are input to machine learning model 112, which generates image 104. For example, when machine learning model 112 generates image 104 from images 101 and 102, each of which is 256×256 pixels, it is necessary to also input images 101 and 102, each of which is 256×256 pixels, to machine learning model 111 to generate motion vector 103.

[0017] On the other hand, if the machine learning model 111 is trained to generate a motion vector by inputting an image pair of 128×128 pixels, the accuracy of the generated motion vector 103 will be significantly impaired. As a result, the machine learning model 112 will not be able to generate the image 104 with high accuracy. Details of this will be described in each embodiment.

[0018] Each embodiment solves the above problem by showing a method for accurately executing a machine learning task using a motion vector generated by a machine learning model without being limited by the image size that the machine learning model used to generate the motion vector has learned. Specifically, a first motion vector is generated using a first machine learning model based on a third image and a fourth image that are obtained by reducing the first image and the second image, respectively.

[0019] In each embodiment, a fifth image is generated using a second machine learning model based on a second motion vector obtained by enlarging the first motion vector, the first image, and the second image. Here, the first image and the second image are both images of the same third size, and the third image and the fourth image are both images of the same first size. The first machine learning model can generate the first motion vector with high accuracy at a first size that is limited based on the image size trained by the first machine learning model. Meanwhile, the second machine learning model can generate the fifth image with high accuracy at a third size that is not limited based on the image size trained by the first machine learning model.

[0020] Each embodiment will be described in detail below. In each embodiment, a first image and a second image are first reduced to generate a third image corresponding to the first image and a fourth image corresponding to the second image. The first image and the second image are a pair of images in which at least a portion of the same object is included in different positions. Note that the first image and the second image may be images extracted from the same video.

[0021] The reduction from the first image to the third image and the reduction from the second image to the fourth image are performed using the same reduction process that converts images, such as downsampling, which extracts only one pixel from multiple pixels, or binning, which uses multiple pixels to generate a new pixel value for one pixel.

[0022] Next, in each embodiment, a first motion vector is generated based on the third image and the fourth image using a first machine learning model. Here, the first motion vector is a vector that represents the movement of corresponding pixels between the third image and the fourth image. For example, the first motion vector is a vector that indicates the movement of each pixel in the third image from the fourth image to the third image. Also, the first motion vector may be generated by inputting the third image and the fourth image into the first machine learning model.

[0023] The first machine learning model is, for example, a convolutional neural network (CNN). However, the first machine learning model may be trained to generate a motion vector based on a first training image set consisting of a plurality of images having a second size. The second size is preferably equal to or larger than the first size (the size of the third and fourth images). As described above, when a motion vector is generated based on an image pair whose size is larger than a threshold multiple of the second size, the accuracy of the generated motion vector is significantly impaired. The threshold value varies depending on the model structure of the first machine learning model, the amount of movement of an object included in the image pair, and other factors, but is a value in the range of 1 to 2. By setting the first size equal to or smaller than the second size (i.e., setting the second size equal to or larger than the first size), the first machine learning model can generate a highly accurate first motion vector regardless of the first machine learning model itself or the third and fourth images.

[0024] Next, in each embodiment, the first motion vector is enlarged by an enlargement process to generate a second motion vector. The enlargement process in each embodiment is a process independent of the first machine learning model. This enlargement process may be an enlargement process using a machine learning model, or may be an enlargement process without using a machine learning model, as long as it is independent of the first machine learning model. The enlargement process using a machine learning model is, for example, a process using one or more deconvolution layers or a subpixel convolution process. The subpixel convolution process is a process that performs enlargement by rearranging pixels after a convolution operation. However, these enlargement processes use weights that are not learned based on the first machine learning model. For enlargement processes without using a machine learning model, a known interpolation method such as nearest neighbor interpolation, bilinear interpolation, or bicubic interpolation is used. Furthermore, the enlargement magnification in this enlargement process may be the same as the reduction magnification in the reduction process that reduces the first image and the second image to a third image and a fourth image, respectively. Furthermore, the second motion vector may have a third size that is the same as that of the first image and the second image.

[0025] Finally, in each embodiment, a fifth image is generated based on the first image, the second image, and the second motion vector using a second machine learning model. The second machine learning model is, for example, a CNN. The fifth image may also be generated by inputting the first image, the second image, and the second motion vector into the second machine learning model. Alternatively, the fifth image may be generated by processing the first image, the second image, or the second motion vector and then inputting them into the second machine learning model. For example, the fifth image may be generated by enlarging the first image and the second image in advance using an interpolation process or the like, and then inputting the enlarged first image, the enlarged second image, and the second motion vector into the second machine learning model.

[0026] Additionally, the second machine learning model may be trained to generate images based on a second training image set of images having a fourth size, where the third size (the size of the first image and the second image) may be greater than or equal to the fourth size (i.e., the fourth size may be less than or equal to the third size).

[0027] The effects obtained by the processing of each embodiment will be described in comparison with the conventional processing shown in FIG. 2. As described above, the conventional processing shown in FIG. 2 is the simplest processing for generating image 104 from image 101 and image 102. In order to generate image 104 with high accuracy using this conventional processing, the image sizes of images 101 and 102 (corresponding to the third size) are limited based on the size of the image used by machine learning model 111 for training (corresponding to the second size). Furthermore, in the conventional processing, since training of machine learning model 112 is also performed according to the flow shown in FIG. 2, the size of the image used by machine learning model 112 for training (corresponding to the fourth size) is also limited based on the size of the image used by machine learning model 111 for training (corresponding to the second size). Regardless of the machine learning model 112 itself or the image used by machine learning model 112 for training, ideally, the fourth size is limited to be equal to or less than the second size in order to perform effective training of machine learning model 112.

[0028] On the other hand, it is preferable that the size of the image used for learning by the machine learning model 112 is as large as possible. This is because the larger the image size, the more accurately the machine learning model 112 can generate image 104 for larger sizes of images 101 and 102. Therefore, the fourth size is set equal to the second size. For the above reasons, in conventional processing, the image sizes of images 101 and 102 (corresponding to the third size) are limited based on the image size (corresponding to the fourth size) learned by the machine learning model 112.

[0029] In each embodiment, the second machine learning model may be trained in a manner similar to that of conventional processing. That is, the second machine learning model may be trained to generate new images based on an image pair and a motion vector generated by inputting the image pair into the first machine learning model. In this case, the size of the image pair used for training by the second machine learning model (fourth size) is limited to the second size trained by the first machine learning model, as in conventional processing, and is set equal to the second size for the same reasons as in conventional processing.

[0030] On the other hand, in each embodiment, as described above, the sizes of the first image and the second image (third size) are not limited based on the second size. Therefore, the third size is not limited by the fourth size, which is equal to the second size. Therefore, the third size may be equal to or larger than the fourth size. By allowing the third size to be equal to or larger than the fourth size, even if the fourth size is limited by the computational resources during learning, it is possible to set an optimal third size for the second machine learning model and the processes before and after the task performed by the second machine learning model. In other words, the degree of freedom in the third size is improved. Details of this will be described in Example 1.

[0031] In the following embodiments, the stage of determining weights for a machine learning model is referred to as a learning phase. Furthermore, the stage of generating a fifth image from a first image and a second image using a first machine learning model and a second machine learning model that use weights determined by learning is referred to as an estimation phase. Examples of machine learning models include neural networks, genetic programming, and Bayesian networks. Examples of neural networks include convolutional neural networks (CNNs), generative adversarial networks (GANs), and recurrent neural networks (RNNs). [Example]

[0032] The first embodiment aims to increase the resolution of a moving image. A plurality of low-resolution frame images included in a low-resolution moving image are increased in resolution, and the increased resolution frame images are combined to generate a high-resolution moving image. To achieve this, in this embodiment, a first image, which is at least a portion of the low-resolution frame image, and a second image, which is at least a portion of a low-resolution frame image adjacent to the first image, are reduced in size to generate a third image and a fourth image. Furthermore, a first motion vector is generated based on the third image and the fourth image using a first machine learning model. Furthermore, a fifth image, which corresponds to the first image and has a higher resolution than the first image, is generated based on a second motion vector generated by enlarging the first motion vector, the first image, and the second image using a second machine learning model. The fifth image is at least a portion of the high-resolution frame image.

[0033] In the first embodiment, a high-resolution video is generated by increasing the resolution of an original low-resolution video, with a size twice as large in both the width and height directions of the video (i.e., four times larger in total). The generated high-resolution video has a sampling pitch twice as small as that of the original low-resolution video. In the present embodiment, increasing the resolution means increasing the pixel count. The fifth image is an upscaled version of the first image, and the size of the fifth image is larger than that of the first image. However, increasing the resolution is not limited to increasing the pixel count. For example, the fifth image may have the same size as the first image, and may be an image obtained by removing noise, blur, shake, etc. from the first image.

[0034] In this embodiment, a fifth image is generated using a first image and a second image that is adjacent in time. This allows a fifth image with higher resolution to be generated compared to generating the fifth image using only the first image. This is because information about objects not present in the first image can be supplemented from the second image, which contains information about the same objects.

[0035] According to this embodiment, the following effects can be obtained. The first machine learning model can generate a first motion vector with high accuracy for the sizes (first size) of the third and fourth images that are restricted based on the image size (second size) on which the first machine learning model has learned. In this embodiment, by using a highly accurate first motion vector, the second image can be accurately aligned within the second machine learning model so that it becomes an image equivalent to the first image (so that each pixel of the second image is aligned with the position of the corresponding pixel in the first image). Therefore, object information contained in the second image can be aggregated with higher accuracy. This is particularly effective when the receptive field of the second machine learning model is small. Furthermore, this embodiment is effective for the sizes (third size) of the first and second images that are not restricted based on the second size.

[0036] In this embodiment, an example will be described in which a fifth image is generated by increasing the resolution of the first image using a second image that is temporally adjacent to the first image in addition to the first image, but more preferably, more images that are temporally adjacent to the first image are used, which makes it possible to supplement information about objects that are not present in the first image from more images and generate a fifth image with higher resolution.

[0037] Fig. 3 shows the configuration of an image processing system 200 in Example 1. Fig. 4 shows the external appearance of the image processing system 200. The image processing system 200 includes a learning device 201, an imaging device 202, and a network 203. The learning device 201 and the imaging device 202 are connected to each other via the wired or wireless network 203.

[0038] The learning device 201 is configured by a computer such as a personal computer, and has a storage unit 211, an acquisition unit 212, a generation unit 213, and an update unit 214, and operates according to a program to determine the weights of the machine learning model.

[0039] The imaging device 202 has an optical system 221, an imaging element 222, an image estimation unit 223 as an image processing device, a memory unit 224, a recording medium 225, a display unit 226, and a system controller 227. The optical system 221 forms an object image by collecting light incident from a space in which an object exists. The optical system 221 has functions such as zoom, aperture, and autofocus as necessary. The imaging element 222 generates a captured video as image data by converting the object image formed by the optical system 221 into an electrical signal (i.e., by capturing an image of the object through the optical system 221). The imaging element 222 is configured with a CCD (Charge Coupled Device) sensor, a CMOS (Complementary Metal-Oxide Semiconductor) sensor, or the like.

[0040] The image estimation unit 223 is configured with a computer such as a CPU or MPU, and operates according to a program to increase the resolution of the captured video generated by the optical system 221 and the image sensor 222. To achieve this, the image estimation unit 223 generates a first motion vector using a first machine learning model based on a third image and a fourth image obtained by reducing a first image and a second image, which are part of the frame images constituting the captured video. Furthermore, the image estimation unit 223 generates a fifth image using a second machine learning model based on a second motion vector obtained by increasing the first motion vector, the first image, and the second image. The image estimation unit 223 then generates a high-resolution video by increasing the resolution of the captured video using the fifth image. To generate the fifth image, weights of the machine learning model determined in advance by the learning device 201 are used. The weights are stored in the storage unit 224. The image estimation unit 223 includes an acquisition unit 223a, a calculation unit 223b, and an estimation unit 223c. Details of the processing performed by the image estimation unit 223 will be described later.

[0041] The recording medium 225 records the high-resolution video. The display unit 226 displays the high-resolution video when a user issues an instruction to output the high-resolution video. The above operations are controlled by the system controller 227.

[0042] The processing performed in this embodiment can be broadly divided into the generation of training data for the first machine learning model, training the weights of the first machine learning model (first training phase), the generation of training data for the second machine learning model, training the weights of the second machine learning model (second training phase), and estimation using the first machine learning model and the second machine learning model using the trained weights (estimation phase).

[0043] First, the process of generating training data for a first machine learning model performed by the learning device 201 will be described using the flowchart in FIG. 5. In this embodiment, the first machine learning model is trained by unsupervised learning, which does not require correct answer data. The training data is a first training image set, which is used to train the first machine learning model. In the next first learning phase, the first training image set is input to the first machine learning model.

[0044] In this embodiment, the learning data for the first machine learning model is generated by the learning device 201, but may be generated by another device.

[0045] In step S101, the acquisition unit 212 acquires a first image set from the storage unit 211. The first image set includes one or more first image pairs. One image of the first image pair includes at least a portion of an object included in the other image of the first image pair, but at a different position from the first image. Each image constituting a first image pair in the first image set may constitute a different first image pair from other images included in the first image set. The first image set may include captured images or CG (Computer Graphics) images. For example, the first image set may include multiple frame images extracted from a captured video. The first image set may also be a public dataset such as a REDS (Realistic and Diverse Scenes) dataset.

[0046] The first image set preferably includes images containing a variety of objects, such as edges, textures, gradients, or flat areas with various intensities and directions, thereby improving the robustness of the first machine learning model with respect to objects contained in the third and fourth images.

[0047] Furthermore, the first image set preferably includes images that have image quality degradations that the third and fourth images have. The image quality degradations include, for example, jaggies in contours and edges, spatial aliasing, compression artifacts, and noise. This improves the robustness of the first machine learning model against the image quality degradations that the third and fourth images have.

[0048] Additionally, it is preferable that one image of the first image pair is an image in which multiple objects included in the other image of the first image pair have moved by different amounts or directions of movement. That is, it is preferable that the two images in the first image pair include multiple objects with different amounts or directions of movement. Furthermore, it is also preferable that the multiple first image pairs included in the first image set include different amounts or directions of movement of the objects. This can improve the robustness of the first machine learning model against movement included between the third image and the fourth image.

[0049] Next, in step S102, the generation unit 213 generates a first training image set. Then, this process ends. The first training image set includes one or more first training image pairs. Each first training image pair is a pair of images having a predetermined size (second size), and in this embodiment, the second size is 128 × 128 pixels. Furthermore, one image of the first training image pair includes at least a portion of an object included in the other image of the first training image pair, but at a different position from the one image.

[0050] In this embodiment, the first training image pair is generated by cropping an area having the second size at the same position from both images of the first image pair. However, the first training image pair may also be generated by resizing at least a portion of the first image pair to the second size. Furthermore, in this embodiment, the first training image set is generated from the first image set. However, if the size of the first image set is the same as the second size, the process of generating the first training image set from the first image set is not necessary.

[0051] Next, the process (learning method) of learning the weights of the first machine learning model performed by the learning device 201 as the first learning phase will be described using the flowchart in FIG. 6. As described above, in this embodiment, the first machine learning model is learned by unsupervised learning that does not require correct answer data. Hereinafter, one image of the first training image pair will be referred to as the first training image, and the other image will be referred to as the second training image. In the first learning phase, the learning device 201 first inputs the first training image pair included in the first training image set, which is training data, to the first machine learning model and obtains a third motion vector indicating the movement of corresponding pixels in the first training image pair.

[0052] Next, the learning device 201 uses the third motion vector to generate an image (a first warped image, described later) in which the second learning image is aligned so that it becomes an image equivalent to the first learning image (so that each pixel of the second learning image approaches the corresponding pixel of the first learning image).Then, the weights of the first machine learning model are determined so as to reduce the difference between the first learning image and the first warped image.In other words, the first machine learning model is trained.

[0053] In step S201, the acquisition unit 212 acquires one or more first training image pairs from the storage unit 211.

[0054] Next, in step S202, the generation unit 213 inputs the first training image pair into the first machine learning model to generate a third motion vector. The third motion vector is a vector representing the movement of corresponding pixels within the first training image pair, i.e., between the first training image and the second training image. In this embodiment, the third motion vector has the same size as the first training image pair, but the size of the third motion vector is not limited to this. In this embodiment, the third motion vector is a vector indicating the movement from the second training image to the first training image for each image in the first training image. Also in this embodiment, the third motion vector is two types of two-dimensional maps, each of which indicates the horizontal or vertical movement amount for each pixel position in the first training image.

[0055] In this embodiment, the first machine learning model is a CNN having multiple convolutional layers. In the first learning run, the weights (filter coefficients and biases) of the convolutional layers are generated by random numbers. However, the first machine learning model is not limited to a CNN, and may be other machine learning models such as a GAN or an RNN.

[0056] Next, in step S203, the generation unit 213 generates a first warped image using the second training image and the third motion vector. The first warped image is an image obtained by aligning the second training image to an image equivalent to the first training image by moving pixels of the second training image using the third motion vector.

[0057] The generation unit 213 calculates each pixel value of the first warped image from the pixel values ​​of the second training image using a known interpolation method (interpolation process) such as nearest neighbor interpolation, bilinear interpolation, or bicubic interpolation. In this case, the interpolation method used to calculate the pixel values ​​of the first warped image is preferably the same as the interpolation method used when aligning the second image within the second machine learning model in the estimation phase so that it becomes an image equivalent to the first image. This makes it possible for the second machine learning model to train a first machine learning model suitable for generating a fifth image with higher accuracy.

[0058] In this embodiment, backward warping is employed to align the second training image to an image equivalent to the first training image using a third motion vector indicating the movement of each pixel of the first training image from the second training image to the first training image. However, forward warping may also be employed to align the first training image to an image equivalent to the second training image using the third motion vector. In this embodiment, because backward warping is employed in the second machine learning model of the estimation phase, backward warping is also employed in step S203. In other words, it is preferable to employ the same alignment method as the alignment method employed in the second machine learning model of the estimation phase. This enables the second machine learning model to train a first machine learning model suitable for generating a fifth image with higher accuracy.

[0059] Next, in step S204, the update unit 214 updates (determines) the weights of the first machine learning model based on the error between the first training image and the first warped image. In this embodiment, the loss function is the Charbonnier loss of the difference in pixel values ​​between the first training image and the first warped image. However, the loss function is not limited to this. If multiple pairs of first training images are acquired in step S201, the update unit 214 calculates the value of the loss function for each pair. Then, the update unit 214 updates the weights using the calculated loss function value by backpropagation or the like.

[0060] Next, in step S205, the update unit 214 determines whether learning of the first machine learning model is complete. Completion of learning can be determined by, for example, whether the number of iterations of weight update has reached a predetermined number, or whether the amount of change in weight during update is smaller than a predetermined value. If it is determined that weight learning is not complete, the process returns to step S201, and the acquisition unit 212 acquires one or more new first training image pairs. If it is determined that weight learning is complete, the update unit 214 ends learning and stores weight information in the storage unit 211.

[0061] In this embodiment, the first machine learning model is trained by unsupervised learning, but the training method of the first machine learning model is not limited to this. For example, the first machine learning model may be trained by supervised learning using ground truth data of the third motion vector corresponding to the first training image pair.

[0062] The flowchart in FIG. 7 shows the process of generating training data for a second machine learning model performed by the training device 201. The training data is a second set of supervised images and a second training image set, and is used to train the second machine learning model. In the next second training phase, the weights of the second machine learning model are determined so as to reduce the difference between a sixth image obtained by inputting the second training image set into the second machine learning model and the second supervised image set. In other words, the second machine learning model is trained.

[0063] In this embodiment, the learning data for the second machine learning model is generated by the learning device 201, but may be generated by another device.

[0064] In step S301, the acquisition unit 212 acquires a first supervised image set from the storage unit 211. The first supervised image set is composed of one or more first supervised images. The first supervised image set may include captured images or CG images. For example, the first supervised image set may include frame images extracted from a captured video. The first supervised image set may also be a public dataset such as the REDS dataset.

[0065] Furthermore, the first set of ground truth images preferably includes images containing a variety of objects, such as edges, textures, gradients, or flat areas with various intensities and directions, thereby improving the robustness of the second machine learning model with respect to objects contained in the first and second images.

[0066] In addition, it is preferable that the first ground truth image has sufficient high-frequency components. For example, if the first training image is a captured image, it is preferable that the first ground truth image is an image captured by an optical system with higher performance than the optical system 221, or a frame image extracted from a video captured by that optical system. It is also preferable that the first ground truth image is an image obtained by reducing the captured image or frame image. This allows the second machine learning model to generate a fifth image with sufficient high-frequency components and high resolution.

[0067] Next, in step S302, acquisition unit 212 acquires a second image set from storage unit 211. The second image set includes one or more second image pairs. The size of the second image pair is smaller than the size of the first ground truth image, and one image of the second image pair is an image that includes the same object as the first ground truth image. In other words, one image of the second image pair is an image that includes the same object as the first ground truth image and has a larger sampling pitch than the first ground truth image. Furthermore, the ratio of the size of the second image pair to the size of the first ground truth image is equal to the ratio of the size of the first image, the second image, and the size of the fifth image in the estimation phase.

[0068] One image of the second image pair includes at least a portion of an object included in the other image of the second image pair at a different location than the other image. Each image constituting the second image pair in the second image set may constitute a separate second image pair from other images included in the second image set. The second image set may include captured images or CG images. The second image set may also include multiple frame images extracted from a captured video. The second image set may also be a public dataset such as the REDS dataset.

[0069] Furthermore, the second image set preferably includes images that have image quality degradation that is the same as that of the third and fourth images described above, thereby improving the robustness of the second machine learning model against image quality degradation that is the same as that of the first and second images.

[0070] Additionally, it is preferable that one image of the second image pair is an image in which multiple objects included in the other image of the second image pair have moved by different amounts or directions of movement. That is, it is preferable that each image in the second image pair includes multiple objects with different amounts or directions of movement. Furthermore, it is preferable that multiple second image pairs included in the second image set also include different amounts or directions of movement of the objects. This can improve the robustness of the second machine learning model against movement included between the first image and the second image.

[0071] Alternatively, the first set of correct images may be used to generate a second set of images. For example, the second set of images may be generated by downscaling the first set of correct images and imparting the image quality degradation that the first and second images have. Alternatively, the first set of correct images and the second set of images may each be generated using a different set of images.

[0072] Next, in step S303, the generation unit 213 generates a second supervised image set and a second training image set. Then, this process ends. The second supervised image set includes one or more second supervised images, and the second training image set includes one or more second training image pairs. The second supervised image is an image having a predetermined size, which is 256 × 256 pixels in this embodiment. The second training image pair is an image pair having a predetermined size (fourth size), which is 128 × 128 pixels in this embodiment. The fourth size is smaller than the size of the second supervised image. One image of the second training image pair is an image that includes the same object as the second supervised image. In other words, one image of the second training image pair is an image that includes the same object as the second supervised image and has a larger sampling pitch than the second supervised image. Furthermore, the ratio between the size of the second training image pair and the size of the second ground truth image is equal to the ratio between the size of the first image and the second image and the size of the fifth image in the estimation phase.

[0073] Furthermore, one image of the second training image pair includes at least a portion of an object included in the other image of the second training image pair, but at a different position from the other image. In this embodiment, the second ground truth image is generated by cropping an area of ​​a predetermined size in the second ground truth image from the first ground truth image. Furthermore, the second training image pair is generated by cropping an area of ​​a fourth size in the same position from both images of the second image pair. The second training image set may have at least a portion in common with the first training image set used to train the first machine learning model.

[0074] In this embodiment, the second supervised image set is generated from the first supervised image set, but if the first supervised image set is the same as the required image size, the process of generating the second supervised image set from the first supervised image set is not necessary. Also, the second training image set is generated from the second image set, but if the size of the second image set is the same as the required image size, the process of generating the second training image set from the second image set is not necessary.

[0075] The flowchart in FIG. 8 illustrates a process (learning method) for learning the weights of the second machine learning model performed in the learning device 201 as the second learning phase. Hereinafter, one image of the second learning image pair is referred to as the third learning image, and the other image is referred to as the fourth learning image. In this process, the second learning image pair included in the second learning image set, which is the learning data, is first input to the first machine learning model trained in the first learning phase, and a fourth motion vector indicating the movement of corresponding pixels in the second learning image pair is obtained. Next, in the second machine learning model, a sixth image is generated using the third learning image, the fourth learning image, and an image obtained by aligning the fourth learning image using the fourth motion vector so that it becomes an image equivalent to the third learning image (a second warped image, described later). Finally, the weights of the second machine learning model are determined so as to reduce the difference between the sixth image and the second ground truth image.

[0076] In step S401, the acquisition unit 212 acquires weight information of the first machine learning model, one or more second ground truth images, and one or more second training image pairs from the storage unit 211. The weight information of the first machine learning model is read out from the storage unit 211 and stored in the storage unit 224 in advance.

[0077] Next, in step S402, the generation unit 213 inputs the second training image pair (the third training image and the fourth training image) into the trained first machine learning model to generate a fourth motion vector. The trained first machine learning model is the first machine learning model whose weights have been determined by training in the first learning phase. The fourth motion vector is a vector representing the movement of corresponding pixels within the second training image pair, i.e., between the third training image and the fourth training image. In this embodiment, the fourth motion vector has the same size as the second training image pair, but the size of the fourth motion vector is not limited to this. In this embodiment, the fourth motion vector is a vector indicating the movement from the fourth training image to the third training image for each image in the third training image. Also, in this embodiment, the fourth motion vector is two types of two-dimensional maps, each of which indicates the horizontal or vertical movement amount for each pixel position in the third training image.

[0078] Next, in step S403, the generation unit 213 inputs the second training image pair and the fourth motion vector into the second machine learning model to generate a sixth image. At this time, the generation unit 213 first generates a second warped image within the second machine learning model using the fourth training image and the fourth motion vector. The second warped image is an image obtained by aligning the fourth training image to an image equivalent to the third training image by moving the pixels of the fourth training image using the fourth motion vector. The generation unit 213 calculates each pixel value of the second warped image from the pixel values ​​of the fourth training image using a known interpolation method such as nearest neighbor interpolation, bilinear interpolation, or bicubic interpolation.

[0079] It is preferable that the interpolation method used to calculate the pixel values ​​of the second warped image is the same as the interpolation method used when aligning the second image within the second machine learning model in the estimation phase to create an image equivalent to the first image. By using the same image interpolation method in the second machine learning model in the second learning phase and the estimation phase, it is possible to train the second machine learning model to generate a fifth image with higher accuracy.

[0080] Furthermore, it is preferable that, within the second machine learning model, an image is generated in which each pixel of the second warped image is more accurately aligned with the corresponding pixel of the third training image based on the second warped image, and this is used as a new second warped image. For example, it is preferable that the third training image and the second warped image are input to a CNN, the amount of deviation of each pixel of the second warped image from the corresponding pixel of the third training image is calculated, and the second warped image is further corrected according to the amount of deviation. By performing a similar process in the estimation phase, the second machine learning model can more accurately align the second image so that it becomes an image equivalent to the first image, thereby generating a fifth image with higher accuracy.

[0081] In this embodiment, backward warping is employed to align the fourth training image to an image equivalent to the third training image using a fourth motion vector indicating the movement of each pixel of the third training image from the fourth training image to the third training image. However, forward warping may also be employed to align the fourth training image to an image equivalent to the third training image using a fourth motion vector indicating the movement of each pixel of the fourth training image from the third training image to the fourth training image. In this embodiment, because backward warping is employed in the second machine learning model in the estimation phase, backward warping is also employed in S403. In this way, by aligning the image alignment methods in the second machine learning model between the second learning phase and the estimation phase, a second machine learning model that generates a fifth image with higher accuracy can be trained.

[0082] Next, the generation unit 213 generates a sixth image using the third training image and the second warped image within the second machine learning model. The third training image and the second warped image are concatenated in the channel direction in a concatenation layer included in the second machine learning model. The sixth image is an upscaled version of the third training image.

[0083] In this embodiment, the second machine learning model is a CNN having multiple convolutional layers. In the first learning run, the weights (filter coefficients and biases) of the convolutional layers are generated by random numbers. However, the second machine learning model is not limited to a CNN, and may be other machine learning models such as a GAN or an RNN.

[0084] Furthermore, it is preferable that the contribution of the third training image and the second warped image to the generation of the sixth image is determined within the second machine learning model, and the sixth image is generated based on the contribution. This contribution is preferably determined for each pixel of the third training image and the second warped image. For example, it is preferable that the third training image and the second warped image are input to a CNN, and the contribution of each pixel of the third training image and the second warped image to the generation of the sixth image is determined. By performing similar processing in the estimation phase, the second machine learning model adjusts the contribution of each of the first image and the second image, thereby enabling the generation of a fifth image with higher accuracy.

[0085] In this embodiment, the sixth image is generated using a second warped image obtained by aligning the fourth training image to an image equivalent to the third training image. Alternatively, the sixth image may be generated through a process of aligning the features of the fourth training image, rather than through a process of aligning the fourth training image. More specifically, in the process of generating the sixth image, for example, the features of the fourth training image may be aligned using a fourth motion vector to become equivalent to the features of the third training image.

[0086] Next, in step S404, the update unit 214 updates the weights of the second machine learning model based on the error between the sixth image and the second ground truth image. In this embodiment, the loss function is the Charbonniellos function of the difference in pixel values ​​between the sixth image and the second ground truth image. However, the loss function is not limited to this. If multiple pairs of second training images are acquired in step S401, the update unit 214 calculates the value of the loss function for each pair. The update unit 214 updates the weights using the calculated loss function value by backpropagation or the like.

[0087] Next, in step S405, the update unit 214 determines whether learning of the second machine learning model is complete. Completion of learning can be determined by, for example, whether the number of iterations of weight update has reached a predetermined number, or whether the amount of change in weight during update is smaller than a predetermined value. If it is determined that weight learning is not complete, the process returns to step S401, and the acquisition unit 212 acquires one or more new second training image pairs and a second supervised image. If it is determined that weight learning is complete, the update unit 214 ends learning and stores weight information in the storage unit 211.

[0088] In this embodiment, after the first machine learning model is trained in the first learning phase, the second machine learning model is trained in the second learning phase. However, this is not limited to this, and the first machine learning model and the second machine learning model may be trained jointly from the beginning. More specifically, the second learning phase may be performed using the first machine learning model whose weights have not been determined, without performing the first learning phase. Then, in step S404, the weights of the first machine learning model and the second machine learning model may be simultaneously updated based on the error between the sixth image and the second ground truth image. Furthermore, the weights of the first machine learning model and the second machine learning model may be simultaneously updated based on the error between the sixth image and the second ground truth image and the error between the third training image and the second warped image. In the estimation phase, the first machine learning model and the second machine learning model are jointly used to generate a fifth image from the first image and the second image. Therefore, by jointly training the first machine learning model and the second machine learning model, the weights of the first machine learning model and the weights of the second machine learning model can be optimized to generate a fifth image with higher accuracy.

[0089] However, after the first and second learning phases, a third learning phase may be provided in which the weights of the first and second machine learning models are jointly updated. This makes it possible to optimize the weights of the first and second machine learning models so as to generate a fifth image with higher accuracy, and also makes it easier to converge the learning of each machine learning model.

[0090] 1 shows the flow of estimation processing (estimation phase) using a trained first machine learning model and a trained second machine learning model performed in the image estimation unit 223 of the imaging device 202. The trained machine learning models are machine learning models whose weights have been determined by learning in the learning phase.

[0091] In the estimation phase, the image estimation unit 223 first extracts a first original image 302 and a second original image 303 from an original video 301. Next, the first original image 302 and the second original image 303 are each divided to generate a first image 304 and a second image 305. Next, the first image 304 and the second image 305 are each reduced to generate a third image 306 and a fourth image 307.

[0092] Next, the image estimation unit 223 inputs the third image 306 and the fourth image 307 into a first machine learning model to generate a first motion vector 308. Next, the first motion vector 308 is enlarged to generate a second motion vector 309. Next, the first image 304, the second image 305, and the second motion vector 309 are input into a second machine learning model to generate a third image 310. Next, the image estimation unit 223 combines the fifth image 310 to generate a target image 311. Finally, the image estimation unit 223 generates a target video 312 from the target image 311.

[0093] The fifth image 310 is an upscaled version of the first image 304, the destination image 311 is an upscaled version of the first original image 302, and the destination video 312 is an upscaled version of the original video 301.

[0094] The flowchart in FIG. 9 illustrates a process (image processing method) performed by the image estimation unit 223 as the estimation phase. First, in step S501, the acquisition unit 223a acquires the original video 301, weight information for the first machine learning model, and weight information for the second machine learning model. In this embodiment, the original video 301 is a captured video generated by the optical system 221 and the image sensor 222. However, the acquired original video 301 may be a portion of the captured video. For example, the captured video may be a video cropped in the spatial or temporal direction, or a video with a lower frame rate than the captured video generated by extracting frame images from the captured video at regular intervals. Furthermore, the original video 301 may be expressed in grayscale or may have multiple channel components. The weight information for the first machine learning model and the weight information for the second machine learning model are read out in advance from the storage unit 211 and stored in the storage unit 224.

[0095] Next, in step S502, the calculation unit 223b extracts a first original image 302 and a second original image 303 from the original video 301. The first original image 302 and the second original image 303 are frame images that constitute the original video 301. The first original image 302 corresponds to the destination image 311 generated in step S509 and is an image to be upscaled. In this embodiment, as shown in FIG. 1, the second original image 303 is a frame image adjacent to the first original image 302 in the original video 301, but it does not necessarily have to be an adjacent frame image. However, in the first image 304 and the second image 305 generated in step S503, the second original image 303 needs to be selected so that the second image 305 includes at least a portion of an object included in the first image 304.

[0096] Next, in step S503, the calculation unit 223b divides the first original image 302 and the second original image 303 to generate a first image 304 and a second image 305. The first image 304 and the second image 305 are both images of the same third size. That is, the first image 304 and the second image 305 are images obtained by cropping areas of the third size from the same positions on the first original image 302 and the second original image 303. In this embodiment, the third size is 256 × 256 pixels. It is preferable to divide the first original image 302 so that multiple first images 304 include a common area. The reason for this will be explained later in step S509. Note that if the sizes of the first original image 302 and the second original image 303 are the same as the third size, the division process of step S503 is unnecessary, and the accompanying merging process of step S509 is also unnecessary.

[0097] Next, in step S504, the calculation unit 223b reduces the first image 304 and the second image 305 to generate a third image 306 and a fourth image 307, respectively. The reduction from the first image 304 to the third image 306 and the reduction from the second image 305 to the fourth image 307 are performed using the same reduction process that converts images. In this embodiment, the reduction process is downsampling, which extracts only one pixel from multiple pixels. The third image 306 and the fourth image 307 are both images of the same first size. In this embodiment, the first size is 128 × 128 pixels. Therefore, the reduction ratio from the first image 304 to the third image 306 and the reduction ratio from the second image 305 to the fourth image 307 are both double in the width direction and double in the height direction of the image.

[0098] Next, in step S505, the estimation unit 223c inputs the third image 306 and the fourth image 307 into a first machine learning model to generate a first motion vector 308. The first motion vector 308 is a vector that represents the movement of corresponding pixels between the third image 306 and the fourth image 307. In this embodiment, the first motion vector 308 has the same first size (128 × 128 pixels) as the third image 306 and the fourth image 307, but the size of the first motion vector 308 is not limited to this. In this embodiment, the first motion vector 308 is a vector that indicates the movement from the fourth image 307 to the third image 306 for each image in the third image 306. Furthermore, in this embodiment, the third motion vector is two types of two-dimensional maps, and each two-dimensional map indicates the horizontal or vertical movement amount for each pixel position in the third image 306.

[0099] Next, in step S506, the calculation unit 223b enlarges the first motion vector 308 to generate a second motion vector 309. The enlargement process of the first motion vector 308 is a process independent of the first machine learning model. This enlargement process may be an enlargement process using a machine learning model or an enlargement process without using a machine learning model, as long as it is independent of the first machine learning model. In this embodiment, the enlargement process is a process using bicubic interpolation. More specifically, in the enlargement process, the second motion vector 309 is generated by multiplying each pixel corresponding to the motion vector obtained by enlarging the first motion vector 308 using bicubic interpolation by an enlargement factor (2x in this embodiment, as described below). In addition, in this embodiment, the enlargement factor in this enlargement process is the same as the reduction factor in the reduction process in S504, i.e., 2x in the width direction and 2x in the height direction of the map. In other words, the size of the second motion vector is 256 x 256 pixels. In this embodiment, the second motion vector 309 has the same size as the first image 304 and the second image 305.

[0100] Next, in step S507, the estimation unit 223c inputs the first image 304, the second image 305, and the second motion vector 309 into a second machine learning model to generate a fifth image 310. Here, the estimation unit 223c first generates a third warped image using the second image 305 and the second motion vector 309 within the second machine learning model. The third warped image is an image obtained by aligning the second image 305 to an image equivalent to the first image 304 by moving the pixels of the second image 305 using the second motion vector 309. At this time, the estimation unit 223c calculates each pixel value of the third warped image from the pixel values ​​of the second image 305 using a known interpolation method such as nearest neighbor interpolation, bilinear interpolation, or bicubic interpolation. In this embodiment, in step S505, the estimation unit 223c generates a first motion vector 308 indicating movement from the fourth image 307 to the third image 306 for each image in the third image 306. At this time, backward warping is employed, in which a second motion vector 309 obtained by enlarging the first motion vector 308 is used to align the second image 305 so that it becomes an image equivalent to the first image 304.

[0101] Next, the estimation unit 223c generates a fifth image 310 using the first image 304 and the third warped image within the second machine learning model. The first image 304 and the third warped image are concatenated in the channel direction in a concatenation layer included in the second machine learning model. The fifth image 310 is an upscaled version of the first image 304. In this embodiment, the fifth image 310 is an image obtained by upscaling the first image 304 by twice the size in the width direction and twice the size in the height direction, and has 512 pixels by 512 pixels.

[0102] Next, in step S508, the calculation unit 223b determines whether or not generation of the fifth images 310 has been completed for all pairs of the first images 304 and the second images 305. If it is determined that generation of all the fifth images 310 has not been completed, the process returns to step S504, and the calculation unit 223b generates a fifth image 310 from a new pair of the first image 304 and the second image 305. If it is determined that generation of all the fifth images 310 has been completed, the process proceeds to step S509.

[0103] In step S509, the calculation unit 223b combines the fifth image 310 to generate the destination image 311. Here, the destination image 311 is generated by combining the fifth image 310 so that the destination image 311 is an upscaled image of the first original image 302. In this embodiment, the destination image 311 is an image obtained by upscaling the first original image 302 by twice in the width direction and twice in the height direction.

[0104] As explained above, in step S503, it is preferable to divide the first original image 302 so that the multiple first images 304 include a common area. This makes it possible to generate the destination image 311 by performing post-processing on the common area included in the multiple fifth images 310 in step S509. If the first images 304 are divided in step S503 so that they do not include a common area, tile-like artifacts may occur in the portion of the destination image 311 where the multiple fifth images 310 are joined. For this reason, artifacts can be reduced in the common area included in the multiple fifth images 310 by dividing the first images 304 so that they include a common area, and generating the destination image 311 based on, for example, a weighted average of the multiple fifth images.

[0105] Furthermore, the accuracy of generation in step S507 may be lower in the peripheral regions of the fifth image 310 than in the central region. This is because fewer pixels of the first image 304 and the second image 305 are taken into account in the peripheral regions of the fifth image 310 than in the central region. Therefore, by dividing the first images 304 so that they include common regions, and then generating the target image 311 in the central regions of the plurality of fifth images 310, it is possible to generate a target image 311 with higher accuracy.

[0106] Next, in step S510, the calculation unit 223b determines whether or not generation of the target image 311 has been completed for all pairs of the first original image 302 and the second original image 303. If it is determined that generation of all target images 311 has not been completed, the process returns to step S503, and the calculation unit 223b generates a target image 311 from a new pair of the first original image 302 and the second original image 303. If it is determined that generation of all target images 311 has been completed, the process proceeds to step S511.

[0107] In step S511, the calculation unit 223b generates the target video 312 from the target image 311. Specifically, the target video 312 is generated so that each of the multiple target images 311 becomes a frame image of the target video 312. Then, this process ends. In this embodiment, the target video 312 is a video obtained by upscaling the source video 301 by twice in the width direction and twice in the height direction.

[0108] The image sizes in the estimation phase and the training phase will now be described. In the estimation phase, the third image 306 and the fourth image 307 input to the first machine learning model both have a first size. In the first training phase, the first training image set input to the first machine learning model has a second size. In this embodiment, to generate a highly accurate first motion vector 208 regardless of the first machine learning model itself or the third image 306 and the fourth image 307, it is preferable that the first size be equal to or smaller than the second size. Note that in this embodiment, both the first size and the second size are 128 × 128 pixels.

[0109] Furthermore, in the second learning phase, the second training image set input to the second machine learning model has a fourth size. In the second learning phase, the second training image set is input to the trained first machine learning model, and a third motion vector is generated. In not only the estimation phase but also in the second learning phase, the third motion vector, which is output from the first machine learning model, needs to be generated with high accuracy. To generate a highly accurate third motion vector regardless of the first machine learning model itself or the second training image set, it is preferable that the fourth size be equal to or smaller than the second size.

[0110] In this embodiment, the fourth size is equal to the second size, 128 x 128 pixels. Thus, it is preferable that the fourth size be as large as possible. This is because the larger the fourth size, the more accurately the second machine learning model can generate the fifth image 310 for the larger-sized third image 306 and fourth image 307.

[0111] Furthermore, in the estimation phase, the first image 304 and the second image 305 input to the second machine learning model are both a third size. The third size is not limited based on the second size, which is the image size trained by the first machine learning model. In other words, it is not limited by the fourth size, which is equal to the second size. Therefore, the third size may be equal to or larger than the fourth size. This is one of the effects of each embodiment. In this embodiment, the third size is 256×256 pixels, which is equal to or larger than the fourth size (128×128 pixels).

[0112] The effect of being able to set the third size equal to or greater than the fourth size in this embodiment will be described. Even when the fourth size is limited by the computational resources available during learning, the degree of freedom in setting an optimal third size for the estimation phase is improved. For example, the degree of freedom in setting an optimal third size is improved depending on the processing speed of the second machine learning model in step S507 and the speed at which the target image 311 is generated from the fifth image 310 in step S509. In other words, the optimal third size can be set to improve the processing speed in the estimation phase.

[0113] Furthermore, in this embodiment, the computational resources required during learning to process the first image 304 and the second image 305, which have the same fourth size, can be reduced compared to the conventional technique. For example, when generating the fifth image 310 from the third image 306 and the fourth image 307, which are 256 × 256 pixels, the second machine learning model would need to be trained with an image size of 256 × 256 pixels in the conventional technique. On the other hand, in this embodiment, learning can be performed with the fourth size of 128 × 128 pixels, so the computational memory required during learning can be reduced to about one-fourth of the conventional technique.

[0114] As described above, according to this embodiment, it is possible to achieve a highly accurate upscaling task without being restricted based on the image size on which the machine learning model that generates the motion vector has been trained. [Example]

[0115] The second embodiment aims to increase the frame rate of a video. Specifically, new frame images are generated between each frame image included in a low-frame-rate video, and a high-frame-rate video is generated by combining the original frame images with the newly generated frame images. More specifically, a first image, which is at least a portion of a frame image included in the low-frame-rate video, and a second image, which is at least a portion of a frame image adjacent to the first frame image, are reduced, respectively, to generate a third image and a fourth image. A first motion vector is generated based on the third image and the fourth image using a first machine learning model. A second motion vector is generated by enlarging the first motion vector, and a fifth image is generated based on the first image and the second image using a second machine learning model. The fifth image is at least a portion of a frame image newly generated between the frame image corresponding to the first image and the frame image corresponding to the second image.

[0116] In this embodiment, as an example, a high-frame-rate video is generated by generating a new frame image in the center of two consecutive frame images included in a low-frame-rate original video. In other words, the generated video has a frame rate approximately twice that of the original video. The fifth image is an image located in the center of a first image and a second image that are consecutive in time. However, this is not limited to this, and a video with a higher frame rate may be generated by generating multiple new fifth images between the first and second images. Furthermore, in this embodiment, the fifth image located between the first and second images is generated based on the first and second images, but the fifth image may also be generated based on three or more images that are at least a portion of three or more frame images included in the original video.

[0117] In this embodiment, the up-frame rate task of generating a fifth image based on a first image and a second image can generate the fifth image based on the average or weighted average of the pixel values ​​of the first image and the pixel values ​​of the second image. By using motion vectors in this task, it is possible to adapt to the movement of objects between the first and second images and generate a fifth image with higher accuracy.

[0118] On the other hand, the generation accuracy of the fifth image is strongly dependent on the generation accuracy of the motion vector. If the generation accuracy of the motion vector is low, the pixels of the fifth image will be generated from pixels of the first image or the second image that are not intended, resulting in the appearance of artifacts in the fifth image. In contrast, the first machine learning model used in this embodiment can generate the first motion vector with high accuracy using the sizes (first size) of the third and fourth images that are restricted based on the image size (second size) learned by the first machine learning model. By using the highly accurate first motion vector, it is possible to generate a fifth image with reduced artifacts. This embodiment is effective for the sizes (third size) of the first and second images that are not restricted based on the second size.

[0119] Fig. 10 shows the configuration of an image processing system 400 in this embodiment. Fig. 11 shows the external appearance of the image processing system 400. The image processing system 400 includes a learning device 401, an imaging device 402, an image estimation device 403 as an image processing device, a display device 404, a storage medium 405, an output device 406, and a network 407.

[0120] The learning device 401 is configured by a computer such as a personal computer, and has a storage unit 401a, an acquisition unit 401b, a generation unit 401c, and an update unit 401d, and operates according to a program to determine the weights of the machine learning model.

[0121] The imaging device 402 has an optical system 402a and an image sensor 402b. The optical system 402a collects light incident from a space in which an object exists to form an object image. The optical system 402a has functions such as zoom, aperture, and autofocus as necessary. The image sensor 402b converts the object image formed by the optical system 402a into an electrical signal and generates a captured video as image data.

[0122] The image estimation device 403 includes a storage unit 403a, an acquisition unit 403b, a generation unit 403c, and an estimation unit 403d. The image estimation device 403 is configured as a personal computer and operates according to a program to up-frame rate the captured video generated by the optical system 221 and the image sensor 222. To this end, the image estimation device 403 generates a first motion vector using a first machine learning model based on a third image and a fourth image obtained by reducing a first image and a second image, which are part of the frame images of the captured video. The image estimation device 403 further generates a fifth image using a second machine learning model based on a second motion vector obtained by enlarging the first motion vector, the first image, and the second image. The fifth image is then used to generate a high frame rate video by up-frame rate the captured video.

[0123] The fifth image is generated using weights determined in advance by the learning device 401. The weights are stored in the storage unit 403a. The processing performed by the image estimation device 403 will be described in detail later.

[0124] The high frame rate video is output to at least one of a display device 404, a storage medium 405, and an output device 406. The display device 404 is a liquid crystal display, a projector, or the like. A user can perform editing work while checking images being processed via the display device 404. The storage medium 405 is a semiconductor memory, a hard disk, a server on a network, or the like, and stores the high frame rate video. The output device 405 is a printer, or the like.

[0125] As with Example 1, the processing performed in this example can be broadly divided into the generation of training data for the first machine learning model, training the weights of the first machine learning model (first training phase), the generation of training data for the second machine learning model, training the weights of the second machine learning model (second training phase), and estimation using the first machine learning model and the second machine learning model using the trained weights (estimation phase).

[0126] The learning device 401 performs a process of generating training data for a first machine learning model according to the flowchart of Fig. 5 described in the first embodiment. In this embodiment, the processes of step S101 and step S102 are performed by an acquisition unit 401b and a generation unit 401c, respectively, instead of the acquisition unit 212 and the generation unit 213 of the first embodiment.

[0127] 6 described in the first embodiment, the learning device 401 performs a process of learning the weights of the first machine learning model as the first learning phase. In this embodiment, the processes of steps S201 to S205 are performed by an acquisition unit 401b, a generation unit 401c, and an update unit 401d, instead of the acquisition unit 212, the generation unit 213, and the update unit 214 of the first embodiment, respectively.

[0128] The flowchart in FIG. 12 shows the process of generating training data for a second machine learning model performed by the learning device 401. The training data is a third supervised image set and a third training image set, and is used for training the second machine learning model. In the next second learning phase, the learning device 401 determines weights for the second machine learning model so as to reduce the difference between a seventh image obtained by inputting the third training image set to the second machine learning model and the third supervised image set. Note that the third training image set corresponds to the second training image set in Example 1 in that it is input to the second machine learning model in the second learning phase. Note that in this example, the process of generating training data for the second machine learning model is performed by the learning device 401, but may be performed by another device.

[0129] In step S601, the acquisition unit 401b acquires a third image set from the storage unit 401a. The third image set includes one or more first image triplets. A first image triplet is composed of three images including the same object at different positions. Each image constituting a first image triplet in the third image set may constitute a different first image triplet from other images included in the third image set. The third image set may include captured images or CG images. For example, the third image set may include frame images extracted from a captured video. The third image set may also be a public dataset such as the REDS dataset.

[0130] In this embodiment, the third image set is generated from a captured video having the same frame rate as the high frame rate video to be ultimately generated. The first image triplet is three consecutive frame images extracted from this captured video. The third image set preferably includes images containing various objects. For example, it preferably includes images containing edges, textures, gradients, or flat areas with various intensities and directions. This improves the robustness of the second machine learning model with respect to objects contained in the first and second images.

[0131] Furthermore, the third image set preferably includes images that include the image quality degradation that the first image and the second image have. The image quality degradation is the same as that described in Example 1. This can improve the robustness of the second machine learning model against the image quality degradation that the first image and the second image have.

[0132] Additionally, it is preferable that each image in the first image triplet contains multiple objects with different amounts and directions of motion. Furthermore, it is preferable that the multiple first image triplets included in the third image set also contain multiple objects with different amounts and directions of motion. This can improve the robustness of the second machine learning model against motion between the first and second images.

[0133] Next, in step S602, the generation unit 401c generates a third supervised image set and a third training image set. The third supervised image set includes one or more third supervised images, and the third training image set includes one or more third training image pairs. One third supervised image corresponds to one third training image pair. The third training image pair is an image pair having a predetermined size (fourth size), and in this embodiment, the fourth size is 128 x 128 pixels. The third supervised image is an image having the same fourth size as the third training image pair.

[0134] Furthermore, the third training image pair and the corresponding third ground truth image pair contain the same object. As described above, in this embodiment, the first image triplet is three consecutive frame images extracted from a captured video having the same frame rate as the high frame rate video to be ultimately generated. In this embodiment, the third training image pair is generated by cropping an area having a fourth size from the first frame image and the last frame image constituting the first image triplet. Also, in this embodiment, the third ground truth image is generated by cropping an area having a fourth size from the middle frame image constituting the first image triplet. In this case, the third training image pair or the third ground truth image is cropped from the same position as the first image triplet. The third ground truth image set and the third training image set may each share at least a portion with the first training image set used to train the first machine learning model.

[0135] The flowchart in Figure 13 shows the process of learning the weights of the second machine learning model, which is performed in the learning device 401 as the second learning phase. Hereinafter, one image of the third learning image pair will be referred to as the fifth learning image, and the other image will be referred to as the sixth learning image. Furthermore, the seventh image, which will be described later, is an image that aims to be the central frame image between the fifth and sixth learning images, assuming that the fifth and sixth learning images are frame images that constitute the same video.

[0136] In the second learning phase, the learning device 401 first inputs a third training image pair included in a third training image set, which serves as learning data, into the first machine learning model learned in the first learning phase to obtain a fifth motion vector indicating the movement of corresponding pixels in the third training image pair. Next, in the second machine learning model, based on the fifth motion vector, a sixth motion vector and a seventh motion vector are generated that respectively indicate the movement of corresponding pixels between the fifth training image and a seventh image to be generated, and between the sixth training image and the seventh image.

[0137] Next, the learning device 401 uses the sixth motion vector in the second machine learning model to generate a fourth warped image, which is an image aligned so that the fifth training image becomes an image equivalent to the seventh image from which the fifth training image is to be generated. Similarly, using the seventh motion vector in the second machine learning model, the learning device 401 generates a fifth warped image, which is an image aligned so that the sixth training image becomes an image equivalent to the seventh image from which the sixth training image is to be generated. Then, the learning device 401 generates a seventh image using the fourth warped image and the fifth warped image.

[0138] Finally, the learning device 401 determines the weights of the second machine learning model so as to reduce the difference between the seventh image and the third correct image.

[0139] In step S701, the acquisition unit 401b acquires weight information of the first machine learning model, one or more third supervised images, and one or more third training image pairs from the storage unit 401a.

[0140] Next, in step S702, the generation unit 401c inputs the third training image pair into the trained first machine learning model to generate a fifth motion vector. The trained first machine learning model is the first machine learning model whose weights have been determined by training in the first learning phase. The fifth motion vector is a vector representing the movement of corresponding pixels within the third training image pair, i.e., between the fifth training image and the sixth training image. In this embodiment, the fifth motion vector has the same size as the third training image pair, but the size of the fifth motion vector is not limited to this. In this embodiment, the fifth motion vector is two types of vectors representing the movement of corresponding pixels within the third training image pair. One type of vector represents the movement of each pixel of the fifth training image from the sixth training image to the fifth training image. The other type of vector represents the movement of each pixel of the sixth training image from the fifth training image to the sixth training image. In this embodiment, the fifth motion vector is four types of two-dimensional maps, each of which indicates the horizontal or vertical displacement for each pixel position in the fifth or sixth learning image.

[0141] Next, in step S703, the generation unit 401c inputs the third training image pair and the fifth motion vector into the second machine learning model to generate a seventh image. To do this, the generation unit 401c first generates a sixth motion vector and a seventh motion vector using the fifth motion vector within the second machine learning model. The sixth motion vector is a vector that indicates the movement of corresponding pixels between the fifth training image and the seventh image.

[0142] In this embodiment, the sixth motion vector indicates the movement from the seventh image to the fifth learning image for each pixel of the fifth learning image. Specifically, the generation unit 401c generates the sixth motion vector by multiplying each pixel of the fifth motion vector, which indicates the movement from the sixth learning image to the fifth learning image for each pixel of the fifth learning image, by 1 / 2.

[0143] On the other hand, the seventh motion vector is a vector that indicates the movement of corresponding pixels between the sixth training image and the seventh image, and indicates the movement from the seventh image to the sixth training image for each pixel of the sixth training image. Specifically, the generation unit 401c generates the seventh motion vector by multiplying each pixel of the fifth motion vector that indicates the movement from the fifth training image to the sixth training image for each pixel of the sixth training image by 1 / 2.

[0144] Next, the generation unit 401c generates a fourth warped image and a fifth warped image within the second machine learning model using the third training image pair and the sixth and seventh motion vectors. The fourth warped image is an image corresponding to the seventh image generated based on the fifth training image and the sixth motion vector. The fourth warped image is generated by moving pixels of the fifth training image using the sixth motion vector. Specifically, the generation unit 401c calculates each pixel value of the fourth warped image from the pixel values ​​of the fifth training image using a known interpolation method such as nearest neighbor interpolation, bilinear interpolation, or bicubic interpolation.

[0145] In this embodiment, forward warping is employed to align the fifth training image to an image equivalent to the seventh image using a sixth motion vector indicating the movement of each pixel of the fifth training image from the seventh image to the fifth training image. However, backward warping may also be employed to align the fifth training image to an image equivalent to the seventh image using a motion vector indicating the movement of each pixel of the seventh image from the fifth training image to the seventh image. In this embodiment, since forward warping is employed in the second machine learning model in the estimation phase, forward warping is also employed in step S703. In this way, by aligning the image alignment methods in the second machine learning model in the second learning phase and the estimation phase, a second machine learning model that generates a fifth image with higher accuracy can be trained. The same applies to the generation of the fifth warped image described below.

[0146] The fifth warped image is an image corresponding to a seventh image generated based on the sixth training image and the seventh motion vector. The fifth warped image is generated by moving pixels of the sixth training image using the seventh motion vector. Specifically, the generation unit 401c calculates each pixel value of the fifth warped image from the pixel values ​​of the sixth training image using a known interpolation method such as nearest neighbor interpolation, bilinear interpolation, or bicubic interpolation.

[0147] Finally, the generation unit 401c generates a seventh image from the fourth warped image and the fifth warped image within the second machine learning model. Specifically, the seventh image is generated based on the average or weighted average of the pixel values ​​of the fourth warped image and the pixel values ​​of the fifth warped image. In this embodiment, the seventh image is generated by setting the average of the pixel values ​​of the fourth warped image and the pixel values ​​of the fifth warped image as the pixel values ​​of the seventh image.

[0148] It is preferable that the contribution of the fourth warped image and the fifth warped image to the generation of the seventh image is determined within the second machine learning model, and the seventh image is generated based on the contribution. The contribution is preferably determined for each pixel of the seventh image. For example, it is preferable that the fourth warped image and the fifth warped image are input to a CNN, and the contribution of each pixel of the fourth warped image and the fifth warped image to the generation of the seventh image is determined. The second machine learning model performs similar processing in the estimation phase, adjusting the contribution of the first image and the second image for each pixel of the first image and the second image, thereby enabling the generation of a fifth image with higher accuracy.

[0149] Furthermore, after generating the seventh image, it is preferable that the second machine learning model calculates a residual component for each pixel of the seventh image based on at least the fourth warped image and the fifth warped image, and then adds the residual component to the seventh image to generate a new seventh image. For example, it is preferable to input the fourth warped image and the fifth warped image to a CNN to determine a residual component for each pixel of the seventh image. In the estimation phase, the second machine learning model performs similar processing, thereby enabling the generation of a fifth image with higher accuracy.

[0150] Next, in step S704, the update unit 401d updates (determines) the weights of the second machine learning model based on the error between the seventh image and the third ground truth image. In this embodiment, the loss function is the Charbonniellos function of the difference in pixel values ​​between the seventh image and the third ground truth image. However, the loss function is not limited to this. If multiple pairs of third training images are acquired in step S701, the value of the loss function is calculated for each pair. The weights are updated using the calculated loss function value by backpropagation or the like.

[0151] Next, in step S705, the update unit 401d determines whether learning of the second machine learning model is complete. Completion of learning can be determined by, for example, whether the number of iterations of weight update has reached a predetermined number, or whether the amount of change in weight during update is smaller than a predetermined value. If it is determined that weight learning is not complete, the process returns to step S701, and the acquisition unit 401b acquires one or more new third training image pairs and a third supervised image. If it is determined that weight learning is complete, the update unit 401d ends learning and stores weight information in the storage unit 401a.

[0152] As in Example 1, in this example, after a first machine learning model is trained in a first learning phase, a second machine learning model is trained in a second learning phase. However, this is not limited to this, and the first machine learning model and the second machine learning model may be trained jointly from the beginning. Furthermore, after the first learning phase and the second learning phase, a third learning phase may be provided in which the weights of the first machine learning model and the weights of the second machine learning model are jointly updated.

[0153] In this embodiment, in the second learning phase, the first machine learning model generates a fifth motion vector indicating the movement of corresponding pixels in the third learning image pair. However, this is not limited thereto, and the first machine learning model may generate both a sixth motion vector and a seventh motion vector in the second learning phase. Specifically, the first machine learning model may be trained to generate both motion vectors between one or the other image of the first learning image pair and the central image of the first learning image pair. The central image of the first learning image pair is an image that is targeted to be the central frame image of the first learning image pair, assuming that the first learning image pair are frame images constituting the same video. In this case, in the estimation phase, the first machine learning model generates both motion vectors indicating the movement of corresponding pixels between the first image or the second image and the fifth image as the first motion vector.

[0154] 14 shows the flow of the estimation process (estimation phase) using a trained first machine learning model and a trained second machine learning model performed in the image estimation device 403. A trained machine learning model is a machine learning model whose weights have been determined by learning in the learning phase.

[0155] In the estimation phase, the image estimation device 403 first extracts a first original image 502 and a second original image 503 from an original video 501. Next, the first original image 502 and the second original image 503 are each divided to generate a first image 504 and a second image 505. Next, the first image 504 and the second image 505 are each reduced to generate a third image 506 and a fourth image 507.

[0156] Next, the image estimation device 403 inputs the third image 506 and the fourth image 507 into a first machine learning model to generate a first motion vector 508. Next, the first motion vector 508 is enlarged to generate a second motion vector 509. Next, the image estimation device 403 inputs the first image 504, the second image 505, and the second motion vector 509 into a second machine learning model to generate a fifth image 510. Next, the fifth image 510 is combined to generate a target image 511.

[0157] Finally, the image estimation device 403 generates a target video 512 from the source video 501 and the target image 511. The fifth image 510 is an image located at the center between the first image 504 and the second image 505. The target image 511 is a frame image located at the center between the first original image 502 and the second original image 503. The target video 512 is a video obtained by approximately doubling the frame rate of the source video 501.

[0158] The flowchart in FIG. 15 shows the processing performed by the image estimation device 403 as the estimation phase. First, in step S801, the acquisition unit 403b acquires the original video 501, weight information for the first machine learning model, and weight information for the second machine learning model. In this embodiment, the original video 501 is a captured video generated by the optical system 302a and the image sensor 302b. However, the acquired original video 501 may be a portion of the captured video. For example, it may be a video obtained by cropping the captured video in the spatial or temporal direction. Furthermore, the original video 501 may be expressed in grayscale or may have multiple channel components. The weight information for the first machine learning model and the weight information for the second machine learning model are read out in advance from the storage unit 401a and stored in the storage unit 403a.

[0159] Next, in step S802, the generation unit 403c extracts a first original image 502 and a second original image 503 from the original video 501. The first original image 502 and the second original image 503 are frame images that make up the original video 501. In this embodiment, as shown in Fig. 14, the second original image 503 is a frame image that is adjacent to the first original image 502 in the original video 501.

[0160] Next, in step S803, the generation unit 403c divides the first original image 502 and the second original image 503 to generate a first image 504 and a second image 505. The first image 504 and the second image 505 are both images of the same third size. That is, the first image 504 and the second image 505 are images obtained by cropping areas of the third size from the same positions on the first original image 502 and the second original image 503. In this embodiment, the third size is 256 × 256 pixels. As in the first embodiment, it is preferable to divide the first original image 502 so that multiple first images 504 include a common area. Note that if the sizes of the first original image 502 and the second original image 503 are the same as the third size, the division process of step S803 is unnecessary, and the accompanying merging process of step S809 is also unnecessary.

[0161] Next, in step S804, the generation unit 403c reduces the first image 504 and the second image 505 to generate a third image 506 and a fourth image 507. The reduction from the first image 504 to the third image 506 and the reduction from the second image 505 to the fourth image 507 are performed using the same reduction process that converts from image to image. The reduction process in this embodiment is downsampling, which extracts only one pixel from multiple pixels.

[0162] The third image 506 and the fourth image 507 are both images of the same first size. In this embodiment, the first size is 128 × 128 pixels. Therefore, the reduction ratio from the first image 504 to the third image 506 and the reduction ratio from the second image 505 to the fourth image 507 are both double in the width direction and double in the height direction of the image.

[0163] Next, in step S805, the estimation unit 403d inputs the third image 506 and the fourth image 507 into a first machine learning model to generate a first motion vector 508. The first motion vector 508 is a vector that represents the movement of corresponding pixels between the third image 506 and the fourth image 507. In this embodiment, the first motion vector 508 has the same first size (128×128 pixels) as the third image 506 and the fourth image 507. However, the size of the first motion vector 508 is not limited to this.

[0164] In this embodiment, the first motion vectors are two types of vectors that indicate the movement of corresponding pixels between the third image 506 and the fourth image 507. One type of vector indicates the movement from the fourth image 507 to the third image 506 for each of the third images 506. The other type of vector indicates the movement from the third image 506 to the fourth image 507 for each of the fourth images 507. Also in this embodiment, the first motion vectors are four types of two-dimensional maps, and each two-dimensional map indicates the amount of horizontal or vertical movement for each pixel position in the third image 506 or the fourth image 507.

[0165] Next, in step S806, the generation unit 403c enlarges the first motion vector 508 to generate a second motion vector 509. The enlargement process for the first motion vector 508 is a process independent of the first machine learning model. This enlargement process may be a process using a machine learning model, or may be a process without using a machine learning model, as long as it is independent of the first machine learning model. In this embodiment, the enlargement process is a process using bicubic interpolation. More specifically, in the enlargement process, the second motion vector 509 is generated by multiplying each pixel of the motion vector obtained by enlarging the first motion vector 508 using bicubic interpolation by an enlargement factor (2x in this embodiment, as described later). In addition, in this embodiment, the enlargement factor in this enlargement process is the same as the reduction factor in the reduction process in step S804, i.e., 2x in the width direction and 2x in the height direction of the map. In other words, the size of the second motion vector is 256 x 256 pixels. In this embodiment, the second motion vector 509 has the same size as the first image 504 and the second image 505.

[0166] Next, in step S807, the estimation unit 403d inputs the first image 504, the second image 505, and the second motion vector 509 into the second machine learning model to generate a fifth image 510. Here, the estimation unit 403d first generates an eighth motion vector and a ninth motion vector using the second motion vector 509 within the second machine learning model. The eighth motion vector is a vector that indicates the movement of corresponding pixels between the first image 504 and the fifth image 510. In this embodiment, the eighth motion vector indicates the movement from the fifth image 510 to the first image 504 for each image in the first image 504. Specifically, the eighth motion vector is generated by multiplying each pixel of the second motion vector 509 that indicates the movement from the second image 505 to the first image 504 for each image in the first image 504 by 1 / 2.

[0167] On the other hand, the ninth motion vector is a vector that indicates the movement of corresponding pixels between the second image 505 and the fifth image 510. In this embodiment, the ninth motion vector indicates the movement from the fifth image 510 to the second image 505 for each of the second images 505. Specifically, the ninth motion vector is generated by multiplying each pixel of the second motion vector 509 that indicates the movement from the first image 504 to the second image 505 for each of the second images 505 by 1 / 2.

[0168] Next, the estimation unit 403d generates a sixth warped image and a seventh warped image using the first image 504, the second image 505, the eighth motion vector, and the ninth motion vector within the second machine learning model. The sixth warped image is an image corresponding to the fifth image 510 generated based on the first image 504 and the eighth motion vector. The sixth warped image is generated by moving pixels of the first image 504 using the eighth motion vector. Specifically, each pixel value of the sixth warped image is calculated from the pixel value of the first image 504 using a known interpolation method such as nearest neighbor interpolation, bilinear interpolation, or bicubic interpolation.

[0169] In this embodiment, forward warping is employed to align the first image 504 to an image equivalent to the fifth image 510, using an eighth motion vector indicating the movement from the fifth image 510 to the first image 504 for each image in the first image 504. The same forward warping is employed for the seventh warped image.

[0170] On the other hand, the seventh warped image is an image equivalent to the fifth image 510, generated based on the second image 505 and the ninth motion vector. The seventh warped image is generated by moving pixels of the second image 505 using the ninth motion vector. Specifically, each pixel value of the seventh warped image is calculated from the pixel value of the second image 505 using a known interpolation method such as nearest neighbor interpolation, bilinear interpolation, or bicubic interpolation.

[0171] Finally, the estimation unit 403d generates a fifth image 510 from the sixth warped image and the seventh warped image within the second machine learning model. Specifically, the fifth image 510 is generated based on the average or weighted average of the pixel values ​​of the sixth warped image and the pixel values ​​of the seventh warped image. In this embodiment, the fifth image 510 is generated by setting the average of the pixel values ​​of the sixth warped image and the pixel values ​​of the seventh warped image as the pixel values ​​of the fifth image 510.

[0172] Next, in step S808, the generation unit 403c determines whether or not generation of the fifth images 510 has been completed for all pairs of the first images 504 and the second images 505. If it is determined that generation of all the fifth images 510 has not been completed, the process returns to step S804, and the generation unit 403c generates a fifth image 510 from a new pair of the first image 504 and the second image 505. If it is determined that generation of all the fifth images 510 has been completed, the process proceeds to step S809.

[0173] In step S809, the generation unit 403c combines the fifth image 510 to generate a target image 511. In this embodiment, the target image 511 is an image located at the center of the first image 504 and the second image 505.

[0174] Next, in step S810, the generation unit 403c determines whether or not the generation of the target image 511 has been completed for all pairs of the first original image 502 and the second original image 503. If it is determined that the generation of all target images 511 has not been completed, the process returns to step S803, and the generation unit 403c generates a target image 511 from a new pair of the first original image 502 and the second original image 503. If it is determined that the generation of all target images 511 has been completed, the process proceeds to step S811.

[0175] In step S811, the generation unit 403c generates a target video 512 from the first original image 502, the second original image 503, and the target image 511. Specifically, the target video 512 is generated by placing the target image 511 in the center of the first original image 502 and the second original image 503. In this embodiment, the target video 512 is a video in which the frame rate of the original video 501 has been increased by approximately twice.

[0176] According to this embodiment, it is possible to achieve a highly accurate up-frame task without being limited by the image size on which the machine learning model that generates the motion vector is trained. [Example]

[0177] The third embodiment aims to increase the resolution of an image. Two images (hereinafter referred to as low-resolution images) containing at least a portion of the same object at different positions are used to generate a single image (a super-resolution image: hereinafter referred to as a high-resolution image) with a higher resolution than the two images. To achieve this, a first image, which is at least a portion of one of the low-resolution images, and a second image, which is at least a portion of the other of the low-resolution images, are reduced to generate a third image and a fourth image. A first motion vector is generated based on the third image and the fourth image using a first machine learning model. A high-resolution fifth image corresponding to the first image is generated based on a second motion vector obtained by enlarging the first motion vector, the first image, and the second image using a second machine learning model. The fifth image is at least a portion of a high-resolution image to be generated. The fifth image may be an upscaled version of the first image, or an image obtained by removing noise, blur, shake, etc. from the first image. The effects obtained in this embodiment are similar to those in the first embodiment.

[0178] In this embodiment, one high-resolution image is generated using two low-resolution images generated by continuous imaging (continuous shooting) using the same optical system and imaging element. However, this is not limited to this, and one high-resolution image may be generated using two low-resolution images generated by imaging using the same optical system and imaging element but at different imaging element positions. Also, one high-resolution image may be generated using two low-resolution images generated by imaging using different optical systems and imaging elements. Note that in this embodiment, one high-resolution image is generated using two low-resolution images, but it is preferable to use a larger number of low-resolution images as in Example 1.

[0179] This embodiment uses the same image processing system 200 as in Embodiment 1. Unlike Embodiment 1, an image estimation unit 223 in this embodiment uses two captured images generated by continuous shooting using an optical system 221 and an image sensor 222 to generate a high-resolution image corresponding to one of the captured images.

[0180] Similar to Example 1, the processing performed in this Example can be broadly divided into generating training data for a first machine learning model, learning weights for the first machine learning model (first training phase), generating training data for a second machine learning model, learning weights for the second machine learning model (second training phase), and estimation by the first machine learning model and the second machine learning model using the trained weights (estimation phase). This Example is similar to Example 1 except for the estimation phase.

[0181] The estimation phase of this embodiment will be described based on the estimation phase of embodiment 1. The steps in the estimation phase of this embodiment are steps S503 to S509 of the steps in the estimation phase of embodiment 1. In this embodiment, the first original image and the second original image in embodiment 1 correspond to two low-resolution images. Also, the target image in embodiment 1 corresponds to one high-resolution image to be generated in this embodiment.

[0182] In this embodiment, too, it is possible to achieve the task of increasing the resolution of an image without being limited by the image size on which the machine learning model that generates the motion vector has been trained.

[0183] The above embodiments include the following methods.

[0184] (Configuration 1) reducing a first image and a second image that include at least a portion of the same object in different positions to generate a third image corresponding to the first image and a fourth image corresponding to the second image; generating a first motion vector based on the third image and the fourth image using a first machine learning model; expanding the first motion vector to generate a second motion vector; and generating a fifth image based on the first image, the second image, and the second motion vector using a second machine learning model. (Configuration 2) 2. The image processing method according to configuration 1, wherein the first image and the second image are images extracted from the same moving image. (Configuration 3) 3. The image processing method according to claim 1, wherein the first image and the second image are images obtained by dividing a first original image and a second original image, respectively. (Configuration 4) 4. The image processing method according to any one of configurations 1 to 3, wherein the fifth image corresponds to the first image and has a higher resolution than the first image. (Configuration 5) 4. The image processing method according to any one of configurations 1 to 3, wherein the fifth image is an upscaled version of the first image. (Configuration 6) The image processing method according to any one of configurations 1 to 3, wherein the fifth image is an image that constitutes a moving image that includes the first and second images and has an up-frame rate. (Configuration 7) 7. The image processing method according to any one of configurations 1 to 6, wherein the first machine learning model is a convolutional neural network. (Configuration 8) 8. The image processing method according to any one of configurations 1 to 7, wherein the second machine learning model is a convolutional neural network. (Configuration 9) 9. The image processing method according to any one of configurations 1 to 8, wherein the first image, the second image, and the second motion vector have the same size. (Configuration 10) 10. The image processing method according to any one of configurations 1 to 9, wherein the expansion of the first motion vector is performed using an interpolation process or a machine learning model that is trained independently of the first machine learning model. (Configuration 11) the third image and the fourth image are both images of a first size; An image processing method according to any one of configurations 1 to 10, characterized in that the first training image set used to train the first machine learning model is images of the first size or larger. (Configuration 12) the first image and the second image are both images of a third size; An image processing method according to any one of configurations 1 to 11, characterized in that the second training image set used to train the second machine learning model is images of a fourth size or smaller. (Configuration 13) A learning method for learning the first and second machine learning models used in the image processing method according to any one of configurations 1 to 5 and configurations 7 to 12, inputting the first training image and the second training image into the first machine learning model to generate a third motion vector; generating a first warped image corresponding to the first training image using the second training image and the third motion vector; training the first machine learning model based on the difference between the first training image and the first warped image; inputting a third training image and a fourth training image into the trained first machine learning model to generate a fourth motion vector; inputting the third training image, the fourth training image, and the fourth motion vector into the second machine learning model, and causing the second machine learning model to generate a second warped image corresponding to the third training image based on the fourth training image and the fourth motion vector, and to generate a sixth image using the third training image and the second warped image; and training the second machine learning model based on the difference between the sixth image and a correct image.

[0185] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0186] The embodiments described above are merely representative examples, and various modifications and changes can be made to each embodiment when implementing the present invention. [Explanation of symbols]

[0187] 223 Image Estimation Unit 223a Acquisition Department 223c Estimation part 403 Image Estimation Device 403b Acquisition Department 403d Estimation Department

Claims

1. reducing a first image and a second image including at least a portion of the same object at different positions to generate a third image corresponding to the first image and a fourth image corresponding to the second image; generating a first motion vector based on the third image and the fourth image using a first machine learning model; generating a second motion vector by scaling up the first motion vector; and generating a fifth image based on the first image, the second image, and the second motion vector using a second machine learning model.

2. 2. The image processing method according to claim 1, wherein the first image and the second image are images extracted from the same moving image.

3. 2. The image processing method according to claim 1, wherein the first image and the second image are images obtained by dividing a first original image and a second original image, respectively.

4. 2. The image processing method according to claim 1, wherein the fifth image corresponds to the first image and has a higher resolution than the first image.

5. 2. The image processing method according to claim 1, wherein the fifth image is an upscaled version of the first image.

6. 2. The image processing method according to claim 1, wherein the fifth image is an image that constitutes a moving image that includes the first and second images and is up-frame-rated.

7. The image processing method according to claim 1 , wherein the first machine learning model is a convolutional neural network.

8. The image processing method according to claim 1 , wherein the second machine learning model is a convolutional neural network.

9. 2. The image processing method according to claim 1, wherein the first image, the second image, and the second motion vector have the same size.

10. The image processing method according to claim 1 , wherein the expansion of the first motion vector is performed using an interpolation process or a machine learning model that is trained independently of the first machine learning model.

11. the third image and the fourth image are both images of a first size; The image processing method according to claim 1 , wherein a first training image set used for training the first machine learning model is images of the first size or larger.

12. the first image and the second image are both images of a third size; The image processing method according to claim 1 , wherein the second training image set used for training the second machine learning model is images of a fourth size or smaller.

13. A learning method for learning the first and second machine learning models used in the image processing method according to any one of claims 1 to 5 and claims 7 to 12, comprising: inputting the first training image and the second training image into the first machine learning model to generate a third motion vector; generating a first warped image corresponding to the first training image using the second training image and the third motion vector; training the first machine learning model based on the difference between the first training image and the first warped image; inputting a third training image and a fourth training image into the trained first machine learning model to generate a fourth motion vector; inputting the third training image, the fourth training image, and the fourth motion vector into the second machine learning model, and causing the second machine learning model to generate a second warped image corresponding to the third training image based on the fourth training image and the fourth motion vector, and to generate a sixth image using the third training image and the second warped image; and training the second machine learning model based on the difference between the sixth image and a correct image.

14. means for reducing a first image and a second image, each of which includes at least a portion of the same object at a different position from each other, to generate a third image corresponding to the first image and a fourth image corresponding to the second image; means for generating a first motion vector based on the third image and the fourth image using a first machine learning model; means for expanding the first motion vector to generate a second motion vector; and means for generating a fifth image based on the first image, the second image, and the second motion vector using a second machine learning model.

15. The image processing device according to claim 13; an imaging element that captures an image of an object through an optical system; An imaging device that generates an original image for obtaining the first and second images.

16. A learning device that performs learning of the first and second machine learning models used in the image processing method according to any one of claims 1 to 5 and claims 7 to 12, a means for inputting a first training image and a second training image into the first machine learning model to generate a third motion vector; means for generating a first warped image corresponding to the first training image using the second training image and the third motion vector; means for training the first machine learning model based on the difference between the first training image and the first warped image; a means for inputting a third training image and a fourth training image into the trained first machine learning model to generate a fourth motion vector; a means for inputting the third training image, the fourth training image, and the fourth motion vector into the second machine learning model, and causing the second machine learning model to generate a second warped image corresponding to the third training image based on the fourth training image and the fourth motion vector, and to generate a sixth image using the third training image and the second warped image; and means for training the second machine learning model based on the difference between the sixth image and a correct image.

17. A program causing a computer to execute a process according to the image processing method of any one of claims 1 to 12.

18. A program causing a computer to execute processing according to the learning method of claim 13.

Images