Image processing method, image processing apparatus, and program

Abstract

To provide an image processing method for enabling highly accurate motion vector estimation using a machine learning model on a high-resolution image.SOLUTION: An image processing method includes: Step (S203) of acquiring a second image set of a second size, which is smaller than a first size, the second image set being a partial region of a first image set, based on the first image set including a first image and a second image of the first size; and Step (S204) of inputting the second image set to a machine learning model to obtain a motion vector. The motion vector is a motion vector in the second image based on the first image. The machine learning model has been trained using a third image set of a third size. The second size is equal to or smaller than a fourth size which is a basis for the third size.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing method, an image processing device, and a program. [Background technology]

[0002] In image processing using a machine learning model, a technique for estimating a motion vector (optical flow) is known. Patent Document 1 discloses a method for training a machine learning model that estimates the optical flow between temporally adjacent frames (images) that make up a video. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2018-156640 Summary of the Invention [Problem to be solved by the invention]

[0004] In image processing using machine learning models, the image size used during training may differ from the image size used during estimation. There are limitations on how large the image size can be during training. One reason is that it is generally difficult to collect a sufficient number of high-resolution training datasets, especially for video datasets. Furthermore, in machine learning, it is desirable to reduce the training time while training on a variety of scenes. Therefore, in order to simultaneously compute many scenes with a limited amount of memory, it is necessary to limit the image size during training to a specified size. Meanwhile, when estimating, it may be desirable to apply image processing to high-resolution content.

[0005] However, in estimating a motion vector using machine learning, if the image size during estimation is larger than the image size during learning, the estimation accuracy decreases.

[0006] The method disclosed in Patent Document 1 does not take into account the image size during learning and the image size during estimation, so accuracy may decrease when estimating optical flow by inputting a high-resolution video.

[0007] Therefore, an object of the present invention is to provide an image processing method that can perform highly accurate motion vector estimation using a machine learning model for high-resolution images. [Means for solving the problem]

[0008] An image processing method as one aspect of the present invention includes the steps of: based on a first image set including a first image and a second image of a first size, acquiring a second image set of a second size that is a partial region of the first image set and is smaller than the first size; and inputting the second image set into a machine learning model to acquire a motion vector, wherein the motion vector is a motion vector in the second image relative to the first image, the machine learning model is trained using a third image set of a third size, and the second size is equal to or smaller than a fourth size that is a reference with respect to the third size.

[0009] Other objects and features of the present invention are illustrated in the following examples. [Effects of the Invention]

[0010] According to the present invention, it is possible to provide an image processing method that can perform highly accurate motion vector estimation using a machine learning model for high-resolution images. [Brief explanation of the drawings]

[0011] [Figure 1] 3 is a flowchart of a learning method in the first embodiment. [Figure 2] 1 is a flowchart of an estimation method according to the first embodiment. [Figure 3] FIG. 10 is a diagram illustrating the relationship between an input image set and divided regions in the first embodiment. [Figure 4]FIG. 10 is an explanatory diagram of image processing in the second embodiment. [Figure 5] 10 is a flowchart of a learning method in the second embodiment. [Figure 6] 10 is a flowchart of an estimation method according to a second embodiment. [Figure 7] FIG. 2 is a block diagram of an image processing system in each embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same components are designated by the same reference numerals, and redundant explanations will be omitted.

[0013] The image processing unit of each embodiment performs motion vector estimation processing using a machine learning model on an input image set. Here, the image set is composed of a plurality of images including at least a first image and a second image, and may be an image pair consisting of two images, the first image and the second image. Furthermore, the motion vector is the motion vector of the second image relative to the first image, and corresponds to the difference in position of the same object included in the images (first image and second image) included in the image set. The motion vector is estimated using two images (still images), such as images (frames) taken at different times in a video (video data), stereo images captured from different viewpoints, or multiple images captured in continuous shooting.

[0014] Motion vectors are also called optical flow. Motion vectors are obtained, for example, as a map corresponding to an image. Each pixel value in the map is a value of the displacement along a specific direction, and represents the displacement in a different image relative to one image. In stereo matching, motion vectors may be obtained as a single map with values ​​in only one direction, or more commonly, as maps corresponding to multiple directions, such as the horizontal and vertical directions of the image.

[0015] By estimating optical flow, it is possible to track objects in video, estimate the amount of parallax between stereo images, and align multiple images. Alignment is used when combining multiple images, and can reduce noise through image synthesis processing, and can sharpen and increase resolution based on differences in sampling of the same object. In addition, position interpolation can be used to increase the frame rate of video.

[0016] To train a machine learning model that performs motion vector estimation processing, correct motion vector data is used for an image set containing multiple images. Motion vector data measured for photographed images may be used, or CG data with known motion vector values ​​may be used. For example, in stereo images, the amount of parallax can be calculated by measuring distance information, making it possible to obtain correct motion vector data. Then, the image set is input to a machine learning model such as a neural network to estimate motion vectors, and the parameters of the machine learning model are optimized to minimize the difference from the correct motion vector. Alternatively, learning may be performed using unsupervised learning, which does not require a correct motion vector. For example, two images may be input to a machine learning model to estimate a motion vector, and a geometric transformation based on the estimated motion vector may be applied to one of the two images, and the parameters of the machine learning model may be optimized to minimize the difference from the other image.

[0017] In each embodiment, a motion vector is estimated between different frames that make up a moving image, but the subject of motion vector estimation is not limited to different frames that make up a moving image.

[0018] Next, the problems of each embodiment will be described in detail. When the input size of a machine learning model used for image processing is variable, the image size (third size) input to the model during learning may differ from the image size used when estimating using the learned model. Meanwhile, weight information (parameters) of the machine learning model is updated based on the image size used for learning. Therefore, if the image size input to the model during estimation is larger than the image size used for learning (or the reference image size), the estimation accuracy of the machine learning model decreases.

[0019] For example, in a convolutional neural network, if the convolution filter uses (references) values ​​outside the image for calculations, the image may be padded with zeros or a fixed value, and the convolution filter is therefore trained based only on the padded values ​​outside the image.

[0020] However, if the image size input to the model during estimation is larger than the image size used during training, pixel values ​​corresponding to the scene contained in the image are input, unlike padding. Inputting an image under different conditions than during training reduces the estimation accuracy. On the other hand, if the image size input to the model during estimation is smaller than the image size used during training, the model has been trained on images of various scenes, so padding the image during estimation does not reduce the estimation accuracy.

[0021] When the convolution filter is 3x3, the only pixels that reference the outside of the image are the pixels at the very periphery of the image. Therefore, it is thought that only the very periphery pixels will suffer from a decrease in estimation accuracy. However, by including multiple layers of convolution processing, the area of ​​the input image that is indirectly referenced becomes larger. In this way, the area of ​​the input image that a machine learning model indirectly references when processing a given pixel is called the receptive field. In a neural network with three convolution layers of 3x3 filters, the receptive field is a 7x7 area.

[0022] As the number of layers in a neural network increases, the receptive field expands, and more pixels outside the image are referenced. While a larger receptive field allows the machine learning model to consider a wider area of ​​the input image, it also increases the area of ​​the image where estimation accuracy decreases.

[0023] However, the aforementioned issues depend on the image processing task performed by the machine learning model. For example, resolution enhancement (upscaling) is a process that corrects degradation during interpolation, but this can be corrected based only on local information. As another example, a process that corrects aberrations in the optical system that captured the image can be corrected based on the local image region affected by the aberration. In this way, in image processing tasks that can be performed based on pixel values ​​of relatively small image regions, even if the receptive field of the machine learning model is large, the parameters of the machine learning model may be trained to emphasize image regions smaller than the image size used during training. Therefore, even if the image size used during estimation is larger than the image size used during training, the aforementioned issues are unlikely to occur because the machine learning model does not emphasize only small image regions.

[0024] Furthermore, in these image correction tasks, when image degradation is expressed as a convolution, the degradation kernel is determined independently of the object. Therefore, the image size required for correction can be estimated in advance and the image size during training can be determined. Therefore, in image correction tasks where the degradation to be corrected does not depend on the global structure of the object, the image size during training can be set large for the image region that the machine learning model prioritizes, making the aforementioned issue less likely to occur.

[0025] On the other hand, in an image processing task for estimating a motion vector of a subject, the upper limit of the size of the motion vector is not determined, and the estimation must be based on a wider area of ​​the image than in the previous task. Therefore, the machine learning model is trained to estimate the motion vector based on pixel values ​​of a wider area. Therefore, if an image size larger than the image size used during training is input, the estimation accuracy decreases. The image size used during training is limited to a size smaller than a certain size depending on the memory capacity of the processing device used during training (e.g., a GPU (Graphics Processing Unit)), the training time, or the size of the training dataset. If a high-resolution image is input during estimation, the image will be larger than the size of the training image, and the estimation accuracy of the motion vector will decrease.

[0026] If the training dataset does not contain large movements, the image regions that the machine learning model focuses on are localized, so accuracy does not decrease even if the input image size is large. On the other hand, accuracy decreases when estimating large movements.

[0027] Each example will be described in detail below.

[0028] Example 1 First, an image processing system 100 in a first embodiment will be described with reference to Fig. 7. Fig. 7 is a block diagram of the image processing system 100. The image processing system 100 includes a learning device (image processing device) 101, an imaging device 102, an image estimation device (image processing device) 103, a display device 104, a recording medium 105, an output device 106, and a network 107. The learning device 101 includes a memory unit (storage means) 101a, an acquisition unit (acquisition means) 101b, a generation unit (generation means) 101c, and an update unit (learning means) 101d.

[0029] The imaging device 102 has an optical system 102a and an imaging element 102b. The optical system 102a collects light incident on the imaging device 102 from the subject space. The imaging element 102b receives (photoelectrically converts) an optical image (subject image) formed via the optical system 102a to obtain a captured image. The imaging element 102b is, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal-Oxide Semiconductor) sensor. The captured image obtained by the imaging device 102 contains blur due to aberration and diffraction of the optical system 102a and noise due to the imaging element 102b.

[0030] The estimation device 103 has a storage unit 103a, an acquisition unit 103b, and an estimation unit 103c. The estimation device 103 acquires a captured image and estimates a motion vector. A neural network is used for motion vector estimation, and weight information (parameters) is read from the storage unit 103a. The weights (weight information) are obtained through learning by the learning device 101, and the estimation device 103 reads the weight information from the storage unit 101a via the network 107 in advance and stores it in the storage unit 103a. The stored weight information may be the weight numerical value itself or in an encoded format. Details regarding weight learning and motion vector estimation processing using the weights will be described later.

[0031] The estimated and output motion vector is output to at least one of a display device 104, a recording medium 105, and an output device 106. The display device 104 is, for example, a liquid crystal display or a projector. The recording medium 105 is, for example, a semiconductor memory, a hard disk, a server on a network, etc. The output device 106 is, for example, a printer. The estimation device 103 has a function of performing other image processing as necessary.

[0032] [Learning Method] Next, a learning method for motion vector estimation processing in this embodiment will be described with reference to Fig. 1. Fig. 1 is a flowchart of the learning method for motion vector estimation processing. The flowchart in Fig. 1 can be embodied as a program for causing a computer to execute the functions of each step. This also applies to the following flowcharts. Each step in Fig. 1 is mainly executed by the acquisition unit 101b, generation unit 101c, or update unit 101d of the learning device 101.

[0033] First, in step S101, the acquisition unit 101b acquires two consecutive images (frames) from a learning dataset of a video as an image set including a plurality of images (first image, second image) used for learning. The acquisition unit 101b also acquires data that is the correct answer for one image (first image) relative to the other image (second image), i.e., the motion vector between the two images.

[0034] The image set may be the entire area of ​​the images included in the training dataset, or a partial area of ​​the image. Here, an area of ​​a predetermined size (third size) in the same image position on two images is randomly cropped to obtain the set. Known data augmentation techniques, such as changing the brightness or color of the image set, may also be used. Here, an area of ​​128 x 128 size is obtained. The same area is also obtained for the ground truth data. If cropping is not performed, the full-pixel image size corresponds to the third size.

[0035] Next, in step S102, the generation unit 101c inputs the image set acquired in step S101 into a machine learning model to acquire estimated motion vectors. The machine learning model may be a known machine learning model such as a convolutional neural network. Here, the motion vectors have the same resolution as the images.

[0036] Next, in step S103, the update unit 101d calculates (obtains) the error (error amount) between the motion vector obtained in step S102 and the correct motion vector obtained in step S101. The error can be calculated using an index such as absolute error or L2 norm, but is not limited to these.

[0037] Subsequently, in step S104, the update unit 101d updates the parameters of the machine learning model by backpropagating the error acquired in step S103.

[0038] Next, in step S105, the update unit 101d determines whether to terminate learning of the machine learning model. For example, it may be determined that learning is to be terminated when a predetermined number of updates is exceeded or the amount of error falls below a reference value. If learning is not to be terminated, the process returns to step S101, and the acquisition unit 101b acquires a new image set and a correct motion vector, and repeats the process. On the other hand, if learning is to be terminated, the learning of this embodiment is terminated and the parameters of the trained machine learning model are obtained.

[0039] In this embodiment, an example in which a correct motion vector is used has been described, but when performing unsupervised learning, it is not necessary to obtain a correct motion vector in step S101. As the error in step S103, a geometric transformation based on the estimated motion vector is applied to one of the two images (e.g., the second image), and the difference with the other image (e.g., the first image) is evaluated. For example, the L1 norm can be used as the evaluation index.

[0040] The image set acquired in step S101 may be acquired for multiple scenes. In this case, a motion vector is estimated for each of the multiple scenes in step S102, and an error is calculated in step S103. The error calculated in step S103 (the acquired error amount) uses the total or average for each scene. Alternatively, three or more consecutive images may be acquired, and a motion vector may be estimated between each adjacent image. In this case, similar processing may be performed for each pair of adjacent images.

[0041] [Estimation method] Next, a motion vector estimation process using a machine learning model trained by the learning method described with reference to Fig. 1 will be described with reference to Fig. 2. Fig. 2 is a flowchart of the motion vector estimation process. Each step in Fig. 2 is mainly executed by the acquisition unit 103b or the estimation unit 103c of the estimation device 103.

[0042] First, in step S201, the acquisition unit 103b acquires a plurality of images for estimating a motion vector as an input image set (first image set). In this embodiment, the size (first size) of the input image set is 4K resolution (3840 × 2160), but is not limited to this. The 4K video is decoded to acquire two adjacent images (frames), a first image and a second image.

[0043] Subsequently, in step S202, the acquisition unit 103b acquires a machine learning model trained by the learning method described with reference to Fig. 1. The machine learning model of this embodiment includes processing by a neural network.

[0044] Next, in step S203, the acquisition unit 103b acquires one input divided image set (second image set) to be input to the machine learning model from the divided regions (partial regions) into which the input image set is divided.

[0045] Here, a method for acquiring an input divided image set will be described in detail with reference to Fig. 3. Fig. 3 is a diagram showing the relationship between an input image set and a plurality of divided regions, where 201 is an input image set (first image set). 202 indicates division positions, indicated by dashed lines, at which the input image set is divided into blocks, and each of the plurality of partial regions enclosed by the dashed lines corresponds to an acquisition region when acquired as a divided image. The division positions and division sizes are set as predetermined values, and the input image set is divided into partial regions a1 to aN.

[0046] The multiple images (first image and second image) included in the input image set are each divided at the same position, and the same partial regions are obtained to form a divided image set (second image set). Since the machine learning model processes input in block units, one partial region is obtained from partial regions a1 to aN in step S203. The partial regions may be set overlapping each other.

[0047] Here, the larger the division size is relative to the image size of 128 × 128 used during training, the lower the accuracy of motion vector estimation. Therefore, in this embodiment, the division size is set to 128 × 128, the same as the image size used during training. However, the division size may be different from the image size used during training. Note that FIG. 6 may further include a step of determining a second size based on at least one of the first size, the fourth size, and the machine learning model.

[0048] Next, in step S204, the estimation unit 103c inputs the set of divided images of a predetermined size (second size) acquired in step S203 into the machine learning model acquired in step S202. The estimation unit 103c then estimates a motion vector (division motion vector) corresponding to the set of divided images. The division motion vector is a map of the same image size (128 x 128) as each image in the set of divided images, and has two channels: a horizontal component and a vertical component.

[0049] Next, in step S205, the estimation unit 103c determines whether all of the partial regions (input partial images) in the divided image set have been processed. If it is determined that the processing of the partial regions has not been completed, the process returns to step S203, and the unprocessed partial regions are acquired as a divided image set, and steps S203 and S204 are executed for the acquired divided image set. On the other hand, if it is determined that the processing of all partial regions has been completed, the process proceeds to step S206.

[0050] In step S206, the acquisition unit 103b acquires an output motion vector of size 3840 x 2160 by arranging and combining the multiple division motion vectors as partial data so that they are in the same positional relationship as before division. In this embodiment, instead of the division motion vectors, images acquired based on the second image set and the division motion vectors may be acquired as partial data. The acquisition unit 103b then combines multiple partial data corresponding to multiple different partial regions. Note that if the division regions overlap, they may be cut out so that they do not overlap, or the overlapping portions may be combined by taking a weighted average. This completes the image processing in this embodiment.

[0051] In this embodiment, the division size (second size) in step S203 does not have to be 128 x 128, but may be, for example, 192 x 192 (i.e., 1.5 times 128) or 160 x 160 (i.e., 1.25 times 128). If the image size during estimation is equal to or smaller than the image size during learning, highly accurate estimation based on learning is possible. However, the accuracy of the estimated motion vector gradually decreases as the image size during estimation increases.

[0052] The inventors have investigated various variations (combinations) of the image size (first size) used in learning and the division size (second size) used in estimation, and have found that there is a reference image size (fourth size) for the image size used in estimation that can be estimated with high accuracy. Here, the fourth size is a reference image size with respect to the third size, which is the image size used in learning. In FIG. 2 or FIG. 6 (described later), a step of acquiring the fourth size may be further included.

[0053] By setting the size of the image (second size) input to the trained machine learning model during estimation to be equal to or smaller than the reference image size (fourth size), it is possible to suppress a decrease in estimation accuracy. Note that the reference image size also depends on the magnitude of the motion vector of the scene. Therefore, the reference image size may be changed depending on the image size (third size) during learning and the magnitude of the motion vector.

[0054] Preferably, the reference image size (fourth size) is 1.5 times or less the image size (third size) used during learning. With this configuration, it is possible to estimate a motion vector with high accuracy. More preferably, the reference image size is 1.25 times or less the image size used during learning. With this configuration, it is possible to estimate a motion vector with even higher accuracy. Even more preferably, the reference image size is 1 time or less the image size used during learning. With this configuration, it is possible to estimate a motion vector with even higher accuracy.

[0055] It should be noted that the important aspect of image size is not the total number of pixels in the image, but the number of horizontal or vertical pixels (number of pixels on one side). If the pixel range referenced by the machine learning model becomes larger than the image size during learning, the estimation accuracy will decrease. For this reason, it is preferable that the number of horizontal and vertical pixels (number of pixels on one side) as the division size be equal to or smaller than the reference image size. It should be noted that the number of pixels on one side is not limited to the number of pixels in the horizontal or vertical direction, but may also be the number of pixels in the diagonal direction.

[0056] When sequentially learning using multiple different datasets, such as in pre-learning, transfer learning, or fine-tuning, it is preferable that at least one dataset meets the image size criterion. More preferably, all datasets meet the image size criterion. To estimate motion vectors with high accuracy, it is preferable to perform learning so that the parameters of the machine learning model are determined mainly by datasets that meet the criterion. For example, when training with a training dataset that does not meet the criterion after sufficient training with datasets that meet the criterion, it is preferable to limit the number of fine-tuning steps.

[0057] Furthermore, in the case of a multitask machine learning model, it is preferable to fix the weights used to estimate motion vectors trained with a dataset that meets the criteria, and train other tasks with a dataset that does not meet the criteria. It is preferable to thoroughly retrain a pre-trained model trained with images smaller than the criteria with a dataset that meets the criteria.

[0058] In a model configuration in which division and reduction are performed within a machine learning model, even if only a portion of the configuration meets the criteria, the estimation accuracy will decrease if there are portions that do not meet the criteria. Therefore, it is preferable that the image size at the time of estimation is equal to or smaller than the standard compared to the image size at the time of learning for the entire configuration that affects the final estimated motion vector. In this case, rule-based processing such as average pooling and bilinear interpolation is not relevant to the present invention, and it is sufficient that the size criteria are met for processing using parameters determined by learning.

[0059] Therefore, after dividing into 256x256 images in step S203, the images (second size images) reduced to 128x128 (reduced number of pixels) may be input to the machine learning model as a divided image set. However, performing the reduction process reduces the estimation accuracy of details. Therefore, in this embodiment, in which motion vectors are estimated with high accuracy even for high-resolution images, the division process is essential, and it is preferable to reduce the image size by about half.

[0060] In this embodiment, the size (first size) of the input image set acquired in step S201 is preferably larger than the reference size (fourth size). When the input image set is larger than the size (second size) input to the machine learning model, the estimation accuracy can be improved by acquiring a segmented image set using the processing of this embodiment. In particular, when the input image set is larger than the fourth size, the estimation accuracy also decreases significantly, so the estimation accuracy can be significantly improved by acquiring a segmented image set using the processing of this embodiment.

[0061] In this embodiment, whether or not to acquire a divided image set may be determined based on the size of the input image set (first size) and a reference size (fourth size). If the input image set is larger than the reference size, a divided image set is acquired and input to the machine learning model. On the other hand, if the input image set is equal to or smaller than the reference size, the input image set is input directly to the machine learning model. This allows optimal processing according to the size of the input image set. If the input image set is equal to or smaller than the reference size, the time required for the division process can be reduced by inputting the input image set directly to the machine learning model.

[0062] In this embodiment, the divided image set (second image set) input to the machine learning model may be an image set obtained by reducing a partial region of the input image set. As described above, the image set reduced after division may be used as the divided image set, or the image set divided after reduction may be used as the divided image set.

[0063] In the estimation process of this embodiment, the image size during learning (third size) may be acquired. For example, by acquiring the image size during learning at the same time as acquiring the machine learning model in step S202, the reference size can be determined based on the image size during learning.

[0064] In this embodiment, the image size (second size) of the divided image set may be determined based on at least one of the size of the input image set, the image size at the time of learning, and the machine learning model. The image size of the divided image set does not need to be fixed to a predetermined value. The time required for the estimation process can be shortened by setting the image size of the divided image set to a size that allows the input image set to be efficiently divided, taking into account overlapping of divided images. Furthermore, when the image size at the time of learning is acquired in association with the machine learning model, it is preferable to determine the image size of the divided image set based on the image size at the time of learning.

[0065] Furthermore, when a machine learning model to be used for the estimation process is selected from multiple models, the image size used during learning may differ for each model, so the second size may be determined based on the model. By linking and acquiring the model and the image size used during learning, it is possible to estimate a motion vector with high accuracy in accordance with the selected model.

[0066] In this embodiment, the machine learning model may be a convolutional neural network. A convolutional neural network can input images larger than the image size used for learning. By applying this embodiment to a convolutional neural network, it is possible to estimate motion vectors with high accuracy for a set of input images of any image size.

[0067] In this embodiment, the receptive field of the machine learning model is preferably larger than the second size. A larger receptive field allows for highly accurate estimation of motion vectors by taking into account the entire area of ​​the divided image set. If the image size of the divided image set is larger than the reference size, the degradation of estimation accuracy due to the problem of this embodiment also occurs over a wide area. However, by applying this embodiment, it is possible to estimate motion vectors by taking into account a wide area of ​​the input image while still performing highly accurate estimation.

[0068] In this embodiment, a motion vector corresponds to the amount of positional displacement of another image (second image) based on one image (first image) included in an image set. However, if the reference image is changed, the value of the motion vector changes. Therefore, a motion vector may be estimated for one reference image, or motion vectors may be estimated for multiple reference images. For example, for two images included in an image set, both a motion vector of one image based on the other image and a motion vector of one image based on the other image may be estimated.

[0069] Example 2 Next, a second embodiment of the present invention will be described. In the first embodiment, the process of estimating optical flow in a moving image has been described. The estimated optical flow may be utilized for other image processing tasks. In this embodiment, the process of increasing the resolution of a moving image based on the estimated optical flow will be described. Note that the image processing in this embodiment is performed by an image processing system having the same configuration as the image processing system 100 described in the first embodiment with reference to FIG. 1.

[0070] First, an overview of image processing in this embodiment will be described with reference to Fig. 4. Fig. 4 is an explanatory diagram of image processing in this embodiment. An image set 301 is an image set input to a machine learning model, and corresponds to the divided image set in the first embodiment. The motion vector estimation process is performed using a machine learning model, as in the first embodiment. Therefore, the image set 301 is input to a motion vector estimation model to obtain a motion vector 302 of an area corresponding to the image set 301. Furthermore, a high-resolution process of the image set 301 is performed using a machine learning model different from that used for motion vector estimation. The image set 301 and the motion vector 302 are input to a high-resolution model to output an image set 303 with a high resolution corresponding to the image set 301.

[0071] When increasing the resolution of a video, images taken at different times show the same subject in different positions, and therefore are sampled differently during capture. Therefore, by using multiple images taken at different times, it is possible to achieve a higher degree of accuracy in the high-resolution process than with a single image. However, the same subject exists in different positions within the images. Therefore, by taking the differences in position into account using a motion vector, it is possible to achieve a high-precision high-resolution process.

[0072] To train a machine learning model that performs high-resolution processing based on multiple images, a correct high-resolution image is used for an image set that includes multiple images. For example, a frame at a certain time in a video may be used as correct high-resolution data, and an image at the same time and an image at an adjacent time that have been reduced by a predetermined factor may be used as an image set to be input to the machine learning model. The image set may then be input to a machine learning model such as a neural network to estimate a high-resolution image, and the parameters of the machine learning model may be optimized so that the difference from the correct high-resolution image is small.

[0073] [Learning Method] Next, a learning method for high resolution processing in this embodiment will be described with reference to Fig. 5. Fig. 5 is a flowchart of the learning method for high resolution processing. Each step in Fig. 5 is mainly executed by the acquisition unit 101b, the generation unit 101c, or the update unit 101d of the learning device 101.

[0074] First, in step S301, the acquisition unit 101b acquires a correct high-resolution image in addition to the image (image set) acquired in step S101. The correct high-resolution image is set to 256 x 256 size for an image set of 128 x 128 size. That is, in this embodiment, the magnification of the high-resolution image is set to 2x, but this is not limited to this. The following step S302 is similar to step S102.

[0075] Next, in step S303, the generation unit 101c acquires a high-resolution image based on the image set and the motion vector. The high-resolution image is acquired by concatenating the image set and the motion vector and inputting the concatenated image into a machine learning model for high resolution. Note that the machine learning model is not limited to a configuration that concatenates the image set and the motion vector. For example, any machine learning model that performs high-resolution processing based on the motion vector may be used, such as a configuration that inputs an image set in which misalignment is compensated using the motion vector. The following step S304 is similar to step S103.

[0076] Next, in step S305, the update unit 101d calculates (obtains) the error (amount of error in the high-resolution image) between the high-resolution image obtained in step S303 and the correct high-resolution image obtained in step S301, as in step S304.

[0077] Next, in step S306, the update unit 101d updates the parameters of the machine learning model for motion vector estimation and the machine learning model for resolution enhancement by backpropagating the errors acquired in steps S303 and S304. The error of the motion vector may be used for backpropagation to the machine learning model for resolution enhancement, or the error of the high-resolution image may be used for backpropagation to the machine learning model for motion vector estimation.

[0078] Next, in step S307, the update unit 101d determines whether to terminate learning of the machine learning model. For example, it may be determined that learning is to be terminated when a predetermined number of updates is exceeded or the amount of error falls below a reference value. If learning is not to be terminated, the process returns to step S301, a new image set, a correct motion vector, and a correct high-resolution image are acquired, and the process is repeated. On the other hand, if learning is to be terminated, the learning of this embodiment is terminated and parameters of the machine learning model that estimates the learned motion vector and the high-resolution machine learning model are obtained.

[0079] In this embodiment, an example has been described in which a machine learning model for motion vector estimation and a machine learning model for high resolution are simultaneously trained, but the present invention is not limited to this. The machine learning model for motion vector estimation and the machine learning model for high resolution may be trained separately. Alternatively, only the training of motion vector estimation may be performed first, and the parameters of the machine learning model for motion vector estimation may be fixed before training the machine learning model for high resolution. Alternatively, the two machine learning models may be trained again and simultaneously using models that have been trained separately.

[0080] In this embodiment, an example in which a motion vector is used for resolution enhancement has been described, but it may also be used for frame rate up-processing or sharpening processing using multiple images. Also, the image set may contain more than two images. Furthermore, the machine learning model for resolution enhancement may output multiple high-resolution images corresponding to frames at multiple different times.

[0081] [Estimation method] Next, with reference to Fig. 6, a description will be given of a high-resolution process using a machine learning model trained by the learning method described with reference to Fig. 5. Fig. 6 is a flowchart of the high-resolution process. Each step in Fig. 6 is mainly executed by the acquisition unit 103b or the estimation unit 103c of the estimation device 103.

[0082] Step S401 is the same as step S201, except that the input image set (first image set) is used not only to estimate a motion vector but also to estimate a high-resolution image.

[0083] Next, in step S402, the acquisition unit 103b acquires machine learning models trained by the learning method described with reference to Fig. 5. As the machine learning models, a model for estimating a motion vector and a model for increasing resolution are acquired. The following step S403 is similar to step S203.

[0084] Next, in step S404, the estimation unit 103c inputs the set of divided images of a predetermined size (second size) acquired in step S403 into the machine learning model for estimating motion vectors acquired in step S402. The estimation unit 103c then estimates motion vectors corresponding to the set of divided images (divided motion vectors). The divided motion vector is a map of the same image size (128 x 128) as each image in the set of divided images, and has two channels: a horizontal component and a vertical component.

[0085] Next, in step S405, the estimation unit 103c inputs the divided image set acquired in step S403 and the divided motion vector acquired in step S404 into a machine learning model for resolution enhancement. Then, the estimation unit 103c estimates a high-resolution image (divided high-resolution image) corresponding to the divided image set. The divided high-resolution image corresponds to the same area as each image in the divided image set, but has been increased in resolution to a size of 256 x 256. The following step S406 is similar to step S205.

[0086] In step S407, the acquisition unit 103b acquires an output high-resolution image with 8K resolution (7680 × 4320) by arranging and combining the multiple divided high-resolution images as partial data so that their positional relationship is the same as before division. Note that if the divided areas overlap with each other, they can be cut out so that they do not overlap, or the overlapping portions can be combined by taking a weighted average. This completes the image processing in this embodiment.

[0087] As in the first embodiment, the division size (second size) in step S403 does not have to be 128×128, but may be, for example, 192×192 (ie, 1.5 times 128) or 160×160 (ie, 1.25 times 128).

[0088] In this embodiment, the divided image set is commonly used as input to the machine learning model for motion vector estimation and the machine learning model for resolution enhancement. In this case, the input size to each machine learning model does not need to be the same. For example, the input image set may be divided into 256 x 256 images and then reduced to 128 x 128 images, and the resulting image set may be input to the machine learning model for motion vector estimation as the divided image set.

[0089] Alternatively, a set of images with a size of 256 x 256 before reduction may be input to the machine learning model for resolution enhancement. In this case, the size (second size) input to the machine learning model for motion vector estimation remains the same as in this embodiment. As described above, the problem of the estimation accuracy decreasing as the image size during estimation increases relative to the image size during learning does not occur in the resolution enhancement process. Therefore, this modification also allows for highly accurate estimation of motion vectors used for resolution enhancement.

[0090] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions. The image processing device of the present invention may be any device that has the image processing function of the present invention, and may be realized in the form of an imaging device or a PC.

[0091] According to each embodiment, it is possible to provide an image processing method, an image processing device, and a program that are capable of performing highly accurate motion vector estimation using a machine learning model for a high-resolution image.

[0092] The disclosure of each embodiment includes the following methods and compositions: (Method 1) acquiring, based on a first image set including a first image and a second image of a first size, a second image set of a partial region of the first image set and having a second size smaller than the first size; and inputting the second set of images into a machine learning model to obtain motion vectors. the motion vector is a motion vector in the second image relative to the first image, the machine learning model is trained using a third set of images having a third size; An image processing method, wherein the second size is equal to or smaller than a fourth size that serves as a reference with respect to the third size. (Method 2) The image processing method according to Method 1, wherein the fourth size is 1.5 times or less than the third size. (Method 3) The image processing method according to Method 1, wherein the fourth size is 1.25 times or less than the third size. (Method 4) The image processing method according to Method 1, wherein the fourth size is equal to or smaller than the third size. (Method 5) 5. The image processing method according to any one of Methods 1 to 4, wherein the first size is the number of pixels on one side of each of the first image and the second image. (Method 6) 6. The image processing method according to any one of methods 1 to 5, wherein the first size is larger than the fourth size. (Method 7) An image processing method according to any one of methods 1 to 6, further comprising a step of determining whether or not to acquire the second image set based on the first size and the fourth size. (Method 8) If the first size is greater than the fourth size, obtaining the second set of images and inputting the second set of images into the machine learning model to obtain the motion vectors; The image processing method of method 7, characterized in that if the first size is smaller than the fourth size, the motion vectors are obtained by inputting the first image set into the machine learning model. (Method 9) 9. An image processing method according to any one of methods 1 to 8, characterized in that the second set of images is obtained by reducing the subregion of the first set of images. (Method 10) An image processing method described in any of methods 1 to 9, further comprising a step of acquiring at least one of the second image set and the image acquired based on the motion vector, or the motion vector, as partial data, and combining multiple pieces of partial data corresponding to each of the different multiple partial areas. (Method 11) 10. The image processing method according to any one of Methods 1 to 9, further comprising the step of estimating a high-resolution image corresponding to the second set of images by inputting the second set of images and the motion vectors into the machine learning model. (Method 12) 12. The image processing method according to any one of Methods 1 to 11, further comprising the step of acquiring the fourth size. (Method 13) 13. An image processing method according to any one of methods 1 to 12, further comprising a step of determining the second size based on at least one of the first size, the fourth size, and the machine learning model. (Method 14) 14. The image processing method according to any one of methods 1 to 13, wherein the machine learning model is a convolutional neural network. (Method 15) An image processing method according to any one of Methods 1 to 14, wherein the first image and the second image are multiple frames at different times in video data. (Method 16) An image processing method according to any one of methods 1 to 15, characterized in that the receptive field of the machine learning model is larger than the second size. (Configuration 1) an acquisition unit that acquires, based on a first image set including a first image and a second image of a first size, a second image set of a partial region of the first image set and a second image of a second size smaller than the first size; an estimation unit that obtains a motion vector by inputting the second image set into a machine learning model; the motion vector is a motion vector in the second image relative to the first image, the machine learning model is trained using a third set of images having a third size; The image processing device is characterized in that the second size is equal to or smaller than a fourth size that serves as a reference with respect to the third size. (Configuration 2) A program causing a computer to execute the image processing method according to any one of Methods 1 to 16.

[0093] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0094] 103b Acquisition Department 103c Estimation part 201 input image set

Claims

1. acquiring, based on a first image set including a first image and a second image of a first size, a second image set of a partial region of the first image set, the second image set having a second size smaller than the first size; and inputting the second set of images into a machine learning model to obtain motion vectors. the motion vector is a motion vector in the second image relative to the first image, the machine learning model is trained using a third set of images having a third size; An image processing method, wherein the second size is equal to or smaller than a fourth size that serves as a reference with respect to the third size.

2. 2. The image processing method according to claim 1, wherein the fourth size is equal to or smaller than 1.5 times the third size.

3. 2. The image processing method according to claim 1, wherein the fourth size is 1.25 times or less the third size.

4. 2. The image processing method according to claim 1, wherein the fourth size is equal to or smaller than the third size.

5. 5. The image processing method according to claim 1, wherein the first size is the number of pixels on one side of each of the first image and the second image.

6. 5. The image processing method according to claim 1, wherein the first size is larger than the fourth size.

7. 5. The image processing method according to claim 1, further comprising the step of determining whether or not to acquire the second image set based on the first size and the fourth size.

8. If the first size is greater than the fourth size, acquiring the second set of images and inputting the second set of images into the machine learning model to acquire the motion vectors; The image processing method according to claim 7 , wherein if the first size is smaller than the fourth size, the motion vector is obtained by inputting the first image set into the machine learning model.

9. 5. The image processing method according to claim 1, wherein the second image set is obtained by reducing the partial region of the first image set.

10. An image processing method according to any one of claims 1 to 4, characterized in that it further comprises a step of acquiring at least one of the image acquired based on the second image set and the motion vector, or the motion vector, as partial data, and combining multiple pieces of partial data corresponding to each of the different multiple partial areas.

11. 5. The image processing method according to claim 1, further comprising a step of estimating a high-resolution image corresponding to the second image set by inputting the second image set and the motion vector into the machine learning model.

12. 5. The image processing method according to claim 1, further comprising the step of acquiring the fourth size.

13. 5. The image processing method according to claim 1, further comprising a step of determining the second size based on at least one of the first size, the fourth size, and the machine learning model.

14. 5. The image processing method according to claim 1, wherein the machine learning model is a convolutional neural network.

15. 5. The image processing method according to claim 1, wherein the first image and the second image are a plurality of frames at different times in video data.

16. The image processing method according to claim 1 , wherein the receptive field of the machine learning model is larger than the second size.

17. an acquisition unit that acquires, based on a first image set including a first image and a second image of a first size, a second image set of a partial region of the first image set and having a second size smaller than the first size; an estimation unit that acquires a motion vector by inputting the second image set into a machine learning model; the motion vector is a motion vector in the second image relative to the first image, the machine learning model is trained using a third set of images having a third size; The image processing device is characterized in that the second size is equal to or smaller than a fourth size that serves as a reference with respect to the third size.

18. A program causing a computer to execute the image processing method according to any one of claims 1 to 4.

Images