Image processing method, image processing device, image processing system, and program

Abstract

To provide an image processing method which can raise accuracy of image processing using machine learning related to resolution performance of an imaging optical system.SOLUTION: An image processing method comprises: an acquisition process of acquiring an input image obtained by imaging using an optical system, and image processing information regarding image processing to be preliminarily applied to the input image; a determination process of determining whether or not first resolution performance information showing resolution performance of the optical system is consistent with the input image on the basis of the image processing information; and a generation process of generating an output image on the basis of the input image and the image processing information, wherein a generation method of the output image is made different according to a determination result in the determination process in the generation process.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing method, an image processing device, an image processing system, and a program. [Background technology]

[0002] Patent Document 1 discloses a method for enlarging an image using machine learning. By training a machine learning model using a pair of a low-pixel image and a high-pixel image (an image with a smaller sampling pitch than the low-pixel image) of the same scene, it is possible to achieve image enlargement with higher accuracy than bicubic interpolation, etc. Patent Document 1 also discloses a method for generating a high-resolution enlarged image by enlarging a low-pixel image by bicubic interpolation to the same number of pixels as the high-pixel image, and then inputting the enlarged image into the trained machine learning model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Application Publication No. 2018 / 0075581 Summary of the Invention [Problem to be solved by the invention]

[0004] The method disclosed in Patent Document 1 has the problem that false structures (artifacts) that do not actually exist appear in the enlarged image, or that moire patterns that existed in the low-resolution image remain in the enlarged image. This problem also occurs in other image enlargement methods that do not use machine learning models (such as bicubic interpolation and sparse coding). This problem also occurs not only in image enlargement, but also in other processes that reduce the sampling pitch of an image (such as demosaicing).

[0005] The cause of this problem is the inability to distinguish whether the structure present in the input low-pixel image is moiré or the original structure of the subject. Because the frequency band in which moiré can occur is determined by the resolution performance of the imaging optical system, it is possible to improve the moiré based on the resolution performance of the optical system that captured the low-pixel image. However, there is a problem in that post-capture image processing performed on the low-pixel image can convert it into an image that does not match the resolution performance of the optical system.

[0006] Therefore, an object of the present invention is to provide an image processing method that can improve the accuracy of image processing using machine learning related to the resolution performance of an imaging optical system. [Means for solving the problem]

[0007] An image processing method according to one aspect of the present invention includes: an input image obtained by imaging using an optical system; Suitable for Used Ta Image Processing 1st an acquisition step of acquiring information; 1st a determination step of determining whether or not first resolution performance information indicating the resolution performance of the optical system matches the input image based on the information; 1st and generating an output image based on the information, wherein in the generating step, a method for generating the output image is changed depending on the determination result in the determining step.

[0008] Other objects and features of the present invention are illustrated in the following examples. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide an image processing method that can improve the accuracy of image processing using machine learning related to the resolution performance of an imaging optical system. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 10 is a diagram illustrating the relationship between the modulation transfer function and the Nyquist frequency in the first and second embodiments. [Figure 2] 1 is a block diagram of an image processing system according to a first embodiment. [Figure 3] 1 is an external view of an image processing system according to a first embodiment. [Figure 4] 1 is a flowchart of training a machine learning model in Example 1. [Figure 5] FIG. 4 is a diagram showing a flow of generating an enlarged image in the first embodiment. [Figure 6] FIG. 1 is a configuration diagram of a machine learning model in Examples 1 to 3. [Figure 7] FIG. 4 is a diagram showing a flow of generating an enlarged image in the first embodiment. [Figure 8] 10 is a flowchart of generating an enlarged image in the first embodiment. [Figure 9] 10 is a flowchart of generating an enlarged image in a modified example of the first embodiment. [Figure 10] FIG. 10 is a block diagram of an image processing system according to a second embodiment. [Figure 11] FIG. 10 is an external view of an image processing system according to a second embodiment. [Figure 12] 10 is a flowchart of training a machine learning model in Example 2. [Figure 13] FIG. 10 is a diagram showing the relationship between the color filter array and the Nyquist frequency in the second embodiment. [Figure 14] FIG. 10 is a diagram showing the flow of generating a demosaic image in the second embodiment. [Figure 15] 10 is a flowchart of generating a demosaic image in the second embodiment. [Figure 16] 11 is a flowchart of generating a blur-corrected image in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same components are designated by the same reference numerals, and redundant explanations will be omitted.

[0012] Before describing each embodiment in detail, a brief summary of the present invention will be provided. In each embodiment, resolution performance information, which is information about the resolution performance of the optical system used to capture the input image (captured image), is used in the process of reducing the sampling pitch of the input image (hereinafter referred to as upsampling). This improves the accuracy of upsampling. To explain the reasons for this, the issues with upsampling and the principles behind their occurrence will be described in detail below.

[0013] When an image of a subject formed by an optical system is converted into a captured image by an image sensor, sampling is performed using the image sensor's pixels. As a result, among the frequency components forming the subject image, components exceeding the image sensor's Nyquist frequency are mixed with low-frequency components due to aliasing, resulting in moiré. When upsampling a captured image, the Nyquist frequency increases as the sampling pitch decreases, so ideally, an image without aliasing up to the increased Nyquist frequency is generated. However, it is difficult to distinguish whether moiré is due to the original structure of the subject from an image that already contains moiré.

[0014] With methods such as bilinear interpolation, moiré remains even when the captured image is upsampled. In contrast, methods using machine learning models can estimate the high frequencies before aliasing occurs from the moiré, which allows some moiré to be removed. However, as mentioned above, it is difficult to distinguish between moiré and the structure of the subject, so some moiré is mistakenly recognized as the subject and remains, and some subjects are mistakenly recognized as moiré, creating false structures.

[0015] Therefore, in each embodiment, a case is considered in which resolution performance information of the optical system used to capture the captured image is used in upsampling the captured image (input image). This case will be explained with reference to Figures 1(A) and 1(B). Figures 1(A) and 1(B) show the frequency characteristics of the modulation transfer function (MTF), which represents the resolution performance of the optical system, and show the relationship between the modulation transfer function and the Nyquist frequency.

[0016] In Figures 1(A) and (B), the horizontal axis represents the spatial frequency in a certain direction, and the vertical axis represents the MTF. For example, as shown in Figure 1(A), if the cutoff frequency 003 of the optical system (here, the frequency above which the MTF becomes 0) is equal to or lower than the Nyquist frequency 001, no moiré will be present in the captured image. This is because even if the MTFs are arranged at a period of the sampling frequency 002, there is no area where the MTFs overlap. Therefore, when the resolution performance corresponds to Figure 1(A), it is not necessary to estimate the high-frequency components before they become moiré from the moiré, which makes it possible to suppress the occurrence of false structures.

[0017] Furthermore, as shown in FIG. 1B, even when the cutoff frequency 003 exceeds the Nyquist frequency 001, it is possible to identify the frequency band in which moiré may occur due to aliasing. Moiré may occur in the band between the frequency 004, which is the sampling frequency 002 minus the cutoff frequency 003, and the Nyquist frequency 001, but moiré will not occur in other bands. Therefore, by using the resolution performance information, it is possible to similarly suppress the occurrence of false structures. For these reasons, the accuracy of upsampling of captured images can be improved.

[0018] When performing image processing using resolution performance information on a captured image, the captured image and the resolution performance information must be consistent. A captured image is acquired by forming an image of a subject through an imaging optical system and sampling it with a sensor. A consistent image is one in which the resolution characteristics of the image are maintained immediately after capture, while an inconsistent image is one in which the resolution characteristics of the image have been changed by image processing that changes the frequency characteristics. Examples of image processing that changes the frequency characteristics include sharpening and blurring. In particular, an inconsistent image is one in which image processing not considered in the training dataset for the machine learning model is applied, or one in which the relationship between the captured image and the resolution performance information is no longer that which is included in the training dataset.

[0019] Image processing during or after capture changes image characteristics, resulting in inconsistency between the input image and resolution performance information, which reduces the accuracy of image processing using resolution performance information. Using resolution performance information that is inconsistent with the input image can increase the frequency of residual moiré and false structures, and can also result in stronger contrast, compared to when resolution performance information is not used. This is because the machine learning model makes an incorrect recognition when distinguishing between moiré and object structure using resolution performance information. If the resolution performance is incorrectly assumed to be higher than the input image, the machine learning model will recognize that moiré occurs in lower frequency bands than actually occurs, thereby incorrectly recognizing objects in frequency bands where moiré does not occur as moiré and generating excessive false structures. If the resolution performance is incorrectly assumed to be lower than the input image, the machine learning model will recognize that moiré does not occur in higher frequency bands than actually occurs, or that moiré does not occur at all, thereby incorrectly recognizing moiré in frequency bands where moiré occurs as objects, resulting in excessive moiré remaining. Therefore, determining whether the resolution performance information and the input image are consistent can improve the accuracy of upsampling the input image. Since a machine learning model using resolution performance information can improve accuracy, the present invention is not limited to upsampling and includes processes such as blur correction, etc. Image enlargement processing also includes cases where the enlargement rate differs for each screen area. [Example]

[0020] First, an image processing system according to a first embodiment of the present invention will be described. In this embodiment, image enlargement (upscaling) is performed as upsampling, but other upsampling methods such as demosaicing can also be applied in the same way. Image enlargement includes increasing the number of sampling points for the entire captured image and increasing the number of sampling points for a partial region of the captured image (such as enlarging a cropped image or digital zooming). A machine learning model is used for image enlargement.

[0021] FIG. 2 is a block diagram of the image processing system 100. FIG. 3 is an external view of the image processing system 100. The image processing system 100 includes a training device 101, an image enlargement device (image processing device, second device) 102, a control device (first device) 103, and an imaging device 104, all of which are connected to one another via a wired or wireless network. The control device 103 includes a storage unit 131, a communication unit 132, a display unit 133, and an image processing unit 134. The control device 103 and the image enlargement device 102 can communicate with each other. In accordance with a user's instruction, the control device 103 acquires a captured image (input image) from the imaging device 104 and transmits the captured image and a request for image enlargement processing to the image enlargement device 102 via the communication unit (transmission means) 132. At this time, the user can perform any image processing while transmitting the captured image acquired from the imaging device 104 to the image enlargement device 102.

[0022] The imaging device 104 has an imaging optical system 141, an imaging element 142, an image processing unit 143, and a storage unit 144. The imaging optical system 141 forms an image of the subject from light in the subject space, and the imaging element 142, which has an array of multiple pixels, converts this image into a captured image. During this process, aliasing occurs in frequency components of the subject image that are higher than the Nyquist frequency of the imaging element 142. As a result, moiré may occur in the captured image. The image processing unit 143 performs predetermined processing (such as pixel defect correction and development) on the captured image as necessary, and the captured image is stored in the storage unit 144. The control device 103 acquires the captured image via communication or a storage medium. The acquired captured image may be the entire captured image or a partial region of the captured image.

[0023] The image enlargement device 102 includes a storage unit 121, a communication unit 122, an acquisition unit (acquisition means) 123, an image enlargement unit (generation means) 124, and a determination unit (determination means) 125. The image enlargement device 102 enlarges a captured image (input image) using a trained machine learning model to generate an enlarged image (output image). At this time, resolution performance information, which is information about the resolution performance of an optical system (such as the imaging optical system 141) used to capture the captured image, is used. Details of this process will be described later. The image enlargement device 102 acquires weight information of the trained machine learning model from the training device 101 and stores it in the storage unit 121. The training device 101 includes a storage unit 111, an acquisition unit 112, a calculation unit 113, and an update unit 114, and pre-trains a machine learning model using a dataset. Weight information of the machine learning model generated by training is stored in the storage unit 111. The control device 103 acquires the enlarged image from the image enlargement device 102 and presents it to the user via the display unit 133 .

[0024] Next, training of the machine learning model (determination of weights) executed by the training device 101 will be described with reference to FIG. 4. FIG. 4 is a flowchart of training of the machine learning model. Each step in FIG. 4 is executed by the storage unit 111, the acquisition unit 112, the calculation unit 113, or the update unit 114 of the training device 101. In this embodiment, training is performed using a GAN (generative adversarial network), but is not limited to this. Furthermore, machine learning models include, for example, neural networks, genetic programming, and Bayesian networks. An example of a neural network is a CNN (convolutional neural network), but is not limited to this. The neural network may also be a GAN (generative adversarial network), an RNN (recurrent neural network), a Transformer, or the like.

[0025] First, in step S101, the acquisition unit 112 acquires one or more pairs of high-resolution images and low-resolution images from the storage unit 111. The storage unit 111 stores a dataset including multiple high-resolution images and low-resolution images. The low-resolution images are input to a machine learning model (a generator in this embodiment) when training the machine learning model. The more accurately the low-resolution images reproduce the properties of the captured image to be actually enlarged, the higher the accuracy of the trained machine learning model. Examples of the properties of the captured image include resolution performance, color representation, and noise characteristics. If the captured image is an RGB image and the low-resolution image is a monochrome or YUV image, the color representations do not match, which may result in a decrease in task accuracy. While the important properties vary depending on the type of task, resolution performance is important for image enlargement because the frequency band in which moiré occurs is important, as described above. Therefore, it is desirable for the resolution performance of the captured image to fall within the range of the resolution performance of the multiple low-resolution images used for training.

[0026] A high-pixel image is an image that serves as the ground truth in training a machine learning model. A high-pixel image is an image of the same scene as a corresponding low-pixel image, but has a smaller sampling pitch (i.e., a larger number of pixels) than the low-pixel image. In this embodiment, the sampling pitch of the high-pixel image is half that of the low-pixel image. Therefore, the machine learning model enlarges the input image by doubling the vertical and horizontal dimensions and four times the number of pixels. However, this embodiment is not limited to this. It is desirable that the multiple low-pixel images and high-pixel images used for training contain various objects (edges with different orientations and strengths, textures, gradients, flat areas, etc.) so that the machine learning model can handle captured images of various objects. At least a portion of the high-pixel image has frequency components equal to or higher than the Nyquist frequency of the low-pixel image.

[0027] In this embodiment, the high-pixel image and the low-pixel image are generated by imaging simulation from the original image. However, the invention is not limited to this, and imaging simulation may be performed using three-dimensional data of the subject space instead of the original image. Furthermore, the high-pixel image and the low-pixel image may be generated by actually capturing images using imaging elements with different pixel pitches.

[0028] The original image is an undeveloped RAW image (where light intensity and signal value have a linear relationship), has a sampling pitch equal to or lower than that of the high-resolution image, and at least some of the image has frequency components equal to or higher than the Nyquist frequency of the low-resolution image. The low-resolution image is generated by using the original image as a subject and reproducing the same imaging process as an image that is actually enlarged using a trained machine learning model. Specifically, the original image is blurred by aberrations and diffractions occurring in the imaging optical system 141, and blurring caused by the optical low-pass filter and pixel aperture of the image sensor 142, etc. If there are multiple types and states of the imaging optical system 141 and the image sensor 142, which may affect the captured image differently, the dataset includes low-resolution images to which these multiple blurs have been added. The blurring can vary depending on the position of each pixel on the image sensor 142 (image height and azimuth relative to the optical axis of the imaging optical system 141) and, if the imaging optical system 141 can assume various states (focal length, F-number, focus distance, etc.), it can also vary depending on the state of the imaging optical system 141. In addition, if there are multiple types of imaging optical system 141, such as an interchangeable lens system, the blur also changes depending on the type. Furthermore, if there are different types of image capture devices 104 and the pixel pitch or optical low-pass filter differs, the blur also changes.

[0029] The blur applied to the original image may be the blur itself generated in the imaging optical system 141 or the image sensor 142, or may be a blur that approximates that blur. For example, the PSF (point spread function) of the blur generated in the imaging optical system 141 or the image sensor 142 may be approximated by a two-dimensional Gaussian distribution function, a mixture of multiple two-dimensional Gaussian distribution functions, a Zernike polynomial, or the like. Furthermore, the OTF (optical transfer function) or MTF (modulation distribution function) may be approximated by a two-dimensional Gaussian distribution function, a mixture of multiple two-dimensional Gaussian distribution functions, a Legendre polynomial, or the like.

[0030] After blurring the original image, it is downsampled at the sampling pitch of the image sensor 142. Furthermore, because the image sensor 142 has RGB (Red, Green, Blue) color filters arranged in a Bayer array, the low-pixel image is also sampled to have a Bayer array. However, the invention is not limited to this, and the image sensor 142 may be monochrome, honeycomb-arrayed, or a three-chip type. If there are multiple types of image sensor 142 and the pixel pitch of the captured image can vary, low-pixel images are generated for multiple sampling pitches to cover the range of variation. In this embodiment, noise generated by the image sensor 142 is also added to the low-pixel image. This is because if noise is not added to the low-pixel image (noise is not taken into account in training the machine learning model), not only the subject but also the noise will be emphasized when the captured image is enlarged. If there is a range of noise intensity generated in the captured image (for example, there are multiple possible ISO sensitivities at the time of capture), multiple low-pixel images with noise intensity varied within the possible range are included in the dataset.

[0031] The high-pixel image is generated by applying blur to the original image using a pixel aperture half the pixel pitch of the low-pixel image, downsampling at half the sampling pitch of the low-pixel image, and Bayer-encoding the image. Note that if the sampling pitch of the original image and the high-pixel image are the same, the original image may be used as the high-pixel image as is. In this embodiment, blur due to aberration and diffraction of the imaging optical system 141 and blur due to the optical low-pass filter of the image sensor 142 are not applied when the high-pixel image is generated. This allows the machine learning model to be trained to correct the blur described above while enlarging the image. However, the invention is not limited to this. The same blur as the low-pixel image may be applied to the high-pixel image, or the blur applied to the low-pixel image may be reduced and applied to the high-pixel image. In this embodiment, noise is not applied when the high-pixel image is generated. This allows the machine learning model to be trained to perform denoising while enlarging the image. However, the invention is not limited to this. Noise of the same intensity as or different from the noise applied to the low-pixel image may be applied. When adding noise to high-resolution images, it is desirable to add noise that is correlated with the noise in low-resolution images. If the noises are uncorrelated, training with multiple images in the dataset may average out the effects of the noise in the high-resolution images, which may not produce the desired effect.

[0032] In this embodiment, image enlargement is performed on a developed captured image. Therefore, the low-pixel image and high-pixel image must also be developed images. Therefore, the same development process as for the captured image is performed on the low-pixel image and high-pixel image in Bayer format, and they are stored in a dataset. However, the invention is not limited to this, and the low-pixel image and high-pixel image may be converted to RAW, and the captured image may also be enlarged in RAW format. Furthermore, if compression noise such as JPEG encoding occurs in the captured image, similar compression noise may be added to the low-pixel image. In this way, the machine learning model is trained to remove compression noise as well as enlarging the image.

[0033] Next, in step S102, the acquisition unit 112 acquires resolution performance information and noise information. The resolution performance information is information related to the resolution performance of the blur added to the low-pixel image. When the resolution performance is low (the MTF is 0 or a sufficiently small value below the Nyquist frequency of the low-pixel image), moiré does not exist in the low-pixel image. When the resolution performance is high (the MTF has a value at frequencies above the Nyquist frequency), moiré does not exist outside the frequency band where aliasing occurs. In this way, the resolution performance information indicates the frequency band where moiré occurs in the low-pixel image. Therefore, the resolution performance information includes information based on the magnitude of the blur added to the low-pixel image. The resolution performance information also includes information based on either the spread of the PSF of the blur or the MTF of the blur. Note that the PTF (phase transfer function) of the blur alone does not constitute resolution performance information. The PTF represents the shift in the imaging position.

[0034] In this embodiment, the resolution performance information used when enlarging a captured image is information regarding blur that combines the effects of all factors, including aberration and diffraction of the imaging optical system 141 and the optical low-pass filter and pixel aperture of the image sensor 142. However, the invention is not limited to this, and resolution performance may be expressed using only a portion of the blur (for example, blur generated by the imaging optical system 141). For example, if the optical low-pass filter and pixel pitch are fixed and do not change, there is no problem in expressing resolution performance using only the blur generated by the imaging optical system 141. In this case, however, the resolution performance of the low-pixel image must also be determined accordingly. It is advisable to determine the resolution performance information for the blur obtained by excluding the effects of the optical low-pass filter and pixel aperture from the blur applied to the low-pixel image.

[0035] The noise information is information about noise added to the low pixel image. The noise information includes information indicating the strength of the noise. The strength of the noise can be expressed by the standard deviation of the noise, the corresponding ISO sensitivity of the image sensor 142, or the like. Furthermore, if denoising has been performed on the captured image before enlargement, it is desirable to perform the same denoising on the low pixel image and include the parameters of the denoising performed (indicating the strength, etc.) in the noise information. This makes it possible to suppress adverse effects and achieve high-precision image enlargement even if the noise or denoising changes.

[0036] Specific examples of resolution performance information and noise information are shown below. In this embodiment, the resolution performance information is generated by the following method, but is not limited to this. The resolution performance information is a map (a map in which values ​​are arranged in a size based on the number of pixels) with the same two-dimensional (horizontal and vertical) pixel count as the low-resolution image. Each pixel in the map indicates the resolution performance of the corresponding pixel in the low-resolution image. The map has multiple channels, with the first channel indicating horizontal resolution performance and the second channel indicating vertical resolution performance. The resolution performance is a value based on the frequency at which the MTF for the white blur added to the low-resolution image reaches a predetermined value in the corresponding direction. More specifically, it is the minimum frequency at which the MTF is equal to or less than a threshold value (0.5 in this embodiment, but this is not limited to this). Furthermore, the resolution performance is indicated by a value obtained by normalizing the minimum frequency by the sampling frequency of the low-resolution image. The sampling frequency used for normalization is the reciprocal of the pixel pitch and is common to RGB. However, the value representing the resolution performance is not limited to this. Furthermore, instead of white, the resolution performance of each of the RGB colors may be expressed using six channels, and the frequencies used for normalization may also be different for each of the RGB colors.

[0037] Other examples of resolution performance information are shown below. The resolution performance direction indicated by the resolution performance information may be the meridional (radial) direction or the sagittal (azimuth) direction. A third channel representing the azimuth of the pixel may also be added. Furthermore, the number of channels may be increased to represent resolution performance in multiple directions, rather than just two. Alternatively, resolution performance may be represented using only one channel, such as by taking the average of a specific direction or all directions. Furthermore, the resolution performance information may be a scalar value or vector rather than a map. When the imaging optical system 141 is a super-telephoto lens or has a large F-number, the change in resolution performance due to image height and azimuth becomes very small. Therefore, in such cases, the effects of the invention can be achieved using a scalar value rather than a map indicating performance for each pixel. Furthermore, the integrated value of the MTF or the like may be used as the resolution performance.

[0038] Resolution performance may be expressed as the spread of the PSF. It may also be expressed as the spatial range where the half-width in multiple directions or the PSF intensity is equal to or greater than a threshold. When expressing resolution performance as a scalar value rather than a map, it is advisable to use a value in a specific direction or an average value in multiple directions, as with MTF.

[0039] Alternatively, the resolution performance may be expressed by coefficients obtained by fitting the MTF or PSF. Fitting may be performed using a power series, Fourier series, a mixed Gaussian model, a Legendre polynomial, a Zernike polynomial, or the like, and each coefficient of the fitting may be expressed using multiple channels.

[0040] The resolution performance information may be generated by calculation from the blur added to the low pixel image, or may be acquired from a storage unit 111 that stores a plurality of blurs and corresponding resolution performance information in advance.

[0041] Like the resolution performance information, the noise information is a two-dimensional map with the same number of pixels as the low-resolution image. The first channel is a parameter that indicates the strength of the noise in the low-resolution image before denoising, and the second channel is a parameter that indicates the strength of the denoising that has been performed. If compression noise is present in the low-resolution image, the strength of the compression noise may be added to an additional channel. Like the resolution performance information, the noise information may be in the form of a scalar value or a vector.

[0042] In this embodiment, the order of execution of steps S101 and S102 is not limited, and step S102 may be executed before step S101.

[0043] Next, in step S103, the calculation unit 113 generates an enlarged image from the low-pixel image, the resolution performance information, and the noise information using a generator that is a machine learning model. The enlarged image is a low-pixel image with a reduced sampling pitch.

[0044] Here, the generation of an enlarged image will be described with reference to FIG. 5 and FIGS. 6(A) and (B). FIG. 5 is a diagram showing the flow of enlarged image generation. FIGS. 6(A) and (B) are configuration diagrams of a machine learning model. In FIG. 5, "sum" indicates the sum of each element (pixel), and "concatenation" indicates concatenation in the channel direction. In this embodiment, the resolution performance information 202 and noise information 203 are maps with the same number of two-dimensional pixels as the low-pixel image 201. The low-pixel image 201, resolution performance information 202, and noise information 203 are concatenated in the channel direction and then input to a generator 211 to generate a residual component 204. The residual component 204 has the same number of two-dimensional pixels as the high-pixel image. The low-pixel image 201 is enlarged to the same number of pixels as the high-pixel image using bilinear interpolation or the like, and then summed with the residual component 204 to generate an enlarged image 205. Note that the enlarged image 205 may be generated directly by the generator 211 without passing through the residual component 204. If the resolution performance information 202 or noise information 203 is a scalar value or vector, or if the number of two-dimensional pixels does not match the low-pixel image 201, they may be converted into a feature map via a convolution layer and then linked to the low-pixel image 201 (or the feature map converted from it) in the channel direction.

[0045] The generator 211 is a CNN with the configuration shown in Figure 6(A), but is not limited to this. The initial weight values ​​of the generator 211 may be generated using random numbers or other methods. In Figure 6(A), conv. represents convolution, ReLU represents rectified linear unit, and sub-pixel conv. represents sub-pixel convolution. In this embodiment, sub-pixel convolution quadruples the number of two-dimensional pixels of the input, thereby making the number of two-dimensional pixels of the residual component 204 the same as that of a high-resolution image. Residual block represents a residual block. The residual block has multiple linear sum layers and an activation function, and is configured to sum the input and output of the block. In this embodiment, the residual block is shown in Figure 6(B). GAP represents global average pooling, dense represents full connection, sigmoid represents a sigmoid function, and multiply represents the product of each element. Generating an attention map using GAP and full connection improves task accuracy. In this embodiment, the generator 211 has 16 residual blocks. However, the number of residual blocks is not limited to this. If it is desired to further improve the performance of the generator 211, it is recommended to increase the number of residual blocks.

[0046] The low pixel image 201 may be enlarged in advance using bilinear interpolation or the like so that the number of pixels matches that of the high pixel image, and then input to the generator 211. In this case, subpixel convolution is not required in the generator 211. However, as the number of two-dimensional pixels in the low pixel image 201 increases, the number of times linear sums are calculated increases, resulting in a heavy calculation load. For this reason, it is desirable to input the low pixel image 201 to the generator 211 without enlarging it, as in this embodiment, and enlarge it internally.

[0047] Next, in step S104 of FIG. 4, the calculation unit 113 inputs the enlarged image 205 and the high-pixel image to a classifier to generate a classification output. The classifier classifies whether the input image is an image generated by the generator 211 (enlarged image 205 in which high-frequency components are estimated from the low-pixel image) or an actual high-pixel image (an image in which frequency components equal to or higher than the Nyquist frequency of the low-pixel image are acquired during imaging). A CNN or the like may be used as the classifier. The initial values ​​of the weights of the classifier are determined using random numbers or the like. Note that the high-pixel image input to the classifier may be an actual high-pixel image, and does not need to be an image corresponding to the low-pixel image 201.

[0048] Next, in step S105, the update unit 114 updates the weights of the classifier based on the classification output and the correct label. In this embodiment, the correct label for the enlarged image 205 is set to 0, and the correct label for the actual high-resolution image is set to 1. Although sigmoid cross entropy is used as the loss function, other functions may also be used. Backpropagation is used to update the weights.

[0049] Next, in step S106, the update unit 114 updates the weights of the generator 211 based on the first loss and the second loss. The first loss is a loss based on the difference between the high-resolution image corresponding to the low-resolution image 201 and the enlarged image 205. In this embodiment, MSE (Mean Square Error) is used, but MAE (Mean Absolute Error) or the like may also be used. The second loss is the sigmoid cross entropy between the classification output when the enlarged image 205 is input to the classifier and the correct label 1. Since the generator 211 wants to train the classifier so that it will erroneously classify the enlarged image 205 as an actual high-resolution image, it sets the correct label to 1 (corresponding to the actual high-resolution image). Note that the order of steps S105 and S106 may be reversed.

[0050] Next, in step S107, the update unit 114 determines whether training of the generator 211 has been completed. If training has not yet been completed, the process returns to step S101, and one or more new pairs of low-pixel image 201 and high-pixel image are acquired. On the other hand, if training has been completed, weight information is stored in the storage unit 111. Note that, since only the generator 211 is used during image enlargement, the weights of only the generator 211 may be stored.

[0051] Note that before training the GAN using the classifier, the generator 211 may be trained using only the first loss. Alternatively, the storage unit 111 may store the first and second datasets, perform training using the first dataset in steps S101 to S107, and then use the weights as initial values ​​to perform training using the second dataset in steps S101 to S107. The first dataset contains fewer high-resolution images with high-frequency components equal to or greater than the Nyquist frequency of low-resolution images (i.e., less moiré in low-resolution images) than the second dataset. Therefore, the generator 211 trained using the first dataset is more likely to retain moiré, but is less likely to produce false structures. In contrast, the generator 211 trained using the second dataset can remove moiré, but is more likely to produce false structures. By storing the progress of the weights of the generator 211 during training using the second dataset, it is possible to later select weights that balance moiré removal and false structures.

[0052] Through the above steps, weight information for the generator 211 (first machine learning model) that generates an enlarged image based on the captured image, resolution performance information, and noise information can be acquired. In this embodiment, a second machine learning model that is not based on resolution performance information is also used. The second machine learning model can be trained by configuring it so that the resolution performance information 202 is not used as input data to the generator 211, as shown in FIG. 7, which shows the flow of generating an enlarged image. The low-pixel image 201 and noise information 203 are equivalent to the first machine learning model. In this embodiment, the configuration of the generator 211 is equivalent to that shown in FIGS. 6(A) and 6(B). However, because the resolution performance information 202 is not linked in the channel direction, the number of parameters in the convolution layer differs from that of the first machine learning model. Therefore, weight information for the second machine learning model is acquired by training it separately from the first machine learning model. Note that the configuration of the generator 211 for the first machine learning model and the second machine learning model may be different.

[0053] Next, the enlargement process (image processing method) of a captured image executed by the image enlargement device 102 and the control device 103 will be described with reference to Fig. 8. Fig. 8 is a flowchart of generating an enlarged image. Each step in Fig. 8 is executed by the storage unit 121, communication unit 122, acquisition unit 123, image enlargement unit 124, or determination unit 125 of the image enlargement device 102, or the storage unit 131, communication unit 132, display unit 133, or image processing unit 134 of the control device 103.

[0054] First, in step S201, the communication unit (transmission means) 132 of the control device 103 transmits a captured image (input image), image processing information, and a request to execute enlargement processing to the image enlargement device 102. The captured image is a developed image, as in the case of training.

[0055] In this embodiment, the image processing information is part of the additional information of the captured image and is attached to the captured image. The additional information is, for example, meta information such as header information, and is information about the data itself attached to the image data. In this embodiment, the additional information is transmitted attached to the captured image. However, it may be transmitted separately from the captured image, or only the information used in the processing described below may be extracted and transmitted from the additional information. The image processing information is information about image processing that can be applied to the captured image. In this embodiment, it is information about image processing applied by the image processing unit 143 of the image capture device 104 during capture. It may also be information about image processing applied by the image processing unit 134 of the control device 103 between capture and transmission to the image enlargement device 102. It may also be information about image processing applied by an external image processing device that is communicated to an external image processing device before transmission to the image enlargement device 102, or a combination of the above information. The information about image processing is, for example, information about sharpening processing, such as whether the corresponding image processing was applied, image processing parameters such as sharpening strength, and which of multiple processing options was selected.

[0056] Subsequently, in step S202, the acquisition unit 123 of the image enlargement device 102 acquires the captured image, image processing information, and execution request transmitted from the control device 103 via the communication unit (reception means) 122.

[0057] Next, in step S203, the determination unit 125 of the image enlargement device 102 determines whether the resolution performance of the optical system (first resolution performance information) matches the captured image obtained in step S202, based on the image processing information obtained in step S202. Note that the process of determining whether the resolution performance of the optical system matches the captured image in step S203 includes a process of obtaining only the possibility that the resolution performance and the captured image do not match, as will be described later. The resolution performance of the optical system corresponds to the optical performance information obtained in step S204 when it is determined that they match. In this embodiment, the obtained image processing information includes information indicating whether sharpening processing has been applied to the captured image (information regarding the sharpening processing), and it is confirmed whether sharpening processing has already been applied to the captured image based on the image processing information.

[0058] In this embodiment, the frequency at which the MTF falls below the threshold is used as the resolution performance information. However, applying sharpening processing changes the contrast of the subject and the contrast of the moiré pattern at frequencies lower than the Nyquist frequency in the captured image. As a result, the frequency characteristics based on the resolution performance of the optical system differ from the frequency characteristics of the captured image. The frequency characteristics based on the resolution performance of the optical system are the frequency characteristics of the captured image calculated from the resolution performance of the optical system, and are the frequency characteristics of the image used to train the machine learning model. Furthermore, the change in moiré contrast results in a difference in intensity from the moiré pattern that occurs when sharpening processing is not applied. In other words, the moiré pattern that occurs in the captured image differs from the moiré pattern that occurs based on the resolution performance of the optical system. The moiré pattern that occurs based on the resolution performance of the optical system is the moiré pattern that should occur in the captured image calculated from the resolution performance of the optical system, and is also moiré that occurs in the training dataset. As a result, moiré pattern remains, and false structures are more likely to occur.

[0059] Therefore, when sharpening processing is applied, the determination unit 125 determines that the resolution performance of the optical system and the captured image do not match. On the other hand, when sharpening processing is not applied, the determination unit 125 determines that the resolution performance of the optical system and the captured image match. It is generally known that the accuracy of machine learning models decreases when there is a difference between the dataset used during training and the data actually used for estimation processing. Therefore, image processing information such as sharpening processing that changes the relationship between the captured image and the resolution performance information to a relationship different from that contained in the dataset is acquired as image processing information. This makes it possible to generate an enlarged image by distinguishing whether the resolution performance and the captured image are consistent, that is, whether they are included in the dataset.

[0060] The above-mentioned sharpening process is preferably a sharpening process that corrects degradation of perceived resolution based on the resolution performance of the optical system. Sharpening processes that are not based on the resolution performance of the optical system are easy to consider when creating a training dataset, such as a uniform frequency gain across the screen. Therefore, by creating a dataset with or without sharpening and varying the intensity, high-resolution images with suppressed moiré and false structures can be obtained. However, when sharpening processes are based on the resolution performance of the optical system, changes in the image frequency characteristics and moiré contrast caused by the sharpening process increase depending on the optical system's shooting conditions and image height, making training to suppress moiré and false structures and obtain high-resolution images difficult. Sharpening processes based on the different optical characteristics of each RGB also change the color components (color moiré) caused by moiré, making training even more difficult.

[0061] To take into account such sharpening processes based on various optical characteristics, a machine learning model with a larger number of parameters is required. This slows down the learning speed and increases the computational load when enlarging an image and the data capacity of the machine learning model. Furthermore, the lower the performance of the optical system, the stronger the sharpening, which results in larger changes in contrast at frequencies close to the cutoff frequency of the optical system and is likely to result in larger changes in moiré.

[0062] The sharpening process may also be a process of generating frequency components above the cutoff frequency of the optical system. In this case, object structures that should not exist based on the resolution performance of the optical system are reproduced. In this case, object structures above the cutoff frequency do not exist in the dataset used for training, so the captured image does not match the resolution performance information, and the accuracy of generating an enlarged image decreases.

[0063] If it is determined in step S203 that the captured image and the resolution performance information are consistent, the process proceeds to step S204. In step S204, the acquisition unit 123 acquires weight information of the first machine learning model, resolution performance information, and noise information from the storage unit 121. The resolution performance information is information indicating the resolution performance of the optical system when the captured image was captured. Here, the optical system in this embodiment includes the imaging optical system 141 and the optical low-pass filter and pixel aperture of the image sensor 142. To acquire the resolution performance information and noise information, necessary information is acquired from the additional information of the captured image. The necessary information includes the type of imaging optical system 141, the state of the imaging optical system 141 at the time of capture (focal length, F-number, focus distance), the pixel pitch of the image sensor 142, etc. The necessary information may also include the optical low-pass filter, the ISO sensitivity (noise intensity) at the time of capture, whether or not the captured image is denoised and the denoising parameters, the cropping position (the position of the optical axis of the imaging optical system 141 relative to the captured image after cropping), etc.

[0064] Resolution performance information (a two-channel map in this embodiment) is generated from the acquired information and a data table related to the resolution performance of the imaging optical system 141 stored in the storage unit 121. Information related to the type, state, image height, and resolution performance corresponding to the azimuth sampling points of the imaging optical system 141 is stored in the storage unit 121 as a data table. From this data table, resolution performance information corresponding to the captured image is generated by interpolation or the like. The resolution performance information in this embodiment is the same as that used during training, and is a map with the same number of two-dimensional pixels as the captured image, with values ​​representing the resolution performance in the horizontal direction stored in the first channel and the vertical direction stored in the second channel for each pixel. The value is the minimum frequency at which the MTF in the corresponding direction falls below a threshold (0.5), normalized by the sampling frequency (the reciprocal of the pixel pitch) of the image sensor 142.

[0065] As in training, the MTF is the MTF for the white of the blur that combines the effects of the optical low-pass filters and pixel apertures of the imaging optical system 141 and the image sensor 142. Note that if the resolution performance of the captured image does not change (the types and states of the imaging optical system 141 and the image sensor 142 are fixed), resolution performance information for the map state may be stored in the storage unit 121 and simply recalled. The noise information is also a map with the same two-dimensional number of pixels as the captured image, with the first channel representing the strength of noise generated during imaging and the second channel representing the denoising parameters applied to the captured image.

[0066] On the other hand, if it is determined in step S203 that the captured image and the resolution performance information do not match, the process proceeds to step S205. In step S205, the acquisition unit 123 acquires generator weight information and noise information from the storage unit 121. This step differs from step S204 in that no resolution performance information is acquired, and the weight information acquired is also different. Since it is determined that the captured image and the resolution performance information that can be acquired in step S204 do not match, an enlarged image is generated in the following step S206 without using the resolution performance information. Therefore, in step S205, weight information of a second machine learning model that is different from the first machine learning model is acquired. With regard to other processing, step S205 is the same as step S204.

[0067] Next, in step S206, the image enlargement unit 124 generates an enlarged image using the captured image and the information acquired in step S204 or step S205. The enlarged image is an image with a sampling pitch that is half that of the captured image (four times the number of pixels). Next, in step S207, the communication unit 122 transmits the enlarged image to the control device 103.

[0068] Next, in step S208, the communication unit 132 of the control device 103 acquires the enlarged image. The acquired enlarged image is stored in the storage unit 131 or displayed on the display unit 133. Alternatively, the image may be stored in another storage device connected to the control device 103 or the image enlargement device 102 via a wired or wireless connection.

[0069] In this embodiment, the captured image is one, but the present invention is not limited to this. An enlarged image may be generated from a plurality of captured images that are displaced by sub-pixels and resolution performance information.

[0070] Next, with reference to FIG. 9, generation of an enlarged image as a modified example of this embodiment will be described. FIG. 9 is a flowchart of generation of an enlarged image in this modified example. In this embodiment, a second machine learning model must be trained separately from the first machine learning model. The second machine learning model has a smaller number of parameters than the first machine learning model and therefore has a faster processing speed, but it must be trained and its weight information must be saved. Therefore, in this modified example, when it is determined that the captured image and the resolution performance of the optical system do not match, the second machine learning model is not used, and instead, second resolution performance information different from the first resolution performance information is used. The second resolution performance information is, for example, commonly used for multiple input images corresponding to different first resolution performance information.

[0071] Steps S301 to S303 in Fig. 9 are the same as steps S201 to S203, respectively, in Fig. 8. Furthermore, steps S304 and S305, which are performed when it is determined that there is a match in step S303 in Fig. 9, are the same as steps S204 and S206, respectively, which are performed when it is determined that there is a match in step S203 in Fig. 8. Furthermore, steps S306 and S307 in Fig. 9 are the same as steps S207 and S208, respectively, in Fig. 8.

[0072] If step S303 determines that the results are inconsistent, step S304 obtains the same weight information and noise information as if the results were inconsistent, but the resolution performance information is second resolution performance information different from the first resolution performance information. If the change in frequency characteristics due to the sharpening process applied to the captured image is known based on the image processing information, the change in frequency characteristics due to the sharpening process can be applied to the resolution performance information of the optical system to obtain resolution performance information that is consistent with the captured image as the second resolution performance information. In this case, if step S303 determines that the results are inconsistent, step S305 obtains the second resolution performance information instead of the first resolution performance information, thereby achieving the desired effect.

[0073] If it is not possible to obtain resolution performance information for sharpening processing, pseudo-resolution performance information may be obtained as the second resolution performance information. Here, a map that stores the maximum possible values ​​of the resolution performance information for all pixels is obtained and used as the second resolution performance information. In this case, the second resolution performance information does not depend on the first resolution performance information.

[0074] Here, it is preferable that the second resolution performance information has a value indicating higher performance than the first resolution performance information. The first reason is that the sharpening process in the captured image increases the contrast of both components below the Nyquist frequency and moiré components, which is considered to correspond to the frequency characteristics of an image acquired with an optical system that has higher resolution performance than the actual one. The second reason is that an image can be generated with higher accuracy by mistakenly assuming that the resolution is higher than the actual one, rather than by mistakenly assuming that the resolution is lower.

[0075] If the resolution performance is mistakenly assumed to be higher than the actual one, the object and moire will be mistakenly recognized as being mixed in a wider frequency band than the actual one, which may result in the generation of a false structure. However, compared to generating an enlarged image without inputting resolution performance information, even with incorrect resolution performance, high accuracy is still achieved in frequency bands where moire is not expected to occur, and the same accuracy can be achieved in frequency bands where moire is expected to occur. This is because generating an enlarged image in a situation where moire and the object are mixed is the same condition as when resolution performance information is not input.

[0076] On the other hand, if the resolution performance is mistakenly assumed to be lower than it actually is, frequencies where moiré is not expected to occur at the incorrect resolution performance will always be judged to be images of the subject, and moiré will inevitably remain. From the above discussion, the accuracy of the enlarged image generated is better when the resolution performance is set higher than the actual performance. In particular, when the resolution performance is set to the maximum, the machine learning model is made to recognize that moiré can occur regardless of the resolution performance of the optical system, and therefore accuracy equivalent to that of a machine learning model that does not use resolution performance information can be obtained.

[0077] Furthermore, in this embodiment, as described above, blur due to aberration and diffraction of the imaging optical system 141 and blur due to the optical low-pass filter of the image sensor 142 are not added when generating a high-resolution image. In other words, the first machine learning model is trained to correct the blur described above in addition to enlarging the image. Therefore, if resolution performance information indicating low performance is used, further sharpening based on the resolution performance information will be applied to a captured image that has already undergone sharpening processing, resulting in over-correction.

[0078] On the other hand, by setting the resolution performance information to the maximum value, an enlarged image can be generated without excessive blur correction. When the sharpening process applied to the captured image is not based on the resolution performance of the optical system, the purpose is to adjust the sharpness of the entire image, so it may overlap with the correction of blur originating from the optical system. On the other hand, when the sharpening process applied to the captured image is based on the resolution performance of the optical system, it is preferable to use resolution performance information that does not depend on the resolution performance information of the optical system and that shows higher performance than the resolution performance information of the optical system to avoid redundant correction. Here, blur correction based on the resolution performance of the optical system has been described, but this does not assume that the image will be enlarged.

[0079] In this embodiment, information on the sharpening process applied to the captured image is obtained from the additional information and used as the image processing information, but this is not limited to this. For example, trimming information on the captured image may be obtained as image processing information, and the consistency between the resolution performance of the optical system and the captured image may be determined based on this. The sharpening process changes the resolution performance indicated by the resolution performance information itself, thereby causing the consistency between the resolution performance of the optical system and the captured image to be lost.

[0080] On the other hand, cropping processing indirectly causes inconsistency by changing the correspondence between the resolution performance of the optical system and pixel positions in the captured image. As described above, the resolution performance information is a map that has the same number of pixels as the low-resolution image and indicates the resolution performance at each pixel in the low-resolution image to which it corresponds. The resolution performance of the optical system depends on the position of the captured image, and cropping the captured image causes inconsistency between the position on the screen and the optical axis and image height of the optical system. Cropping a position that is shifted from the center of the captured image causes the optical axis to be positioned off-center, and extracting a partial region disrupts the correspondence between the image height and the position on the image for the optical system.

[0081] Therefore, if information indicating that the captured image is a cropped image remains as image processing information, the determination unit 125 determines that the resolution performance of the optical system and the captured image do not match. If no direct cropping information remains, the number of pixels of the captured image and information identifying the imaging device may be used as image processing information. By identifying the imaging device, the number of pixels of the imaging device before cropping stored in the storage unit 121 of the image enlargement device 102 can be obtained. If the number of pixels is compared with the number of pixels of the captured image actually input and there is a mismatch, it can be determined that the resolution performance of the optical system and the captured image do not match. Note that if information on the cropping position can be obtained from the captured image, resolution performance information for the corresponding pixel position can be obtained. In this case, resolution performance information consistent with the captured image can be obtained, so it may be determined that there is a match.

[0082] Furthermore, information on image processing that changes the number of pixels, such as image enlargement, may be used as image processing information. This is because changing the number of pixels changes the correspondence between the resolution performance of the optical system and the pixel positions of the captured image, similar to cropping. Also, this is because enlarging or reducing an image changes the frequency characteristics and Nyquist frequency of the image.

[0083] In this embodiment, the control device 103 acquires the captured image directly from the imaging device 104, but the captured image may be acquired via an external image processing device that acquires the captured image from the imaging device 104.

[0084] In this embodiment, the captured image (input image) may also be a developed image. It is preferable that the developed image is one in which the processing performed during development is recorded as image processing information. Image processing applied during development can cause the resolution performance information of the optical system to lose consistency with the captured image. Therefore, it is preferable to acquire development information indicating the development conditions as image processing information. Furthermore, since developed images are stored in a commonly used data format, image processing is often applied after development. Therefore, when the captured image is a developed image, the present invention can improve the accuracy of the output image.

[0085] In this embodiment, a RAW image may also be used as the input image. When upsampling captured image data that has not been subjected to image processing that would cause the image data to lose consistency with the resolution performance of the optical system, the image data and the resolution performance of the optical system may be used. On the other hand, when upsampling a RAW image that has been subjected to sharpening processing as the input image, or when upsampling a RAW image that has already been upsampled as the input image, the input image and the resolution performance information of the optical system may not be consistent. When applying multiple image processing processes to a RAW image in this manner, it is preferable to obtain information on whether image processing has been applied before the input image is upsampled, and then perform upsampling based on that information. In this case, the image processing information may not be additional information, but may be information temporarily stored during a series of processes. The image processing information to be applied to the input image may also be obtained before the image processing is actually applied. In this case, the image processing information may be obtained in advance.

[0086] The image processing information may also include the name of the editing software or the name of the editing device that edited the image. Because some editing software does not record information about the image processing performed as additional information, acquiring the name of the editing software as image processing information can determine the possibility that the image processing performed by the editing software will result in a mismatch between the resolution performance of the optical system and the captured image. Some editing software can change (edit) the additional information of the input image, so editing the additional information can determine the possibility that the captured image will be mismatched with the resolution performance of the optical system assumed from the additional information. Furthermore, image processing performed by software other than the editing software can determine the possibility that the resolution performance of the optical system will be mismatched with the captured image. When a possibility that the resolution performance of the optical system will be mismatched with the captured image is obtained in this way, it is preferable to determine that the resolution performance of the optical system is mismatched with the captured image to avoid erroneous processing.

[0087] As described above, the image enlargement unit 124 changes the method of generating an enlarged image (output image) depending on the determination result by the determination unit 125. Therefore, according to this embodiment, it is possible to determine whether the resolution performance information and the input image are consistent with each other and generate an output image in which moire and false structures are effectively suppressed. [Example]

[0088] Next, an image processing system according to a second embodiment of the present invention will be described with reference to Fig. 10 and Fig. 11. In this embodiment, demosaicing is performed as upsampling, but the present invention can be similarly applied to other upsampling. In addition, a machine learning model is used for demosaicing.

[0089] Figure 10 is a block diagram of image processing system 300. Figure 11 is an external view of image processing system 300. Image processing system 300 has a training device 301 and an imaging device 302. The imaging device 302 has an imaging optical system 321, an imaging element 322, an image processing unit (image processing device) 323, a storage unit 324, a communication unit 325, and a display unit 326. The image processing unit 323 has an acquisition unit (acquisition means) 323a, a demosaic unit (generation means) 323b, and a determination unit (determination means) 323c.

[0090] The imaging optical system 321 forms a subject image from light in the subject space, and the image sensor 322 captures the subject image to generate a captured image. The captured image is an image in which RGB pixels are arranged in a Bayer array. The captured image is acquired as a live view of the subject space before capture or when the user presses the release button, and after undergoing development processing in the image processing unit 323, is stored in the storage unit 324 or displayed on the display unit 326. When developing the captured image, demosaicing is performed using a machine learning model, and a demosaiced image (output image) is generated. The machine learning model is trained in advance by the training device 301, and information on the trained weights is acquired via the communication unit 325. In demosaicing the captured image, resolution performance information, which is information on the resolution performance of the imaging optical system 321, is used. This processing will be described in detail.

[0091] First, the training of the machine learning model executed by the training device 301 will be described with reference to Fig. 12. Fig. 12 is a flowchart of the training of the machine learning model. Each step in Fig. 12 is executed by the storage unit 311, the acquisition unit 312, the calculation unit 313, or the update unit 314 of the training device 301.

[0092] First, in step S401, the acquisition unit 312 acquires one or more pairs of a mosaic image and a correct image from the storage unit 311. The mosaic image is a Bayer image of the same RGB (red, green, blue) as the captured image. FIGS. 13A and 13B are diagrams showing the relationship between a color filter array (Bayer array) and a Nyquist frequency, with FIG. 13A showing the Bayer array and FIG. 13B showing the Nyquist frequency of each color in the Bayer array. In FIG. 13A, G (green) has a sampling pitch that is the square root of 2 times the pixel pitch in the diagonal direction, and has a Nyquist frequency 402. R (red) and B (blue) have sampling pitches that are twice the pixel pitch in the horizontal and vertical directions, and have a Nyquist frequency 403.

[0093] The correct image has the same number of two-dimensional pixels as the mosaic image and three RGB channels. The correct image has a sampling pitch equal to the pixel pitch of each RGB color, and all colors have a Nyquist frequency 401. The correct image is generated as an original image captured by CG (Computer Graphics) or a three-chip image sensor. Alternatively, an image captured using a Bayer array may be reduced to generate an image with RGB signal values ​​for each pixel, and this may be used as the original image. At least a portion of the original image has frequency components equal to or greater than the Nyquist frequencies 402 and 403 of each color in the Bayer array. The correct image is generated by adding blur due to aberration and diffraction generated in the imaging optical system 321, and blur due to the optical low-pass filter and pixel aperture of the image sensor 322, to the original image. The mosaic image can be generated by sampling the correct image in a Bayer array. Multiple mosaic images and the correct image with different added blurs are generated, and the blur of the actual captured image is set within the range of the blur. Note that the mosaic image is not limited to the Bayer array.

[0094] 12, the calculation unit 313 acquires resolution performance information. In this embodiment, the resolution performance information is generated for each of RGB. As in this embodiment, the minimum frequency at which the MTF is equal to or less than a threshold in the horizontal and vertical directions for each of RGB is normalized by the Nyquist frequency of each of RGB, and this value is used as the resolution performance.

[0095] Next, in step S403, the calculation unit 313 inputs the mosaic image and the resolution performance information into a machine learning model to generate a demosaic image. In this embodiment, the demosaic image is generated using the configuration shown in FIG. 14. FIG. 14 is a diagram showing the flow of generating a demosaic image. An RGGB image 502 is generated by rearranging the mosaic image 501 into four channels: R, G1, G2, and B. The RGGB image 502 and resolution performance information 503, which is a map of 8 (4 × 2) channels indicating the resolution performance of each pixel of each RGGB color, are linked in the channel direction and input to a machine learning model 511 to generate a demosaic image 504. The machine learning model 511 has the same configuration as that shown in FIG. 6, but is not limited to this. Alternatively, the mosaic image 501 may be input to the machine learning model as is in the Bayer array without being rearranged into four channels.

[0096] 12, the update unit 314 updates the weights of the machine learning model 511 based on the error between the correct image and the demosaic image 504. Then, in step S405, the update unit 314 determines whether training of the machine learning model 511 is complete. If it is determined that training is not complete, the process returns to step S401. On the other hand, if it is determined that training is complete, the training is terminated and weight information is stored in the storage unit 311.

[0097] Next, the demosaic of a captured image performed by the image processing unit 323 will be described with reference to Fig. 15. Fig. 15 is a flowchart showing the generation of a demosaic image. Each step in Fig. 15 is performed by the acquisition unit 323a, demosaic unit 323b, or determination unit 323c of the image processing unit 323.

[0098] First, in step S501, the acquisition unit 323a acquires a captured image and image processing information. The captured image is a Bayer array image. Next, in step S502, similar to step S203, the determination unit 323c determines whether the resolution performance of the optical system (first resolution performance information) and the captured image acquired in step S501 are consistent based on the image processing information acquired in step S501.

[0099] Next, in step S503, the acquisition unit 323a acquires weight information of the machine learning model (first machine learning model) from the storage unit 324. Furthermore, if it is determined in step S502 that the resolution performance of the optical system and the captured image match, the acquisition unit 323a acquires resolution performance information of the optical system (first resolution performance information) from the storage unit 324 based on the state of the imaging optical system at the time of image capture, etc. If it is determined in step S502 that the resolution performance of the optical system and the captured image do not match, the acquisition unit 323a acquires a map (second resolution performance information) that holds the maximum possible values ​​of the resolution performance information for all pixels. Note that the order of acquisition is not limited to the above, as long as the image processing information is acquired before step S502, and the captured image and weight information are acquired before step S504. Next, in step S504, the demosaic unit 323b generates a demosaic image (output image) from the captured image and resolution performance information using the configuration shown in FIG. 14. The demosaic image is an image obtained by demosaicing the captured image.

[0100] The image processing unit 323 may perform other processes such as denoising and gamma correction as necessary. Furthermore, the image enlargement of this embodiment may be performed simultaneously with demosaicing. In this embodiment, similar to the modification of the first embodiment, when it is determined that the resolution performance information and the input image do not match, the machine learning model uses the second resolution performance information instead of the first resolution performance information without changing it. However, this is not limited to this. Similar to the first embodiment, a second machine learning model that does not input resolution performance information may be used instead of the first machine learning model.

[0101] As described above, the demosaic unit 323b changes the method of generating the output image depending on the determination result by the determination unit 323c. Therefore, according to this embodiment, it is possible to determine whether the resolution performance information and the input image are consistent, and generate an output image in which moire and false structures are effectively suppressed. [Example]

[0102] Next, an image processing system according to a third embodiment of the present invention will be described. The present invention is not limited to image upsampling, and can also be applied to other image processing using resolution performance information of an optical system. More specifically, any image processing may be used as long as it corrects image components that occur in an input image and are caused by the resolution characteristics of the optical system, based on the resolution characteristics of the optical system. In the first and second embodiments, moiré is mainly corrected as an image component caused by the resolution characteristics of the optical system. On the other hand, in this embodiment, a machine learning model is used to correct blur components caused by the resolution characteristics of the optical system.

[0103] The image processing system of this embodiment is the same as that of the second embodiment. When a captured image is developed, blur correction using a machine learning model is performed to generate a corrected image (output image). When aberrations of the captured image are corrected, resolution performance information, which is information relating to the resolution performance of the imaging optical system, is used. This processing will be described.

[0104] In the first embodiment, blurring due to aberration and diffraction of the imaging optical system and blurring due to the optical low-pass filter of the image sensor are not added when generating a high-resolution image. When generating a low-resolution image, these blurring effects are added and the image is generated by downsampling. This low-resolution image and resolution performance information are input and the system is trained to estimate a high-resolution image, thereby learning to simultaneously perform image enlargement and optical system blur correction. On the other hand, in this embodiment, the high-resolution image of the first embodiment is used as a blur-free ground truth image, and downsampling is eliminated from the low-resolution image generation process to generate a blurred image. By inputting the blurred image and resolution performance information into a machine learning model and training the system to output a ground truth image, it is possible to obtain information on the weights of the machine learning model that corrects blurring due to the optical system. The resolution performance information and the machine learning model are the same as those of the first embodiment, but are not limited thereto.

[0105] Blur correction of a captured image executed by the image processing unit 323 will be described with reference to Fig. 16. Fig. 16 is a flowchart showing generation of a blur-corrected image. Steps S601 to S604 in Fig. 16 are basically equivalent to steps S501 to S504 in Fig. 15, respectively, but instead of generating a demosaic image from a captured image and resolution performance information, a blur-corrected image is generated.

[0106] The image processing unit 323 may perform other processing such as denoising and gamma correction as necessary. In addition, the image enlargement in the first embodiment or the demosaic in the second embodiment may be used in combination with the blur correction.

[0107] In blur correction, if the resolution performance of the optical system and the captured image do not match, excessive correction can occur, resulting in overshoot or undershoot, or insufficient correction can leave blur, reducing the accuracy of the output image. Therefore, by determining whether the resolution performance information and the input image match and changing the processing accordingly, a more accurate blur-corrected image (output image) can be generated. In this embodiment, excessive blur correction can be avoided by setting the resolution performance information to its maximum value when it is determined that there is no match. In this embodiment, matching refers to the frequency characteristics determined by the imaging optical system or the optical low-pass filter of the image sensor matching the frequency characteristics of the captured image. In particular, matching refers to the frequency characteristics of the image used in training the machine learning model matching the frequency characteristics of the captured image, and mismatch refers to inconsistency. However, if the blur correction processing only corrects a portion of the blur (e.g., blur generated by the imaging optical system), only the corresponding portion of the blur is considered when determining whether there is match.

[0108] In this embodiment, similar to the modified example of the first embodiment, when it is determined that the resolution performance information and the input image do not match, the machine learning model is left as is and the second resolution performance information is used instead of the first resolution performance information, but this is not limited to this. As in the first embodiment, a second machine learning model that does not input resolution performance information may be used instead of the first machine learning model.

[0109] With the above configuration, this embodiment can determine whether the resolution performance information and the input image are consistent, suppress excessive correction, and generate an output image in which blur is well corrected.

[0110] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions. The image processing device of the present invention may be any device that has the image processing function of the present invention, and may be realized in the form of an imaging device or a PC.

[0111] According to each embodiment, it is possible to provide an image processing method, an image processing device, and a program that can improve the accuracy of image processing using machine learning related to the resolution performance of an imaging optical system.

[0112] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0113] 102 Image enlargement device (image processing device) 123 Acquisition unit (acquisition means) 124 Image enlargement unit (generation means) 125 Judgment unit (judgment means)

Claims

1. an acquisition step of acquiring an input image obtained by imaging using an optical system and first information related to image processing applied to the input image; a determination step of determining whether or not first resolution performance information indicating the resolution performance of the optical system matches the input image based on the first information; a generating step of generating an output image based on the input image and the first information, An image processing method, wherein in the generating step, a method for generating the output image is changed depending on the determination result in the determining step.

2. In the producing step, generating the output image by inputting the input image and the first resolution performance information into a first machine learning model when it is determined in the determination step that the first resolution performance information and the input image are consistent; The image processing method described in claim 1, characterized in that if it is determined that the first resolution performance information and the input image are not consistent, the output image is generated by inputting the input image into a second machine learning model different from the first machine learning model.

3. In the producing step, generating the output image by inputting the input image and the first resolution performance information into a first machine learning model when it is determined in the determination step that the first resolution performance information and the input image are consistent; The image processing method described in claim 1, characterized in that if it is determined that the first resolution performance information and the input image are not consistent, the output image is generated by inputting second resolution performance information indicating a resolution performance different from the resolution performance of the optical system into the first machine learning model.

4. 4. The image processing method according to claim 3, wherein the second resolution performance information is used in common for a plurality of input images corresponding to different first resolution performance information.

5. 5. The image processing method according to claim 3, wherein the resolution performance indicated by the second resolution performance information is higher than the resolution performance indicated by the first resolution performance information.

6. 6. The image processing method according to claim 3, wherein the second resolution performance information is acquired using information related to a sharpening process applied to the input image.

7. 7. The image processing method according to claim 6, wherein the sharpening process is based on the resolution performance of the optical system.

8. The image processing method according to any one of claims 1 to 7, characterized in that in the generation process, if it is determined in the judgment process that the first resolution performance information and the input image are not consistent, the first resolution performance information is not input.

9. 9. The image processing method according to claim 1, wherein the matching between the first resolution performance information and the input image is a matching between a frequency characteristic based on the resolution performance of the optical system and a frequency characteristic of the input image.

10. 9. The image processing method according to claim 1, wherein the matching between the first resolution performance information and the input image is a matching between a moire based on the resolution performance of the optical system and a moire of the input image.

11. 11. The image processing method according to claim 1, wherein the first information is acquired based on additional information of the input image.

12. 12. The image processing method according to claim 1, wherein the first information is acquired based on the number of pixels of the input image.

13. 13. The image processing method according to claim 1, wherein the input image is a developed image.

14. 14. The image processing method according to claim 1, wherein the first information is development information indicating development conditions for the input image.

15. 15. The image processing method according to claim 1, wherein the first information is information indicating a possibility that additional information of the input image has been changed.

16. 16. The image processing method according to claim 1, wherein the output image is an image in which image components contained in the input image and caused by the resolution characteristics of the optical system have been corrected.

17. 17. The image processing method according to claim 1, wherein the output image is an image obtained by enlarging or demosaicing a captured image, or an image obtained by correcting blurring that occurs in the optical system in the captured image.

18. 18. The image processing method according to claim 1, wherein the first resolution performance information is information based on the magnitude of blur occurring in the optical system.

19. 19. The image processing method according to claim 1, wherein the first resolution performance information is information based on the spread of a point spread function of the optical system or a modulation transfer function of the optical system.

20. 20. The image processing method according to claim 1, wherein the first resolution performance information is information based on the position of a pixel of the input image.

21. 21. The image processing method according to claim 1, wherein the first resolution performance information is a map in which values ​​are arranged in a size based on the number of pixels of the input image.

22. an acquisition means for acquiring an input image obtained by imaging using an optical system and first information relating to image processing applied to the input image; a determination means for determining whether or not first resolution performance information indicating the resolution performance of the optical system matches the input image based on the first information; generating means for generating an output image based on the input image and the first information; The image processing device is characterized in that the generating means changes the method of generating the output image depending on the determination result by the determining means.

23. An image processing system comprising the image processing device according to claim 22 and a control device capable of communicating with the image processing device, the control device has a transmission means for transmitting a request to cause the image processing device to execute processing on the input image, The image processing system is characterized in that the image processing device has a receiving means for receiving the request, and executes processing on the input image in response to the request.

24. A program causing a computer to execute the image processing method according to any one of claims 1 to 21.

Images