Image processing method, image processing device, image processing system, and program

Abstract

To provide an image processing method which can raise accuracy of image processing using machine learning related to resolution performance of an imaging optical system.SOLUTION: An image processing method comprises: an acquisition process of acquiring an input image obtained by imaging using an optical system, and image processing information regarding image processing to be preliminarily applied to the input image; a determination process of determining whether or not first resolution performance information showing resolution performance of the optical system is consistent with the input image on the basis of the image processing information; and a generation process of generating an output image on the basis of the input image and the image processing information, wherein a generation method of the output image is made different according to a determination result in the determination process in the generation process.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image processing method, an image processing apparatus, an image processing system, and a program.

Background Art

[0002] Patent Document 1 discloses a method for enlarging an image using machine learning. By training a machine learning model using a pair of a low-pixel image and a high-pixel image (an image with a smaller sampling pitch than the low-pixel image) of the same scene, it is possible to achieve higher-precision image enlargement than bicubic interpolation or the like. Further, Patent Document 1 discloses a method for generating a high-resolution enlarged image by enlarging a low-pixel image to the same number of pixels as a high-pixel image by bicubic interpolation and then inputting it into a trained machine learning model.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the method disclosed in Patent Document 1, there is a problem that a pseudo structure (artifact) that does not actually exist appears in the enlarged image, or moire that existed in the low-pixel image remains in the enlarged image. This problem also occurs in other image enlargement methods (such as bicubic interpolation and sparse coding) that do not use a machine learning model. Further, this problem occurs not only in image enlargement but also in processing for reducing the sampling pitch of other images (for example, demosaicing).

[0005] The root cause of this problem lies in the inability to distinguish between moiré patterns and the inherent structure of the subject in the input low-resolution image. Since the frequency range in which moiré patterns can occur is determined by the resolution performance of the imaging optical system, improvement is possible by basing the image on the resolution performance of the optical system used to capture the low-resolution image. On the other hand, there is a problem that post-processing applied to the low-resolution image can transform it into an image that does not match the resolution performance of the optical system.

[0006] Therefore, the present invention aims to provide an image processing method that can improve the accuracy of image processing using machine learning related to the resolution performance of an imaging optical system. [Means for solving the problem]

[0007] An image processing method as one aspect of the present invention includes an acquisition step of acquiring an input image obtained by imaging using an optical system and image processing information relating to image processing to be applied to the input image in advance; a determination step of determining whether the input image matches a first resolution performance information indicating the resolution performance of the optical system based on the image processing information; and a generation step of generating an output image based on the input image and the image processing information, wherein the method of generating the output image is varied in the generation step according to the determination result in the determination step.

[0008] Other objects and features of the present invention are described in the following examples. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide an image processing method that can improve the accuracy of image processing using machine learning related to the resolution performance of an imaging optical system. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows the relationship between the modulation transfer function and the Nyquist frequency in Examples 1 and 2. [Figure 2] This is a block diagram of the image processing system in Example 1. [Figure 3] This is an external view of the image processing system in Example 1. [Figure 4] This is a flowchart of the machine learning model training in Example 1. [Figure 5] This diagram shows the process for generating an enlarged image in Example 1. [Figure 6] This is a diagram illustrating the configuration of the machine learning models in Examples 1-3. [Figure 7] This diagram shows the process for generating an enlarged image in Example 1. [Figure 8] This is a flowchart for generating an enlarged image in Example 1. [Figure 9] This is a flowchart for generating an enlarged image in a modified example of Example 1. [Figure 10] This is a block diagram of the image processing system in Example 2. [Figure 11] This is an external view of the image processing system in Example 2. [Figure 12] This is a flowchart of the machine learning model training in Example 2. [Figure 13] This figure shows the relationship between the color filter array and the Nyquist frequency in Example 2. [Figure 14] This diagram shows the process for generating demosaicing images in Example 2. [Figure 15] This is a flowchart for generating demosaicing images in Example 2. [Figure 16] This is a flowchart for generating a blur-corrected image in Example 3. [Modes for carrying out the invention]

[0011] The embodiments of the present invention will be described in detail below with reference to the drawings. In each figure, the same reference numerals are used for the same components, and redundant explanations are omitted.

[0012] Before explaining each embodiment in detail, the gist of the present invention will be briefly described. In each embodiment, in the process of reducing the sampling pitch of an input image (hereinafter referred to as upsampling), resolution performance information, which is information regarding the resolution performance of the optical system used for imaging the input image (captured image), is used. As a result, the accuracy of upsampling is improved. To explain the reason, the problems of upsampling and its principle of occurrence will be described in detail below.

[0013] When converting the subject image formed by the optical system into a captured image by the image sensor, sampling (sampling) is performed by the pixels of the image sensor. Therefore, among the frequency components forming the subject image, components exceeding the Nyquist frequency of the image sensor are mixed with low-frequency components by aliasing (folding), and moire occurs. In the upsampling of the captured image, since the Nyquist frequency increases as the sampling pitch decreases, ideally, it is desired to generate an image in which aliasing has not occurred up to the increased Nyquist frequency. However, it is difficult to distinguish whether it is moire or the original structure of the subject from a captured image in which moire has already occurred.

[0014] In a method such as bilinear interpolation, moire remains as it is even when the captured image is upsampled. On the other hand, in a method using a machine learning model or the like, since it is possible to estimate high frequencies before aliasing occurs from moire, some moire can be removed. However, as described above, since it is difficult to distinguish between moire and the structure of the subject, some moire is misrecognized as the subject and remains, and some subjects are misrecognized as moire and false structures are generated.

[0015] Therefore, in each embodiment, consider the case where resolution performance information of the optical system used for imaging the captured image (input image) is used in the upsampling of the captured image. This case will be described with reference to FIGS. 1(A) and (B). FIGS. 1(A) and (B) are the frequency characteristics of the modulation transfer function (MTF) representing the resolution performance of the optical system, and show the relationship between the modulation transfer function and the Nyquist frequency.

[0016] In Figures 1(A) and (B), the horizontal axis represents spatial frequency in a certain direction, and the vertical axis represents MTF. For example, as shown in Figure 1(A), if the cutoff frequency 003 of the optical system (here, meaning the frequency above which the MTF becomes 0) is less than or equal to the Nyquist frequency 001, then no moiré patterns are present in the captured image. This is because even if the MTFs are arranged with a period of sampling frequency 002, there is no region where the MTFs overlap with each other. Therefore, if the resolution performance corresponds to Figure 1(A), it is clear that there is no need to estimate the high-frequency components before moiré patterns appear, thereby suppressing the generation of false structures.

[0017] Furthermore, as shown in Figure 1(B), even when the cutoff frequency 003 exceeds the Nyquist frequency 001, it is possible to identify the frequency band in which moiré patterns may occur due to aliasing. Moiré patterns may occur in the band between the frequency 004 (obtained by subtracting the cutoff frequency 003 from the sampling frequency 002) and the Nyquist frequency 001, but not in other bands. Therefore, by using resolution performance information, the occurrence of false structures can be similarly suppressed. For these reasons, the accuracy of upsampling of captured images can be improved.

[0018] When performing image processing using resolution performance information on captured images, the captured image and the resolution performance information must be consistent. Captured images are acquired when the subject is imaged through the imaging optical system and sampled by the sensor. Consistency is defined as maintaining the resolution characteristics of the image immediately after capture, while consistency is not defined as changing the resolution characteristics of the image due to image processing that alters the frequency characteristics. Image processing that alters the frequency characteristics includes, for example, sharpening and blurring. In particular, consistency is defined as applying image processing that was not considered in the training dataset for the machine learning model, or when the relationship between the captured image and the resolution performance information is no longer one that is included in the training dataset.

[0019] Image processing during or after imaging alters the characteristics of the image, leading to a mismatch between the input image and the resolution performance information, which reduces the accuracy of image processing using the resolution performance information. Using resolution performance information that does not match the input image can increase the frequency of moiré persistence and the occurrence of false structures, and may also increase contrast compared to not using resolution performance information. This is because the machine learning model misinterprets moiré and the structure of the subject when using the resolution performance information. If the resolution performance is incorrectly perceived as higher than the input image, the machine learning model perceives moiré as occurring in a lower frequency range than it actually is, leading to the misidentification of subjects in frequency ranges where moiré does not occur as moiré and the excessive generation of false structures. If the resolution performance is incorrectly perceived as lower than the input image, the machine learning model perceives that moiré does not occur in a higher frequency range than it actually is, or that moiré does not occur at all, leading to the misidentification of moiré as the subject in frequency ranges where moiré does occur, resulting in excessive moiré persistence. Therefore, determining whether the resolution performance information matches the input image can improve the accuracy of upsampling the input image. Since machine learning models using resolution performance information can improve accuracy, the present invention is not limited to upsampling and includes processes such as blur correction. It also includes image enlargement processing where the magnification ratio differs for each screen area. [Examples]

[0020] First, the image processing system in Embodiment 1 of the present invention will be described. In this embodiment, image enlargement (upscaling) is performed as upsampling, but it can be similarly applied to other upsampling methods such as demosaicing. Image enlargement includes increasing the number of sampling points for the entire captured image and increasing the number of sampling points for a partial region of the captured image (such as enlarging a cropped image or digital zoom). A machine learning model is also used for image enlargement.

[0021] Figure 2 is a block diagram of the image processing system 100. Figure 3 is an external view of the image processing system 100. The image processing system 100 includes a training device 101, an image magnification device (image processing device, second device) 102, a control device (first device) 103, and an imaging device 104, all connected to each other by a wired or wireless network. The control device 103 includes a storage unit 131, a communication unit 132, a display unit 133, and an image processing unit 134. The control device 103 and the image magnification device 102 can communicate with each other. The control device 103 acquires an image (input image) from the imaging device 104 according to the user's instructions and transmits the image and a request to perform image magnification processing to the image magnification device 102 via the communication unit (transmission means) 132. At this time, the user can perform any image processing while transmitting the image acquired from the imaging device 104 to the image magnification device 102.

[0022] The imaging device 104 includes an imaging optical system 141, an image sensor 142, an image processing unit 143, and a storage unit 144. The imaging optical system 141 forms an image of the subject from light in the subject space, and the image sensor 142, which has multiple pixels arranged in it, converts this image into a captured image. At this time, aliasing occurs in the frequency components of the subject image that are higher than the Nyquist frequency of the image sensor 142. As a result, moiré patterns may occur in the captured image. The image processing unit 143 performs predetermined processing on the captured image as needed (such as correction of pixel defects and development), and the captured image is stored in the storage unit 144. The control device 103 acquires the captured image via communication or a storage medium. The captured image to be acquired may be the entire image or a partial region of the captured image.

[0023] The image magnification device 102 includes a storage unit 121, a communication unit 122, an acquisition unit (acquisition means) 123, an image magnification unit (generation means) 124, and a determination unit (determination means) 125. The image magnification device 102 magnifies an captured image (input image) using a trained machine learning model and generates a magnified image (output image). In this process, it uses resolution performance information, which is information about the resolution performance of the optical system (imaging optical system 141, etc.) used to capture the captured image. Details of this process will be described later. The image magnification device 102 acquires weight information of the trained machine learning model from the training device 101 and stores it in the storage unit 121. The training device 101 includes a storage unit 111, an acquisition unit 112, a calculation unit 113, and an update unit 114, and pre-trains a machine learning model using a dataset. The weight information of the machine learning model generated by training is stored in the storage unit 111. The control device 103 acquires the enlarged image from the image enlargement device 102 and presents it to the user via the display unit 133.

[0024] Next, with reference to Figure 4, the training (weight determination) of the machine learning model performed by the training device 101 will be explained. Figure 4 is a flowchart of the training of the machine learning model. Each step in Figure 4 is performed by the memory unit 111, acquisition unit 112, calculation unit 113, or update unit 114 of the training device 101. In this embodiment, training is performed using a GAN (Generative Adversarial Network), but it is not limited to this. The machine learning model includes, for example, neural networks, genetic programming, and Bayesian networks. The neural network is, for example, a CNN (Convolutional Neural Network), but it is not limited to this. The neural network may also be a GAN (Generative Adversarial Network), an RNN (Recurrent Neural Network), a Transformer, etc.

[0025] First, in step S101, the acquisition unit 112 acquires one or more pairs of high-resolution and low-resolution images from the storage unit 111. The storage unit 111 stores a dataset containing multiple high-resolution and low-resolution images. The low-resolution images are the images input to the machine learning model (generator in this embodiment) during training of the machine learning model. The more accurately the low-resolution images reproduce the properties of the actual enlarged captured image, the higher the accuracy of the trained machine learning model. Properties of the captured image include, for example, resolution performance, color representation, and noise characteristics. If the captured image is represented in RGB, while the low-resolution images are represented in monochrome or YUV, the color representations of the two images will not match, which may reduce the accuracy of the task. The properties that are important differ depending on the type of task, but in image enlargement, as mentioned above, the frequency band in which moiré occurs is important, so resolution performance is important. Therefore, it is desirable that the resolution performance of the captured image falls within the range of the resolution performance of the multiple low-resolution images used for training.

[0026] High-resolution images are ground truth images used in training machine learning models. High-resolution images are images of the same scene as the corresponding low-resolution images, but with a smaller sampling pitch (i.e., more pixels) than the low-resolution images. In this embodiment, the sampling pitch of the high-resolution image is half that of the low-resolution image. Therefore, the machine learning model enlarges the input image to twice the number of pixels in both the horizontal and vertical directions, quadrupling the number of pixels. However, this embodiment is not limited to this. To enable the machine learning model to handle images of various subjects, it is desirable that the multiple low-resolution and high-resolution images used for training include various subjects (edges with different orientations and strengths, textures, gradients, flat areas, etc.). At least a portion of the high-resolution image has frequency components above the Nyquist frequency of the low-resolution image.

[0027] In this embodiment, the high-resolution and low-resolution images used are those generated from the original image through imaging simulation. However, the invention is not limited to this, and imaging simulation may be performed using 3D data of the subject space instead of the original image. Alternatively, the high-resolution and low-resolution images may be generated by actual shooting with image sensors having different pixel pitches.

[0028] The original image is an undeveloped RAW image (where light intensity and signal value have a linear relationship), has a sampling pitch lower than or equal to that of a high-resolution image, and contains frequency components above the Nyquist frequency of the low-resolution image, at least in part. The low-resolution image is generated by reproducing the same imaging process as the image that is actually enlarged by a trained machine learning model, using the original image as the subject. Specifically, the original image is given blur due to aberrations and diffraction occurring in the imaging optical system 141, and blur due to the optical low-pass filter and pixel aperture of the image sensor 142. If there are multiple types and states of the imaging optical system 141 and the image sensor 142, and different blurs may act on the captured image depending on them, the dataset is made to include low-resolution images with these multiple blurs. The blur can change depending on the position of each pixel of the image sensor 142 (image height and azimuth relative to the optical axis of the imaging optical system 141), and also depending on the state of the imaging optical system 141, if it can take on various states (focal length, F-number, focus distance, etc.). Furthermore, if the imaging optical system 141 can take on multiple types, such as in an interchangeable lens system, the blur will also change depending on the type. In addition, if there are different types of imaging devices 104, and the pixel pitch and optical low-pass filter differ, the blur will also change.

[0029] The blur applied to the original image may be the blur generated by the imaging optical system 141 or the image sensor 142 itself, or a blur that approximates it. For example, the PSF (point image distribution function) of the blur generated by the imaging optical system 141 or the image sensor 142 may be approximated by a two-dimensional Gauss distribution function, a mixture of multiple two-dimensional Gauss distribution functions, or a Zernike polynomial. Alternatively, the OTF (optical transfer function) or MTF (modulation distribution function) may be approximated by a two-dimensional Gauss distribution function, a mixture of multiple two-dimensional Gauss distribution functions, or a Legendre polynomial.

[0030] After blurring the original image, it is downsampled using the sampling pitch of the image sensor 142. Furthermore, since the image sensor 142 has RGB (Red, Green, Blue) color filters arranged in a Bayer array, the low-resolution image is also sampled so that it is arranged in a Bayer array. However, the invention is not limited to this, and the image sensor 142 may be monochrome, honeycomb array, 3-chip type, etc. If there are multiple types of image sensors 142 and the pixel pitch of the captured image may change, low-resolution images are generated for multiple sampling pitches to cover the range of change. In this embodiment, noise generated by the image sensor 142 is also added to the low-resolution image. This is because if noise is not added to the low-resolution image (noise is not considered when training the machine learning model), when the captured image is enlarged, not only the subject but also the noise will be emphasized. If there is a range in the strength of the noise generated in the captured image (for example, if there are multiple ISO sensitivities during imaging), the dataset should include multiple low-resolution images with varying noise strengths within the possible range.

[0031] High-resolution images are generated by adding blur to the original image using a pixel aperture with half the pixel pitch of the low-resolution image, downsampling with half the sampling pitch of the low-resolution image, and then performing Bayer processing. Note that if the sampling pitch of the original image and the high-resolution image are the same, the original image may be used directly as the high-resolution image. In this embodiment, blur due to aberrations and diffraction of the imaging optical system 141, and blur due to the optical low-pass filter of the image sensor 142, are not added during high-resolution image generation. This allows the machine learning model to be trained to correct the aforementioned blur along with image magnification. However, the invention is not limited to this; the same blur as the low-resolution image may be added to the high-resolution image, or the blur added to the low-resolution image may be reduced and applied to the high-resolution image. In this embodiment, no noise is added during high-resolution image generation. This allows the machine learning model to be trained to perform denoising along with image magnification. However, the invention is not limited to this; noise of the same or different intensity as the noise added to the low-resolution image may be added. Furthermore, when adding noise to high-resolution images, it is desirable to add noise that is correlated with the noise in low-resolution images. This is because if the noises are uncorrelated, the effect of the noise in high-resolution images may be averaged out when training with multiple images in the dataset, and the desired effect may not be achieved.

[0032] In this embodiment, image enlargement is performed on a developed captured image. Therefore, both the low-resolution and high-resolution images must also be developed images. Accordingly, the same development process as the captured image is performed on the low-resolution and high-resolution images in Bayer state and stored in the dataset. However, the invention is not limited to this, and the low-resolution and high-resolution images may be in RAW format, and the captured image may also be enlarged in RAW format. Furthermore, if compression noise such as JPEG encoding occurs in the captured image, similar compression noise may be applied to the low-resolution image. This trains the machine learning model to remove compression noise along with image enlargement.

[0033] Next, in step S102, the acquisition unit 112 acquires resolution performance information and noise information. The resolution performance information is information about the resolution performance of the blur applied to the low-pixel image. If the resolution performance is low (MTF is 0 or a sufficiently small value below the Nyquist frequency of the low-pixel image), there is no moiré in the low-pixel image. If the resolution performance is high (MTF has a value at frequencies above the Nyquist frequency), there is no moiré outside the frequency band where aliasing occurs. Thus, the resolution performance information represents the frequency band in which moiré occurs in the low-pixel image. Therefore, the resolution performance information includes information based on the magnitude of the blur applied to the low-pixel image. Furthermore, the resolution performance information includes information based on either the spread of the blur's PSF or the blur's MTF. Note that the blur's PTF (phase transfer function) alone does not constitute resolution performance information. This is because the PTF represents the shift in the imaging position.

[0034] In this embodiment, the resolution performance information used when enlarging the captured image is information on blur that integrates all the effects of the aberrations and diffraction of the imaging optical system 141, the optical low-pass filter and pixel aperture of the image sensor 142. However, the invention is not limited to this, and the resolution performance may be expressed using only a part of the blur (for example, the blur generated in the imaging optical system 141). For example, if the optical low-pass filter and pixel pitch are fixed and do not change, there is no problem in expressing the resolution performance using only the blur generated in the imaging optical system 141. However, in this case, it is necessary to determine the resolution performance of low-resolution images accordingly. It is preferable to determine the resolution performance information for blur obtained by excluding the effects of the optical low-pass filter and pixel aperture from the blur applied to low-resolution images.

[0035] Noise information refers to information about noise added to a low-resolution image. This noise information includes information representing the noise intensity. Noise intensity can be expressed using the standard deviation of the noise or the corresponding ISO sensitivity of the image sensor 142. Furthermore, if denoising has been performed on the image before enlargement, it is desirable to perform similar denoising on the low-resolution image and include the denoising parameters (representing intensity, etc.) in the noise information. This allows for high-precision image enlargement while suppressing negative effects even when noise or denoising changes.

[0036] The following provides specific examples of resolution performance information and noise information. In this embodiment, resolution performance information is generated by the following method, but is not limited to this. The resolution performance information is a map (a map in which values ​​are arranged in a size based on the number of pixels) with the same number of pixels in two dimensions (horizontal and vertical) as the low-resolution image. Each pixel in the map indicates the resolution performance of the corresponding pixel in the low-resolution image. The map has multiple channels, with the first channel indicating the horizontal resolution performance and the second channel indicating the vertical resolution performance. The resolution performance is a value based on the frequency at which the MTF for the white of the blur applied to the low-resolution image reaches a predetermined value in the relevant direction. More specifically, it is the minimum frequency among the frequencies at which the MTF becomes less than or equal to a threshold (0.5 in this embodiment, but is not limited to this). Furthermore, the resolution performance is shown as a value normalized by the sampling frequency of the low-resolution image using the aforementioned minimum frequency. The sampling frequency used for normalization is the reciprocal of the pixel pitch and is common to RGB. However, the value representing the resolution performance is not limited to this. Furthermore, instead of using white light, the resolution performance of each RGB channel may be represented using 6 channels, and the frequencies used for standardization may also differ for each RGB channel.

[0037] Examples of other resolution performance information are shown below. The resolution performance shown in the resolution performance information may be in the meridional (radial) direction and the sagittal (azimuth) direction. Furthermore, a third channel representing the azimuth of the pixel may be added. In addition to two directions, the number of channels may be increased to represent resolution performance in multiple directions. Alternatively, resolution performance may be represented by only one channel, such as by taking the average of a specific direction or all directions. Also, the resolution performance information may be a scalar value or vector instead of a map. When the imaging optical system 141 is a super telephoto lens or when the F number is large, the change in resolution performance due to image height and azimuth becomes very small. Therefore, in the above cases, the effect of the invention can be obtained even with a scalar value instead of a map showing the performance for each pixel. Furthermore, the integral value of MTF may be used as the resolution performance.

[0038] Resolution performance can also be expressed as the spread of the PSF. Alternatively, resolution performance can be represented by the spatial range where the full width at half maximum (FWHM) or PSF intensity in multiple directions exceeds a threshold value. Furthermore, when representing resolution using scalar values ​​instead of a map, it is advisable to use values ​​in a specific direction or the average value across multiple directions, similar to the MTF.

[0039] Alternatively, resolution performance may be expressed using coefficients fitted to MTF or PSF. It is advisable to use power series, Fourier series, mixed Gaussian models, Legendre polynomials, Zernike polynomials, etc., to fit the model and express the fitting coefficients across multiple channels.

[0040] Resolution performance information may be generated by calculation from the blur applied to a low-resolution image, or it may be obtained from a storage unit 111 that stores in advance multiple blurs and corresponding resolution performance information.

[0041] Noise information, like resolution performance information, is a map with the same number of pixels as the low-resolution image and the 2D image. The first channel represents the strength of the noise in the low-resolution image before denoising, and the second channel represents the strength of the denoising performed. If compression noise exists in the low-resolution image, the strength of the compression noise may be added to the channel. Noise information, like resolution performance information, can also be in the form of scalar values ​​or vectors.

[0042] In this embodiment, the execution order of steps S101 and S102 is not limited, and step S102 may be executed before step S101.

[0043] Next, in step S103, the calculation unit 113 uses a generator, which is a machine learning model, to generate an enlarged image from the low-resolution image, resolution performance information, and noise information. The enlarged image is a low-resolution image with a reduced sampling pitch.

[0044] The generation of the enlarged image will now be explained with reference to Figures 5 and 6(A) and (B). Figure 5 is a diagram showing the flow of the enlarged image generation. Figures 6(A) and (B) are diagrams of the machine learning model configuration. In Figure 5, sum represents the sum of each element (pixel), and concatenation represents concatenation in the channel direction. In this embodiment, the resolution performance information 202 and the noise information 203 are maps with the same number of pixels in two dimensions as the low-resolution image 201. The low-resolution image 201, the resolution performance information 202, and the noise information 203 are concatenated in the channel direction and then input to the generator 211 to generate the residual component 204. The residual component 204 has the same number of pixels in two dimensions as the high-resolution image. The enlarged image 205 is generated by enlarging the low-resolution image 201 to the same number of pixels as the high-resolution image using bilinear interpolation or the like, and then summing it with the residual component 204. Alternatively, the enlarged image 205 may be generated directly by the generator 211 without going through the residual component 204. If the resolution performance information 202 or noise information 203 are scalar values ​​or vectors, or if the number of pixels in the two-dimensional plane does not match that of the low-resolution image 201, they may be converted to a feature map via a convolutional layer and then concatenated in the channel direction with the low-resolution image 201 (or the feature map obtained by converting it).

[0045] The generator 211 is a CNN configured as shown in Figure 6(A), but is not limited to this configuration. The initial values ​​of the weights of the generator 211 can be generated using random numbers or similar methods. In Figure 6(A), conv. represents convolution, ReLU represents Rectified Linear Unit, and sub-pixel conv. represents subpixel convolution. In this embodiment, by quadrupling the number of pixels in the 2D input using subpixel convolution, the number of pixels in the 2D residual component 204 becomes the same as that of a high-resolution image. residual block represents a residual block. A residual block has multiple linear sum layers and an activation function, and is configured to sum the inputs and outputs of the block. In this embodiment, the residual block is shown in Figure 6(B). GAP represents global average pooling, dense represents fully connected, sigmoid represents the sigmoid function, and multiply represents element-wise multiplication. By generating an attention map using GAP and fully connected, the accuracy of the task is improved. In this embodiment, the generator 211 has 16 residual blocks. However, the number of residual blocks is not limited to this. If the performance of the generator 211 is to be further improved, the number of residual blocks can be increased.

[0046] Alternatively, the low-resolution image 201 may be pre-scaled using bilinear interpolation or similar methods to match the pixel count of the high-resolution image before being input to the generator 211. In this case, subpixel convolution is not required in the generator 211. However, if the number of 2D pixels in the low-resolution image 201 increases, the number of times linear sums are taken increases, leading to a higher computational load. Therefore, it is preferable to input the low-resolution image 201 to the generator 211 without scaling it up, as in this embodiment, and scale it internally.

[0047] Next, in step S104 of Figure 4, the calculation unit 113 inputs the enlarged image 205 and the high-resolution image to the classifier and generates a classification output. The classifier identifies whether the input image is the image generated by the generator 211 (the enlarged image 205 from which high-frequency components were estimated from the low-resolution image) or the actual high-resolution image (an image from which frequency components above the Nyquist frequency of the low-resolution image were acquired during imaging). A CNN or similar architecture may be used for the classifier. The initial values ​​of the classifier weights are determined by random numbers or the like. Note that the high-resolution image input to the classifier can be any actual high-resolution image and does not need to be the image corresponding to the low-resolution image 201.

[0048] Next, in step S105, the update unit 114 updates the classifier weights based on the identification output and the correct label. In this embodiment, the correct label for the enlarged image 205 is 0, and the correct label for the actual high-resolution image is 1. Sigmoid cross-entropy is used as the loss function, but other functions may also be used. Backpropagation is used to update the weights.

[0049] Next, in step S106, the update unit 114 updates the weights of the generator 211 based on the first loss and the second loss. The first loss is the loss based on the difference between the high-resolution image corresponding to the low-resolution image 201 and the enlarged image 205. In this embodiment, MSE (Mean Square Error) is used, but MAE (Mean Absolute Error) or other methods may also be used. The second loss is the sigmoid cross-entropy between the identification output when the enlarged image 205 is input to the classifier and the correct label 1. The generator 211 is trained so that the classifier misidentifies the enlarged image 205 as the actual high-resolution image, so the correct label is set to 1 (corresponding to the actual high-resolution image). Note that the order of steps S105 and S106 may be reversed.

[0050] Next, in step S107, the update unit 114 determines whether or not training of the generator 211 is complete. If training is not yet complete, the process returns to step S101 and acquires one or more new sets of low-resolution images 201 and high-resolution images. On the other hand, if training is complete, the weight information is stored in the storage unit 111. Note that since only the generator 211 is used during image enlargement, only the weight of the generator 211 may be stored.

[0051] Alternatively, the generator 211 may be trained using only the first loss before training the GAN with the discriminator. Alternatively, the first and second datasets may be stored in the memory unit 111, training steps S101-S107 may be performed on the first dataset, and then the resulting weights may be used as initial values ​​for training steps S101-S107 on the second dataset. The first dataset contains fewer high-resolution images with high-frequency components above the Nyquist frequency compared to the second dataset (i.e., fewer moiré patterns in low-resolution images). Therefore, a generator 211 trained on the first dataset is more likely to retain moiré patterns but less likely to produce false structures. In contrast, a generator 211 trained on the second dataset can remove moiré patterns but is more likely to produce false structures. By storing the intermediate weights of the generator 211 during training with the second dataset, weights that balance moiré removal and false structure removal can be selected later.

[0052] Through the above process, it is possible to obtain weight information for the generator 211 (first machine learning model) that generates an enlarged image based on the captured image, resolution performance information, and noise information. In this embodiment, a second machine learning model that does not rely on resolution performance information is also used. The second machine learning model can be trained by configuring it so that resolution performance information 202 is not used as input data to the generator 211, as shown in Figure 7 which illustrates the flow of generating an enlarged image. The low-resolution image 201 and noise information 203 are the same as those of the first machine learning model. In this embodiment, the configuration of the generator 211 is the same as in Figures 6(A) and (B). However, since the resolution performance information 202 is not concatenated in the channel direction, the number of parameters in the convolutional layer is different from that of the first machine learning model. Therefore, the weight information for the second machine learning model is obtained by training it separately from the first machine learning model. Note that the configuration of the generator 211 may differ between the first and second machine learning models.

[0053] Next, with reference to Figure 8, the image magnification process (image processing method) performed by the image magnification device 102 and the control device 103 will be described. Figure 8 is a flowchart of the generation of a magnified image. Each step in Figure 8 is performed by the storage unit 121, communication unit 122, acquisition unit 123, image magnification unit 124, or determination unit 125 of the image magnification device 102, or by the storage unit 131, communication unit 132, display unit 133, or image processing unit 134 of the control device 103.

[0054] First, in step S201, the communication unit (transmission means) 132 of the control device 103 transmits the captured image (input image), image processing information, and a request to perform the enlargement process to the image enlargement device 102. The captured image is the developed image, as in the training session.

[0055] Image processing information is part of the additional information of the captured image in this embodiment and is attached to the captured image. Additional information is metadata such as header information, and is information about the data itself that is attached to the image data. In this embodiment, additional information is transmitted attached to the captured image, but it may also be transmitted separately from the captured image, or only the information used in the processing described later may be extracted from the additional information and transmitted. Image processing information is information about image processing that can be applied to the captured image, and in this embodiment, it is information about the image processing applied by the image processing unit 143 of the imaging device 104 at the time of capture. It may also be information about image processing applied by the image processing unit 134 of the control device 103 between the time of capture and transmission to the image magnification device 102. It may also be information about image processing that is communicated to an external image processing device before transmission to the image magnification device 102 and applied by the external image processing device, or it may be a combination of the above-mentioned information. Information about image processing is, for example, information about sharpening processing, and includes information such as whether the corresponding image processing was applied, image processing parameters such as sharpening intensity, and which of multiple processing options was selected.

[0056] Next, in step S202, the acquisition unit 123 of the image magnification device 102 acquires the captured image, image processing information, and execution request transmitted from the control device 103 via the communication unit (receiving means) 122.

[0057] Next, in step S203, the determination unit 125 of the image magnification device 102 determines whether the resolution performance of the optical system (first resolution performance information) and the captured image acquired in step S202 are consistent, based on the image processing information acquired in step S202. Note that the process of determining whether the resolution performance of the optical system and the captured image are consistent in step S203 includes a process to acquire only the possibility that the resolution performance and the captured image are not consistent, as will be described later. The resolution performance of the optical system corresponds to the optical performance information acquired in step S204 if it is determined that they are consistent. In this embodiment, the acquired image processing information includes information indicating whether or not sharpening processing has been applied to the captured image (information related to sharpening processing), and it is confirmed whether or not sharpening processing has already been applied to the captured image based on the image processing information.

[0058] In this embodiment, the frequency at which the MTF falls below a threshold is used as resolution performance information. However, when sharpening processing is applied, the contrast of subjects with frequencies lower than the Nyquist frequency in the captured image and the contrast of moiré patterns change. As a result, the frequency characteristics based on the resolution performance of the optical system and the frequency characteristics of the captured image differ. The frequency characteristics based on the resolution performance of the optical system are the frequency characteristics of the captured image calculated from the resolution performance of the optical system, and are the frequency characteristics of the image used when training the machine learning model. Also, because the contrast of the moiré patterns changes, the intensity differs from the moiré patterns when sharpening processing is not applied. In other words, the moiré patterns that occur in the captured image and the moiré patterns based on the resolution performance of the optical system are different. The moiré patterns based on the resolution performance of the optical system are the moiré patterns that should occur in the captured image calculated from the resolution performance of the optical system, and are also present in the training dataset. As a result, moiré patterns remain and false structures are more likely to occur.

[0059] Therefore, if sharpening is applied, the determination unit 125 determines that the resolution performance of the optical system and the captured image do not match. On the other hand, if sharpening is not applied, the determination unit 125 determines that the resolution performance of the optical system and the captured image do match. Generally, it is known that in machine learning models, accuracy decreases if there is a difference between the dataset used for training and the data actually used for estimation. Therefore, information from image processing that changes the relationship between the captured image and the resolution performance information to a different relationship than that included in the dataset, such as sharpening, is acquired as image processing information. This makes it possible to distinguish whether the resolution performance and the captured image match, that is, whether or not they are in a relationship included in the dataset, and to generate an enlarged image.

[0060] The sharpening process described above is preferably one that corrects the degradation of resolution based on the resolution performance of the optical system. In the case of sharpening that is not based on the resolution performance of the optical system, the frequency gain is uniform across the screen, making it easy to consider when creating a training dataset. Therefore, by creating a dataset with varying levels of sharpening and intensity, high-resolution images with suppressed moiré and false structures can be obtained. However, in the case of sharpening based on the resolution performance of the optical system, the changes in the frequency characteristics of the image and the changes in moiré contrast due to the sharpening process increase in accordance with the shooting conditions of the optical system and the image height, making it difficult to train the system to obtain high-resolution images with suppressed moiré and false structures. In the case of sharpening based on different optical characteristics for each RGB channel, the color components (color moiré) caused by moiré also change, making training even more difficult.

[0061] To learn and process sharpening based on various optical properties in this way requires a machine learning model with a larger number of parameters. As a result, the learning speed becomes slower, and the computational load during image enlargement and the data size of the machine learning model increase. Also, the lower the performance of the optical system, the stronger the sharpening becomes, resulting in larger contrast changes and more pronounced moiré patterns at frequencies close to the optical system's cutoff frequency.

[0062] Furthermore, the sharpening process may also be a process that generates frequency components above the optical system's cutoff frequency. In this case, subject structures that should not exist based on the optical system's resolution performance will be reproduced. However, since subject structures above the cutoff frequency do not exist in the training dataset, the captured image will not match the resolution performance information, and the accuracy of generating the enlarged image will decrease.

[0063] If it is determined in step S203 that the captured image and the resolution performance information are consistent, the process proceeds to step S204. In step S204, the acquisition unit 123 acquires information on the weights of the first machine learning model, resolution performance information, and noise information from the storage unit 121. The resolution performance information is information indicating the resolution performance of the optical system when the captured image was taken. Here, the optical system in this embodiment includes the imaging optical system 141, the optical low-pass filter of the image sensor 142, and the pixel aperture. In order to acquire the resolution performance information and noise information, necessary information is acquired from the additional information of the captured image. The necessary information includes the type of imaging optical system 141, the state of the imaging optical system 141 at the time of imaging (focal length, F-number, focus distance), or the pixel pitch of the image sensor 142. The necessary information may also include the optical low-pass filter, the ISO sensitivity at the time of imaging (noise strength), whether or not the captured image is denoised and the denoising parameters, and the trimming position (position of the optical axis of the imaging optical system 141 relative to the captured image after trimming).

[0064] Resolution performance information (a 2-channel map in this embodiment) is generated from the acquired information and a data table regarding the resolution performance of the imaging optical system 141 stored in the memory unit 121. The memory unit 121 stores information regarding the resolution performance corresponding to the type, state, image height, and azimuth sampling point of the imaging optical system 141 as a data table. Resolution performance information corresponding to the captured image is generated from this data table by interpolation or other means. The resolution performance information in this embodiment is the same as during training, with a 2D map having the same number of pixels as the captured image, and the first channel of each pixel stores a value representing the horizontal resolution performance, and the second channel stores a value representing the vertical resolution performance. The value is the minimum frequency at which the MTF in the corresponding direction falls below a threshold (0.5), normalized by the sampling frequency of the image sensor 142 (the reciprocal of the pixel pitch).

[0065] The MTF is the MTF for white blur, combining the effects of the imaging optical system 141, the optical low-pass filter and pixel aperture of the image sensor 142, as in training. If the resolution performance of the captured image does not change (the type and state of the imaging optical system 141 and the image sensor 142 are fixed), the resolution performance information of the map state can be stored in the storage unit 121 and simply recalled. The noise information is also a map with the same number of pixels in two dimensions as the captured image, with the first channel being the strength of the noise generated during imaging and the second channel being the denoising parameters applied to the captured image.

[0066] On the other hand, if it is determined in step S203 that the captured image and the resolution performance information do not match, the process proceeds to step S205. In step S205, the acquisition unit 123 acquires the generator weight information and noise information from the storage unit 121. The difference from step S204 is that resolution performance information is not acquired, and the weight information acquired is also different. Because it was determined that the captured image and the resolution performance information that could be acquired in step S204 do not match, an enlarged image is generated in the following step S206 without using the resolution performance information. Therefore, in step S205, weight information for a second machine learning model, which is different from the first machine learning model, is acquired. Other processing in step S205 is the same as in step S204.

[0067] Next, in step S206, the image enlargement unit 124 generates an enlarged image using the captured image and the information acquired in step S204 or step S205. The enlarged image is an image in which the sampling pitch is halved (the number of pixels is quadrupled) compared to the captured image. Next, in step S207, the communication unit 122 transmits the enlarged image to the control device 103.

[0068] Next, in step S208, the communication unit 132 of the control device 103 acquires the enlarged image. The acquired enlarged image is stored in the storage unit 131 or displayed in the display unit 133. Alternatively, it may be stored in another storage device connected via wired or wireless connection from the control device 103 or the image enlargement device 102.

[0069] In this embodiment, only one image is captured, but the invention is not limited to this. An enlarged image may be generated from multiple images that are sub-pixel-shifted and from resolution performance information.

[0070] Next, with reference to Figure 9, the generation of an enlarged image as a modification of this embodiment will be described. Figure 9 is a flowchart of the generation of an enlarged image in this modification. In this embodiment, it is necessary to train a second machine learning model separately from the first machine learning model. The second machine learning model has fewer parameters than the first machine learning model and therefore has a faster processing speed, but it requires separate training and storage of weight information. Therefore, in this modification, if it is determined that the captured image and the resolution performance of the optical system do not match, instead of not using the second machine learning model, a second resolution performance information different from the first resolution performance information is used. The second resolution performance information is used in common for multiple input images corresponding to different first resolution performance information, for example.

[0071] Steps S301 to S303 in Figure 9 are the same as steps S201 to S203 in Figure 8, respectively. Also, steps S304 and S305 in Figure 9, when a match is determined in step S303, are the same as steps S204 and S206 in Figure 8, when a match is determined in step S203, respectively. Furthermore, steps S306 and S307 in Figure 9 are the same as steps S207 and S208 in Figure 8, respectively.

[0072] In step S304, if it is determined in step S303 that there is no match, the weight information and noise information are obtained in the same way as when it is determined that there is a match, but the resolution performance information is second resolution performance information, which is different from the first resolution performance information. If the change in frequency characteristics due to the sharpening process applied to the captured image can be determined based on the image processing information, the change in frequency characteristics due to the sharpening process can be applied to the resolution performance information of the optical system, thereby obtaining second resolution performance information that matches the captured image. In this case, in step S305, if it is determined in step S303 that there is a match, the desired effect can be achieved by inputting the second resolution performance information instead of the first resolution performance information.

[0073] If resolution performance information from the sharpening process cannot be obtained, pseudo-resolution performance information may be obtained as second resolution performance information. In this case, a map is obtained that holds the maximum possible value for resolution performance information for all pixels and is used as second resolution performance information. In this case, the second resolution performance information does not depend on the first resolution performance information.

[0074] Here, it is preferable that the second resolution performance information takes a value that indicates higher performance than the first resolution performance information. The first reason is that, in the captured image, sharpening processing makes both the components below the Nyquist frequency and the moiré components high contrast, so it is considered to correspond to the frequency characteristics of an image acquired with an optical system that has higher resolution performance than the actual one. The second reason is that it is possible to generate a more accurate image by misreporting a higher resolution performance than the actual one, rather than misreporting a lower resolution performance.

[0075] If the resolution performance is incorrectly assumed to be higher than the actual performance, a false structure may be generated because the subject and moiré patterns are misrecognized as being mixed in a wider frequency band than intended. However, compared to generating an enlarged image without inputting resolution performance information, the accuracy remains high in the frequency band where moiré patterns are not expected to occur even with the incorrect resolution performance, and the same level of accuracy can be achieved even in the frequency band where moiré patterns are expected to occur. This is because generating an enlarged image in a situation where moiré patterns and the subject are mixed is the same condition as when resolution performance information is not input.

[0076] On the other hand, if the resolution performance is incorrectly set lower than it actually is, the image will always be judged as the subject at frequencies where moiré is not expected to occur at the incorrect resolution performance, inevitably resulting in the persistence of moiré. From the above discussion, it is clear that setting the resolution performance higher than it actually is will result in better accuracy of the generated magnified image. In particular, when the resolution performance is maximized, the machine learning model is made to recognize that moiré can occur regardless of the resolution performance of the optical system, thus achieving accuracy equivalent to that of a machine learning model that does not use resolution performance information.

[0077] Furthermore, in this embodiment, as described above, blurring due to aberrations and diffraction of the imaging optical system 141, and blurring due to the optical low-pass filter of the image sensor 142 are not added when generating high-resolution images. In other words, the first machine learning model is trained to correct the aforementioned blurring along with image enlargement. Therefore, if resolution performance information showing low performance is used, further sharpening based on the resolution performance information will be applied to the captured image to which sharpening processing has been applied, resulting in overcorrection.

[0078] On the other hand, by setting the resolution performance information to its maximum value, it is possible to generate an enlarged image without excessive blur correction. If the sharpening process applied to the captured image is not based on the resolution performance of the optical system, the purpose is to adjust the overall sharpness of the image, so it may overlap with the blur correction originating from the optical system. On the other hand, if the sharpening process applied to the captured image is based on the resolution performance of the optical system, it is best to use resolution performance information that does not depend on the resolution performance information of the optical system and shows higher performance than the resolution performance information of the optical system, in order to avoid overlapping corrections. Here, we have described blur correction based on the resolution performance of the optical system, but this does not presuppose image enlargement.

[0079] In this embodiment, information on the sharpening process applied to the captured image was obtained from the supplementary information and used as image processing information, but this is not limited to this. For example, cropping information of the captured image may be obtained as image processing information, and the consistency between the resolution performance of the optical system and the captured image may be determined based on that. The sharpening process causes the consistency between the resolution performance of the optical system and the captured image to be lost by changing the resolution performance itself indicated by the resolution performance information.

[0080] On the other hand, cropping indirectly disrupts the consistency between the resolution performance of the optical system and the pixel positions in the captured image by altering the correspondence between the optical system's resolution performance and the captured image's pixel positions. As mentioned above, resolution performance information is a map that shows the resolution performance of the corresponding pixels in the low-resolution image, where the number of pixels is the same as that of the low-resolution image. The resolution performance of the optical system depends on the position in the captured image, but cropping the captured image disrupts the consistency between the position on the screen and the optical axis and image height of the optical system. Cropping a position in the captured image that is offset from the center of the screen causes the optical axis to no longer be at the center, and extracting a partial region disrupts the correspondence between the image height for the optical system and the position on the image.

[0081] Therefore, if the image processing information contains information indicating that the captured image is a cropped image, the determination unit 125 determines that the resolution performance of the optical system and the captured image do not match. If there is no direct cropping information, the number of pixels in the captured image and information identifying the imaging device may be used together as image processing information. By identifying the imaging device, the number of pixels of the imaging device before cropping, stored in the storage unit 121 of the image magnification device 102, can be obtained. If there is a discrepancy when comparing the number of pixels with the actual number of pixels in the input captured image, it can be determined that the resolution performance of the optical system and the captured image do not match. If information on the cropping position can be obtained from the captured image, the resolution performance information for the corresponding pixel position can be obtained. In this case, since resolution performance information that matches the captured image can be obtained, it may be determined that they match.

[0082] Furthermore, image processing information that changes the number of pixels, such as image enlargement, may also be used. This is because changing the number of pixels alters the correspondence between the resolution performance of the optical system and the pixel positions in the captured image, similar to cropping. Also, enlarging or reducing the image changes the frequency characteristics and Nyquist frequency of the image.

[0083] In this embodiment, the control device 103 acquires captured images directly from the imaging device 104, but it may also acquire captured images via an external image processing device that has acquired the captured images from the imaging device 104.

[0084] In this embodiment, the captured image (input image) may also be a developed image. It is preferable that the developed image retains the processing performed during development as image processing information. Image processing applied during development can cause inconsistencies between the optical system's resolution performance information and the captured image. Therefore, it is preferable to acquire development information indicating development conditions as image processing information. Also, since developed images are stored in a commonly used data format, image processing is often applied after development. Therefore, if the captured image is a developed image, the accuracy of the output image can be improved by the present invention.

[0085] In this embodiment, a RAW image may also be used as the input image. When upsampling captured image data that has not been subjected to image processing that would cause inconsistency with the resolution performance of the optical system, the resolution performance of the image data and the optical system may be used. On the other hand, when upsampling a RAW image that has been sharpened as the input image, or when upsampling a RAW image that has already been upsampled as the input image, the resolution performance information of the input image and the optical system will not be consistent. When applying multiple image processing steps to a RAW image in this way, it is preferable to obtain information on whether or not image processing has been applied to the input image before it is upsampled, and to upsample based on that information. In this case, the image processing information may not be additional information but information temporarily stored during a series of processes. The image processing information applied to the input image may also be obtained before the image processing is actually applied. In this case, the image processing information may be obtained in advance.

[0086] Furthermore, image processing information such as the name of the editing software or editing device used to edit the image may be obtained. Some editing software does not record information about the image processing performed as additional information, so by obtaining the name of the editing software as image processing information, it is possible to obtain the possibility that the resolution performance of the optical system and the captured image may not match due to the image processing performed by the editing software. Also, some editing software allows the modification (editing) of additional information of the input image, so it is possible to obtain the possibility that the resolution performance of the optical system may not match the image captured image and the resolution performance of the optical system expected from the additional information due to the editing of the additional information. In addition, it is possible to obtain the possibility that the resolution performance of the optical system and the captured image may not match due to image processing performed by software other than the editing software in question. When the possibility of a mismatch between the resolution performance of the optical system and the captured image is obtained in this way, it is preferable to determine that the resolution performance of the optical system and the captured image do not match in order to avoid incorrect processing.

[0087] As described above, the image enlargement unit 124 varies the method of generating the enlarged image (output image) according to the determination result by the determination unit 125. Therefore, according to this embodiment, it is possible to determine whether the resolution performance information and the input image are consistent and generate an output image that effectively suppresses moiré and false structures. [Examples]

[0088] Next, with reference to Figures 10 and 11, the image processing system in Embodiment 2 of the present invention will be described. In this embodiment, demosaicing is performed as an upsample, but it can be similarly applied to other upsamples. Furthermore, a machine learning model is used for demosaicing.

[0089] Figure 10 is a block diagram of the image processing system 300. Figure 11 is an external view of the image processing system 300. The image processing system 300 includes a training device 301 and an imaging device 302. The imaging device 302 includes an imaging optical system 321, an image sensor 322, an image processing unit (image processing device) 323, a storage unit 324, a communication unit 325, and a display unit 326. The image processing unit 323 includes an acquisition unit (acquisition means) 323a, a demosaicing unit (generation means) 323b, and a determination unit (determination means) 323c.

[0090] The imaging optical system 321 forms a subject image from the light in the subject space, and the image sensor 322 captures the subject image to generate an image. The image is an image in which RGB pixels are arranged in a Bayer array. The image is acquired from a live view of the subject space before imaging or when the shutter release is pressed by the user, and after development processing is performed by the image processing unit 323, it is stored in the storage unit 324 or displayed on the display unit 326. During the development processing of the image, demosaicing is performed using a machine learning model, and a demosaiced image (output image) is generated. The machine learning model is pre-trained by the training device 301, and information on the trained weights is acquired via the communication unit 325. In demosaicing of the image, resolution performance information, which is information regarding the resolution performance of the imaging optical system 321, is used. This process will be explained in detail.

[0091] First, with reference to Figure 12, the training of the machine learning model performed in the training device 301 will be explained. Figure 12 is a flowchart of the training of the machine learning model. Each step in Figure 12 is performed by the storage unit 311, acquisition unit 312, calculation unit 313, or update unit 314 of the training device 301.

[0092] First, in step S401, the acquisition unit 312 acquires one or more mosaic images and ground truth images from the storage unit 311. The mosaic image is a Bayer image with the same RGB (red, green, blue) colors as the captured image. Figures 13(A) and (B) show the relationship between the color filter array (Bayer array) and the Nyquist frequency, with Figure 13(A) showing the Bayer array and Figure 13(B) showing the Nyquist frequency of each color in the Bayer array. In Figure 13(A), G (green) has a sampling pitch that is the square root of 2 times the pixel pitch in the diagonal direction and has a Nyquist frequency of 402. R (red) and B (blue) have a sampling pitch that is twice the pixel pitch in the horizontal and vertical directions and have a Nyquist frequency of 403.

[0093] The ground truth image is a 2D image with the same number of pixels as the mosaic image and has three RGB channels. The ground truth image has a sampling pitch equal to the pixel pitch for each of the RGB channels, and all colors have a Nyquist frequency of 401. The ground truth image is generated using an image captured by a computer graphics (CG) or a three-chip image sensor as the source image. Alternatively, an image captured with a Bayer array may be scaled down to generate an image with RGB signal values ​​for each pixel, and this can be used as the source image. At least a portion of the source image has frequency components of Nyquist frequencies of 402, 403 or higher for each color in the Bayer array. The ground truth image is generated by adding blur due to aberrations and diffraction generated in the imaging optical system 321, and blur due to the optical low-pass filter and pixel aperture of the image sensor 322, to the source image. The mosaic image can be generated by sampling the ground truth image with a Bayer array. Multiple mosaic images with different blurs and the ground truth image are generated, and the blur of the actual captured image is made to fit within the range of these blurs. Note that the mosaic image is not limited to a Bayer array.

[0094] Next, in step S402 of Figure 12, the calculation unit 313 acquires resolution performance information. In this embodiment, resolution performance information is generated for each RGB channel. For each RGB channel, the resolution performance is defined as the minimum frequency at which the MTF in the horizontal and vertical directions falls below a threshold, normalized by the Nyquist frequency of each RGB channel, similar to this embodiment.

[0095] Next, in step S403, the calculation unit 313 inputs the mosaic image and resolution performance information to the machine learning model to generate a demosaiced image. In this embodiment, the demosaiced image is generated using the configuration shown in Figure 14. Figure 14 is a diagram showing the flow of demosaiced image generation. The mosaic image 501 is rearranged into four channels, R, G1, G2, and B, to generate an RGGB image 502. The RGGB image 502 and the resolution performance information 503, which is an 8 (4×2) channel map showing the resolution performance of each pixel of each RGGB color, are concatenated in the channel direction and input to the machine learning model 511 to generate a demosaiced image 504. The machine learning model 511 is similar to the configuration shown in Figure 6, but is not limited to this. Alternatively, the mosaic image 501 may be input to the machine learning model in its Bayer array form without being rearranged into four channels.

[0096] Next, in step S404 of Figure 12, the update unit 314 updates the weights of the machine learning model 511 based on the error between the ground truth image and the demosaiced image 504. Then, in step S405, the update unit 314 determines whether or not the training of the machine learning model 511 is complete. If it is determined that the training is not complete, the process returns to step S401. On the other hand, if it is determined that the training is complete, the training is terminated and the weight information is stored in the storage unit 311.

[0097] Next, with reference to Figure 15, the demosaicing of captured images performed by the image processing unit 323 will be explained. Figure 15 is a flowchart of the generation of a demosaiced image. Each step in Figure 15 is performed by the acquisition unit 323a, the demosaicing unit 323b, or the determination unit 323c of the image processing unit 323.

[0098] First, in step S501, the acquisition unit 323a acquires the captured image and image processing information. The captured image is a Bayer array image. Next, in step S502, similar to step S203, the determination unit 323c determines whether the resolution performance of the optical system (first resolution performance information) and the captured image acquired in step S501 are consistent, based on the image processing information acquired in step S501.

[0099] Next, in step S503, the acquisition unit 323a acquires information on the weights of the machine learning model (first machine learning model) from the storage unit 324. Also, if it is determined in step S502 that the resolution performance of the optical system matches the captured image, the optical system's resolution performance information (first resolution performance information) is acquired from the storage unit 324 based on the state of the imaging optical system at the time of imaging. If it is determined in step S502 that the optical system's resolution performance does not match the captured image, a map (second resolution performance information) is acquired that holds the maximum value of the resolution performance information for all pixels. Note that if the image processing information is acquired before step S502, the order in which the captured image and weight information are acquired is not limited to that described, as long as they are acquired before step S504. Next, in step S504, the demosaicing unit 323b generates a demosaiced image (output image) from the captured image and resolution performance information using the configuration shown in Figure 14. The demosaiced image is an image obtained by demosaicing the captured image.

[0100] The image processing unit 323 may perform other processing such as denoising and gamma correction as needed. Furthermore, image enlargement in this embodiment may be performed simultaneously with demosaicing. Also, in this embodiment, similar to the modification in Embodiment 1, if it is determined that the resolution performance information and the input image do not match, the machine learning model uses the second resolution performance information instead of the first resolution performance information, but it is not limited to this. Similar to Embodiment 1, a second machine learning model that does not take resolution performance information as input may be used instead of the first machine learning model.

[0101] As described above, the demosaicing unit 323b varies the method of generating the output image according to the determination result by the determination unit 323c. Therefore, according to this embodiment, it is possible to determine whether the resolution performance information and the input image are consistent and generate an output image that effectively suppresses moiré and false structures. [Examples]

[0102] Next, the image processing system in Embodiment 3 of the present invention will be described. The present invention is not limited to image upsampling, but can also be applied to image processing using resolution performance information of other optical systems. More specifically, any image processing that corrects image components caused by the resolution characteristics of the optical system in the input image based on the resolution characteristics of the optical system is acceptable. In Embodiments 1 and 2, moiré patterns are mainly corrected as image components caused by the resolution characteristics of the optical system. On the other hand, in this embodiment, a machine learning model is used to correct blur components caused by the resolution characteristics of the optical system.

[0103] The image processing system in this embodiment is the same as in Embodiment 2. During the development process of the captured image, blur correction is performed using a machine learning model, and a corrected image (output image) is generated. In the aberration correction of the captured image, resolution performance information, which is information about the resolution performance of the imaging optical system, is used. This process will be explained below.

[0104] In Example 1, when generating high-resolution images, blurring due to aberrations and diffraction of the imaging optical system, as well as blurring due to the optical low-pass filter of the image sensor, is not added. When generating low-resolution images, these blurs are added and the images are generated by downsampling. By learning to estimate high-resolution images by inputting these low-resolution images and resolution performance information, the system is trained to simultaneously perform image enlargement and optical system blur correction. On the other hand, in this example, a blurred image is generated by using the high-resolution image from Example 1 as the ground truth image without blur, and excluding downsampling from the low-resolution image generation process. By inputting the blurred image and resolution performance information into the machine learning model and training it to output the ground truth image, it is possible to obtain information on the weights of the machine learning model that corrects blur caused by the optical system. The resolution performance information and machine learning model are the same as in Example 1, but are not limited to these.

[0105] Referring to Figure 16, the blur correction of the captured image, performed by the image processing unit 323, will be explained. Figure 16 is a flowchart for generating a blur-corrected image. Steps S601 to S604 in Figure 16 are basically equivalent to steps S501 to S504 in Figure 15, respectively, but instead of generating a demosaicing image from the captured image and resolution performance information, a blur-corrected image is generated.

[0106] The image processing unit 323 may perform other processing such as denoising and gamma correction as needed. Furthermore, image enlargement from Example 1 or demosaicing from Example 2 may be performed simultaneously with blur correction.

[0107] In blur correction, if the resolution performance of the optical system and the captured image do not match, overcorrection can occur, resulting in overshoot or undershoot, or insufficient correction can leave blur, thus reducing the accuracy of the output image. Therefore, by determining whether the resolution performance information and the input image match and changing the processing accordingly, a more accurate blur-corrected image (output image) can be generated. In this embodiment, if it is determined that they do not match, the resolution performance information is set to its maximum value, thereby avoiding overcorrection of blur. In this embodiment, matching refers to the frequency characteristics determined by the optical low-pass filter of the imaging optical system and the image sensor matching the frequency characteristics of the captured image. In particular, matching refers to the frequency characteristics of the image used during training of the machine learning model matching with the frequency characteristics of the captured image, and not matching refers to the frequency characteristics not matching. However, if the blur correction process corrects only a portion of the blur (for example, blur generated in the imaging optical system), only the corresponding portion of the blur is considered when determining whether or not they match.

[0108] In this embodiment, similar to the modification in Example 1, if it is determined that the resolution performance information and the input image do not match, the machine learning model uses the second resolution performance information instead of the first resolution performance information without modification, but it is not limited to this. Similar to Example 1, a second machine learning model that does not take resolution performance information as input may be used instead of the first machine learning model.

[0109] With the above configuration, according to this embodiment, it is possible to determine whether the resolution performance information and the input image are consistent, thereby suppressing overcorrection and generating an output image with good blur correction.

[0110] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. The image processing device in the present invention can be any device having the image processing function of the present invention, and can be implemented in the form of an imaging device or a PC.

[0111] According to each embodiment, it is possible to provide an image processing method, an image processing device, and a program that can improve the accuracy of image processing using machine learning related to the resolution performance of an imaging optical system.

[0112] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its essence. [Explanation of Symbols]

[0113] 102 Image Magnification Device (Image Processing Device) 123 Acquisition unit (acquisition means) 124 Image enlargement unit (generation means) 125 Judgment unit (judgment means)

Claims

1. an acquisition step of acquiring an input image obtained by imaging using an optical system and first information related to image processing applied to the input image; a determination step of determining whether or not first resolution performance information indicating a resolution performance of the optical system matches the input image based on the first information; generating an output image based on the input image and the first information; An image processing method, wherein in the generating step, a method for generating the output image is changed depending on a result of the determination in the determining step.

2. In the generating step, generating the output image by inputting the input image and the first resolution performance information into a first machine learning model when it is determined in the determination step that the first resolution performance information and the input image are consistent; The image processing method according to claim 1, characterized in that when it is determined that the first resolution performance information and the input image are not consistent, the output image is generated by inputting the input image to a second machine learning model different from the first machine learning model.

3. In the generating step, generating the output image by inputting the input image and the first resolution performance information into a first machine learning model when it is determined in the determination step that the first resolution performance information and the input image are consistent; The image processing method described in claim 1, characterized in that when it is determined that the first resolution performance information and the input image are not consistent, the output image is generated by inputting second resolution performance information indicating a resolution performance different from the resolution performance of the optical system into the first machine learning model.

4. 4. The image processing method according to claim 3, wherein the second resolution performance information is used in common for a plurality of input images corresponding to different first resolution performance information.

5. 5. The image processing method according to claim 3, wherein the resolution performance indicated by the second resolution performance information is higher than the resolution performance indicated by the first resolution performance information.

6. 6. The image processing method according to claim 3, wherein the second resolution performance information is obtained using information related to a sharpening process applied to the input image.

7. 7. The image processing method according to claim 6, wherein the sharpening process is based on the resolution performance of the optical system.

8. An image processing method according to any one of claims 1 to 7, characterized in that in the generation process, if it is determined in the judgment process that the first resolution performance information and the input image are not consistent, the first resolution performance information is not input.

9. 9. The image processing method according to claim 1, wherein the matching between the first resolution performance information and the input image is a matching between a frequency characteristic based on the resolution performance of the optical system and a frequency characteristic of the input image.

10. 9. The image processing method according to claim 1, wherein the matching between the first resolution performance information and the input image is matching between a moire based on the resolution performance of the optical system and a moire of the input image.

11. 11. The image processing method according to claim 1, wherein the first information is acquired based on additional information of the input image.

12. 12. The image processing method according to claim 1, wherein the first information is acquired based on a number of pixels of the input image.

13. 13. The image processing method according to claim 1, wherein the input image is a developed image.

14. 14. The image processing method according to claim 1, wherein the first information is development information indicating development conditions of the input image.

15. 15. The image processing method according to claim 1, wherein the first information is information indicating a possibility that additional information of the input image has been changed.

16. 16. The image processing method according to claim 1, wherein the output image is an image in which image components contained in the input image and caused by the resolution characteristics of the optical system have been corrected.

17. 17. The image processing method according to claim 1, wherein the output image is an image obtained by enlarging or demosaicing a captured image, or an image in which blurring generated in the optical system in the captured image has been corrected.

18. 18. The image processing method according to claim 1, wherein the first resolution performance information is information based on a magnitude of blur occurring in the optical system.

19. 19. The image processing method according to claim 1, wherein the first resolution performance information is information based on a spread of a point spread function of the optical system or a modulation transfer function of the optical system.

20. 20. The image processing method according to claim 1, wherein the first resolution performance information is information based on a position of a pixel of the input image.

21. 21. The image processing method according to claim 1, wherein the first resolution performance information is a map in which values ​​are arranged in a size based on the number of pixels of the input image.

22. an acquisition means for acquiring an input image obtained by imaging using an optical system and first information related to image processing applied to the input image; a determination means for determining whether or not first resolution performance information indicating a resolution performance of the optical system matches the input image based on the first information; generating means for generating an output image based on the input image and the first information; The image processing device according to claim 1, wherein the generating means changes a method for generating the output image depending on a result of the determination by the determining means.

23. An image processing system comprising the image processing device according to claim 22 and a control device capable of communicating with the image processing device, the control device has a transmission means for transmitting a request for causing the image processing device to execute processing on the input image; The image processing system according to the present invention is characterized in that the image processing device has a receiving means for receiving the request, and executes processing on the input image in response to the request.

24. A program causing a computer to execute the image processing method according to any one of claims 1 to 21.

Images