Image processing method, image processing system, program, production method of trained machine learning model, processor and image processing system

Abstract

To improve an accuracy of processing for reducing a sampling pitch of a picked-up image.SOLUTION: An image processing method includes: a step of acquiring a picked-up image and resolution performance information being information showing a resolution performance of an optical instrument used for imaging the picked-up image; and a step of generating an output image with a reduced sampling pitch of the picked-up image according to the picked-up image and the resolution performance information.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to image processing for reducing the sampling pitch of a captured image. [Background technology]

[0002] Patent Document 1 discloses a method for generating a high-resolution enlarged image by enlarging a low-resolution image to the same number of pixels as a high-resolution image by bicubic interpolation and then inputting the image to a trained machine learning model. By using a trained machine learning model for image enlargement processing, it is possible to achieve image enlargement with higher accuracy than with general methods such as bicubic interpolation. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Application Publication No. 2018 / 0075581 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the method disclosed in Patent Document 1 has a problem that false structures (artifacts) that do not actually exist appear in the enlarged image, or moire that existed in the low-pixel image remains in the enlarged image. This problem also occurs in other image enlargement methods that do not use machine learning models (such as bicubic interpolation and sparse coding). This problem also occurs not only in image enlargement, but also in other processes that reduce the sampling pitch of images (such as demosaic).

[0005] SUMMARY OF THE PRESENT EMBODIMENT An object of the present invention is to improve the accuracy of processing for reducing the sampling pitch of a captured image. [Means for solving the problem]

[0006] The image processing method of the present invention is characterized by comprising the steps of acquiring an captured image and resolution performance information, which is information representing the resolution performance of an optical device used to capture the captured image, and generating an output image in which the sampling pitch of the captured image is reduced based on the captured image and the resolution performance information. Effect of the Invention

[0007] According to the present invention, it is possible to improve the accuracy of the process of reducing the sampling pitch of a captured image. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing the relationship between the modulation transfer function and the Nyquist frequency in the first and second embodiments. [Diagram 2] 1 is a block diagram of an image processing system according to a first embodiment. [Diagram 3] 1 is an external view of an image processing system according to a first embodiment. [Figure 4] 1 is a flowchart of training a machine learning model in Example 1. [Diagram 5] 4 is a diagram showing a flow of generating an enlarged image in the first embodiment. [Figure 6] FIG. 2 is a diagram illustrating a configuration of a machine learning model in the first and second embodiments. [Figure 7] 4 is a flowchart of generating an enlarged image in the first embodiment. [Figure 8] FIG. 11 is a block diagram of an image processing system according to a second embodiment. [Figure 9] FIG. 11 is an external view of an image processing system according to a second embodiment. [Figure 10] 11 is a flowchart of training a machine learning model in Example 2. [Figure 11] FIG. 11 is a diagram showing the relationship between the color filter arrangement and the Nyquist frequency in the second embodiment. [Figure 12] FIG. 11 is a diagram showing a flow of generating a demosaic image in the second embodiment. [Figure 13]11 is a flowchart of generation of a demosaic image in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In each drawing, the same members are given the same reference numerals, and duplicated explanations will be omitted.

[0010] Before describing the embodiments in detail, the gist of the present invention will be briefly described. In the present invention, in a process of reducing the sampling pitch of a captured image (hereinafter referred to as upsampling), resolution performance information, which is information on the resolution performance of an optical device used to capture the captured image, is used. This improves the accuracy of upsampling. To explain the reason, the issues with upsampling and the principles behind their occurrence will be described in detail below.

[0011] When the subject image formed by the optical system is converted into a captured image by the image sensor, sampling is performed by the pixels of the image sensor. Therefore, among the frequency components forming the subject image, components exceeding the Nyquist frequency of the image sensor are mixed with low-frequency components due to aliasing, resulting in moire. In upsampling of a captured image, the Nyquist frequency increases as the sampling pitch becomes smaller, so ideally it is desired to generate an image that is free of aliasing up to the increased Nyquist frequency. However, it is generally difficult to perform image processing that distinguishes whether a structure included in a captured image containing moire is a moire or an original structure of the subject.

[0012] In general upsampling, such as bilinear interpolation, moire remains even after upsampling a captured image. In contrast, in upsampling using a machine learning model, it is possible to estimate high frequencies from moire to some extent before aliasing occurs, so it is expected that some moire can be removed. However, as mentioned above, it is difficult to distinguish between moire and the structure of the subject, so even if a machine learning model is used, some moire may be mistakenly recognized as the subject and remain, and some subjects may be mistakenly recognized as moire, generating false structures.

[0013] Therefore, in the present invention, in upsampling a captured image, information on the resolution performance of the optical device used to capture the captured image is used, which will be further explained with reference to Figures 1(A) and 1(B).

[0014] 1(A) and (B) show frequency characteristics of the modulation transfer function (MTF) that represents the resolution performance of an optical device. The horizontal axis represents the spatial frequency in a certain direction, and the vertical axis represents the MTF. FIG. 1(A) shows a state in which the cutoff frequency 003 (in this specification, the cutoff frequency refers to a frequency above which the MTF becomes 0) of the optical device is equal to or lower than the Nyquist frequency 001. In this case, no moiré is present in the captured image. This is because even if the MTFs are arranged at a period of the sampling frequency 002, there is no area where the MTFs overlap each other. Therefore, when the resolution performance corresponds to FIG. 1(A), this fact can be given (input) to the algorithm, so that the algorithm can determine that it is not necessary to estimate the high-frequency components before the moiré occurs from the structure of the moiré. This makes it possible to suppress the occurrence of false structures in the image processing results.

[0015] FIG. 1B shows a state where the cutoff frequency 003 exceeds the Nyquist frequency 001. In this case, by providing this information to the algorithm, the algorithm can identify the frequency band where moiré may occur due to aliasing. In the example of FIG. 1B, moiré may occur in the band between the frequency 004, which is obtained by subtracting the cutoff frequency 003 from the sampling frequency 002, and the Nyquist frequency 001, and moiré will not occur in other bands. Therefore, by providing the resolution performance information to the algorithm, it is possible to suppress the occurrence of false structures. Therefore, it is possible to improve the accuracy of upsampling of captured images.

[0016] [Example 1] An image processing system according to a first embodiment of the present invention will be described. In the first embodiment, image enlargement (upscaling) is performed as upsampling, but other upsampling such as demosaicing can be similarly applied. Image enlargement includes an increase in sampling points for the entire captured image and an increase in sampling points for a partial region of the captured image (enlargement of a cropped image, digital zoom, etc.). In the first embodiment, a machine learning model is used for image enlargement, but other methods such as sparse coding can be similarly applied.

[0017] 2 and 3 are a block diagram and an external view, respectively, of the image processing system 100. The image processing system 100 has a training device 101, an image enlargement device 102, a control device 103, and an imaging device 104, which are connected to each other via a wired or wireless network. The control device 103 has a storage unit 131, a communication unit 132, and a display unit 133, and acquires a captured image from the imaging device 104 in accordance with a user's instruction, and transmits the captured image and a request to perform image enlargement to the image enlargement device 102 via the communication unit 132.

[0018] The imaging device 104 has an imaging optical system 141, an imaging element 142, an image processing unit 143, and a storage unit 144. The imaging optical system 141 forms an image of the subject from light in the subject space, and the imaging element 142, in which a plurality of pixels are arranged, converts the formed image into a captured image. At this time, aliasing occurs in frequency components of the subject image that are higher than the Nyquist frequency of the imaging element 142. As a result, moire may occur in the captured image. The image processing unit 143 executes predetermined processing (such as correction of pixel defects and development) on the captured image as necessary. The captured image or the captured image processed by the image processing unit 143 is stored in the storage unit 144.

[0019] The control device 103 acquires a captured image via communication or a storage medium. The captured image to be acquired may be the entire captured image or only a part of the captured image (a partial region).

[0020] The image enlargement device 102 has a storage unit 121, a communication unit (acquisition means) 122, an acquisition unit 123, and an image enlargement unit (generation means) 124, and enlarges a captured image using a trained machine learning model to generate an enlarged image (output image). At this time, it uses resolution performance information, which is information on the resolution performance of an optical device (such as the imaging optical system 141) used to capture the captured image. Details of this process will be described later. The image enlargement device 102 acquires information on the weight of the trained machine learning model from the training device 101 and stores it in the storage unit 121.

[0021] The training device 101 includes a storage unit 111, an acquisition unit 112, a calculation unit 113, and an update unit 114, and trains a machine learning model in advance using a data set. Information on the weights of the machine learning model generated by training is stored in the storage unit 111.

[0022] When the image enlargement device 102 generates an enlarged image, the control device 103 acquires the enlarged image from the image enlargement device 102 and presents it to the user via the display unit 133 .

[0023] Here, a method for training (determining weights) a machine learning model (a method for manufacturing a trained model) executed by the training device 101 will be described with reference to the flowchart in FIG. 4. In the first embodiment, a machine learning model using a GAN (generative adversarial network) is trained, but the present invention is not limited to this. In addition, machine learning models include, for example, neural networks, genetic programming, Bayesian networks, etc. Neural networks include CNNs (Convolutional Neural Networks), GANs (Generative Adversarial Networks), RNNs (Recurrent Neural Networks), etc.

[0024] Each step in FIG. 4 is performed by a training device 101 .

[0025] In step S101, the acquisition unit 112 acquires one or more pairs of a high pixel image and a low pixel image from the storage unit 111. A data set including a plurality of high pixel images and low pixel images is stored in the storage unit 111. That is, as will be described in detail later, the acquisition unit 112 functions as a data acquisition means for acquiring a first image (low pixel image) and a second image (high pixel image) having a smaller sampling pitch than the first image.

[0026] The low-pixel image is an image input to the machine learning model (generator in the first embodiment) during training of the machine learning model, and is an image with a relatively low number of pixels (an image with a large sampling pitch). The more accurately the low-pixel image reproduces the properties of the captured image that is actually enlarged using the trained machine learning model, the higher the accuracy of the trained machine learning model. Examples of the properties of the captured image include resolution performance, color expression, and noise characteristics. For example, if the captured image is an image expressed in RGB, while the low-pixel image is an image expressed in monochrome or YUV, the color expressions do not match, so the accuracy of the task (the accuracy of upsampling) may decrease. Although the properties of the captured image that are important vary depending on the type of task by the machine learning model, in the task of enlarging an image, as described above, information on the frequency band in which moire occurs is important, so the resolution performance is particularly important. Therefore, it is desirable that the resolution performance of the captured image that is actually enlarged using the trained machine learning model (the resolution performance of the optical device used to obtain the captured image that is actually enlarged) falls within the range of the resolution performance of the multiple low-pixel images used for training.

[0027] The high-pixel image is an image that is the ground truth in training the machine learning model. The high-pixel image is an image that captures the same scene as the corresponding low-pixel image, and has a smaller sampling pitch (i.e., a larger number of pixels) than the low-pixel image. In the first embodiment, the sampling pitch of the high-pixel image is half that of the low-pixel image. Therefore, the machine learning model expands the number of pixels of the input image by four times (twice vertically and horizontally). However, the present invention is not limited to this. It is desirable that the multiple low-pixel images and high-pixel images used for training include various subjects (edges, textures, gradations, flat parts, etc. with different orientations and strengths) so that the machine learning model can handle captured images of various subjects. In addition, at least a part of the high-pixel image has frequency components equal to or higher than the Nyquist frequency of the low-pixel image.

[0028] In the first embodiment, the high-pixel image and the low-pixel image are generated from an original image by imaging simulation. However, the present invention is not limited to this, and the high-pixel image and the low-pixel image may be generated using an image obtained by imaging simulation using three-dimensional data of the subject space instead of the original image. Also, the high-pixel image and the low-pixel image may be generated by actually capturing images using imaging elements with different pixel pitches.

[0029] The original image is an undeveloped RAW image (an image in which the light intensity and the signal value have a linear relationship), has a sampling pitch equal to or lower than that of the high-resolution image, and at least a part of the image has a frequency component equal to or higher than the Nyquist frequency of the low-resolution image. The low-resolution image is generated by reproducing the same imaging process as that of the captured image that is actually enlarged by the trained machine learning model, using the original image as the subject. Specifically, the original image is given blurring due to aberration and diffraction occurring in the imaging optical system 141, and blurring due to the optical low-pass filter and pixel aperture of the image sensor 142, etc. If there are multiple types and states of optical equipment used to obtain the captured image that is enlarged by the trained machine learning model, and these types of optical equipment can affect the captured image in different ways, it is preferable to include low-resolution images to which these multiple blurrings have been added in the data set. The blurring can change depending on the position of each pixel of the image sensor 142 (image height and azimuth relative to the optical axis of the imaging optical system 141), and also depending on the state of the imaging optical system 141 when the imaging optical system 141 can take various states (for example, focal length, F-number, focus distance). In addition, if the imaging device 104 is a lens-interchangeable camera and multiple types of optical systems can be used as the imaging optical system 141, the blur also changes depending on the type of optical system. Furthermore, if there are different types of imaging devices 104 and the pixel pitch or optical low-pass filter is different, the blur also changes.

[0030] The blur to be added to the original image may be the blur generated in the imaging optical system 141 or the image sensor 142 itself, or may be an approximation of that blur. For example, the PSF (point spread function) of the blur generated in the imaging optical system 141 or the image sensor 142 may be approximated by a two-dimensional Gaussian distribution function, a mixture of multiple two-dimensional Gaussian distribution functions, a Zernike polynomial, or the like. Also, the OTF (optical transfer function) or MTF (modulation distribution function) may be approximated by a two-dimensional Gaussian distribution function, a mixture of multiple two-dimensional Gaussian distribution functions, a Legendre polynomial, or the like. In this case, blur may be added to the original image using the approximated PSF, OTF, MTF, or the like.

[0031] After blurring the original image, the image sensor 142 downsamples the image at a sampling pitch. Since the image sensor 142 has RGB (Red, Green, Blue) color filters arranged in a Bayer array, it is preferable to sample the low-pixel image so that it is arranged in a Bayer array. However, the present invention is not limited to this, and the image sensor 142 may be monochrome, honeycomb arrayed, or three-plate type. When there are multiple types of image sensor 142 used to obtain the captured image to be enlarged by the trained machine learning model, and the pixel pitch of the captured image can change, low-pixel images can be generated for multiple sampling pitches to cover the range of change. In addition, in the first embodiment, noise generated by the image sensor 142 is also added to the low-pixel image. This is because if noise is not added to the low-pixel image (noise is not considered in the training of the machine learning model), not only the subject but also the noise may be considered as the structure of the subject when the captured image is enlarged, and may be emphasized. When there is a range of noise intensity generated in the captured image (e.g., there are multiple possible ISO sensitivities at the time of capture), it is preferable to include multiple low-pixel images with noise intensity changed within the possible range in the data set.

[0032] The high-pixel image is generated by applying blurring due to a pixel aperture half the pixel pitch of the low-pixel image to the original image, downsampling at half the sampling pitch of the low-pixel image, and Bayer-coding. Note that when the sampling pitch of the original image and the high-pixel image are equal, the original image may be used as the high-pixel image as it is. In the first embodiment, blurring due to aberration and diffraction of the imaging optical system 141 and blurring due to the optical low-pass filter of the image sensor 142 are not applied when the high-pixel image is generated. As a result, the machine learning model is trained to correct the above-mentioned blurring as well as enlarging the image. However, the present invention is not limited to this, and the high-pixel image may also be given a blurring similar to that of the low-pixel image, or the blurring given to the low-pixel image may be reduced and applied to the high-pixel image. Also, in the first embodiment, noise is not added when the high-pixel image is generated. As a result, the machine learning model is trained to perform denoising as the image is enlarged. However, the present invention is not limited to this, and noise of the same intensity as or different from the noise given to the low-pixel image may be added. When adding noise to a high-resolution image, it is desirable to add noise that is correlated with the noise in the low-resolution image (for example, noise generated by the same random numbers as the noise added to the low-resolution image). If the noises are uncorrelated, the effect of the noise in the high-resolution image is averaged by training with multiple images in the data set, and the desired effect may not be obtained.

[0033] In the first embodiment, the image enlargement is performed on the developed captured image. Therefore, the low pixel image and the high pixel image must also be developed images. Therefore, the low pixel image and the high pixel image in the Bayer state are subjected to the same development process as the captured image and stored in the data set. However, the invention is not limited to this, and the low pixel image and the high pixel image may be configured as RAW, and the captured image may also be enlarged in the RAW state. In addition, when compression noise such as JPEG encoding occurs in the captured image, the same compression noise may be added to the low pixel image. In this way, the machine learning model is trained to perform removal of compression noise as well as image enlargement.

[0034] In step S102, the acquisition unit 112 acquires resolution performance information and noise information. That is, the acquisition unit 112 also functions as a data acquisition unit that acquires the resolution performance information.

[0035] The resolution performance information is information on the resolution performance according to the blur given to the low pixel image. When the resolution performance is low (MTF is 0 or a sufficiently small value below the Nyquist frequency of the low pixel image), no moire is present in the low pixel image. On the other hand, when the resolution performance is high (MTF has a value at a frequency above the Nyquist frequency), no moire is present except in the frequency band where aliasing occurs. Therefore, information on the frequency band where moire occurs in the low pixel image can be obtained from the resolution performance information. Therefore, the resolution performance information may include information based on the magnitude of the blur given to the low pixel image. In addition, the resolution performance information may include information based on the spread of the PSF of the blur or the MTF of the blur. Note that the PTF (phase transfer function) of the blur alone does not correspond to the resolution performance information. This is because the PTF only represents the shift in the imaging position.

[0036] In the first embodiment, the resolution performance information used when enlarging a captured image is information on blur that integrates all the influences of the aberration and diffraction of the imaging optical system 141, the optical low pass filter and pixel aperture of the image sensor 142, etc. However, the present invention is not limited to this, and the resolution performance may be expressed only by a part of the blur (for example, the blur generated in the imaging optical system 141). For example, if the optical low pass filter and pixel pitch are fixed and do not change, there is no problem in expressing the resolution performance only by the blur generated in the imaging optical system 141. In this case, however, it is necessary to determine the resolution performance of the low pixel image so that it corresponds to the blur. It is preferable to determine the resolution performance information for the blur obtained by excluding the influences of the optical low pass filter and pixel aperture from the blur applied to the low pixel image.

[0037] The noise information is information about noise added to the low pixel image. The noise information includes information representing the strength of the noise. The strength of the noise can be represented by the standard deviation of the noise or the corresponding ISO sensitivity of the image sensor 142. If denoising has been performed on the captured image before enlargement, the low pixel image may also be similarly denoised, and the parameters of the denoising performed (representing strength, etc.) may be used as the noise information. Information about the noise strength and information about denoising may also be used together as the noise information. This makes it possible to suppress adverse effects and achieve high-precision image enlargement even if the noise or denoising changes.

[0038] Specific examples of the resolution performance information and noise information are shown below. In the first embodiment, the resolution performance information is generated by the following method, but the present invention is not limited to this.

[0039] The resolution performance information in the first embodiment is a map having the same number of pixels (size) in two dimensions (horizontal and vertical) as the low pixel image. Each pixel of the map indicates the resolution performance at the corresponding pixel of the low pixel image. That is, the resolution performance information in the first embodiment is information that differs depending on the position of the low pixel image. The map has multiple channels, with the first channel indicating the resolution performance in the horizontal direction and the second channel indicating the resolution performance in the vertical direction. That is, the resolution performance information in the first embodiment is information having multiple channel components that indicate components of different resolution performance for the same pixel of the low pixel image.

[0040] Moreover, the resolution performance is a value based on the frequency at which the MTF for the white blur applied to the low pixel image becomes a preset value (predetermined value) in the corresponding direction. More specifically, the "frequency at which the preset value is reached" is the minimum frequency among the frequencies at which the MTF becomes equal to or less than a threshold value (0.5 in the first embodiment, but is not limited to this). Furthermore, the resolution performance is indicated by a value obtained by normalizing the minimum frequency with the sampling frequency of the low pixel image. The sampling frequency used for normalization is the reciprocal of the pixel pitch, and is common to RGB. That is, the resolution performance information in the first embodiment is information obtained by using information on the pixel pitch corresponding to the low pixel image. However, the value representing the resolution performance is not limited to this. Moreover, instead of white, the resolution performance of each of RGB may be expressed by six channels, and the frequency used for normalization may be different for each of RGB.

[0041] Further, other examples of the resolution performance information are shown below. The direction of the resolution performance indicated by the resolution performance information may be the meridional (radial) direction and the sagittal (azimuth) direction. A third channel may be added to indicate the azimuth of the pixel. The number of channels may be increased to indicate the resolution performance in multiple directions, not just two directions. Conversely, the resolution performance may be indicated by only one channel, such as by taking an average in a specific direction or in all directions. Furthermore, the resolution performance information may be a scalar value or vector, rather than a map. For example, when the imaging optical system 141 is a super telephoto lens or when the F-number is large, the change in the resolution performance due to the image height and azimuth becomes very small. Therefore, in the above-mentioned case, the effect of the invention can be sufficiently obtained even with a scalar value, rather than a map indicating the performance for each pixel. Furthermore, the integral value of the MTF, etc., may be used as the resolution performance, rather than a value based on the frequency at which the MTF becomes a default value.

[0042] Furthermore, the resolution performance may be expressed by the spread of the PSF. The resolution performance may be expressed by the half-width of the PSF in multiple directions, or the spatial range in which the intensity of the PSF is equal to or greater than a threshold value. In this case, when expressing the resolution performance as a scalar value rather than a map, it is advisable to take a specific direction or an average of directions, as described for the MTF.

[0043] Alternatively, the resolution performance may be expressed by coefficients obtained by fitting the MTF or PSF. For example, the MTF or PSF may be fitted with a power series, a Fourier series, a mixed Gaussian model, a Legendre polynomial, a Zernike polynomial, or the like, and each coefficient of the fitting may be expressed by multiple channels.

[0044] Furthermore, the resolution performance information may be generated by calculation from the blur added to the low pixel image, or a plurality of blurs and corresponding resolution performance information may be stored in advance in the storage unit 111 and acquired from there.

[0045] Similarly to the resolution performance information, the noise information is a map with the same two-dimensional number of pixels as the low pixel image. In this embodiment, the first channel is a parameter representing the strength of noise before denoising the low pixel image, and the second channel is a parameter representing the strength of the denoising performed. If compression noise is present in the low pixel image, the strength of the compression noise may be added to an additional channel. Similarly to the resolution performance information, the noise information may be in the form of a scalar value or a vector.

[0046] The order of steps S102 and S101 may be reversed, or steps S102 and S101 may be performed simultaneously.

[0047] In step S103, the calculation unit 113 generates an enlarged image from the low pixel image, the resolution performance information, and the noise information using a generator that is a machine learning model. The enlarged image is an image in which the sampling pitch of the low pixel image is reduced. In other words, the calculation unit 113 has a function as a calculation means that uses the machine learning model to generate an enlarged image in which the sampling pitch of the low pixel image is reduced based on the low pixel image and the resolution performance information.

[0048] The generation of the enlarged image will be described with reference to FIG. 5. In FIG. 5, "sum" indicates the sum of each element (pixel), and "concatenation" indicates concatenation in the channel direction. As described above, in the first embodiment, the resolution performance information 202 and the noise information 203 are maps having the same number of two-dimensional pixels as the low pixel image 201. The low pixel image 201, the resolution performance information 202, and the noise information 203 are concatenated in the channel direction, and then input as input data to the generator 211 to generate the residual component 204. The residual component 204 has the same number of two-dimensional pixels as the high pixel image. The low pixel image 201 is enlarged to the same number of pixels as the high pixel image by bilinear interpolation or the like, and the enlarged image 205 is generated by taking the sum of the low pixel image 201 and the residual component 204. That is, in the first embodiment, the enlarged image 205 is generated by adding a second intermediate image (residual component 204) generated using the low pixel image and the resolution performance information to a first intermediate image in which the sampling pitch of the low pixel image is reduced without using the resolution performance information. The second intermediate image has a smaller sampling pitch than the low pixel image.

[0049] The enlarged image 205 may be generated directly by the generator 211 without passing through the residual component 204. When information such as a scalar value or a vector whose two-dimensional pixel count does not match that of the low pixel image 201 is used as the resolution performance information 202 or the noise information 203, the resolution performance information 202 or the noise information 203 may be converted into a feature map via a convolution layer. In this case, the resolution performance information 202 or the noise information 203 converted into the feature map may be linked in the channel direction to the low pixel image 201 (or the feature map converted therefrom). When the low pixel image 201 is converted into a feature map and then linked to the resolution performance information 202 or the noise information 203 (or information obtained by converting them into a feature map) in the channel direction, the number of pixels in the feature map of the low pixel image 201 does not necessarily match the number of pixels of the low pixel image 201. In this case, the number of two-dimensional pixels of the resolution performance information 202 and noise information 203 (or information expressing them as a feature map) may be adjusted to the number of two-dimensional pixels of the feature map into which the low pixel image 201 is converted.

[0050] The generator 211 in this embodiment is a CNN having the configuration shown in Fig. 6(A), although the present invention is not limited to this.

[0051] 6A, conv. stands for convolution, ReLU stands for Rectified Linear Unit, and sub-pixel conv. stands for sub-pixel convolution. The initial values ​​of the weights of the generator 211 may be generated by random numbers or the like.

[0052] In the first embodiment, the number of two-dimensional pixels of the input is quadrupled by subpixel convolution, so that the number of two-dimensional pixels of the residual component 204 is made the same as the number of pixels of the high resolution image.

[0053] The residual block is a residual block. The residual block has a plurality of linear sum layers and an activation function, and is configured to take the sum of the input and output of the block. The residual block in the first embodiment is shown in FIG. 6(B). In the first embodiment, the generator 211 has 16 residual blocks. However, the number of residual blocks is not limited to this. If it is desired to further improve the performance of the generator 211, it is recommended to increase the number of residual blocks.

[0054] GAP stands for global average pooling, dense stands for full connection, sigmoid stands for sigmoid function, and multiply stands for multiplication by element. By generating an attention map using GAP and full connection, we aim to improve the accuracy of the task.

[0055] Note that the low pixel image 201 may be enlarged in advance by bilinear interpolation or the like so that the number of pixels matches that of the high pixel image, and then input to the generator 211. In this case, subpixel convolution is not required in the generator 211. However, as the number of two-dimensional pixels of the low pixel image 201 increases, the number of times linear sums are calculated increases, resulting in a large calculation load. For this reason, it is desirable to input the low pixel image 201 to the generator 211 without enlarging it as in the first embodiment, and to enlarge it internally.

[0056] In step S104 of FIG. 4, the calculation unit 113 inputs the enlarged image 205 and the high pixel image to a classifier to generate a classification output. The classifier classifies whether the input image is an image generated by the generator 211 (enlarged image 205 in which high frequency components are estimated from the low pixel image) or an actual high pixel image (an image in which frequency components equal to or higher than the Nyquist frequency of the low pixel image are acquired during imaging). The classifier may be a CNN or the like. The initial values ​​of the weights of the classifier are determined by random numbers or the like. Note that the high pixel image input to the classifier may be any actual high pixel image, and does not have to be an image corresponding to the low pixel image 201.

[0057] In step S105, the update unit 114 updates the weights of the classifier based on the classification output and the correct label so that a correct classification output is generated. In the first embodiment, the correct label for the enlarged image 205 is set to 0, and the correct label for the actual high-pixel image is set to 1. Although sigmoid cross entropy is used as the loss function, other functions may be used. Backpropagation is used to update the weights.

[0058] In step S106, the update unit 114 updates the weights of the generator 211 based on the first loss and the second loss. The first loss is a loss based on the difference between the high-pixel image corresponding to the low-pixel image 201 and the enlarged image 205. In the first embodiment, MSE (Mean Square Error) is used, but MAE (Mean Absolute Error) or the like may be used. The second loss is a sigmoid cross entropy between the classification output when the enlarged image 205 is input to the classifier and the correct label 1. The generator 211 is trained so that the classifier erroneously determines that the enlarged image 205 is an actual high-pixel image. For this reason, the correct label is set to 1 (corresponding to the actual high-pixel image). Note that the execution order of steps S105 and S106 may be reversed. That is, the update unit 114 has a function as an update means for updating the weights of the machine learning model using the enlarged image and the high-pixel image.

[0059] In step S107, the update unit 114 determines whether the training of the generator 211 is complete. If it is determined that the training is incomplete, the process returns to step S101 to acquire one or more new pairs of low pixel image 201 and high pixel image. If the training is complete, information on the weights of the trained machine learning model produced by this flow is stored in the storage unit 111. Note that since only the generator 211 is used during actual image enlargement, it is possible to store the weights of only the generator 211 and not the weights of the classifier.

[0060] Note that, before training of the GAN using the discriminator, the generator 211 may be trained using only the first loss. Also, the first data set and the second data set may be stored in the storage unit 111, training may be performed in steps S101 to S107 using the first data set, and training may be performed in steps S101 to S107 using the second data set with the weights as initial values. The first data set has fewer high-pixel images having high-frequency components equal to or higher than the Nyquist frequency of the low-pixel images (i.e., less moire in the low-pixel images) than the second data set. Therefore, the generator 211 trained with the first data set is likely to leave moire, but false structures are also unlikely to appear. In contrast, the generator 211 trained with the second data set can remove moire, but false structures are also likely to appear. By storing the progress of the weights of the generator 211 during training with the second data set, it is possible to select weights that balance between removing moire and false structures later.

[0061] Next, the enlargement process of the captured image will be described with reference to the flowchart of Fig. 7. Each step is executed by the image enlargement device 102 or the control device 103.

[0062] In step S201, the communication unit 132 of the control device 103 transmits a captured image and a request to execute enlargement processing of the captured image to the image enlargement device 102. That is, the communication unit 132 has a function as a transmission means for transmitting a request to cause the image enlargement device 102 to execute processing on the captured image. However, if the image enlargement device 102 can obtain an imaging device from a source other than the control device 103, the control device 103 does not necessarily need to transmit the captured image to the image enlargement device 102. The captured image is a developed image, as in training.

[0063] In step S202, the communication unit 122 of the image enlargement device 102 acquires the captured image and a request to execute an enlargement process on the captured image transmitted from the control device 103. That is, the communication unit 122 has a function as a receiving means for receiving a request from the control device 103. The communication unit 122 also has a function as an acquiring means for acquiring the captured image.

[0064] In step S203, the acquisition unit 123 acquires information on the weight of the generator, resolution performance information, and noise information from the storage unit 121. That is, the acquisition unit 123 has a function as an acquisition means for acquiring the resolution performance information. The resolution performance information is information indicating the resolution performance of the optical device when the captured image is captured. Here, the optical device in the first embodiment includes the imaging optical system 141, the optical low-pass filter of the image sensor 142, and the pixel aperture. In order to acquire the resolution performance information and the noise information, the image enlargement device 102 acquires necessary information from the meta information of the captured image. The necessary information is, for example, the type of the imaging optical system 141, the state of the imaging optical system 141 at the time of capturing (focal length, F value, focus distance), the pixel pitch of the image sensor 142, the optical low-pass filter, and the ISO sensitivity at the time of capturing (the intensity of noise). In addition, the presence or absence of denoising of the captured image, denoising parameters, the cropping position (the position of the optical axis of the imaging optical system 141 with respect to the captured image after cropping), and the like may be acquired. The image enlargement device 102 generates resolution performance information (a map of two channels in the first embodiment) from the acquired information and a data table related to the resolution performance of the imaging optical system 141 stored in the storage unit 121. The storage unit 121 stores information related to the type, state, image height, and resolution performance corresponding to the sampling points of the imaging optical system 141 as a data table. From the data table, the resolution performance information corresponding to the captured image can be generated by interpolation or the like. Note that the resolution performance information in the first embodiment is the same as that in training, and is a map with the same number of two-dimensional pixels as the captured image, and is a value representing the resolution performance in the horizontal direction in the first channel and in the vertical direction in the second channel of each pixel. As a value representing the resolution performance, a value obtained by normalizing the minimum frequency at which the MTF in the corresponding direction falls below a threshold value (0.5) by the sampling frequency (the reciprocal of the pixel pitch) of the image sensor 142 is used. As in training, the MTF is the MTF for the white color of the blur that combines the effects of the imaging optical system 141, the optical low-pass filter of the image sensor 142, and the pixel aperture. In addition, when the resolution performance of the captured image does not change (the type and state of the imaging optical system 141 and the image sensor 142 are fixed), the resolution performance information of the map state may be stored in the storage unit 121 and simply called up.The noise information is also a two-dimensional map with the same number of pixels as the captured image, with the first channel representing the strength of noise generated during imaging, and the second channel representing the denoising parameters applied to the captured image.

[0065] In step S204, the image enlargement unit 124 generates an enlarged image from the captured image, resolution performance information, and noise information using the generator shown in Fig. 5. The enlarged image is an image in which the sampling pitch is half that of the captured image (the number of pixels is four times larger). In other words, the image enlargement unit 124 functions as a generating means for generating an output image in which the sampling pitch of the captured image is reduced.

[0066] In step S205, the communication unit 122 transmits the enlarged image to the control device 103. After that, the processing of the image enlargement device 102 ends.

[0067] In step S206, the communication unit 132 of the control device 103 acquires the enlarged image, and the process of the control device 103 ends. The acquired enlarged image is stored in the storage unit 131, or displayed on the display unit 133. Alternatively, the image may be stored in another storage device connected from the control device 103 or the image enlargement device 102 via a wired or wireless connection.

[0068] In the first embodiment, a machine learning model is used for image enlargement, but other methods may be used. For example, in the case of sparse coding, a first dictionary set is generated from a low pixel image in which moiré does not occur and a high pixel image corresponding to the low pixel image. Furthermore, a second dictionary set is generated from a low pixel image in which moiré occurs and a high pixel image corresponding to the low pixel image. Based on the resolution performance information of the captured image, the image may be enlarged using the first dictionary in an area in which moiré does not occur, and the image may be enlarged using the second dictionary in other areas. In the first embodiment, the captured image is one, but the invention is not limited to this. An enlarged image may be generated from multiple captured images that are shifted by sub-pixels and resolution performance information.

[0069] With the above configuration, it is possible to provide an image processing system capable of improving the accuracy of upsampling a captured image.

[0070] [Example 2] An image processing system according to a second embodiment of the present invention will be described. In the second embodiment, demosaicing is performed as upsampling, but other upsampling methods can be applied in the same way. In addition, a machine learning model is used for demosaicing, but other methods can be applied in the same way.

[0071] 8 and 9 are a block diagram and an external view of the image processing system 300, respectively. The image processing system 300 includes a training device 301 and an imaging device 302. The imaging device 302 includes an imaging optical system 321, an imaging element 322, an image processing unit 323, a storage unit 324, a communication unit 325, and a display unit 326. The imaging optical system 321 forms a subject image from light in the subject space, and the imaging element 322 captures the subject image to generate a captured image. The captured image is an image in which RGB pixels are arranged in a Bayer array. The captured image is acquired as a live view of the subject space before capture or when the user presses the release button, and after a development process is performed by the image processing unit 323, it is stored in the storage unit 324 or displayed on the display unit 326. When the captured image is developed, demosaic is performed using a machine learning model, and a demosaic image (output image) is generated. The machine learning model is trained in advance by the training device 301, and information on the trained weights is acquired via the communication unit 325. However, the trained weights trained by the training device 301 may be stored in advance (for example, at the time of shipment) in the storage unit 324 of the imaging device. In demosaicing of the captured image, resolution performance information, which is information on the resolution performance of the imaging optical system 321, is used. This process will be described in detail.

[0072] First, training of the machine learning model will be described with reference to the flowchart in Fig. 10. Each step is executed by the training device 301.

[0073] In step S301, the acquisition unit 312 acquires one or more pairs of a mosaic image and a correct image from the storage unit 311. The mosaic image is a Bayer image of RGB, the same as the captured image. FIG. 11(A) illustrates the Bayer array, and FIG. 11(B) illustrates the Nyquist frequency of each color in the Bayer array. G has a sampling pitch that is the square root of 2 times the pixel pitch in the diagonal direction, and has a Nyquist frequency 402. R and B have a sampling pitch that is twice the pixel pitch in the horizontal and vertical directions, and has a Nyquist frequency 403. The correct image is an image having the same number of two-dimensional pixels as the mosaic image and three channels of RGB. The correct image has a sampling pitch equal to the pixel pitch for each of RGB, and all colors have a Nyquist frequency 401. The correct image is generated as an original image from an image captured by CG (Computer Graphics) or a three-chip image sensor. Alternatively, an image captured in the Bayer array may be reduced to generate an image having RGB signal values ​​for each pixel, and used as the original image. At least a part of the original image has frequency components equal to or higher than the Nyquist frequencies 402 and 403 of each color in the Bayer array. A correct image is generated by adding blurring caused by aberration and diffraction occurring in the imaging optical system 321 and blurring caused by the optical low-pass filter and pixel aperture of the image sensor 322 to the original image. A mosaic image can be generated by sampling the correct image in the Bayer array. A plurality of mosaic images and a correct image with different blurring are generated, and the blurring of the actual captured image is made to fall within the range of the blurring. Note that the mosaic image is not limited to the Bayer array.

[0074] In step S302, the calculation unit 313 acquires resolution performance information. In the second embodiment, the resolution performance information is generated for each of RGB. As in the first embodiment, the minimum frequency at which the MTF is equal to or less than a threshold in the horizontal and vertical directions for each of RGB is normalized by the Nyquist frequency of each of RGB, and the result is regarded as the resolution performance.

[0075] In step S303, the calculation unit 313 inputs the mosaic image and the resolution performance information to the machine learning model to generate a demosaic image. In the second embodiment, the demosaic image is generated in the flow shown in FIG. 12. The mosaic image 501 is rearranged into four channels, R, G1, G2, and B, to generate an RGGB image 502. The RGGB image 502 and the resolution performance information 503, which is a map of 8 (4×2) channels indicating the resolution performance of each pixel of each RGGB color, are linked in the channel direction and input to the machine learning model 511 to generate a demosaic image 504. The machine learning model 511 has the same configuration as that shown in FIG. 6, but the present invention is not limited thereto. Also, the mosaic image 501 may be configured to be input to the machine learning model as it is in the Bayer array without being rearranged into four channels.

[0076] In step S304, the update unit 314 updates the weights of the machine learning model 511 based on the error between the correct image and the demosaic image 504.

[0077] In step S305, the update unit 314 determines whether training of the machine learning model 511 is complete. If it is determined that the training is not complete, the process returns to step S301. If it is determined that the training is complete, the training is terminated and the weight information is stored in the storage unit 311.

[0078] Next, demosaicing of a captured image will be described with reference to the flowchart of Fig. 13. Each step is executed by the image processing unit 323.

[0079] In step S401, the acquisition unit (acquisition means) 323a acquires a captured image and resolution performance information. The captured image is an image of a Bayer array, and the resolution performance information is acquired from the storage unit 324 based on the state of the imaging optical system at the time of capturing the image.

[0080] In step S402, the acquisition unit 323a acquires information on the weights of the machine learning model from the storage unit 324. Note that the order in which steps S401 and S402 are performed does not matter.

[0081] In step S403, the demosaic unit (generation means) 323b generates a demosaic image from the captured image and the resolution performance information in accordance with the flow shown in Fig. 12. The demosaic image is an image obtained by demosaicing the captured image.

[0082] The image processing unit 323 may execute other processes such as denoising and gamma correction as necessary. In addition, the image enlargement in the first embodiment may be used in combination with the demosaic.

[0083] With the above configuration, it is possible to provide an image processing system capable of improving the accuracy of upsampling a captured image.

[0084] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0085] According to each embodiment, it is possible to provide an image processing device, an imaging device, an image processing method, an image processing program, and a storage medium capable of improving upsampling of a captured image.

[0086] Although the preferred embodiments of the present invention have been described, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0087] 102 Image enlargement device (image processing device) 122 Communication unit (acquisition means) 123 Acquisition unit (acquisition means) 124 Image enlargement unit (generation means)

Claims

1. acquiring a captured image and resolution performance information that is information representing the resolution performance of an optical device used to capture the captured image; generating an output image in which a sampling pitch of the captured image is reduced based on the captured image and the resolution performance information.

2. 2. The image processing method according to claim 1, wherein the output image is an image obtained by enlarging or demosaicing the captured image.

3. 3. The image processing method according to claim 1, wherein the resolution performance information is information regarding the magnitude of blur occurring in the optical device.

4. 4. The image processing method according to claim 1, wherein the resolution performance information is information based on a spread of a point spread function of the optical device or a modulation transfer function of the optical device.

5. 5. The image processing method according to claim 1, wherein the resolution performance information is information that differs depending on a position of a pixel in the captured image.

6. 6. The image processing method according to claim 1, wherein the resolution performance information is a map in which values ​​are arranged in a size according to the number of pixels of the captured image.

7. 7. The image processing method according to claim 6, wherein the value is based on a frequency at which a modulation transfer function of the optical device takes a predetermined value.

8. 8. The image processing method according to claim 6, wherein the resolution performance information has a plurality of channel components that represent components of different resolution performances for the same pixel of the captured image.

9. the resolution performance information is acquired using a type of the optical device or a state of the optical device when the captured image is captured, 9. The image processing method according to claim 1, wherein the state is information relating to at least one of a focal length, an F-number, and a focus distance.

10. 10. The image processing method according to claim 1, wherein the resolution performance information is acquired using information on a pixel pitch of an image pickup device used to capture the captured image.

11. 11. The image processing method according to claim 1, wherein the output image is an image obtained by correcting blurring caused by the optical device from the captured image.

12. 12. The image processing method according to claim 1, wherein in the step of generating an output image, the output image is generated further using information about noise in the captured image.

13. 13. The image processing method according to claim 12, wherein the information regarding noise includes at least one of information regarding the intensity of noise generated when the captured image was captured and information regarding denoising performed on the captured image.

14. The image processing method according to claim 1 , wherein the output image is generated using a machine learning model that processes the captured image and the resolution performance information.

15. the resolution performance information is a map in which values ​​are arranged in a size corresponding to the number of pixels of the captured image, The image processing method of claim 14 , wherein the machine learning model generates the output image by processing input data in which the resolution performance information and the captured image are linked in a channel direction.

16. The image processing method according to claim 14 or 15, wherein the machine learning model comprises one or more residual blocks.

17. The image processing method according to any one of claims 14 to 16, characterized in that the machine learning model generates the output image by adding a first intermediate image in which the sampling pitch of the captured image is reduced without using the resolution performance information to a second intermediate image in which the captured image and a sampling pitch of the captured image generated using the resolution performance information are smaller than the sampling pitch of the captured image.

18. A program causing a computer to execute the image processing method according to any one of claims 1 to 17.

19. An acquisition means for acquiring a captured image and resolution performance information that is information representing the resolution performance of an optical device used to capture the captured image; a generating unit configured to generate an output image in which a sampling pitch of the captured image is reduced based on the captured image and the resolution performance information.

20. acquiring a first image, resolution performance information representing a resolution performance corresponding to the first image, and a second image having a sampling pitch smaller than that of the first image; generating an output image in which the sampling pitch of the first image is reduced based on the first image and the resolution performance information using a machine learning model; updating weights of the machine learning model using the output image and the second image; A method for producing a trained machine learning model, comprising:

21. a data acquisition means for acquiring a first image, resolution performance information representing a resolution performance corresponding to the first image, and a second image having a smaller sampling pitch than the first image; a computing means for generating an output image in which the sampling pitch of the first image is reduced based on the first image and the resolution performance information by using a machine learning model; an update means for updating weights of the machine learning model using the output image and the second image; A processing device comprising:

22. An image processing system including a first device and a second device, the first device has a transmission means for transmitting a request for causing the second device to execute processing on a captured image; The second device comprises: A receiving means for receiving the request; An acquisition means for acquiring a captured image and resolution performance information that is information representing the resolution performance of an optical device used to capture the captured image; a generating means for generating an output image in which a sampling pitch of the captured image is reduced based on the captured image and the resolution performance information; 1. An image processing system comprising:

Images