Image processing method, image processing device, program, method for manufacturing a trained machine learning model, processing device, image processing system

Abstract

To improve an accuracy of processing for reducing a sampling pitch of a picked-up image.SOLUTION: An image processing method includes: a step of acquiring a picked-up image and resolution performance information being information showing a resolution performance of an optical instrument used for imaging the picked-up image; and a step of generating an output image with a reduced sampling pitch of the picked-up image according to the picked-up image and the resolution performance information.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to image processing for reducing the sampling pitch of a captured image.

Background Art

[0002] Patent Document 1 discloses a method for generating a high-resolution enlarged image by enlarging a low-pixel image to the same number of pixels as a high-pixel image by bicubic interpolation and then inputting it into a trained machine learning model. By using a machine learning model trained for image enlargement processing, it is possible to achieve higher-precision image enlargement compared to general methods such as bicubic interpolation.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the method disclosed in Patent Document 1 has problems that a false structure (artifact) that does not actually exist appears in the enlarged image, or moire that existed in the low-pixel image remains in the enlarged image. This problem also occurs in other image enlargement methods (such as bicubic interpolation and sparse coding) that do not use a machine learning model. Further, this problem occurs not only in image enlargement but also in other processes for reducing the sampling pitch of an image (for example, demosaicing).

[0005] Therefore, an object of the present invention is to improve the accuracy of the process for reducing the sampling pitch of a captured image.

Means for Solving the Problems

[0006] An image processing method according to one aspect of the present disclosure isPhotography The image obtained from the image, The imaging used Resolution performance of optical instruments The spread or modulation transfer function of the point image distribution function that represents this. The process includes the steps of acquiring information and generating an output image with a smaller sampling pitch than the captured image using a machine learning model based on the captured image and the information. do It is characterized by the following: [Effects of the Invention]

[0007] According to the present invention, the accuracy of the process for reducing the sampling pitch of captured images can be improved. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows the relationship between the modulation transfer function and the Nyquist frequency in Examples 1 and 2. [Figure 2] This is a block diagram of the image processing system in Example 1. [Figure 3] This is an external view of the image processing system in Example 1. [Figure 4] This is a flowchart of the machine learning model training in Example 1. [Figure 5] This diagram illustrates the process of generating an enlarged image in Example 1. [Figure 6] This figure shows the configuration of the machine learning models in Examples 1 and 2. [Figure 7] This is a flowchart for generating an enlarged image in Example 1. [Figure 8] This is a block diagram of the image processing system in Example 2. [Figure 9] This is an external view of the image processing system in Example 2. [Figure 10] This is a flowchart of the machine learning model training in Example 2. [Figure 11] This figure shows the relationship between the color filter array and the Nyquist frequency in Example 2. [Figure 12]This diagram illustrates the process of generating demosaicing images in Example 2. [Figure 13] This is a flowchart for generating demosaicing images in Example 2. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described in detail below with reference to the drawings. In each figure, the same reference numerals are used for the same components, and redundant descriptions are omitted.

[0010] Before describing the embodiments in detail, the gist of the present invention will be briefly explained. In the present invention, in the process of reducing the sampling pitch of an captured image (hereinafter referred to as upsampling), resolution performance information, which is information regarding the resolution performance of the optical equipment used to capture the image, is used. This improves the accuracy of upsampling. To explain the reason for this, the problems of upsampling and the principle of its occurrence will be described in detail below.

[0011] When an optical system forms an image of a subject, and this image is converted into a captured image by an image sensor, sampling is performed at the pixels of the image sensor. Therefore, among the frequency components that make up the subject image, those exceeding the Nyquist frequency of the image sensor are mixed with low-frequency components due to aliasing, resulting in moiré patterns. In upsampling of a captured image, the Nyquist frequency increases as the sampling pitch decreases, so ideally, it is desirable to generate an image in which aliasing does not occur up to that increased Nyquist frequency. However, it is generally difficult to perform image processing that distinguishes whether a structure in a captured image containing moiré patterns is a moiré pattern or the original structure of the subject.

[0012] In a general upsampling represented by bilinear interpolation, moiré remains as it is even when the captured image is upsampled. In contrast, in upsampling using a machine learning model, it is expected that some moiré can be removed because it is possible to estimate high frequencies before aliasing occurs to some extent from the moiré. However, as described above, since it is difficult to distinguish between moiré and the structure of the subject, even when using a machine learning model, some moiré may be misrecognized as the subject and remain, and some subjects may be misrecognized as moiré and false structures may be generated.

[0013] Therefore, in the present invention, in upsampling of a captured image, resolution performance information of an optical device used for capturing the captured image is used. This will be further described using FIGS. 1(A) and (B).

[0014] FIGS. 1(A) and (B) show the frequency characteristics of the modulation transfer function (MTF) representing the resolution performance of an optical device. The horizontal axis represents the spatial frequency in a certain direction, and the vertical axis represents the MTF. FIG. 1(A) shows a state where the cut-off frequency 003 of the optical device (in this specification, the cut-off frequency means that the MTF becomes 0 at frequencies higher than that) is below the Nyquist frequency 001. In this case, moiré does not exist in the captured image. This is because even when the MTF is arranged at the period of the sampling frequency 002, there is no region where the MTFs overlap each other. Therefore, when the resolution performance corresponds to FIG. 1(A), by giving (inputting) this to the algorithm, the algorithm can be made to determine that there is no need to estimate the high-frequency components before moiré occurs from the moiré structure. Thereby, it is possible to suppress the generation of false structures in the image processing result.

[0015] Figure 1(B) shows a state where the cut-off frequency 003 exceeds the Nyquist frequency 001. Also in this case, by providing this information to the algorithm, the algorithm can identify the frequency band in which moiré may occur due to aliasing. In the example of Figure 1(B), moiré may occur in the band between the frequency 004 obtained by subtracting the cut-off frequency 003 from the sampling frequency 002 and the Nyquist frequency 001, and moiré does not occur in other bands. Therefore, by providing resolution performance information to the algorithm, it is possible to suppress the generation of false structures. Thus, it becomes possible to improve the accuracy of upsampling of the captured image.

[0016] [Example 1] The image processing system in Example 1 of the present invention will be described. In Example 1, image enlargement (upscaling) is performed as upsampling, but it can be similarly applied to other upsamplings such as demosaicing. Image enlargement includes an increase in sampling points for the entire captured image and an increase in sampling points for a partial region of the captured image (such as enlargement of a trimmed image or digital zoom). Also in Example 1, a machine learning model is used for image enlargement, but it can be similarly applied to other methods such as sparse coding.

[0017] Figures 2 and 3 are respectively a block diagram and an external view of the image processing system 100. The image processing system 100 includes a training device 101, an image enlargement device 102, a control device 103, and an imaging device 104 connected to each other by a wired or wireless network. The control device 103 includes a storage unit 131, a communication unit 132, and a display unit 133. According to a user's instruction, the control device 103 acquires a captured image from the imaging device 104 and transmits the captured image and a request for executing image enlargement to the image enlargement device 102 via the communication unit 132.

[0018] The imaging device 104 includes an imaging optical system 141, an image sensor 142, an image processing unit 143, and a storage unit 144. The imaging optical system 141 forms an image of the subject from light in the subject space, and the image sensor 142, which has multiple pixels arranged in it, converts the formed image into a captured image. At this time, aliasing occurs in the frequency components of the subject image that are higher than the Nyquist frequency of the image sensor 142. As a result, moiré patterns may occur in the captured image. The image processing unit 143 performs predetermined processing on the captured image as needed (such as correction of pixel defects and development). The captured image or the captured image processed by the image processing unit 143 is stored in the storage unit 144.

[0019] The control device 103 acquires the captured image via communication or a storage medium. The acquired image may be the entire image or only a part of the image (partial region).

[0020] The image magnification device 102 includes a storage unit 121, a communication unit (acquisition means) 122, an acquisition unit 123, and an image magnification unit (generation means) 124. It magnifies an captured image using a trained machine learning model and generates a magnified image (output image). In this process, it uses resolution performance information, which is information regarding the resolution performance of the optical equipment (such as the imaging optical system 141) used to capture the image. Details of this process will be described later. The image magnification device 102 acquires weight information of the trained machine learning model from the training device 101 and stores it in the storage unit 121.

[0021] The training device 101 includes a storage unit 111, an acquisition unit 112, a calculation unit 113, and an update unit 114, and pre-trains a machine learning model using a dataset. The weight information of the machine learning model generated by the training is stored in the storage unit 111.

[0022] When the image magnification device 102 generates an enlarged image, the control device 103 acquires the enlarged image from the image magnification device 102 and presents it to the user via the display unit 133.

[0023] Here, the method for training (determining weights) the machine learning model executed by the training device 101 (method for manufacturing a trained model) will be explained using the flowchart in Figure 4. In Example 1, a machine learning model is trained using a GAN (Generative Adversarial Network), but the present invention is not limited to this. Furthermore, machine learning models include, for example, neural networks, genetic programming, and Bayesian networks. Neural networks include CNNs (Convolutional Neural Networks), GANs (Generative Adversarial Networks), and RNNs (Recurrent Neural Networks).

[0024] Each step in Figure 4 is performed on the training device 101.

[0025] In step S101, the acquisition unit 112 acquires one or more pairs of high-resolution and low-resolution images from the storage unit 111. The storage unit 111 stores a dataset containing multiple high-resolution and low-resolution images. That is, as will be described in detail later, the acquisition unit 112 functions as a data acquisition means that acquires a first image (low-resolution image) and a second image (high-resolution image) with a smaller sampling pitch than the first image.

[0026] Low-resolution images are images input to the machine learning model (generator in Example 1) during training, and are relatively low-resolution images (images with a large sampling pitch). The more accurately the low-resolution images reproduce the properties of the captured image that is actually enlarged using the trained machine learning model, the higher the accuracy of the trained machine learning model will be. Properties of the captured image include, for example, resolution, color representation, and noise characteristics. For example, if the captured image is represented in RGB, while the low-resolution image is represented in monochrome or YUV, the color representations will not match, which may reduce the accuracy of the task (upsampling accuracy). Although the properties of the captured image that are important differ depending on the type of task performed by the machine learning model, in the task of image enlargement, as mentioned above, information on the frequency band in which moiré occurs is important, so resolution performance is particularly important. Therefore, it is desirable that the resolution performance of the captured image that is actually enlarged using the trained machine learning model (the resolution performance of the optical equipment used to obtain the captured image that is actually enlarged) falls within the range of resolution performance of the multiple low-resolution images used for training.

[0027] High-resolution images are ground truth images in training machine learning models. High-resolution images depict the same scene as the corresponding low-resolution images, but with a smaller sampling pitch (i.e., more pixels) than the low-resolution images. In Example 1, the sampling pitch of the high-resolution image is half that of the low-resolution image. Therefore, the machine learning model quadruples the number of pixels in the input image (twice in both the vertical and horizontal directions). However, the present invention is not limited to this. To enable the machine learning model to handle images of various subjects, it is desirable that the multiple low-resolution and high-resolution images used for training include various subjects (edges with different orientations and strengths, textures, gradients, flat areas, etc.). Furthermore, at least a portion of the high-resolution image has frequency components above the Nyquist frequency of the low-resolution image.

[0028] In Example 1, the high-resolution and low-resolution images used are those generated from the original image through imaging simulation. However, the present invention is not limited to this, and the high-resolution and low-resolution images may be generated using images obtained from imaging simulation using 3D data of the subject space instead of the original image. Alternatively, the high-resolution and low-resolution images may be generated by actual shooting with image sensors with different pixel pitches.

[0029] The original image is an undeveloped RAW image (an image in which the light intensity and signal value have a linear relationship), has a sampling pitch lower than or equal to that of the high-resolution image, and has frequency components above the Nyquist frequency of the low-resolution image, at least in part. The low-resolution image is generated by reproducing the same imaging process as the image that is actually magnified by the trained machine learning model, using the original image as the subject. Specifically, the original image is given blur due to aberrations and diffraction occurring in the imaging optical system 141, and blur due to the optical low-pass filter and pixel aperture of the image sensor 142. If there are multiple types and states of optical instruments used to obtain the image magnified by the trained machine learning model, and different blurs may act on the image depending on these, it is desirable to include low-resolution images with these multiple blurs in the dataset. The blur can change depending on the position of each pixel of the image sensor 142 (image height and azimuth with respect to the optical axis of the imaging optical system 141), and also depending on the state of the imaging optical system 141 if it can take on various states (e.g., focal length, F-number, focus distance). Furthermore, if the imaging device 104 is an interchangeable-lens camera and multiple types of optical systems can be used as the imaging optical system 141, the blur will also change depending on the type of optical system. In addition, if there are different types of imaging devices 104, and the pixel pitch and optical low-pass filter differ, the blur will also change.

[0030] The blur applied to the original image may be the blur generated by the imaging optical system 141 or the image sensor 142 itself, or a blur that approximates it. For example, the PSF (point image distribution function) of the blur generated by the imaging optical system 141 or the image sensor 142 may be approximated by a two-dimensional Gauss distribution function, a mixture of multiple two-dimensional Gauss distribution functions, or a Zernike polynomial. Alternatively, the OTF (optical transfer function) or MTF (modulation distribution function) may be approximated by a two-dimensional Gauss distribution function, a mixture of multiple two-dimensional Gauss distribution functions, or a Legendre polynomial. In this case, the blur can be applied to the original image using the approximated PSF, OTF, MTF, etc.

[0031] After blurring the original image, it is downsampled using the sampling pitch of the image sensor 142. Furthermore, since the image sensor 142 has RGB (Red, Green, Blue) color filters arranged in a Bayer array, it is preferable to sample the low-resolution image so that it also has a Bayer array. However, the present invention is not limited to this, and the image sensor 142 may be monochrome, honeycomb array, 3-chip type, etc. If there are multiple types of image sensors 142 used to obtain the captured image that will be enlarged by the trained machine learning model, and the pixel pitch of the captured image may change, then low-resolution images should be generated for multiple sampling pitches to cover the range of change. In addition, in Example 1, noise generated by the image sensor 142 is also added to the low-resolution image. This is because if noise is not added to the low-resolution image (noise is not considered when training the machine learning model), there is a risk that when the captured image is enlarged, not only the subject but also the noise may be considered as part of the structure of the subject and be emphasized. If there is a range in the strength of the noise generated in the captured image (for example, if there are multiple ISO sensitivities during imaging), it is preferable to include multiple low-resolution images with varying noise strengths within the possible range in the dataset.

[0032] High-resolution images are generated by adding blur to the original image using a pixel aperture with half the pixel pitch of the low-resolution image, downsampling with half the sampling pitch of the low-resolution image, and then performing Bayer processing. Note that if the sampling pitch of the original image and the high-resolution image are the same, the original image may be used directly as the high-resolution image. In Example 1, blur due to aberrations and diffraction of the imaging optical system 141, and blur due to the optical low-pass filter of the image sensor 142 are not added during high-resolution image generation. This allows the machine learning model to correct for the aforementioned blur along with image magnification. However, the present invention is not limited to this; the same blur as the low-resolution image may be added to the high-resolution image, or the blur added to the low-resolution image may be reduced and applied to the high-resolution image. Furthermore, in Example 1, no noise is added during high-resolution image generation. This allows the machine learning model to perform denoising along with image magnification. However, the present invention is not limited to this; noise of the same or different intensity as the noise added to the low-resolution image may be added. Furthermore, when adding noise to high-resolution images, it is desirable to add noise that is correlated with the noise in low-resolution images (for example, noise generated by the same random numbers as the noise added to the low-resolution images). This is because if the noises are uncorrelated, training on multiple images in the dataset may average out the effect of the noise in high-resolution images, potentially preventing the desired effect from being achieved.

[0033] In Example 1, image enlargement is performed on a developed captured image. Therefore, both the low-resolution and high-resolution images must also be developed images. Accordingly, the same development process as the captured image is performed on the low-resolution and high-resolution images in Bayer state and stored in the dataset. However, the invention is not limited to this, and the low-resolution and high-resolution images may be in RAW format, and the captured image may also be enlarged in RAW format. Furthermore, if compression noise such as JPEG encoding occurs in the captured image, similar compression noise may be applied to the low-resolution image. This trains the machine learning model to remove compression noise along with image enlargement.

[0034] In step S102, the acquisition unit 112 acquires resolution performance information and noise information. In other words, the acquisition unit 112 also functions as a data acquisition means for acquiring resolution performance information.

[0035] Resolution performance information is information about the resolution performance in relation to the blur applied to a low-resolution image. When the resolution performance is low (MTF is 0 or a sufficiently small value below the Nyquist frequency of the low-resolution image), no moiré patterns are present in the low-resolution image. On the other hand, when the resolution performance is high (MTF has a value at frequencies above the Nyquist frequency), no moiré patterns are present except in the frequency band where aliasing occurs. Therefore, information about the frequency band in which moiré patterns occur in a low-resolution image can be obtained from the resolution performance information. Consequently, resolution performance information may include information based on the magnitude of the blur applied to the low-resolution image. Furthermore, resolution performance information may include information based on either the spread of the blur's PSF or the blur's MTF. Note that the blur's PTF (phase transfer function) alone does not constitute resolution performance information, as the PTF only represents the shift in the imaging position.

[0036] In Example 1, the resolution performance information used when enlarging the captured image is information on blur that integrates all the effects of the aberration and diffraction of the imaging optical system 141, the optical low-pass filter and pixel aperture of the image sensor 142, etc. However, the present invention is not limited to this, and the resolution performance may be expressed using only a part of the blur (for example, blur generated in the imaging optical system 141). For example, if the optical low-pass filter and pixel pitch are fixed and do not change, there is no problem in expressing the resolution performance using only the blur generated in the imaging optical system 141. However, in this case, it is necessary to determine the resolution performance of low-resolution images accordingly. It is preferable to determine the resolution performance information for blur obtained by excluding the effects of the optical low-pass filter and pixel aperture from the blur applied to low-resolution images.

[0037] Noise information refers to information about noise added to a low-resolution image. This noise information includes information representing the noise intensity. Noise intensity can be expressed using the standard deviation of the noise or the corresponding ISO sensitivity of the image sensor 142. Furthermore, if denoising has been performed on the image before enlargement, the same denoising can be performed on the low-resolution image, and the parameters of the denoising (representing intensity, etc.) can be used as noise information. Additionally, information regarding noise intensity and denoising can be used together as noise information. This allows for high-precision image enlargement while suppressing adverse effects even when noise or denoising changes.

[0038] The following are specific examples of resolution performance information and noise information. In Example 1, the resolution performance information is generated by the following method, but the present invention is not limited thereto.

[0039] The resolution performance information in Example 1 is a map with the same number of pixels (size) in two dimensions (horizontal and vertical) as the low-resolution image. Each pixel in the map represents the resolution performance of the corresponding pixel in the low-resolution image. In other words, the resolution performance information in Example 1 is different depending on the position in the low-resolution image. The map has multiple channels, with the first channel representing the horizontal resolution performance and the second channel representing the vertical resolution performance. In other words, the resolution performance information in Example 1 is information having multiple channel components representing different resolution performance components for the same pixel in the low-resolution image.

[0040] Furthermore, the resolution performance is a value based on the frequency at which the MTF for the white blur applied to the low-resolution image reaches a predetermined value in the relevant direction. More specifically, the "frequency at which the default value is reached" is the minimum frequency among the frequencies at which the MTF falls below a threshold (0.5 in Example 1, but not limited to this). Furthermore, the resolution performance is shown as a value normalized by the sampling frequency of the low-resolution image using the aforementioned minimum frequency. The sampling frequency used for normalization is the reciprocal of the pixel pitch and is common to RGB. In other words, the resolution performance information in Example 1 is information obtained using information about the pixel pitch corresponding to the low-resolution image. However, the value representing the resolution performance is not limited to this. Also, the resolution performance of each RGB channel may be expressed in 6 channels instead of white, and the frequencies used for normalization may differ for each RGB channel.

[0041] Furthermore, other examples of resolution performance information are shown below. The direction of resolution performance shown in the resolution performance information may be the meridional (radial) direction and the sagittal (azimuth) direction. A third channel representing the pixel azimuth may also be added. In addition to two directions, the number of channels may be increased to represent resolution performance in multiple directions. Conversely, resolution performance may be represented by only one channel, such as by taking the average of a specific direction or all directions. Also, the resolution performance information may be a scalar value or vector instead of a map. For example, when the imaging optical system 141 is a super telephoto lens or when the F number is large, the change in resolution performance due to image height and azimuth becomes very small. Therefore, in the above case, the effect of the invention can be fully obtained even with a scalar value instead of a map showing the performance for each pixel. Also, instead of a value based on the frequency at which the MTF becomes a predetermined value, the integral value of the MTF may be used as the resolution performance.

[0042] Furthermore, resolution performance may also be represented by the spread of the PSF. Resolution performance may be represented by the full width at half maximum of the PSF in multiple directions, or by the spatial range in which the intensity of the PSF is above a threshold value. In this case as well, when representing resolution performance as a scalar value rather than a map, it is advisable to take the average in a specific direction, similar to what was described for MTF.

[0043] Alternatively, resolution performance may be expressed using coefficients fitted to the MTF or PSF. For example, the MTF or PSF can be fitted using power series, Fourier series, mixed Gauss models, Legendre polynomials, Zernike polynomials, etc., and the fitting coefficients can be expressed for multiple channels.

[0044] Furthermore, resolution performance information may be generated by calculation from the blur applied to the low-resolution image, or multiple blurs and corresponding resolution performance information may be stored in the storage unit 111 in advance and retrieved from there.

[0045] Furthermore, the noise information, like the resolution performance information, is a map with the same number of pixels as the low-resolution image and the 2D image. In this embodiment, the first channel is a parameter representing the strength of the noise in the low-resolution image before denoising, and the second channel is a parameter representing the strength of the denoising performed. If compression noise exists in the low-resolution image, the strength of the compression noise may be added to the channel. The noise information, like the resolution performance information, may also be in the form of a scalar value or a vector.

[0046] Note that the execution order of steps S102 and S101 can be reversed or performed simultaneously.

[0047] In step S103, the arithmetic unit 113 uses a machine learning model, which is a generator, to generate an enlarged image from the low-resolution image, resolution performance information, and noise information. The enlarged image is an image in which the sampling pitch of the low-resolution image has been reduced. In other words, the arithmetic unit 113 functions as an arithmetic means that uses a machine learning model to generate an enlarged image in which the sampling pitch of the low-resolution image has been reduced, based on the low-resolution image and resolution performance information.

[0048] Figure 5 will be used to explain the generation of the enlarged image. In Figure 5, `sum` represents the sum of each element (pixel), and `concatenation` represents concatenation in the channel direction. As mentioned above, in Example 1, the resolution performance information 202 and the noise information 203 are maps with the same number of pixels in two dimensions as the low-resolution image 201. The low-resolution image 201, the resolution performance information 202, and the noise information 203 are concatenated in the channel direction and then input as input data to the generator 211, generating the residual component 204. The residual component 204 has the same number of pixels in two dimensions as the high-resolution image. The low-resolution image 201 is enlarged to the same number of pixels as the high-resolution image by bilinear interpolation or the like, and the enlarged image 205 is generated by summing it with the residual component 204. In other words, in Example 1, the enlarged image 205 is generated by adding a second intermediate image (residual component 204) generated using the low-resolution image and the resolution performance information to a first intermediate image obtained by reducing the sampling pitch of the low-resolution image without using the resolution performance information. The second intermediate image has a sampling pitch smaller than that of the low-resolution image.

[0049] Alternatively, the enlarged image 205 may be generated directly by the generator 211 without going through the residual component 204. Furthermore, when using information such as scalar values ​​or vectors where the number of pixels in the low-resolution image 201 does not match the number of pixels in the two-dimensional array as resolution performance information 202 or noise information 203, the resolution performance information 202 or noise information 203 may be converted to a feature map via a convolutional layer. In this case, the resolution performance information 202 or noise information 203 converted to a feature map can be concatenated with the low-resolution image 201 (or the converted feature map) in the channel direction. Note that when concatenating the resolution performance information 202 or noise information 203 (or the information converted from them into a feature map) in the channel direction after converting the low-resolution image 201 to a feature map, the number of pixels in the feature map of the low-resolution image 201 does not necessarily match the number of pixels in the low-resolution image 201. In this case, the two-dimensional pixel count of the resolution performance information 202 and noise information 203 (or information representing them as a feature map) should be matched to the two-dimensional pixel count of the feature map obtained by converting the low-resolution image 201.

[0050] In this embodiment, the generator 211 is a CNN with the configuration shown in Figure 6(A). However, the present invention is not limited thereto.

[0051] In Figure 6(A), conv. represents convolution, ReLU represents Rectified Linear Unit, and sub-pixel conv. represents subpixel convolution. The initial values ​​of the weights of generator 211 should be generated using random numbers or similar methods.

[0052] In Example 1, the number of pixels in the 2D input is quadrupled by subpixel convolution, making the number of pixels in the 2D residual component 204 the same as the number of pixels in the high-resolution image.

[0053] A residual block is a residual block. A residual block has multiple linear sum layers and an activation function, and is configured to take a sum at the input and output of the block. The residual block in Example 1 is shown in Figure 6(B). In Example 1, the generator 211 has 16 residual blocks. However, the number of residual blocks is not limited to this. If you want to improve the performance of the generator 211, you can increase the number of residual blocks.

[0054] GAP represents global average pooling, dense represents fully connected, sigmoid represents the sigmoid function, and multiply represents element-wise multiplication. By generating an attention map using GAP and fully connected, the accuracy of the task is improved.

[0055] Alternatively, the low-resolution image 201 may be pre-scaled using bilinear interpolation or similar methods to match the pixel count of the high-resolution image before being input to the generator 211. In this case, subpixel convolution is not required in the generator 211. However, if the number of 2D pixels in the low-resolution image 201 increases, the number of times linear sums are taken increases, leading to a higher computational load. Therefore, it is preferable to input the low-resolution image 201 to the generator 211 without scaling it up, as in Example 1, and scale it internally.

[0056] In step S104 of Figure 4, the arithmetic unit 113 inputs the enlarged image 205 and the high-resolution image to the classifier and generates a classification output. The classifier identifies whether the input image is an image generated by the generator 211 (enlarged image 205 from which high-frequency components were estimated from the low-resolution image) or an actual high-resolution image (an image from which frequency components above the Nyquist frequency of the low-resolution image were acquired during imaging). A CNN or similar architecture may be used for the classifier. The initial values ​​of the classifier weights are determined by random numbers or the like. Note that the high-resolution image input to the classifier can be any actual high-resolution image and does not need to be the image corresponding to the low-resolution image 201.

[0057] In step S105, the update unit 114 updates the classifier weights based on the identification output and the correct label so that the correct identification output is generated. In Example 1, the correct label for the enlarged image 205 is 0, and the correct label for the actual high-resolution image is 1. Sigmoid cross-entropy is used as the loss function, but other functions may also be used. Backpropagation is used to update the weights.

[0058] In step S106, the update unit 114 updates the weights of the generator 211 based on the first loss and the second loss. The first loss is the loss based on the difference between the high-resolution image corresponding to the low-resolution image 201 and the enlarged image 205. In Example 1, MSE (Mean Square Error) is used, but MAE (Mean Absolute Error) or other methods may also be used. The second loss is the sigmoid cross-entropy between the discrimination output when the enlarged image 205 is input to the classifier and the correct label 1. The generator 211 is trained so that the classifier misclassifies the enlarged image 205 as the actual high-resolution image. For this reason, the correct label is set to 1 (corresponding to the actual high-resolution image). Note that the execution order of steps S105 and S106 may be reversed. That is, the update unit 114 functions as an update means that updates the weights of the machine learning model using the enlarged image and the high-resolution image.

[0059] In step S107, the update unit 114 determines whether the training of the generator 211 is complete. If it determines that it is not complete, it returns to step S101 and acquires one or more new sets of low-resolution images 201 and high-resolution images. If it is complete, it stores the weight information of the trained machine learning model produced by this flow in the storage unit 111. Note that since only the generator 211 is used when actually enlarging images, it is also possible to store only the weights of the generator 211 and not the weights of the classifier.

[0060] Alternatively, the generator 211 may be trained using only the first loss before training the GAN with the discriminator. Alternatively, the first and second datasets may be stored in the memory unit 111, training steps S101 to S107 may be performed on the first dataset, and the resulting weights may be used as initial values ​​for training steps S101 to S107 on the second dataset. The first dataset contains fewer high-resolution images with high-frequency components above the Nyquist frequency compared to the second dataset (i.e., fewer moiré patterns in low-resolution images). Therefore, a generator 211 trained on the first dataset is more likely to retain moiré patterns but less likely to produce false structures. In contrast, a generator 211 trained on the second dataset can remove moiré patterns but is more likely to produce false structures. By storing the intermediate weights of the generator 211 during training with the second dataset, weights that balance moiré removal and false structure removal can be selected later.

[0061] Next, the image magnification process will be explained using the flowchart in Figure 7. Each step is performed by the image magnification device 102 or the control device 103.

[0062] In step S201, the communication unit 132 of the control device 103 transmits the captured image and a request to the image magnification device 102 to perform the image magnification process. That is, the communication unit 132 functions as a transmission means that transmits a request to the image magnification device 102 to perform processing on the captured image. However, if the image magnification device 102 can acquire the imaging device from a source other than the control device 103, the control device 103 does not necessarily have to transmit the captured image to the image magnification device 102. The captured image is the developed image, as in the training.

[0063] In step S202, the communication unit 122 of the image magnification device 102 receives the captured image and a request to perform magnification processing on the captured image transmitted from the control device 103. That is, the communication unit 122 functions as a receiving means for receiving requests from the control device 103. The communication unit 122 also functions as an acquisition means for acquiring the captured image.

[0064] In step S203, the acquisition unit 123 acquires generator weight information, resolution performance information, and noise information from the storage unit 121. That is, the acquisition unit 123 functions as an acquisition means for acquiring resolution performance information. Resolution performance information is information indicating the resolution performance of the optical equipment when the captured image was taken. Here, the optical equipment in Embodiment 1 includes the imaging optical system 141, the optical low-pass filter of the image sensor 142, and the pixel aperture. To acquire resolution performance information and noise information, the image magnification device 102 acquires the necessary information from the metadata of the captured image. The necessary information includes, for example, the type of imaging optical system 141, the state of the imaging optical system 141 at the time of imaging (focal length, F-number, focus distance), the pixel pitch of the image sensor 142, the optical low-pass filter, and the ISO sensitivity (noise strength) at the time of imaging. In addition, the presence or absence of denoising of the captured image and the denoising parameters, the trimming position (position of the optical axis of the imaging optical system 141 relative to the captured image after trimming), etc., may also be acquired. The image magnification device 102 generates resolution performance information (a 2-channel map in Example 1) from the acquired information and a data table of resolution performance information of the imaging optical system 141 stored in the memory unit 121. The memory unit 121 stores information regarding the type, state, image height, and resolution performance corresponding to the azimuth sampling point of the imaging optical system 141 as a data table. Resolution performance information corresponding to the captured image can be generated from this data table by interpolation or other means. In Example 1, the resolution performance information is the same as during training, with a 2D map having the same number of pixels as the captured image, and the first channel of each pixel represents the resolution performance in the horizontal direction, and the second channel represents the resolution performance in the vertical direction. As a value representing resolution performance, the minimum frequency at which the MTF in the relevant direction falls below the threshold (0.5) is normalized by the sampling frequency of the image sensor 142 (the reciprocal of the pixel pitch). The MTF is the same as during training, with respect to the white of the blur, combining the effects of the optical low-pass filter and pixel aperture of the imaging optical system 141 and the image sensor 142. Furthermore, if the resolution performance of the captured image does not change (the type and state of the imaging optical system 141 and the image sensor 142 are fixed), it is sufficient to simply store the resolution performance information of the map state in the storage unit 121 and recall it.The noise information also uses a 2D map with the same number of pixels as the captured image, where the first channel represents the strength of the noise generated during imaging, and the second channel represents the denoising parameters applied to the captured image.

[0065] In step S204, the image enlargement unit 124 generates an enlarged image from the captured image, resolution performance information, and noise information using the generator shown in Figure 5. The enlarged image is an image in which the sampling pitch is halved (the number of pixels is quadrupled) compared to the captured image. In other words, the image enlargement unit 124 functions as a generation means that generates an output image in which the sampling pitch of the captured image is reduced.

[0066] In step S205, the communication unit 122 transmits the enlarged image to the control device 103. After that, the image enlargement device 102 terminates its processing.

[0067] In step S206, the communication unit 132 of the control device 103 acquires the enlarged image and terminates the processing of the control device 103. The acquired enlarged image is stored in the storage unit 131 or displayed on the display unit 133. Alternatively, it may be stored in another storage device connected via wired or wireless connection from the control device 103 or the image enlargement device 102.

[0068] In Example 1, a machine learning model was used to enlarge the image, but other methods may be used. For example, in the case of sparse coding, a first dictionary set is generated using low-pixel images where moiré patterns do not occur and high-pixel images corresponding to the low-pixel images. Furthermore, a second dictionary set is generated using low-pixel images where moiré patterns occur and their corresponding high-pixel images. Based on the resolution performance information of the captured image, image enlargement can be performed using the first dictionary in areas where moiré patterns do not occur, and image enlargement can be performed using the second dictionary in other areas. In addition, although only one captured image was used in Example 1, the invention is not limited to this, and an enlarged image may be generated from multiple captured images that are subpixel-shifted and their resolution performance information.

[0069] With the above configuration, it is possible to provide an image processing system that can improve the accuracy of upsampling of captured images.

[0070] [Example 2] The image processing system in Embodiment 2 of the present invention will now be described. In Embodiment 2, demosaicing is performed as an upsampling, but the same method can be applied to other upsampling methods. Furthermore, a machine learning model is used for demosaicing, but the same method can be applied to other methods as well.

[0071] Figures 8 and 9 are a block diagram and an external view of the image processing system 300, respectively. The image processing system 300 includes a training device 301 and an imaging device 302. The imaging device 302 includes an imaging optical system 321, an image sensor 322, an image processing unit 323, a storage unit 324, a communication unit 325, and a display unit 326. The imaging optical system 321 forms a subject image from light in the subject space, and the image sensor 322 captures the subject image to generate an image. The image is an image in which RGB pixels are arranged in a Bayer array. The image is acquired from a live view of the subject space before imaging or when the shutter release is pressed by the user. After development processing is performed by the image processing unit 323, it is stored in the storage unit 324 or displayed in the display unit 326. During the development processing of the image, demosaicing is performed using a machine learning model, and a demosaiced image (output image) is generated. The machine learning model is pre-trained by the training device 301, and information on the trained weights is acquired via the communication unit 325. However, the trained weights trained by the training device 301 may be stored in the storage unit 324 of the imaging device in advance (for example, at the time of shipment). In demosaicing of the captured image, resolution performance information, which is information regarding the resolution performance of the imaging optical system 321, is used. This process will be explained in detail.

[0072] First, the training of the machine learning model will be explained using the flowchart in Figure 10. Each step is performed by the training device 301.

[0073] In step S301, the acquisition unit 312 acquires one or more mosaic images and ground truth images from the storage unit 311. The mosaic image is a Bayer image with the same RGB colors as the captured image. Figure 11(A) shows the Bayer array, and Figure 11(B) shows the Nyquist frequencies of each color in the Bayer array. G has a sampling pitch that is the square root of 2 times the pixel pitch in the diagonal direction and has a Nyquist frequency of 402. R and B have a sampling pitch that is twice the pixel pitch in the horizontal and vertical directions and have a Nyquist frequency of 403. The ground truth image is an image with the same number of pixels in two dimensions as the mosaic image and has three RGB channels. In the ground truth image, each of RGB has a sampling pitch equal to the pixel pitch, and all colors have a Nyquist frequency of 401. The ground truth image is generated using an image captured with CG (Computer Graphics) or a three-chip image sensor as the source image. Alternatively, an image captured with a Bayer array may be reduced to generate an image with RGB signal values ​​for each pixel, and this image may be used as the source image. At least a portion of the original image has frequency components of Nyquist frequencies 402, 403 or higher for each color in the Bayer array. A ground truth image is generated by adding blur due to aberrations and diffraction generated in the imaging optical system 321, and blur due to the optical low-pass filter and pixel aperture of the image sensor 322, to the original image. A mosaic image can be generated by sampling the ground truth image with a Bayer array. Multiple mosaic images with different added blurs and the ground truth image are generated, and the blur of the actual captured image is made to fit within the range of these blurs. Note that the mosaic image is not limited to a Bayer array.

[0074] In step S302, the calculation unit 313 acquires resolution performance information. In Embodiment 2, resolution performance information is generated for each RGB channel. For each RGB channel, the resolution performance is defined as the minimum frequency at which the MTF in the horizontal and vertical directions falls below a threshold, normalized by the Nyquist frequency of each RGB channel, similar to Embodiment 1.

[0075] In step S303, the calculation unit 313 inputs the mosaic image and resolution performance information to the machine learning model to generate a demosaiced image. In Example 2, the demosaiced image is generated in the manner shown in Figure 12. The mosaic image 501 is rearranged into four channels, R, G1, G2, and B, to generate an RGGB image 502. The RGGB image 502 and the resolution performance information 503, which is an 8 (4×2) channel map showing the resolution performance of each pixel of each RGGB color, are concatenated in the channel direction and input to the machine learning model 511 to generate a demosaiced image 504. The machine learning model 511 is the same configuration as shown in Figure 6, but the present invention is not limited thereto. Alternatively, the mosaic image 501 may not be rearranged into four channels, but may be input to the machine learning model in its Bayer array form.

[0076] In step S304, the update unit 314 updates the weights of the machine learning model 511 based on the error between the ground truth image and the demosaiced image 504.

[0077] In step S305, the update unit 314 determines whether the training of the machine learning model 511 is complete. If it is determined that the training is not complete, the process returns to step S301. If it is determined that the training is complete, the training is terminated and the weight information is stored in the storage unit 311.

[0078] Next, the demosaicing of the captured image will be explained using the flowchart in Figure 13. Each step is performed by the image processing unit 323.

[0079] In step S401, the acquisition unit (acquisition means) 323a acquires the captured image and resolution performance information. The captured image is a Bayer array image, and the resolution performance information is acquired from the storage unit 324 based on the state of the imaging optical system at the time of imaging.

[0080] In step S402, the acquisition unit 323a acquires information on the weights of the machine learning model from the storage unit 324. The execution order of steps S401 and S402 does not matter.

[0081] In step S403, the demosaicing unit (generation means) 323b generates a demosaiced image from the captured image and resolution performance information in the manner shown in Figure 12. The demosaiced image is an image obtained by demosaicing the captured image.

[0082] The image processing unit 323 may perform other processing such as denoising and gamma correction as needed. Furthermore, image enlargement from Example 1 may be performed simultaneously with demosaicing.

[0083] With the above configuration, it is possible to provide an image processing system that can improve the accuracy of upsampling of captured images.

[0084] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0085] According to each embodiment, it is possible to provide an image processing device, an imaging device, an image processing method, an image processing program, and a storage medium that can improve the upsampling of captured images.

[0086] Furthermore, although preferred embodiments of the present invention have been described, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its essence. [Explanation of Symbols]

[0087] 102 Image Magnification Device (Image Processing Device) 122 Communications Department (Means of Acquisition) 123 Acquisition unit (acquisition means) 124 Image enlargement unit (generation means)

Claims

1. A step of acquiring an image obtained by imaging and information relating to the spread of the point image distribution function or the modulation transfer function that represents the resolution performance of the optical instrument used for imaging, An image processing method characterized by comprising the steps of generating an output image with a smaller sampling pitch than the captured image using a machine learning model based on the captured image and the information.

2. The image processing method according to claim 1, characterized in that the output image is an image obtained by enlarging or demosaicing the captured image.

3. The image processing method according to claim 1 or 2, characterized in that the information is different depending on the position of the pixels in the captured image.

4. The image processing method according to any one of claims 1 to 3, characterized in that the information is a map in which values ​​are arranged to a size corresponding to the number of pixels in the captured image.

5. The image processing method according to claim 4, characterized in that the value is a value based on the frequency at which the modulation transfer function becomes a predetermined value.

6. The image processing method according to claim 4 or 5, characterized in that the information has a plurality of channel components representing different resolution performance components for the same pixel of the captured image.

7. The aforementioned information is obtained using the type of optical instrument or the state of the optical instrument in the imaging process. The image processing method according to any one of claims 1 to 6, characterized in that the aforementioned state is information relating to at least one of the focal length, F-number, and focus distance.

8. The image processing method according to any one of claims 1 to 7, characterized in that the aforementioned information is obtained using information relating to the pixel pitch of the image sensor used for imaging.

9. The image processing method according to any one of claims 1 to 8, characterized in that the output image is an image in which the blur caused by the optical device in the captured image has been corrected.

10. The image processing method according to any one of claims 1 to 9, characterized in that the step of generating the output image further generates the output image based on information regarding the noise of the captured image.

11. The image processing method according to claim 10, characterized in that the noise information includes at least one of the information regarding the intensity of the noise generated during imaging or the information regarding the denoising performed on the captured image.

12. The image processing method according to any one of claims 1 to 11, characterized in that the step of generating the output image generates the output image based on the captured image and the information linked in the channel direction.

13. The image processing method according to any one of claims 1 to 12, characterized in that the machine learning model has one or more residual blocks.

14. The image processing method according to any one of claims 1 to 13, characterized in that, in the step of generating the output image, the output image is generated by adding a second intermediate image, which is generated using the captured image and the information and has a smaller sampling pitch than the captured image, to a first intermediate image generated by reducing the sampling pitch of the captured image without using the information.

15. A program characterized by causing a computer to execute the image processing method described in any one of claims 1 to 14.

16. Acquisition means for acquiring an image obtained by imaging and information relating to the spread of the point image distribution function or the modulation transfer function that represents the resolution performance of the optical instrument used for imaging, An image processing apparatus characterized by having a generation means that generates an output image having a smaller sampling pitch than the captured image using a machine learning model based on the captured image and the information.

17. A step of acquiring a first image, information regarding the spread or modulation transfer function of the point image distribution function representing the resolution performance of the optical instrument corresponding to the first image, and a second image having a smaller sampling pitch than the first image. A step of generating an output image with a smaller sampling pitch than the first image using a machine learning model based on the first image and the information, A method for generating a trained machine learning model, characterized by comprising the step of updating the weights of the machine learning model using the output image and the second image.

18. A data acquisition means that acquires a first image, information regarding the spread or modulation transfer function of the point image distribution function representing the resolution performance of an optical instrument corresponding to the first image, and a second image having a smaller sampling pitch than the first image. A computation means that generates an output image with a smaller sampling pitch than the first image using a machine learning model based on the first image and the information, A processing apparatus characterized by having an update means for updating the weights of the machine learning model using the output image and the second image.

19. An image processing system comprising an image processing device according to claim 16 and a control device capable of communicating with the image processing device, The control device has means for transmitting a request to perform processing on the captured image, The image processing system is characterized in that the image processing device has means for performing processing on the captured image in response to the request.

Images