Image processing methods, image processing devices, image processing systems, image processing programs

Abstract

To provide an image processing method that highly accurately increases image resolution and reduces noise while suppressing a training load of a machine learning model.SOLUTION: An image processing method comprises steps of: generating a first noise reduction output; generating a high-resolution output based on first input data according to the first noise reduction output using a first machine learning model; generating a second noise reduction output based on second input data according to the high-resolution output using a second machine learning model; and generating an output image based on the second noise reduction output. The second input data is an image that has a larger difference from a second addition image than difference from a first addition image. The first addition image is an image obtained by adding the image and a high-resolution residual map based on the high-resolution output. The second addition image is an image obtained by adding the high-resolution residual map and the noise-reduced image based on the first noise reduction output.SELECTED DRAWING: Figure 4

Description

Technical Field

[0006]

[0001] The present invention relates to an image processing method for performing high-resolution and noise reduction on a captured image.

Background Art

[0002] Patent Document 1 discloses a method of performing image noise reduction and upscaling with one neural network by using weights corresponding to the noise reduction level specified by the user.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the method of Patent Document 1, the dataset becomes extremely large, increasing the training load of the neural network.

[0005] When high-resolution and noise reduction are performed with separate machine learning models, the final image quality deteriorates due to the adverse effects on image quality caused by each other's processing. When high-resolution is performed first, the noise is amplified, and even if noise reduction is performed, the amplified noise remains. When noise reduction is performed first, the blurred image changes, resulting in overcorrection or undercorrection during high-resolution.

[0006] An object of the present invention is to provide an image processing method that performs high-resolution and noise reduction of a captured image with high precision while suppressing the training load of a machine learning model.

Means for Solving the Problems

[0007] One aspect of the present invention is an image processing method comprising the steps of generating a first noise reduction output using an captured image, and using a first machine learning model to process first input data corresponding to the first noise reduction output. Blur correction The steps involve generating output related to and using a second machine learning model, Blur correction The process includes the steps of generating a second noise reduction output based on a second input data corresponding to the output related to the first noise reduction, and generating an output image based on the second noise reduction output, wherein the second input data is an image in which the difference between the second and first added image is greater than the difference between the first and first added image, and the first added image is Blur correction Based on the output related to Blur correction This image is obtained by adding the residual map and the captured image, and the second added image is Blur correction The image is characterized by being obtained by adding the residual map and a noise-reduced captured image based on the output for the first noise reduction. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide an image processing method that can perform high-resolution enhancement and noise reduction of captured images with high accuracy while suppressing the training load of machine learning models. [Brief explanation of the drawing]

[0009] [Figure 1] This is an external view of the image processing system in Example 1. [Figure 2] This is a block diagram of the image processing system in Example 1. [Figure 3] This is a flowchart showing the training of a machine learning model. [Figure 4] This is a flowchart showing the generation of the output image in Example 1. [Figure 5] This is an external view of the image processing system in Example 2. [Figure 6] This is a block diagram of the image processing system in Example 2. [Figure 7] This is a flowchart showing the generation of the output image in Example 2. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described in detail below with reference to the drawings. In each figure, the same reference numerals are used for the same components, and redundant descriptions are omitted.

[0011] Before describing this embodiment in detail, the gist of the present invention will be briefly explained. In this invention, the captured image is subjected to high resolution and noise reduction. High resolution includes blur correction or upscaling. Upscaling includes enlarging the entire captured image (increasing the number of pixels) and enlarging a part of the captured image (digital zoom, etc.). Noise reduction includes reducing noise generated by the image sensor and reducing compression noise when the image is irreversibly compressed (JPEG compression, etc.).

[0012] When training a neural network that only performs noise reduction, the dataset only needs to cover variations in noise. The same applies when performing only upscaling. However, when performing both noise reduction and upscaling with a single neural network, the dataset must include both noise and degradation of resolution performance, and the variations increase in combination. As a result, the dataset becomes enormous, and the training load increases.

[0013] Therefore, in this invention, high-resolution enhancement and noise reduction of captured images are performed using individual machine learning models. This reduces the number of datasets used for training, thereby reducing the training load on the machine learning models.

[0014] In the present invention, high-resolution processing is also performed on the captured image after noise reduction to generate a high-resolution image without noise reduction. For example, a required high-resolution residual map is added to the captured image without noise reduction to generate a high-resolution image without noise reduction. Noise reduction is performed again on the high-resolution image without noise reduction to generate a final output image. With this configuration, it is possible to suppress a decrease in the image quality of the final output image due to the adverse effects of either noise reduction or high-resolution processing. The above processing will be described below.

[0015] The key point is that the estimation of noise reduction highly depends on the signal distribution of the input image, while the estimation of high-resolution processing has a small dependence on the signal distribution of the input image. To simplify the explanation, noise reduction targets Gaussian noise with a specific variance, and high-resolution processing targets blur represented by a Gaussian distribution function with a specific variance. At this time, in high-resolution processing, since the same blur always acts on the input image, similar corrections are performed for various signal distributions. However, in noise reduction, although the variance of the noise in the input image is the same, the distribution of the noise is completely unknown. Therefore, in noise reduction, it is necessary to distinguish the subject from the noise based on the signal distribution of the input image, and the estimation result is greatly affected by the signal distribution of the input image.

[0016] If noise reduction is performed after high-resolution processing on the captured image, some strong noise (or all noise) is emphasized due to the adverse effects of high-resolution processing. The emphasized noise is larger than the specific variance assumed for noise reduction, so it is regarded as a subject rather than noise and remains without being reduced.

[0017] Conversely, when performing high-resolution processing on a captured image after noise reduction, some blurred images are deformed due to the adverse effects of noise reduction. However, as described above, since the estimation of high-resolution processing has little dependence on the signal distribution of the input image, even if the blurred image changes, the correction for high-resolution processing does not change significantly. Therefore, if the blurred image spreads due to noise reduction, there will be insufficient correction, and if the blurred image becomes smaller, there will be excessive correction instead.

[0018] Therefore, in the present invention, for example, by adding the residual map of high-resolution processing obtained for the captured image after noise reduction to the captured image that has not been noise-reduced, a captured image on which only high-resolution processing has been performed is generated. The residual map is a component obtained by subtracting the captured image on which only noise reduction has been performed from the captured image on which noise reduction and high-resolution processing have been performed. Since high-resolution processing is being performed on the captured image after noise reduction, the component of noise amplification included in the residual map of high-resolution processing becomes small. Also, since the estimation of high-resolution processing has little dependence on the signal distribution of the input image, even if the residual map of high-resolution processing obtained for the captured image after noise reduction is added to the captured image, the decrease in correction accuracy is small. As a result, it is possible to suppress the excess or deficiency of high-resolution processing and noise amplification, and generate a high-resolution captured image. Since the resolution performance and contrast of the subject are improved by high-resolution processing of the blurred image in the high-resolution captured image, it is easy to distinguish between the subject and the noise. Therefore, by performing noise reduction again on the high-resolution captured image, it is possible to suppress deformation of the subject and reduce noise.

[0019] For the above reasons, it is possible to perform high-resolution processing and noise reduction of a captured image with high accuracy while suppressing the training load of the machine learning model.

Embodiment

[0020] FIG. 1 is an external view of an image processing system 100 of the present embodiment. FIG. 2 is a block diagram of the image processing system 100 of the present embodiment. The image processing system 100 includes a training device 101, an image processing device 102, and an imaging device 103 that are connected to each other by a wired or wireless network.

[0021] The imaging device 103 comprises an imaging optical system 131, an image sensor 132, a storage unit 133, and a communication unit 134. The imaging optical system 131 forms an image of the subject from light in the subject space. The image sensor 132 includes multiple pixels and converts the image of the subject into an image by photoelectric conversion. Because the image of the subject is blurred due to aberrations and diffraction generated in the imaging optical system 131, the image also suffers from degradation due to this blur. In this embodiment, the blur is corrected as a way to increase resolution. In addition, the image sensor 132 generates shot noise, dark current noise, and readout noise, so noise exists in the image. In this embodiment, the noise is reduced as a way to reduce noise. The storage unit 133 stores the image. In this embodiment, blur is corrected as a way to increase resolution, but upscaling or the like may also be performed. Also, in this embodiment, the target of noise reduction is noise generated by the image sensor, but it may also be compression noise or the like.

[0022] The image processing device 102 comprises a storage unit 121, a communication unit 122, an acquisition unit 123, a noise reduction unit (first generation unit, third generation unit) 124, a high-resolution unit (second generation unit) 125, a calculation unit (output unit) 126, and a display unit 127. The communication unit 122 acquires the captured image via the communication unit 134 of the imaging device 103. Alternatively, the captured image may be recorded on a recording medium, and the image processing device 102 may acquire the captured image by connecting the recording medium to the image processing device 102. The image processing device 102 generates an output image by performing high-resolution enhancement and noise reduction on the captured image using a first machine learning model and a second machine learning model. The first and second machine learning models use a set of weights that have been previously generated by the training device 101. The image processing device 102 acquires the set of weights from the training device 101 via the communication unit 122 and stores it in the storage unit 121. The generated output image is either presented to the user via the display unit 127 or stored in the storage unit 121.

[0023] The training device 101 comprises a storage unit 111, an acquisition unit 112, an arithmetic unit 113, and an update unit 114. The training device 101 uses the dataset to train a first machine learning model that performs high-resolution processing and generates a set of trained first weights. The training device 101 also uses the dataset to train a second machine learning model that performs noise reduction and generates a set of trained second weights.

[0024] The training of the machine learning model (determination of the weight set) performed on the training device 101 will be described below with reference to Figure 3. Figure 3 is a flowchart of the training of the machine learning model. In this embodiment, a CNN (Convolutional Neural Network) is used as the machine learning model. However, the present invention is not limited to this. Machine learning models include neural networks, genetic programming, and Bayesian networks. Neural networks include CNNs, GANs (Generative Adversarial Networks), and RNNs (Recurrent Neural Networks).

[0025] First, we will explain the training of the second machine learning model that performs noise reduction.

[0026] In step S101, the acquisition unit 112 acquires one or more sets of ground truth images and training images from the storage unit 111. Since the second machine learning model aims to achieve noise reduction through training, the training images are images with noise, and the ground truth images are images of the same scene as the training images, but with less noise (or no noise) than the training images. The storage unit 111 stores a dataset containing multiple ground truth images and training images. The noise present in the training images is similar to the noise generated by the image sensor 132. To enable the first machine learning model to handle images of various subjects, it is desirable that the multiple ground truth images and training images used for training include various subjects (edges with different orientations and strengths, textures, gradients, and flat areas, etc.).

[0027] The ground truth images and training images used in this embodiment are images generated by adding noise to the original image. However, the same scene may also be captured in real life using the image sensor 132 and under conditions with a better signal-to-noise ratio, and these may be used as training and ground truth images. The original image is an undeveloped RAW image (where light intensity and signal value have a linear relationship), and the training image is generated by adding noise generated by the image sensor 132 to it. If the image sensor 132 can take on multiple types and ISO sensitivities, the dataset should include training images with various levels of noise generated by them. The ground truth image may be the original image as is, or it may be an image with less noise added than the noise added to the training image. It is desirable that the added noise is correlated with the training image. If uncorrelated noise is added, the effect of the noise on the ground truth image will be averaged out by training with multiple images in the dataset, and as a result, it will be almost the same as if no noise had been added. The original image may be a real-life RAW image or a CG (Computer Graphics) image. While RAW images already contain noise, this is not a problem in machine learning model training because the noise is treated as part of the subject. In this embodiment, the second machine learning model performs noise reduction on the RAW images. Therefore, both the ground truth image and the training image are RAW images. If you want to perform noise reduction on the developed image, you can develop both the ground truth image and the training image.

[0028] In step S102, the calculation unit 113 generates a second noise reduction output (information) based on the training image using the second machine learning model. Specifically, the second noise reduction output is generated by inputting the training image into the second machine learning model. The second noise reduction output includes the noise-reduced training image and the residual map of the noise reduction applied to the training image (the residual map of the second noise reduction), etc. When the residual map of the noise reduction is added to the training image, it becomes a noise-reduced image. In this embodiment, the second noise reduction output is the noise-reduced training image (estimated image). The second machine learning model is a CNN and has multiple weights (a set of second weights). The initial value of each weight can be determined by random numbers, etc.

[0029] In step S103, the update unit 114 updates each weight of the second machine learning model based on the difference between the ground truth image and the estimated image. In this embodiment, the mean squared error between the ground truth image and the estimated image is used as the loss function, and the weights are updated using backpropagation. However, the present invention is not limited thereto. By optimizing to reduce the difference between the ground truth image and the estimated image, the second machine learning model acquires the effect of noise reduction on the input image. If the output of the second noise reduction is a residual map, it is sufficient to minimize the difference between the training image and the ground truth image.

[0030] In step S104, the update unit 114 determines whether the training of the second machine learning model is complete. If it is determined that the training is not complete (incomplete), the process returns to step S101, and one or more new sets of ground truth images and training images are acquired. If it is determined that the training is complete, the information of the trained set of second weights is stored in the storage unit 111.

[0031] Next, we will explain the training of the first machine learning model that performs high-resolution enhancement. Sections that are the same as those in the training of the second machine learning model will be omitted from the explanation.

[0032] In step S101, the training image is an image in which blurring occurs due to aberrations and diffraction generated by the imaging optical system 131. The ground truth image is an image of the same scene as the training image, but with less (or no) blurring than the training image. In this embodiment, the training image and ground truth image are generated by adding blurring to the original image. Blurring due to aberrations and diffraction generated by the imaging optical system 131 is added to the original image to generate the training image. If necessary, blurring may be added by an optical low-pass filter or the pixel aperture of the image sensor 132. If there are multiple types and states (focal length, aperture value, focus distance, etc.) of the imaging optical system 131, and different blurring may occur in the captured image depending on them, the dataset will include training images with these multiple blurrings. The blur can vary depending on the position of each pixel in the image sensor 132 (image height and azimuth relative to the optical axis of the imaging optical system 131), and also depending on the state of the imaging optical system 131, which can take on various states (focal length, aperture value, and focus distance, etc.). Furthermore, if the imaging optical system 131 can take on multiple types, such as interchangeable lenses, the blur will also vary depending on the type. The blur applied to the original image may be the blur generated by the imaging optical system 131 itself, or a blur that approximates it. The ground truth image is generated by either not applying blur to the original image, or by applying a blur smaller than the blur applied to the training image.

[0033] In this embodiment, no noise is added to the training images and ground truth images used to train the first machine learning model. This is because the present invention aims to enhance the resolution of the noise-reduced captured images. However, correlated noise may be added to both. If correlated noise is added, the first machine learning model can, through training, acquire the effect of suppressing noise changes while simultaneously correcting blur.

[0034] In this embodiment, the first machine learning model performs blur correction on RAW images. Therefore, the ground truth image and training images are RAW images. If you want to perform blur correction on developed images, you can develop the ground truth image and training images.

[0035] In step S102, the calculation unit 113 generates output (information) regarding high-resolution based on the training image using the first machine learning model. Specifically, it generates output regarding high-resolution by inputting the training image into the first machine learning model. The output regarding high-resolution includes the high-resolution (blur-corrected) training image and the high-resolution residual map for the training image. When the high-resolution residual map is added to the training image, it becomes a high-resolution image. In this embodiment, the output regarding high-resolution is the high-resolution training image (estimated image). The first machine learning model is a CNN and has multiple weights (a set of first weights).

[0036] In step S103, the update unit 114 updates each weight of the first machine learning model based on the difference between the ground truth image and the estimated image.

[0037] In step S104, the update unit 114 determines whether the training of the first machine learning model is complete. If it is determined that the training is not complete (incomplete), the process returns to step S101. If it is determined that the training is complete, the information of the trained set of first weights is stored in the storage unit 111.

[0038] In this embodiment, the training of the first and second machine learning models is performed independently. When processing captured images, the first machine learning model is used to increase resolution, and then the second machine learning model is used to reduce noise. Therefore, it is conceivable to train the second machine learning model with training images based on the output related to resolution. However, if the two models are trained with correlation, the generation of training images requires calculations using the first machine learning model, and any changes to the first machine learning model necessitate retraining of the second machine learning model, thus increasing the training load.

[0039] The following describes the noise reduction and high-resolution processing of captured images performed by the image processing device 102, with reference to the flowchart in Figure 4. Figure 4 is a flowchart showing the generation of the output image in this embodiment.

[0040] In step S201, the acquisition unit 123 acquires the captured image and a set of weights for the machine learning model. In this embodiment, three sets of weights are acquired. Specifically, a first set of weights used in the first machine learning model, a second set of weights used in the second machine learning model, and a third set of weights used in the third machine learning model. The third machine learning model performs noise reduction before high-resolution enhancement by the first machine learning model. In this embodiment, the third machine learning model is identical to the second machine learning model. Therefore, the third set of weights is also the same as the second set of weights. However, the present invention is not limited thereto.

[0041] In step S202, the noise reduction unit 124 generates a first noise reduction output (information) based on the captured image using a third machine learning model. Specifically, the first noise reduction output is generated by inputting the captured image into the third machine learning model. In this embodiment, the first noise reduction output is the noise-reduced captured image (first image). Alternatively, a residual map of noise reduction for the captured image (first noise reduction residual map) may be generated instead of the first image. When the noise reduction residual map is added to the captured image, it becomes the first image. Therefore, the first noise reduction output is either the noise-reduced captured image or the noise reduction residual map for the captured image. In this step, a noise reduction method that does not use a machine learning model, such as NLM (Non-local means filter) or BM3D (Block-matching and 3D filtering), may also be used.

[0042] In step S203, the high-resolution unit 125 uses a first machine learning model to generate an output (information) related to high resolution based on first input data corresponding to the output related to first noise reduction. In this embodiment, the first input data is a first image. However, the present invention is not limited thereto. For example, the first input data may be data obtained by adding the residual map of the first noise reduction (output related to the first noise reduction) and the captured image, or by combining them in the channel direction. In this case, the first machine learning model is also trained using data in the same format as the first input data. By inputting the first input data into the first machine learning model, an output related to high resolution is generated. In this embodiment, the output related to high resolution is a first image (second image) that has been blur-corrected (high-resolution). Alternatively, a residual map for high resolution for the first image may be generated instead of the second image. When the residual map for high resolution is added to the first image, it becomes the second image. Therefore, the output related to high-resolution enhancement is either the first high-resolution image (the captured image with noise reduction and high resolution) or the residual map of the high-resolution enhancement applied to the first image.

[0043] In step S204, the calculation unit 126 generates a noise reduction residual map (first noise reduction residual map) for the captured image using the captured image and the first image. The first noise reduction residual map is generated by subtracting the captured image from the first image. Note that if the residual map has been generated in step S202, step S204 does not need to be executed. Also, step S204 may be executed before step S203.

[0044] In step S205, the calculation unit 126 generates a third image using the second image and the residual map of the first noise reduction. The third image is obtained by subtracting the residual map of the first noise reduction from the second image, and is an image in which only high-resolution enhancement has been performed on the captured image, canceling out the noise reduction in step S202. The third image becomes the second input data for the second machine learning model. As a result, as described before the explanation of this embodiment, it is possible to generate a high-resolution captured image while suppressing noise amplification. Since the noise in the third image has hardly changed from the captured image, the machine learning model can effectively reduce the noise. Also, since the blurred image has already been enhanced to high resolution, it is easier to distinguish between noise and the subject, and deformation of the subject due to noise reduction can also be suppressed. The third image may also be generated by adding the high-resolution residual map to the captured image. Alternatively, the third image may be generated by adding the high-resolution residual map to the first image and subtracting the residual map of the first noise reduction. Therefore, in this embodiment, the second input data (third image) is generated based on the output related to high resolution and the output related to the captured image or the first noise reduction.

[0045] When generating the third image, it is not necessary to completely negate the effect of noise reduction in step S202; some of the effect may be retained. Therefore, the third image is one in which the difference between the captured image in which high resolution was performed after noise reduction (the second added image) is greater than the difference between the captured image in which high resolution was performed only after noise reduction (the first added image) where the effect of noise reduction was negated and only high resolution was performed. The first added image is an image obtained by adding the captured image to the high resolution residual map (a map obtained by subtracting the first image from the second image) based on the output related to high resolution, and is equivalent to the image obtained by subtracting the noise reduction residual map from the second image. The second added image is an image obtained by adding the high resolution residual map to the captured image (the first image) which has been noise-reduced using the output related to the first noise reduction (the second image). The difference between the two images can be evaluated using MSE (Mean Squared Error) or MAE (Mean Absolute Error), etc. The characteristic of the third image can also be rephrased as the noise in the third image being closer to the noise in the captured image than the noise in the first image.

[0046] The third image is generated, for example, by multiplying the residual map of the first noise reduction by a coefficient greater than 0.5 and less than or equal to 1, and subtracting it from the second image. Note that in step S203, if the output related to high resolution is the residual map of high resolution for the first image, the third image can be generated by adding the high resolution residual map to the captured image.

[0047] In step S206, the noise reduction unit 124 generates an output related to the second noise reduction based on the second input data using a second machine learning model. Specifically, the output related to the second noise reduction is generated by inputting the second input data into the second machine learning model. In this embodiment, the output related to the second noise reduction is a fourth image, which is a noise-reduced and high-resolution captured image. However, the output related to the second noise reduction may also be a residual map of the noise reduction for the third image (second input data) (a residual map of the second noise reduction).

[0048] In step S207, the calculation unit 126 generates an output image based on the captured image, the output related to high resolution, and the output related to the second noise reduction. In this embodiment, the output related to high resolution is the third image, and the output related to the second noise reduction is the fourth image. By subtracting the fourth image from the third image, a residual map for the second noise reduction is generated, and the intensity of noise reduction can be adjusted by multiplying the residual map for the second noise reduction by a coefficient representing the intensity of noise reduction and adding it to the captured image. Similarly, by subtracting the captured image from the third image, a residual map for high resolution is generated, and the intensity of high resolution can be adjusted by multiplying the residual map for high resolution by a coefficient representing the intensity of high resolution and adding it to the captured image. The output image, after adjusting the intensity of both noise reduction and high resolution, is displayed on the display unit 127, and the user can freely adjust the intensity of both noise reduction and high resolution while checking the displayed output image. The intensity can be adjusted by changing the coefficients that represent the intensity of noise reduction and high resolution, respectively. With the above configuration, it is not necessary to recalculate the machine learning model each time the intensity of noise reduction and high resolution is changed, and intensity adjustment can be performed quickly with lightweight computation. If intensity adjustment is not necessary, the fourth image can be output as is.

[0049] As described above, the configuration of this embodiment makes it possible to increase the resolution and reduce the noise of captured images with high accuracy while suppressing the training load of the machine learning model.

[0050] The following describes desirable configurations that enhance the effects of the present invention.

[0051] The output regarding high resolution generated in step S203 should preferably be generated based on information about the optical system used to acquire the captured image. When various types of resolution degradation are mixed in the dataset during the training of the first machine learning model that performs high resolution, the first machine learning model will acquire an average high resolution for that degradation. Therefore, in order to perform high resolution that corresponds to degradation, the first machine learning model needs to generate an output regarding high resolution based on information about the optical system used to capture the captured image. The information about the optical system is information about the type of optical system used to acquire the captured image, the focal length, aperture value, focus distance at the time of imaging, the position of the pixels in the captured image relative to the optical axis, and the resolution performance of the optical system at the pixels of the captured image. The type of optical system is the type of imaging optical system 131, the type of optical low-pass filter, etc. Once the type of optical system, focal length, aperture value, focus distance, and the position of the pixels in the captured image relative to the optical axis are identified, the aberrations and blurring due to diffraction that occur in the optical system at that pixel position are uniquely determined. During the training and execution of the first machine learning model, by inputting information about the optical system as well as the image, it is possible to perform high-resolution enhancement appropriate for each pixel of the image. A map (with the same number of pixels in 2D as the image) containing values ​​representing information about the optical system in its channel components can be input to the first machine learning model by concatenating these maps in the channel direction of the image. Since the type of optical system, focal length, aperture value, and focus distance do not change with pixel position, they form a flat map with identical values. The position of pixels in the captured image relative to the optical axis will be represented by a map with gradients in two directions (horizontal and vertical, etc.) in different channel components. Alternatively, information about the resolution performance of the optical system at each pixel of the image may be used. A map representing resolution performance can be generated at frequencies where the blur acting on each pixel has an MTF (Modulation Transfer Function) greater than or equal to a predetermined value. Furthermore, information about resolution performance in two or more different directions (horizontal and vertical, meridional and sagittal) may be contained in different channel components.Furthermore, information regarding the optical system may be obtained directly from the metadata of the captured image, or it may be generated by performing calculations based on the metadata information to obtain the desired data format.

[0052] Furthermore, in step S201, it is desirable to select a set of first weights to be used in the first machine learning model from a set of multiple first weights based on information regarding high resolution. In addition, it is desirable to select a set of second weights to be used in the second machine learning model from a set of multiple second weights based on information regarding noise in the captured image. Rather than reducing all the noise across the ISO sensitivities (e.g., ISO 100 to 25600) that the image sensor 132 can take on with a single machine learning model, it is more accurate to divide the ISO sensitivity range into multiple ranges and reduce the noise using machine learning models trained individually for each range. The same applies to high resolution. If the imaging device 103 is an interchangeable lens camera, the imaging optical system 131 can take on multiple types. For example, accuracy can be improved by training a second machine learning model for each type of imaging optical system 131. For example, suppose a second machine learning model is trained individually according to the noise level, such as ISO 100-400, 400-1600, 1600-6400, and 6400-25600, generating four sets of second weights. Similarly, suppose a first machine learning model is trained individually according to the type of imaging optical system 131 (e.g., eight types), generating eight sets of first weights. In this case, the number of training runs for the machine learning model and the number of weight sets to retain are both 12. However, if noise reduction and resolution enhancement are performed with a single machine learning model, the number of training runs and the number of weight sets become 4 and 8, respectively, which is 32. Therefore, by separating noise reduction and resolution enhancement and performing them with separate machine learning models, the training load and the amount of data to retain can be reduced. The second weight sets are selected based on information about the noise in the captured image. Information regarding noise in the captured image includes information such as the type of image sensor 132, the ISO sensitivity at the time of imaging, the signal distribution (dispersion, etc.) of the optical black region of the captured image, and the compression quality of the captured image (Q value of JPEG compression, etc.). Similarly, the first set of weights is selected based on information regarding high resolution.Information regarding high resolution relates to the resolution performance of the captured image and includes information on the type of imaging optical system 131, the state of the imaging optical system 131 during imaging (focal length, aperture value, and focus distance, etc.), and the magnification ratio of the upscaling.

[0053] Furthermore, in step S205, it is desirable to modify the residual map of the first noise reduction based on the captured image and the output related to high resolution, and to generate the second input data based on the modified residual map of the first noise reduction. Since the noise generated by the image sensor 132 includes shot noise, the intensity of the noise changes depending on the amount of light. When flare in the blurred image is corrected by high resolution, regions where the brightness changes locally are created within the captured image. If the residual map of the first noise reduction is not modified, noise corresponding to the amount of light in the blurred state will exist in the region where the amount of light has changed after high resolution, resulting in a mismatch between the brightness of the subject and the intensity of the noise. Since this mismatch may reduce the effectiveness of the noise reduction of the second machine learning model, it is desirable to modify the residual map of the noise reduction in accordance with the local brightness changes in the captured image due to high resolution, and to resolve the mismatch between brightness and noise intensity. [Examples]

[0054] In this embodiment, only the configurations that differ from those in Embodiment 1 will be described, and the configurations that are the same as those in Embodiment 1 will not be described. In this embodiment, the target of high resolution is blurring generated in the optical system, and the target of noise reduction is noise generated in the image sensor.

[0055] Figure 5 is an external view of the image processing system 300 in this embodiment. Figure 6 is a block diagram of the image processing system 300 in this embodiment. The image processing system 300 includes a training device 301, an image processing device (second device) 302, a control device (first device) 303, and an imaging device 304.

[0056] The imaging device 304 comprises an imaging optical system 341, an image sensor 342, a storage unit 343, and a communication unit 344. The image captured by the imaging device 304 is transmitted to the control device 303.

[0057] The control device 303 comprises a storage unit 331, a communication unit (transmission unit) 332, a calculation unit (acquisition unit) 333, and a display unit 334. The control device 303 transmits the captured image and a request to perform noise reduction and high-resolution enhancement on the captured image to the image processing device 302. The control device 303 also receives the output of the image processing device 302 (the high-resolution captured image described later and the residual map of the second noise reduction) and generates an output image according to the user's instructions.

[0058] The image processing device 302 comprises a storage unit 321, a communication unit 322, an acquisition unit 323, a noise reduction unit 324, a high-resolution unit 325, and a calculation unit 326. The image processing device 302 performs noise reduction and high-resolution processing on captured images using a machine learning model that utilizes a set of weights trained by the training device 301.

[0059] The training device 301 comprises a storage unit 311, an acquisition unit 312, a calculation unit 313, and an update unit 314. The training device 301 performs training of a machine learning model according to the flowchart in Figure 3. In this embodiment, the training image and ground truth image used to train the first machine learning model differ from those in Embodiment 1. The training image is an image to which blur and noise have been added to the original image. However, the first machine learning model is input data in which the training image and a map obtained by multiplying the added noise by -1 (corresponding to the residual map of the first noise reduction) are coupled in the channel direction. The ground truth image is an image to which a smaller blur than that of the training image has been added (or no blur has been added) and noise of a similar strength correlated with the noise added to the training image has been added. By revealing the noise present in the training image and inputting it into the first machine learning model, the first machine learning model can easily separate the subject and noise in the training image, and thus obtain the effect of high resolution with suppressed noise changes. Note that the noise map does not necessarily need to be concatenated in the channel direction of the training image. Alternatively, the training image and the noise map can be input into separate convolutional layers, and the resulting feature maps can be concatenated in the channel direction.

[0060] The following describes the noise reduction and resolution enhancement of captured images performed by the control device 303 and the image processing device 302, with reference to Figure 7. Figure 7 is a flowchart showing the generation of the output image in this embodiment.

[0061] In step S301, the communication unit 332 transmits the captured image and a request to perform noise reduction and high-resolution enhancement on the captured image to the image processing device 302.

[0062] In step S302, the communication unit 322 acquires the captured image and a request to perform noise reduction and high-resolution enhancement on the captured image. The captured image may be stored in the storage unit 321 in advance, or it may be read from another recording medium.

[0063] In step S303, the acquisition unit 323 acquires a set of weights for the machine learning model. Specifically, the acquisition unit 323 acquires a set of first to third weights to be used for each of the first to third machine learning models.

[0064] In step S304, the noise reduction unit 324 generates an output related to the first noise reduction based on the captured image using a third machine learning model. In this embodiment, the output related to the first noise reduction is a residual map of the first noise reduction. The residual map of the first noise reduction is a component that cancels out the noise in the captured image.

[0065] In step S305, the calculation unit 326 combines the captured image and the first noise reduction residual map in the channel direction to generate the first input data.

[0066] In step S306, the high-resolution unit 325 generates an output related to high-resolution based on the first input data using the first machine learning model. In this embodiment, the output related to high-resolution is a third image, which is a captured image with high resolution without noise reduction. Unlike in Embodiment 1, in this embodiment, the first machine learning model can directly generate the third image by using the first input data, which includes the captured image and the residual map of the first noise reduction, and training the first machine learning model. The first input data may also be data obtained by combining an image obtained by adding the captured image and the residual map of the first noise reduction (the first image) and the residual map of the first noise reduction in the channel direction. Alternatively, it may be data obtained by combining the captured image and the first image in the channel direction. In these cases, the same format of input is used when training the first machine learning model. In this embodiment, the output for high resolution is generated using at least two of the following: the captured image, the captured image with noise reduction based on the output for the first noise reduction, and the residual map of the first noise reduction based on the output for the first noise reduction.

[0067] In step S307, the noise reduction unit 324 generates an output related to the second noise reduction based on the second input data using a second machine learning model. In this embodiment, the second input data is the third image, and the output related to the second noise reduction is the residual map of the second noise reduction. By adding the third image and the residual map of the second noise reduction, an image is obtained in which noise reduction and high resolution have been performed.

[0068] In step S308, the communication unit 322 transmits an output related to high resolution (third image) and an output related to second noise reduction (second noise reduction residual map) to the control device 303.

[0069] In step S309, the communication unit 332 acquires an output related to high resolution (third image) and an output related to second noise reduction (second noise reduction residual map).

[0070] In step S310, the calculation unit 333 generates an output image based on the captured image, the output related to high resolution (third image), and the output related to second noise reduction (second noise reduction residual map). The intensity of high resolution can be adjusted by changing the weights of the weighted average of the captured image and the third image. The intensity of noise reduction can be adjusted by adding the second noise reduction residual map, which is obtained by multiplying the weighted average of the captured image and the third image by a coefficient representing the intensity of noise reduction. The user can quickly adjust the intensity of high resolution and noise reduction while checking the output image displayed on the display unit 334.

[0071] As described above, the configuration of this embodiment makes it possible to increase the resolution and reduce the noise of captured images with high accuracy while suppressing the training load of the machine learning model.

[0072] This embodiment includes the following methods and configurations. (Method 1) A step of generating a first noise reduction output using the captured image, A step of using a first machine learning model to generate an output for high resolution based on first input data corresponding to the output for first noise reduction, A step of generating a second noise reduction output based on second input data corresponding to the output for high resolution using a second machine learning model, The process includes the step of generating an output image based on the output related to the second noise reduction, The second input data is an image in which the difference between the second and the first added image is greater than the difference between the second and the first added image. The first added image is an image obtained by adding the high-resolution residual map based on the output related to the high-resolution enhancement and the captured image. The image processing method is characterized in that the second added image is an image obtained by adding the high-resolution residual map and the noise-reduced captured image based on the output for the first noise reduction. (Method 2) The image processing method according to Method 1, characterized in that the output relating to the high-resolution enhancement is generated based on information about the optical system used when acquiring the captured image. (Method 3) The image processing method according to Method 2, characterized in that the information relating to the optical system is any of the following: the type of optical system, the focal length of the optical system, the aperture value of the optical system, the focus distance of the optical system, the position of the pixels of the captured image with respect to the optical axis of the optical system, and the resolution performance of the optical system at the pixels of the captured image. (Method 4) The image processing method according to any one of configurations 1 to 3, characterized in that the aforementioned high-resolution enhancement includes at least one of blur correction or upscaling. (Method 5) The image processing method according to any one of methods 1 to 4, characterized in that the high-resolution enhancement includes correction of blurring that occurs in the optical system used when acquiring the captured image. (Method 6) The image processing method according to any one of methods 1 to 5, characterized in that the output image is generated based on the captured image and the output related to high resolution. (Method 7) A step of selecting a first set of weights to be used in the first machine learning model based on information regarding high resolution, An image processing method according to any one of methods 1 to 6, further comprising the step of selecting a second set of weights to be used in the second machine learning model based on information regarding the noise of the captured image. (Method 8) The image processing method according to any one of methods 1 to 7, characterized in that the output relating to the first noise reduction is generated using a machine learning model. (Method 9) The image processing method according to method 8, characterized in that the machine learning model is identical to the second machine learning model. (Method 10) The image processing method according to any one of methods 1 to 9, characterized in that the second input data is generated using a first noise reduction residual map based on the first noise reduction output, which has been modified based on the captured image and the output related to high resolution. (Method 11) The image processing method according to any one of methods 1 to 10, characterized in that the second input data is generated based on the output relating to high resolution and the captured image or the output relating to the first noise reduction. (Method 12) The image processing method according to any one of methods 1 to 11, characterized in that the output for high resolution is generated using at least two of the captured image, the captured image with noise reduction based on the output for the first noise reduction, and the residual map of the first noise reduction based on the output for the first noise reduction. (Composition 1) A first generation unit that generates a first noise reduction output using an captured image, A first generation unit generates an output for high resolution based on first input data corresponding to the output for first noise reduction, using a first machine learning model that performs high resolution enhancement. A third generation unit generates a second noise reduction output based on second input data corresponding to the output for high resolution, using a second machine learning model for noise reduction. The system includes an output unit that outputs an output image based on the output related to the second noise reduction, The second input data is an image in which the difference between the second added image and the first added image is greater than the difference between the second added image and the first added image. The first added image is an image obtained by adding the high-resolution residual map based on the output related to the high-resolution enhancement and the captured image. The image processing apparatus is characterized in that the second added image is an image obtained by adding the high-resolution residual map and the noise-reduced captured image based on the output for the first noise reduction. (Configuration 2) An image processing system having a first device and a second device, The first apparatus described above, A transmitting unit that transmits a request for the execution of processing to the second device, The system includes an acquisition unit that acquires an output image using the output acquired from the second device, The second apparatus described above is A first generation unit that generates a first noise reduction output using an captured image, A first generation unit generates an output for high resolution based on first input data corresponding to the output for first noise reduction, using a first machine learning model that performs high resolution enhancement. A third generation unit generates a second noise reduction output based on second input data corresponding to the output for high resolution, using a second machine learning model for noise reduction. The system comprises an output unit that outputs the output based on the output relating to the second noise reduction, The second input data is an image in which the difference between the second added image and the first added image is greater than the difference between the second added image and the first added image. The first added image is an image obtained by adding the high-resolution residual map based on the output related to the high-resolution enhancement and the captured image. The image processing system is characterized in that the second added image is an image obtained by adding the high-resolution residual map and the noise-reduced captured image based on the output for the first noise reduction. (Composition 3) A program characterized by causing a computer to execute the image processing method described in any one of the configurations of Method 1 to 12. [Other examples] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0073] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its essence. [Explanation of Symbols]

[0074] 102 Image Processing Device 124 Noise Reduction Section 125 High-resolution section 126 Arithmetic section

Claims

1. A step of generating a first noise reduction output using the captured image, A step of using a first machine learning model to generate an output for blur correction based on first input data corresponding to the output for the first noise reduction, A step of using a second machine learning model to generate a second noise reduction output based on second input data corresponding to the output for blur correction, The process includes the step of generating an output image based on the output related to the second noise reduction, The second input data is an image in which the difference between the second added image and the first added image is greater than the difference between the second added image and the first added image. The first added image is an image obtained by adding the residual map of the blur correction based on the output related to the blur correction and the captured image. The image processing method is characterized in that the second added image is an image obtained by adding the residual map of the blur correction and the noise-reduced captured image based on the output of the first noise reduction.

2. The image processing method according to claim 1, characterized in that the output relating to the correction of blur is generated based on information about the optical system used when acquiring the captured image.

3. The aforementioned captured image is an image obtained by imaging using an optical system. The image processing method according to claim 2, characterized in that the information relating to the optical system is any of the following: the type of optical system, the focal length of the optical system, the aperture value of the optical system, the focus distance of the optical system, the position of the pixels of the captured image with respect to the optical axis of the optical system, and the resolution performance of the optical system at the pixels of the captured image.

4. The image processing method according to claim 1 or 2, characterized in that the blur correction includes correction of blur occurring in the optical system used when acquiring the captured image.

5. The image processing method according to claim 1 or 2, characterized in that the output image is generated based on the captured image and the output related to the correction of blur.

6. A step of selecting a first set of weights to be used in the first machine learning model based on information regarding blur correction, The image processing method according to claim 1 or 2, further comprising the step of selecting a second set of weights to be used in the second machine learning model based on information regarding the noise of the captured image.

7. The image processing method according to claim 1 or 2, characterized in that the output for the first noise reduction is a first image generated by noise reduction from the captured image.

8. The image processing method according to claim 7, characterized in that the first input data is the first image.

9. The second input data is generated using a first noise reduction residual map based on the output of the first noise reduction, The image processing method according to claim 1 or 2, characterized in that the first noise reduction residual map is modified based on the output for blur correction and the captured image so as to resolve the mismatch between the brightness and the noise intensity corresponding to the change in brightness of the captured image due to the blur correction.

10. The image processing method according to claim 1 or 2, characterized in that the second input data is generated based on the output relating to blur correction and the captured image or the output relating to the first noise reduction.

11. The image processing method according to claim 1 or 2, characterized in that the output for blur correction is generated using at least two of the following: the captured image, the captured image with noise reduction based on the output for the first noise reduction, and the residual map of the first noise reduction based on the output for the first noise reduction.

12. The image processing method according to claim 7, characterized in that the output for the blur correction is a second image generated by performing blur correction on the first image.

13. A first generation unit that generates an output related to first noise reduction using an captured image, A second generation unit generates an output for blur correction based on first input data corresponding to the output for noise reduction, using a first machine learning model that performs blur correction, A third generation unit generates a second noise reduction output based on second input data corresponding to the output for blur correction, using a second machine learning model for noise reduction. The system includes an output unit that outputs an output image based on the output related to the second noise reduction, The second input data is an image in which the difference between the second added image and the first added image is greater than the difference between the second added image and the first added image. The first added image is an image obtained by adding the residual map of the blur correction based on the output related to the blur correction and the captured image. The image processing apparatus is characterized in that the second added image is an image obtained by adding the residual map of the blur correction and the noise-reduced captured image based on the output of the first noise reduction.

14. An image processing system comprising an image processing device according to claim 13 and a control device capable of communicating with the image processing device, The control device has a transmission means for transmitting a request to the image processing device to perform processing on the captured image, The image processing system is characterized in that the image processing device has receiving means for receiving the request and performs processing on the captured image in response to the request.

15. A program characterized by causing a computer to execute the image processing method described in claim 1 or 2.

Images