Image processing method and image processing device

Abstract

To suppress changes in brightness and color in image estimation using a machine learning model.SOLUTION: An image processing method includes a first step for generating a first residual image based on a first image using a machine learning model and a second step for generating a third image on the basis of the absolute value of the first residual image and a fourth image. When the sum of the fourth image and the first residual image is set as a second image, in the second step, a signal value of each pixel in the third image is made to be closer to the signal values of the corresponding pixels in the fourth image than to those of the corresponding pixels in the second image, as the non-zero absolute values of the signal values of the corresponding pixels in the first residual image become smaller.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to an image processing method for estimating an image using a machine learning model. [Background technology]

[0002] Patent Document 1 discloses a method for generating a high-resolution image by up-sampling from a low-resolution image using a neural network (NN). The NN generates multiple high-resolution residual sub-images and rearranges their pixels to obtain a high-resolution residual image. Then, a high-resolution image with a high sense of resolution is generated by taking the sum of a base high-resolution image obtained by bicubic up-sampling the low-resolution image and the high-resolution residual image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2022-536807 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the method disclosed in Patent Document 1 has a problem in that unintended luminance changes unrelated to upsampling occur in the high-resolution image. For example, periodic patterns may occur in the high-resolution image. The mechanism behind this problem will be described later.

[0005] The present invention provides an image processing method and an image processing device that can suppress unintended changes in luminance and color in image estimation using a machine learning model. [Means for solving the problem]

[0006] An image processing method according to one aspect of the present invention includes a first step of generating a first residual image based on a first image using a machine learning model, and a second step of generating a third image based on the absolute value of the first residual image and a fourth image. When the sum of the fourth image and the first residual image is the second image, in the second step, the signal value of each pixel in the third image is made closer to the signal value of the pixel corresponding to each pixel in the fourth image than to the signal value of the pixel corresponding to each pixel in the second image as the non-zero absolute value of the signal value of the pixel corresponding to each pixel in the first residual image is smaller. Note that a program for causing a computer to execute processing according to the image processing method also constitutes another aspect of the present invention.

[0007] An image processing device according to another aspect of the present invention includes a first generating means for generating a first residual image based on a first image using a machine learning model, and a second generating means for generating a third image based on an absolute value of the first residual image and a fourth image. When the sum of the fourth image and the first residual image is the second image, the second generating means makes the signal value of each pixel in the third image closer to the signal value of the pixel corresponding to each pixel in the fourth image than the signal value of the pixel corresponding to each pixel in the second image as the non-zero absolute value of the signal value of the pixel corresponding to each pixel in the first residual image becomes smaller. Note that an imaging device including the image processing device also constitutes another aspect of the present invention. Effect of the Invention

[0008] According to the present invention, unintended changes in luminance and color can be suppressed in image estimation using a machine learning model. [Brief description of the drawings]

[0009] [Figure 1] 8A to 8C are diagrams showing a flow of generating a third image in the first embodiment. [Diagram 2] FIG. 1 is a block diagram showing a configuration of an image processing system according to a first embodiment. [Diagram 3]FIG. 1 is an external view of an image processing system according to a first embodiment. [Figure 4] 1 is a flowchart showing training of a machine learning model in Examples 1 and 2. [Diagram 5] FIG. 2 is a diagram showing the structure of a machine learning model in the first embodiment. [Figure 6] 10 is a flowchart showing a process for generating a third image in the first embodiment. [Figure 7] FIG. 4 is a diagram showing an example of nonlinear processing in the first embodiment. [Figure 8] FIG. 11 is a block diagram showing the configuration of an image processing system according to a second embodiment. [Figure 9] FIG. 11 is an external view of an image processing system according to a second embodiment. [Figure 10] FIG. 11 is a diagram showing the structure of a machine learning model in the second embodiment. [Figure 11] FIG. 10 is a diagram showing a flow of generating a third image in the second embodiment. [Figure 12] 10 is a flowchart showing a process for generating a third image in the second embodiment. [Figure 13] FIG. 11 is a diagram showing an example of nonlinear processing in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Before describing the embodiments in detail, an overview of the embodiments will be given.

[0011] In training a machine learning model that performs image estimation, the difference between the estimated image and the desired ground truth image is expressed by a loss function such as MSE (Mean Square Error). By minimizing the value of the loss function, multiple model parameters of the machine learning model (weights and biases of each layer, etc.: hereinafter simply referred to as model parameters) are determined. In general, in image estimation tasks, a solution is not uniquely determined for the input data, so even if the loss function is minimized, the value of the loss function does not become 0, and a finite error remains. For example, if the task is to correct the blur of an image, the uncorrected or overcorrected components of the blur remain as finite values ​​of the loss function even after training. Blurring affects areas where structures such as edges exist, but does not change the appearance of the image in flat areas where the signal does not change. In other words, the desired behavior of a machine learning model that performs blur correction is to not change the signal value in flat areas of the image.

[0012] However, image processing using machine learning models can generate brightness and color variations that do not actually exist in flat areas of the estimated image. If the variation is an order of magnitude smaller than the components related to the task (such as the blur residuals and overcorrection described above), the variation will remain even after minimization. This can lead to unintended brightness and color variations that are unrelated to the task in image estimation using machine learning models.

[0013] In order to suppress this change, in the embodiment, the absolute value of the first residual image (the blur correction component when the task is blur correction) generated using a machine learning model is used. Since unintended changes in brightness and color have a small effect on the loss function, the absolute value of the residual is inevitably small. Therefore, by weakening the effect of pixels with small absolute values ​​in the first residual image, it is possible to obtain an estimated image (third image) in which unintended changes in brightness and color are suppressed. Specific examples of the method will be described in the following embodiments 1 and 2.

[0014] [Example 1] An image processing system according to the first embodiment will be described. The task executed by the machine learning model in the first embodiment is upsampling (high resolution). In the first embodiment, the entire image is upsampled, but a part of the image may be cropped and upsampled like digital zoom.

[0015] Similar effects can also be obtained for tasks other than upsampling. Examples of other tasks include correction of degradation, increasing contrast, demosaicing, expanding dynamic range, increasing gradation, changing lighting, changing the shape of defocus blur, changing focus distance, changing depth of field, etc. Degradation here refers to blur, noise, haze, turbulence (air fluctuation), etc. Blur includes optical system aberration, diffraction, defocus blur, and shaking. Noise includes noise generated by the image sensor and noise generated by data compression. Changing the shape of defocus blur means changing a blurred image with double line blur to a blurred image with flat light distribution in the defocus area of ​​an image, changing a blurred image with a chipped shape due to vignetting to a circular blurred image, etc.

[0016] Fig. 2 shows the configuration of the image processing system 100 in the embodiment 1. Fig. 3 shows the external appearance of the image processing system 100. The image processing system 100 includes a training device 101 and an imaging device 102.

[0017] The imaging device 102 includes an optical system 121, an image sensor 122, a storage unit 123, a communication unit 124, an acquisition unit 125, a calculation unit 126, and a display unit 127. The optical system 121 collects light incident from a subject space to form a subject image. The image sensor 122 receives the subject image and performs photoelectric conversion to generate a captured image.

[0018] The calculation unit 126 performs predetermined image processing on the captured image generated by the imaging element 122. The storage unit 123 stores the captured image after image processing.

[0019] Furthermore, the calculation unit 126 generates an output image (corresponding to a third image described later) by performing upsampling using a machine learning model on a captured image (corresponding to a first image described later) as an input image generated by the image sensor 122. The captured image to be upsampled may be an undeveloped RAW image or a developed image, but here only the luminance component of the developed image is upsampled. Upsampling may be performed automatically when capturing an image, or may be performed at any timing designated by the user for a saved captured image.

[0020] The model parameters of the machine learning model are trained in advance by the training device 101, and are acquired in advance by the imaging device 102 via the communication unit 112 of the training device 101 and the communication unit 124 of the imaging device 102, and stored in the storage unit 123. Details regarding training of the machine learning model and upsampling by the trained machine learning model will be described later. An output image generated by upsampling is displayed on the display unit 127, and is stored in the storage unit 123.

[0021] A method for training a machine learning model (a method for determining model parameters) executed by the training device 101 will be described with reference to the flowchart in Fig. 4. The training device 101, which is configured by a computer, has a storage unit 111, a communication unit 112, an acquisition unit 113, a calculation unit 114, and an update unit 115. The acquisition unit 113, the calculation unit 114, and the update unit 115 execute the following processes (processing steps and update steps) according to a program. The acquisition unit 113 and the calculation unit 114 configure a processing means, and the update unit 115 corresponds to an update means.

[0022] In step S101, the acquisition unit 113 acquires one or more pairs of training input data and a correct answer image from the storage unit 111. The training input data is data to be input to a machine learning model during training. The training input data in the first embodiment is a low-resolution image (an input image during training). However, the training input data may include other images and maps in addition to the low-resolution image. For example, the training input data may include a map indicating the strength of degradation (noise, etc.) acting on the low-resolution image. The case where a map is included will be described in the second embodiment.

[0023] The low-resolution image has the number of pixels in the vertical, horizontal and channel directions (h, w, c). In the first embodiment, since only the luminance component is upsampled, c = 1. When the image has multiple color components, for example, when the image has three color components of RGB (Red, Green, Blue), c = 3.

[0024] The correct image is a high-resolution image in which the same subject as the low-resolution image exists. In the first embodiment, the number of pixels is quadrupled (twice vertically and twice horizontally) by upsampling, so the number of pixels of the high-resolution image is (2h, 2w, c=1). However, the magnification of upsampling is not limited to this. A pair of low-resolution and high-resolution images can be prepared by real photography, imaging simulation, CG (Computer Graphics), etc.

[0025] In step S102, the calculation unit 114 inputs training input data to the machine learning model to generate an upsampled estimated image. In the first embodiment, the CNN (Convolutional Neural Network) shown in Figs. 5(A) and (B) is used as the machine learning model. However, the type of the machine learning model is not limited to this, and MLP (Multi-Layer Perceptron), ViT (Vision Transformer), GAN (Generative Adversarial Network), diffusion model, etc. may be used. In addition, the model parameters refer to multiple variables determined by training in the machine learning model. Specifically, they are the weights and biases of the kernel (filter) of the convolution layer, the weights and biases of the full connection layer, etc.

[0026] In the first embodiment, a first residual image 211 is generated from training input data 201 by the CNN shown in FIG. 5(A). The training input data 201 is converted into a feature map through a convolution layer and an activation function. In the first embodiment, ReLU (Rectified Linear Unit) is used as the activation function, but Leaky ReLU, a sigmoid function, or the like may be used. In the convolution layer, there are 48 kernels and biases with vertical, horizontal, and channel sizes of (3, 3, 1). As a result, a feature map having the number of pixels of (h, w, 48) is generated. The feature map is input to a residual block. Note that the feature map means output data of each layer generated during the computation of the machine learning model. Also, the size of the kernel and the number of kernels and biases are not limited to those described above.

[0027] The residual block is shown in Fig. 5(B). The input 203 of the residual block is passed through a convolutional layer twice (with ReLU applied along the way) that has 48 kernels and biases of size (3,3,48), and the element-by-element sum of the result and the input 203 is obtained. The output 204 of the residual block is then generated by applying ReLU to the obtained sum.

[0028] There are multiple residual blocks, and in Example 1, 10 residual blocks are connected. However, the number of residual blocks is not limited to this. Each feature map generated by the residual block has a pixel count of (h, w, 48). The final convolutional layer has four kernels and biases of size (3, 3, 48).

[0029] As a result of the calculation by the calculation unit 114, a residual map 202 having the number of pixels of (h, w, 4) is generated. The number of channels of the residual map 202 corresponds to the upsampling magnification. The calculation unit 114 rearranges the pixels of the residual map 202 to generate a first residual image 211 having the number of pixels of (2h, 2w, 1). The components of the first channel of the residual map 202 are stored in pixels of the first residual image 211 with odd-numbered vertical and horizontal indices. The components of the second channel are stored in pixels with odd-numbered vertical indices and even-numbered horizontal indices. The components of the third channel are stored in pixels with even-numbered vertical indices and odd-numbered horizontal indices. The components of the fourth channel are stored in pixels with even-numbered vertical and horizontal indices.

[0030] The calculation unit 114 generates an estimated image by taking the sum of a bicubic enlarged image obtained by enlarging the low-resolution image to (2h, 2w, 1) by four times the number of pixels by bicubic interpolation and the first residual image 211. However, the low-resolution image may be enlarged by a rule-based process that does not use other machine learning models, such as bilinear interpolation or nearest neighbor interpolation. Note that the sum of A and B includes multiplying A or B by -1 and taking the difference. Also, the target to be estimated by the machine learning model may be an estimated image directly, rather than a residual.

[0031] In step S103, the update unit 115 updates the model parameters of the machine learning model based on the difference between the estimated image and the correct image (high-resolution image). In the first embodiment, the MSE (Mean Square Error) between the estimated image and the high-resolution image is used as the loss function, but is not limited thereto, and the MAE (Mean Absolute Error) or the Perceptual Loss may be used. For the update, the backpropagation method or the like is used.

[0032] In step S104, the update unit 115 determines whether the training of the machine learning model is complete. If the training is not complete, the process returns to step S101 to obtain one or more new sets of training input data and a ground truth image. If the training is complete, the model parameters are stored in the storage unit 111.

[0033] Next, upsampling using a trained CNN (machine learning model) executed in the imaging device 102 will be described with reference to Fig. 1 showing the flow of the process and the flowchart in Fig. 6. In the imaging device 102, an acquisition unit 125 and a calculation unit 126 configured by a computer (image processing device) execute the following processes (first step and second step) according to a program. The acquisition unit 125 and the calculation unit 126 configure a first generation means and a second generation means.

[0034] In step S201, the acquisition unit 125 acquires a first image 221. The first image 221 is a luminance component of a developed captured image, as in training. The number of pixels in the vertical, horizontal, and channel directions in the first image 221 is represented by (H, W, C), and in the first embodiment, C=1.

[0035] A partial region of the captured image may be the first image 221. The entire captured image can be upsampled by repeating a series of processes described below while changing the position of the partial region and synthesizing the results. When the partial region is the first image 221, only step S202 may be executed for each partial region, and the results may be synthesized before executing step S203 and subsequent steps. It is preferable that multiple partial regions have overlapping areas so that the division boundary line is not visible after upsampling. It is preferable to average the overlapping areas after upsampling, or to adopt one of the results.

[0036] In step S202, the calculation unit 126 generates a first residual image 231 based on the first image 221 using a machine learning model (CNN). As in training, the calculation unit 126 uses a CNN having the structure shown in Figs. 5(A) and (B), and acquires model parameters trained by the training device 101 from the storage unit 123 for use. The input data input to the CNN is the first image 221. The output data output from the CNN is the first residual image 231. The number of pixels in the vertical, horizontal, and channel directions of the first residual image 231 is (2H, 2W, C), and C=1 in the first embodiment. The calculation unit 126 obtains a second image, which is a high-resolution upsampled image of the first image 221, by adding a fourth image 224, which is obtained by enlarging the first image 221 to (2H, 2W, C=1) using bicubic interpolation (the same process as that performed on the low-resolution image during training), and a first residual image 231.

[0037] Here, the second image may have unintended luminance changes that are unrelated to the upsampling, and may also have color changes if the first image 221 has multiple color components. As described before in the first embodiment, components in the loss function during training that are sufficiently smaller in order than the components related to the task performed by the machine learning model (high resolution in the first embodiment) may remain after training. These small components cause unintended luminance and color changes.

[0038] The CNN with the structure shown in FIG. 5 rearranges the residual map 202 with the number of pixels (H, W, 4) to generate the first residual image 211 with (2H, 2W, 1). In this way, in a configuration in which the number of pixels in the vertical or horizontal direction changes, unintended changes in brightness and color are likely to occur. Since the estimation error is likely to vary from channel to channel, minute errors δ1, δ2, δ3, and δ4 may occur in each channel of the residual map 202. When the residual map 202 with such errors is rearranged to generate the first residual image 211, pixels with errors δ1 to δ4 are arranged in adjacent 2×2 pixel units, so a periodic pattern occurs. This pattern is particularly noticeable in flat areas where the signal value does not change and in gradation areas where the change is gradual. In order to suppress such unintended changes in brightness and color, the following process is executed.

[0039] In step S203, the calculation unit 126 generates the second residual image 232 by nonlinear processing (nonlinear transformation) based on the absolute value of the first residual image 231. The nonlinear processing executed here is a process in which, among the multiple pixels included in the first residual image 231, the pixel with a signal value other than 0 has a larger residual attenuation rate as the absolute value of the signal value decreases.

[0040] In the first embodiment, threshold processing is used as the nonlinear processing. Examples include hard thresholding shown in Fig. 7(A) and soft thresholding shown in Fig. 7(B). In Fig. 7(A) and (B), the horizontal axis indicates the input signal value, the vertical axis indicates the output signal value, the dashed line indicates the identity transformation, the solid line indicates the hard or soft thresholding, and the dashed line indicates the threshold of the thresholding. When the input signal value is multiplied by s in , the output signal value is s out , the threshold value is s thr Then, hard thresholding is expressed by the following equation (1). [Number 1] JPEG2025052803000002.jpg22127

[0041] Note that the presence or absence of an equal sign between the two conditional expressions may be reversed.

[0042] Moreover, the soft thresholding is expressed by the following equation (2). [Number 2] JPEG2025052803000003.jpg36162

[0043] The signal value of each pixel of the third image 223, which is the sum of the second residual image 232 generated by hard or soft thresholding and the fourth image 224, is a value between the signal values ​​of pixels in the second image and the fourth image 224 that correspond to each pixel of the third image 223. This prevents the occurrence of new problems other than unintended changes in brightness and color suppressed by nonlinear processing.

[0044] However, in hard thresholding, residuals whose absolute values ​​are greater than the threshold value remain unchanged, so while the effect of the task (high resolution in the first embodiment) is unlikely to decrease, there is a possibility that discontinuous steps in signal values ​​may occur in the second residual image 232. In soft thresholding, no steps occur in the second residual image 232, but the residual approaches 0 by the amount of the threshold value, so the effect of the task decreases slightly.

[0045] For this reason, when using soft thresholding, it is desirable to further scale the residual as shown in FIG. 7(C). st Then, the soft thresholding with scaling is given by the following equation (3). [Number 3] JPEG2025052803000004.jpg17124

[0046] Here, s0 is a value that controls the scaling factor that is applied (multiplied) to the residual image obtained by performing soft thresholding on the first residual image 231.

[0047] The scaling factor is the threshold s thrIt is a coefficient greater than 1 based on. For example, s0 may be set to the signal value range that the first image 221 can have (the difference between the maximum signal value and the minimum signal value). For example, the signal value range is 255 for an 8-bit developed image, and 30719 for a RAW image with a maximum signal value of 32767 and an optical black level of 2048. If the signal value is normalized to the range from 0 to 1, the signal value range is 1.

[0048] Setting s0 to the signal value width ensures that the effect of the task by the second residual image 232 is equal to or smaller than that of the first residual image 231, thereby preventing the effect from becoming excessive. This causes the signal value of each pixel of the third image 223 to be a value between the signal values ​​of the pixels in the second image and the fourth image 224 that correspond to each pixel of the third image 223.

[0049] Other processing may be executed as long as it is nonlinear processing in which the residual attenuates at a greater rate as the absolute value of the component of the first residual image 231 decreases. The other processing will be described in Example 2. By executing such nonlinear processing on the first residual image 231, it is possible to generate the second residual image 232 in which unintended changes in brightness and color are suppressed while suppressing a decrease in the effect of the task (high resolution).

[0050] In step S204, the calculation unit 126 generates the third image 223 by taking the sum of the fourth image 224 and the second residual image 232. The third image 223 is a high-resolution image obtained by upsampling the first image 221, and unintended changes in brightness and color are suppressed. By using the second residual image 232, the signal value of each pixel of the third image 223 becomes closer to the signal value of the pixel corresponding to each pixel of the third image in the fourth image 224 than the signal value of the pixel corresponding to each pixel of the third image in the second image, as the absolute value of the signal value of the pixel corresponding to each pixel of the third image in the first residual image 231 that is not 0 becomes smaller. The number of pixels of the third image 223 is (2H, 2W, C). In the first embodiment, only the brightness component is upsampled, so C=1.

[0051] By enlarging the color difference components of the captured image by four times the number of pixels using bicubic interpolation or the like and combining it with the third image 223, an upsampled image having a plurality of color components can be obtained.

[0052] In the first embodiment, such a configuration is adopted because human vision is insensitive to color resolution and the processing can be accelerated by limiting the upsampling by CNN to luminance only. However, an image having multiple color components may be upsampled by CNN. In the first embodiment, the number of pixels in the vertical and horizontal directions is increased in the machine learning model, but the first image input to the machine learning model may be an image enlarged by bicubic interpolation or the like.

[0053] In addition, since unintended changes in brightness or color are noticeable in areas where the change in the signal value of the image is small, the effect of the nonlinear processing in step S203 may be limited to flat parts or gradation parts. This can be achieved by acquiring information about the edges of the fourth image 224 by spatial differentiation, edge extraction, or the like, and generating the third image 223 based on the information about the edges. It is preferable to generate the second residual image 232 by limiting the nonlinear processing in step S203 only to non-edge areas. Alternatively, in step S204, the third image 223 may be generated by adding the second residual image 232 to only the non-edge areas of the fourth image 224 and the first residual image 231 to the edge areas.

[0054] Furthermore, since unintended changes in luminance and color are noticeable in areas (dark areas) where the luminance value of the image is low, the effect of the nonlinear processing in step S203 may be limited to dark areas. This can be achieved by generating the third image 223 based on the magnitude of the signal value of each pixel of the fourth image 224. The nonlinear processing in step S203 may be limited to only the dark areas of the fourth image 224 to generate the second residual image 232. Alternatively, in step S204, the third image 223 may be generated by adding the second residual image 232 only to the dark areas of the fourth image 224 and the first residual image 231 to the other areas. In particular, when the first image 221 is an undeveloped RAW image, unintended changes in luminance and color in dark areas are emphasized by gamma correction during development, making them more noticeable.

[0055] Furthermore, if the number of bits of the model parameters is smaller during image estimation using a trained machine learning model than during training of the machine learning model, unintended changes in brightness and color may worsen due to quantization errors. For this reason, if the number of bits of at least one or more model parameters of the machine learning model that generates the first residual image 231 is smaller than during training, it is easier to obtain an effect in the processing from step S203 onwards. Reducing the number of bits during image estimation is performed when it is desired to perform lightweight image estimation processing using an ASIC (Application Specific Integrated Circuit) or the like.

[0056] In addition, the image estimation process may be made lighter by reducing the bit depth of some of the model parameters in each layer of the machine learning model or some of the operations such as convolution executed in each layer, but in this case as well, unintended changes in brightness and color may worsen. Therefore, when the machine learning model has a configuration including a plurality of first layers having a first bit number and a plurality of second layers having a second bit number smaller than the first bit number, the effects are likely to be obtained in the processes from step S203 onwards.

[0057] Furthermore, the non-linear processing based on the absolute value of the first residual image 231 in step S203 includes processing that can obtain results similar to those of the non-linear processing based on absolute values, even if the absolute values ​​are not directly taken.

[0058] For example, instead of taking the absolute value, the positive and negative components of the first residual image 231 may be divided into two and nonlinearly processed. This will be described using an example in which soft thresholding is performed on the first residual image 231. The second residual image 232 is obtained by subtracting a threshold (positive here) from the first residual image 231, then performing ReLU, and subtracting a threshold from -1 times the first residual image 231, then performing ReLU, and multiplying by -1. In this manner, the second residual image 232 can be generated without directly taking the absolute value of the first residual image 231.

[0059] As described above, according to this embodiment, it is possible to provide an image processing method and an image processing device (acquisition unit 125 and calculation unit 126) capable of suppressing unintended changes in brightness and color in image estimation using a machine learning model.

[0060] [Example 2] An image processing system according to a second embodiment of the present invention will be described. The task of image estimation executed by the machine learning model in the second embodiment is blur correction. The correction targets are blur caused by aberration and diffraction occurring in the optical system. However, the same effect can be obtained for other blurs such as defocus blur and shaking. The same effect can also be obtained for tasks other than blur correction.

[0061] Fig. 8 shows the configuration of an image processing system 300 in Example 2. Fig. 9 shows the external appearance of the image processing system 300. The image processing system 300 includes a training device 301, a blur correction device 302, an imaging device 303, and a lens device 304.

[0062] The imaging device 303 is a lens-interchangeable imaging device, and multiple types of lens devices 304 can be connected. The subject image formed by the lens device 304 is photoelectrically converted by the imaging element 331 to obtain a captured image. The captured image is affected by aberration and blurring due to diffraction occurring in the optical system 341. The calculation unit 334 performs necessary processing on the captured image, such as removing defective pixels. The storage unit 332 stores the captured image that has been processed by the calculation unit 334.

[0063] The blur correction device 302 as an image processing device corrects blur in a captured image using a machine learning model to generate a blur-corrected third image. In the second embodiment, blur correction is performed on an undeveloped RAW image. However, blur correction may be performed on a developed image. The model parameters of the machine learning model are trained by the training device 301, and are acquired in advance by the blur correction device 302 via the communication unit 312 of the training device 301 and the communication unit 322 of the blur correction device 302, and are stored in the storage unit 321. Training of the machine learning model and blur correction using the trained model parameters will be described later.

[0064] In addition, the third image that has been subjected to blur correction undergoes other necessary processing in the calculation unit 324, and is stored in the memory unit 321 of the blur correction device 302 or the memory unit 332 of the imaging device 303, or is displayed on the display unit 325 of the blur correction device 302 or the display unit 335 of the imaging device 303.

[0065] Next, a method for training a machine learning model executed by the training device 301 will be described with reference to the flowchart of Fig. 4, which is also used in the first embodiment. The training device 301, which is configured by a computer, has a storage unit 311, a communication unit 312, an acquisition unit 313, a calculation unit 314, and an update unit 315. The acquisition unit 313, the calculation unit 314, and the update unit 315 execute the following processes (processing step and update step) according to a program. The acquisition unit 313 and the calculation unit 314 configure a processing means, and the update unit 315 corresponds to an update means. In this embodiment, a CNN is used as the machine learning model, but other methods may be used.

[0066] In step S101, the acquisition unit 313 acquires one or more pairs of training input data and a correct image from the storage unit 311. The training input data includes a blurred image (input image during training) and a first map. The blurred image is an image that is degraded by blur to be corrected. The correct image is an image that includes the same subject as the corresponding blurred image and has less blur (or no blur) than the blurred image. The blurred image and the correct image are undeveloped RAW images. The blurred image and the correct image can be prepared by actual shooting, imaging simulation, CG, or the like.

[0067] In the second embodiment, correction for all blurring occurring in the optical system 341 is learned all at once. For this reason, blurred images including various blurring occurring in the optical system 341 and corresponding correct images are stored in the storage unit 311. The blurring of the optical system 341 is determined by the state of the optical system 341 (focal length, aperture value, focus distance, etc.) and the vertical and horizontal image plane coordinates, etc. The focus distance is the distance between the subject on which the optical system 341 is focused and the image sensor 331 or the optical system 341.

[0068] Furthermore, since the imaging device 303 in the second embodiment is of an interchangeable lens type, the optical system 341 may be connected to a plurality of types of imaging devices having imaging elements of different sizes and numbers of pixels. For this reason, the color acquired by the imaging element, the pixel pitch of the imaging element, and the action of the optical low pass filter may also change. In the second embodiment, as an example, the imaging elements of the imaging devices connectable to the optical system 341 are all in a Bayer array of RGB, and the colors are fixed to RGB. However, the color filter array is not limited to this.

[0069] The multiple blurred images stored in the memory unit 311 include blurs that vary in various ways within the range of values ​​that the image plane coordinates, pixel pitch of the image sensor 331, and optical low-pass filter can take, in addition to the focal length, aperture value, and focus distance of the optical system 341.

[0070] In the second embodiment, the first map is a map that specifies the blur that acts on the blurred image. The first map has six values ​​in each channel, the focal length of the optical system 341, the aperture value, the focus distance, the pixel pitch of the image sensor 331, the vertical image plane coordinate, and the horizontal image plane coordinate, each corresponding to the blur. The first map may further have a channel that indicates information such as the separation method and cutoff frequency of the optical low pass filter, and the manufacturing error of the lens. By inputting the first map to the machine learning model together with the blurred image, the machine learning model can specify the blur that acts on the blurred image from the first map, thereby improving the blur correction effect. When only the blurred image is input, the machine learning model estimates the blur from the blurred image and corrects it, or averagely corrects the blur of various shapes that it has learned, so that a high correction effect cannot be obtained.

[0071] The first map may be a map representing information other than blur, as long as the information is related to the task to be performed, such as a map representing the intensity of noise present in the blurred image, or a depth map or segmentation map corresponding to the blurred image.

[0072] In step S102, the calculation unit 314 inputs training input data to a machine learning model (CNN) and generates an estimated image by correcting the blur of the blurred image. FIG. 10 shows an example of the structure of a CNN in the second embodiment. When the number of pixels of the blurred image and the correct image are (h, w, c), c=1 because both are RAW images in a Bayer array. In the second embodiment, a blurred image 401 rearranged to (h / 2, w / 2, 4) is input to the CNN. The R component of Bayer is the first channel, G adjacent to R in the horizontal direction is the second channel, G adjacent to R in the vertical direction is the third channel, and B is the fourth channel. However, the input method is not limited to this.

[0073] Since the first map 411 is connected to the blurred image 401 in the channel direction, the number of pixels is (h / 2, w / 2, 6). However, the blurred image 401 or the first map 411 may be input to a different layer and converted into a feature map, and then connected to the other (or a feature map generated therefrom). In this case, the number of pixels in the vertical and horizontal directions of the blurred image 401 and the first map 411 do not necessarily need to match. Note that the above-mentioned CNN structure is merely an example, and other structures may be used.

[0074] The calculation unit 314 inputs the result of concatenating the blurred image 401 and the first map 411 in the channel direction to the convolution layer 421. The convolution layer 421 has 64 kernels and biases. The kernel size is (3, 3, 10). The structures of the activation function 422 and the residual block 423 are the same as those in the first embodiment. The convolution layer of the residual block 423 has 64 kernels and biases, and the kernel size is (3, 3, 64). The first feature map 412 is generated by passing through five residual blocks 423. The number of pixels of the first feature map 412 is (h / 2, w / 2, 64).

[0075] The calculation unit 314 rearranges the first feature map 412 into (h / 4, w / 4, 256) to generate a first reduced feature map 413. The first reduced feature map 413 is converted into a second reduced feature map 414 by passing through five residual blocks 424. The convolution layer of the residual block 424 has 256 kernels and biases, and the kernel size is (3, 3, 256). The number of pixels of the second reduced feature map 414 is (h / 4, w / 4, 256).

[0076] The calculation unit 314 rearranges the second reduced feature map 414 to (h / 2, w / 2, 64) to generate a second feature map 415. The calculation unit 314 also concatenates the first feature map 412 and the second feature map 415 in the channel direction and inputs them to the convolution layer 425. The convolution layer 425 has 64 kernels and biases, and the kernel size is (3, 3, 128). As a result of passing through the activation function 426, five residual blocks 427, and the convolution layer 428, the estimated image 402 is generated. The convolution layer of the residual block 427 has 64 kernels and biases, and the kernel size is (3, 3, 64). The convolution layer 428 has four kernels and biases, and the kernel size is (3, 3, 64). An estimated image 402, which is output data of the machine learning model, is an image obtained by correcting the blur of the blurred image 401, and has the number of pixels (h / 2, w / 2, 4).

[0077] In step S103, the update unit 315 updates the CNN model parameters based on the difference between the estimated image and the ground truth image. When calculating the difference, the estimated image or the ground truth image is rearranged to match the number of pixels in each direction.

[0078] In step S104, the update unit 315 determines whether the training of the machine learning model is complete. If the training is not complete, the process returns to step S101 to obtain one or more new sets of training input data and a ground truth image. If the training is complete, the model parameters are stored in the storage unit 311.

[0079] Next, blur correction using a trained CNN (machine learning model) executed by the blur correction device 302 will be described with reference to FIG. 11 showing the flow of the process and the flowchart in FIG. 12. The blur correction device 302 configured by a computer has a storage unit 321, a communication unit 322, an acquisition unit 323, a calculation unit 324, and a display unit 325. The acquisition unit 323 and the calculation unit 324 execute the following processes (first step and second step) according to a program. The acquisition unit 323 and the calculation unit 324 configure a first generation means and a second generation means.

[0080] In step S301, the acquisition unit 323 acquires an undeveloped RAW image (first image) 451. The first image 451 has blurring due to aberration and diffraction occurring in the optical system 341.

[0081] In step S302, the acquisition unit 323 acquires CNN model parameters used for blur correction of the first image 451. In the second embodiment, it is preferable that a plurality of CNN model parameters are prepared for each type of lens device. However, the model parameters may be prepared in common for a plurality of types of lens devices, or a plurality of model parameters may be prepared for one type of lens device. In the second embodiment, a CNN with a common structure is used for all lens devices, but the structure of the CNN may be changed depending on the type of lens device.

[0082] Information regarding the lens device 304 used to capture the first image 451 may be acquired, for example, from metadata of the first image 451. Identification information of the lens device 304 is stored in the storage unit 342. When the lens device 304 and the imaging device 303 are connected or when imaging is performed by the imaging device 303, the identification information is acquired by the imaging device 303 via the communication unit 333 of the imaging device 303 and the communication unit 343 of the lens device 304, and is written into the metadata of the first image 451.

[0083] In step S303, the calculation unit 324 generates a first map 461 that is a map for identifying the blur acting on the first image 451. As in training, each channel of the first map 461 has the focal length, aperture value, and focus distance of the optical system 341, the pixel pitch of the image sensor 331, and the values ​​of the vertical and horizontal image plane coordinates when the first image 451 was captured. Information regarding each value is obtained from the metadata of the first image 451.

[0084] The first image 451 input to the CNN is rearranged in the same manner as during training. The first image 451 has a Bayer array and the number of pixels in each of the vertical, horizontal and channel directions is (W, H, C=1). By performing the rearrangement, the number of pixels of the first image 451 becomes (W / 2, H / 2, 4). The first map 461 is connected to the first image 451 in the channel direction as in training, and therefore the number of pixels becomes (W / 2, H / 2, 6). In the second embodiment, the first map 461 is generated based on the number of pixels of the first image 451.

[0085] In the first map 461, the channels representing the focal length, aperture value, and focus distance of the optical system 341 and the pixel pitch of the image sensor 331 each have the same values ​​in the vertical and horizontal directions. This is because these pieces of information do not change spatially. On the other hand, the image plane coordinates in the vertical and horizontal directions change for each pixel of the first image 451, so the channels representing them have non-uniform values ​​in the vertical or horizontal directions.

[0086] In the second embodiment, the values ​​of the vertical and horizontal image plane coordinates are the coordinates of the corresponding R pixel in the first image 451. However, the coordinates of pixels of other colors may be used. Note that steps S302 and S303 can be executed in any order.

[0087] In step S304, the calculation unit 324 inputs the input data including the first image 451 and the first map 461 to a CNN (machine learning model) to generate a blur-corrected second image 452. The CNN has the same structure as in Fig. 10, and uses the model parameters acquired in step S302.

[0088] The second image 452, which is the output data of the machine learning model, may have unintended changes in brightness or color. In the CNN shown in FIG. 10, the number of pixels in the vertical and horizontal directions changes when the first feature map 412 is rearranged to the first reduced feature map 413 and when the second reduced feature map 414 is rearranged to the second feature map 415. In image processing using a machine learning model including a change in the number of pixels in the vertical or horizontal direction, unintended changes in brightness or color are likely to occur in the second image 452 due to variations in values ​​for each channel. In addition, a first map 461 having multiple channels or having uneven values ​​in the vertical or horizontal directions within a channel is input to the CNN together with the first image 451. In this configuration, the CNN performs different processes depending on the value of the first map 461, so that the task performed by the CNN becomes complicated and unintended changes in brightness or color are likely to occur in the second image 452.

[0089] In step S305, the calculation unit 324 generates a first residual image 462 by taking the difference between the second image 452 and the fourth image 454. In the second embodiment, the fourth image 454 is an image obtained by performing an identity transformation on the first image 451 (that is, the same image as the first image 451). However, the first image 451 may be used as the fourth image 454 without performing the identity transformation. The calculation unit 324 generates the first residual image 462 by subtracting the fourth image 454 from the second image 452.

[0090] In step S306, the calculation unit 324 generates a weight map 463 by nonlinear processing based on the absolute value of the first residual image 462. The weight map 463 represents the weight of the second image 452 in the weighted average of the second image 452 and the fourth image 454. The weight map 463 is generated by, for example, nonlinear processing as shown in Figure 13 (A) or (B). The horizontal axis of Figure 13 (A) and (B) is the absolute value of the first residual image 462, and the vertical axis is the weight of the weight map 463.

[0091] In the nonlinear processing of FIG. 13(A), weight map 463 is generated with a binary value of 0 or 1. When taking the weighted average, the boundary between 0 and 1 may appear discontinuous, so if nonlinear processing such as that of FIG. 13(A) is performed, it is advisable to perform a spatial blurring process on weight map 463. On the other hand, in the nonlinear processing of FIG. 13(B), the weight changes continuously according to the absolute value of the residual, so such a problem is unlikely to occur. The weight of each pixel in weight map 463 becomes smaller as the absolute value of the corresponding pixel in the first residual image 462 becomes smaller.

[0092] In step S307, the calculation unit 324 generates a third image 453 based on the second image 452, the fourth image 454, and the weight map 463. The third image 453 is an estimated image in which unintended changes in luminance and color are suppressed and blur is corrected. The calculation unit 324 calculates a weighted average by taking the sum (A+B) of the product (A) of the weight map 463 and the second image 452 and the product (B) of the result of subtracting the weight map 463 from 1 and the fourth image 454, thereby generating the third image 453. Alternatively, the third image 453 may be generated by adding the product of the weight map 463 and the first residual image 462 to the fourth image 454.

[0093] The third image 453 is generated by a weighted average of the second image 452 and the fourth image 454, so that the signal value of each pixel of the third image 453 has a value between the signal values ​​of the pixels corresponding to each pixel of the third image 453 in each of the second image 452 and the fourth image 454.

[0094] By using the weight map 463, the signal value of each pixel of the third image 453 will be closer to the signal value of the pixel corresponding to each pixel of the third image in the fourth image 454 than to the signal value of the pixel corresponding to each pixel of the third image in the second image 452, the smaller the non-zero absolute value of the signal value of the corresponding pixel in the first residual image 462.

[0095] According to the present embodiment described above, it is possible to provide an image processing method and an image processing device (blur correction device 302) capable of suppressing unintended changes in luminance and color in image estimation using a machine learning model.

[0096] The above embodiments include the following methods and configurations.

[0097] (Method 1) A first step of generating a first residual image based on a first image using a machine learning model; a second step of generating a third image based on the absolute value of the first residual image and a fourth image; the fourth image is the first image or an image based on the first image; When a sum of the fourth image and the first residual image is a second image, an image processing method characterized in that, in the second step, the signal value of each pixel in the third image is made closer to the signal value of the pixel corresponding to each pixel in the fourth image than to the signal value of the pixel corresponding to each pixel in the second image, the smaller the absolute value of the signal value of the pixel corresponding to each pixel in the first residual image that is not 0. (Method 2) 2. The image processing method according to claim 1, characterized in that, in the second step, the signal value of each of the pixels in the third image is set to a value between the signal value of a pixel corresponding to each of the pixels in the fourth image and the signal value of a pixel corresponding to each of the pixels in the second image. (Method 3) 2. The image processing method according to claim 1, wherein in the second step, the third image is generated using a result of a nonlinear transformation based on an absolute value of the first residual image. (Method 4) 2. The image processing method according to claim 1, wherein in the second step, the third image is generated using a result of a threshold process based on an absolute value of the first residual image. (Method 5) 2. The image processing method according to claim 1, wherein in the second step, the third image is generated based on a second residual image generated from the first residual image using soft thresholding. (Method 6) 6. The image processing method according to claim 5, wherein in the second step, the second residual image is generated by applying a coefficient greater than 1 based on a threshold value of the soft thresholding to a residual image obtained by performing the soft thresholding on the first residual image. (Method 7) 2. The image processing method according to claim 1, wherein in the second step, the third image is generated based on a weight map based on an absolute value of the first residual image, the fourth image, and one of the first residual image or the second image. (Method 8) the weight map represents weights of the first residual image or the second image; 8. The image processing method according to claim 7, wherein in the weight map, the weight of a pixel in the first residual image that has a smaller non-zero absolute value is smaller. (Method 9) The image processing method according to claim 1, characterized in that in the first step, the number of pixels in the vertical or horizontal direction in any of the input data input to the machine learning model, the output data output from the machine learning model, and the multiple feature maps generated within the machine learning model is different from the others. (Method 10) In the first step, input data input to the machine learning model includes the first image and a first map; 2. The method of claim 1, wherein the first map has a plurality of channels. (Method 11) In the first step, input data input to the machine learning model includes the first image and a first map; 2. The method of claim 1, wherein the first map has non-uniform values ​​in a vertical or horizontal direction. (Method 12) The machine learning model has a plurality of model parameters determined in advance by training; 2. The image processing method according to claim 1, wherein the number of bits of at least one of the plurality of model parameters is smaller when generating the first residual image than when training. (Method 13) 2. The image processing method according to claim 1, wherein in the second step, the third image is generated based on information about edges of the fourth image or on the magnitude of the signal value of each pixel in the fourth image. (Method 14) The image processing method according to claim 1, characterized in that the second image is an image obtained by performing at least one of correction of degradation, enhancement of resolution, enhancement of contrast, demosaicing, expansion of dynamic range, enhancement of gradation, change of lighting, change of shape of defocus blur, change of focus distance, and change of depth of field on the first image. (Method 15) 2. The image processing method according to claim 1, wherein the machine learning model has a plurality of first layers having a first number of bits and a plurality of second layers having a second number of bits smaller than the first number of bits. (Method 16) A program causing a computer to execute the image processing method according to any one of claims 1 to 15. (Configuration 1) a first generating means for generating a first residual image based on a first image by using a machine learning model; a second generating means for generating a third image based on the absolute value of the first residual image and a fourth image; the fourth image is the first image or an image based on the first image; When a sum of the fourth image and the first residual image is a second image, The image processing device is characterized in that the second generation means makes the signal value of each pixel in the third image closer to the signal value of the pixel corresponding to each pixel in the fourth image than to the signal value of the pixel corresponding to each pixel in the second image, the smaller the absolute value of the signal value of the pixel corresponding to each pixel in the first residual image that is not zero. (Configuration 2) An image processing device according to claim 17; and an imaging element for acquiring a captured image as the first image. (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0098] The embodiments described above are merely representative examples, and various modifications and alterations are possible for each embodiment when implementing the present invention. [Explanation of symbols]

[0099] 101,301 Training equipment 102,303 Imaging device 125 Acquisition Department 126 Arithmetic section 302 Blur correction device

Claims

1. A first step of generating a first residual image based on a first image using a machine learning model; a second step of generating a third image based on the absolute value of the first residual image and a fourth image; the fourth image is the first image or an image based on the first image; When a sum of the fourth image and the first residual image is a second image, An image processing method characterized in that, in the second step, the signal value of each pixel in the third image is made closer to the signal value of the pixel corresponding to each pixel in the fourth image than to the signal value of the pixel corresponding to each pixel in the second image, the smaller the absolute value of the signal value of the pixel corresponding to each pixel in the first residual image that is not zero.

2. 2. The image processing method according to claim 1, wherein in the second step, the signal value of each pixel in the third image is set to a value between the signal value of a pixel corresponding to each pixel in the fourth image and the signal value of a pixel corresponding to each pixel in the second image.

3. 2. The image processing method according to claim 1, wherein in the second step, the third image is generated using a result of a non-linear transformation based on an absolute value of the first residual image.

4. 2. The image processing method according to claim 1, wherein in the second step, the third image is generated using a result of a threshold process based on an absolute value of the first residual image.

5. 2. The image processing method according to claim 1, wherein in the second step, the third image is generated based on a second residual image generated from the first residual image using soft thresholding.

6. 6. The image processing method according to claim 5, wherein in the second step, the second residual image is generated by applying a coefficient greater than 1 based on a threshold value of the soft thresholding to a residual image obtained by performing the soft thresholding on the first residual image.

7. 2. The image processing method according to claim 1, wherein in the second step, the third image is generated based on a weight map based on an absolute value of the first residual image, the fourth image, and one of the first residual image or the second image.

8. the weight map represents weights of the first residual image or the second image; 8. The image processing method according to claim 7, wherein in the weight map, the weight of a pixel in the first residual image having a smaller non-zero absolute value is smaller.

9. The image processing method according to claim 1, characterized in that in the first step, the number of pixels in the vertical or horizontal direction in any of the input data input to the machine learning model, the output data output from the machine learning model, and the multiple feature maps generated within the machine learning model is different from the others.

10. In the first step, input data input to the machine learning model includes the first image and a first map; 2. The method of claim 1, wherein the first map has multiple channels.

11. In the first step, input data input to the machine learning model includes the first image and a first map; 2. The method of claim 1, wherein the first map has non-uniform values ​​in the vertical or horizontal direction.

12. The machine learning model has a plurality of model parameters determined in advance by training; 2. The image processing method according to claim 1, wherein the number of bits of at least one of the plurality of model parameters is smaller when generating the first residual image than when training.

13. 2. The image processing method according to claim 1, wherein in the second step, the third image is generated based on information about edges of the fourth image or the magnitude of the signal value of each pixel in the fourth image.

14. The image processing method according to claim 1, characterized in that the second image is an image obtained by performing at least one of correction of degradation, enhancement of resolution, enhancement of contrast, demosaicing, expansion of dynamic range, enhancement of gradation, change of lighting, change of shape of defocus blur, change of focus distance, and change of depth of field on the first image.

15. The image processing method according to claim 1 , wherein the machine learning model has a plurality of first layers having a first number of bits and a plurality of second layers having a second number of bits smaller than the first number of bits.

16. A program causing a computer to execute the image processing method according to any one of claims 1 to 15.

17. a first generating means for generating a first residual image based on a first image by using a machine learning model; a second generating means for generating a third image based on the absolute value of the first residual image and a fourth image; the fourth image is the first image or an image based on the first image; When a sum of the fourth image and the first residual image is a second image, The image processing device is characterized in that the second generation means makes the signal value of each pixel in the third image closer to the signal value of the pixel corresponding to each pixel in the fourth image than to the signal value of the pixel corresponding to each pixel in the second image, the smaller the absolute value of the signal value of the pixel corresponding to each pixel in the first residual image that is not zero.

18. The image processing device according to claim 17 ; and an imaging element for acquiring a captured image as the first image.

Images