Image processing method, image processing apparatus, image pickup apparatus, and storage medium

Abstract

The invention relates to an image processing method, an image processing apparatus, an image pickup apparatus, and a storage medium. An image processing method includes generating a first residual image based on a first image using a machine learning model, and generating a third image based on an absolute value of the first residual image and a fourth image. The fourth image is the first image or an image based on the first image. When the second image is the sum of the fourth image and the first residual image, the second image is obtained as the non-zero absolute value of the signal value of the pixel corresponding to each pixel in the third image in the first residual image decreases. A signal value of each pixel in the third image is set to be closer to a value of a signal value of a pixel corresponding to each pixel in the third image in the fourth image than a signal value of a pixel corresponding to each pixel in the third image in the second image.

Description

Technical Field

[0001] One of the aspects of the embodiments relates to an image processing method, an image processing device, and a storage medium (program), each of which estimates an image using a machine learning model. Background Art

[0002] Japanese Patent Laid-Open No. 2022-536807 discloses a method for generating an upsampled high-resolution image from a low-resolution image using a neural network (NN). The NN generates a plurality of high-resolution residual sub-images and obtains a high-resolution residual image by rearranging these pixels. Then, a high-resolution image with a high sense of resolution is generated by adding a basic high-resolution image obtained by bicubic upsampling the low-resolution image to the high-resolution residual image.

[0003] However, the method disclosed in Japanese Patent Laid-Open No. 2022-536807 has the following problem: unintended luminance changes unrelated to upsampling occur in high-resolution images. For example, a periodic pattern may appear in a high-resolution image. The principle of this problem will be described later. Summary of the invention

[0004] An image processing method includes: a first step of generating a first residual image based on a first image using a machine learning model; and a second step of generating a third image based on the absolute value of the first residual image and a fourth image. The fourth image is the first image or an image based on the first image. In the case where the second image is the sum of the fourth image and the first residual image, as the non-zero absolute value of the signal value of the pixel corresponding to each pixel in the third image in the first residual image decreases, the second step sets the signal value of each pixel in the third image to a value closer to the signal value of the pixel corresponding to each pixel in the third image in the fourth image than the signal value of the pixel corresponding to each pixel in the third image in the second image. An image processing device corresponding to the above image processing method and an image pickup device having the above image processing device also constitute another aspect of the present disclosure.

[0005] Further features of various embodiments of the present disclosure will become apparent from the following description of the embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 The process of generating the third image in Example 1 is illustrated.

[0007] Figure 2 is a block diagram illustrating the configuration of an image processing system according to Example 1.

[0008] Figure 3 is an external view of the image processing system according to Example 1.

[0009] Figure 4 is a flowchart illustrating the training of the machine learning models in Examples 1 and 2.

[0010] Figure 5A and Figure 5B Illustrate the configuration of the machine learning model in Example 1.

[0011] Figure 6 is a flowchart illustrating a generation process of a third image according to Example 1.

[0012] Fig. 7A , Figure 7B and Figure 7C An example of nonlinear processing according to Example 1 is illustrated.

[0013] Figure 8 is a block diagram illustrating the configuration of an image processing system according to Example 2.

[0014] Fig. 9 is an external view of the image processing system according to Example 2.

[0015] Fig.10 Illustrate the configuration of the machine learning model in Example 2.

[0016] Fig.11 The process of generating the third image in Example 2 is illustrated.

[0017] Fig.12 is a flowchart illustrating a generation process of a third image according to Example 2.

[0018] Fig.13A and Fig. 13B An example of nonlinear processing according to Example 2 is illustrated. DETAILED DESCRIPTION

[0019] In the following, the term "unit" may refer to a software context, a hardware context, or a combination of software and hardware contexts. In the software context, the term "unit" refers to functionality, application, software module, function, routine, instruction set, or program, which can be executed by a programmable processor (such as a microprocessor), a central processing unit (CPU), or a specially designed programmable device or controller. The memory contains instructions or programs, which, when executed by the CPU, cause the CPU to perform operations corresponding to the unit or function. In the hardware context, the term "unit" refers to a hardware element, circuit, component, physical structure, system, module, or subsystem. Depending on the specific embodiment, the term "unit" may include mechanical, optical or electrical components, or any combination thereof. The term "unit" may include active (e.g., transistors) or passive (e.g., capacitors) components. The term "unit" may include a semiconductor device having a substrate and other material layers having various conductive concentrations. It may include a CPU or a programmable processor that can execute a program stored in a memory to perform a specified function. The term "unit" may include a logic element (e.g., AND, OR) implemented by a transistor circuit or any other switching circuit. In the combination of software and hardware contexts, the term "unit" or "circuit" refers to any combination of software and hardware contexts as described above. In addition, the term "element", "component", "part", or "device" may also refer to a "circuit" with or without integration with packaging materials.

[0020] Now, with reference to the drawings, a description will be given of an embodiment according to the present disclosure.Before describing the details of the embodiment, a description of an overview of the embodiment will now be given.

[0021] In training a machine learning model configured to perform image estimation, the difference between the estimated image and the expected ground truth image is expressed by a loss function such as mean square error (MSE). By minimizing the value of the loss function, multiple model parameters of the machine learning model (weights and biases of each layer, etc.: referred to as model parameters hereinafter) are determined. Generally, in image estimation tasks, the solution is not uniquely determined for the input data, so even if the loss function is minimized, the value of the loss function does not become 0, and there is still a finite error. For example, in the case where the task is to correct image blur, the uncorrected or overcorrected components of the blur remain as a finite value of the loss function even after training. Blur affects areas where structures such as edges exist, but does not change the image appearance in flat areas where the signal does not change. In other words, the desired behavior of a machine learning model configured to correct blur is to not change the signal value in flat areas of the image.

[0022] However, image processing using machine learning models can generate brightness and color changes that do not actually exist in flat areas of the estimated image. In the case where the change is sufficiently smaller than the task-related (blur correction residual and overcorrection) components in the order of magnitude of the loss function, the change still exists even after minimization. As a result, unintended changes in brightness and color that are not related to the task occur in image estimation using machine learning models.

[0023] In order to suppress this change, this example uses the absolute value of the first residual image (the blur correction component in the case where the task is blur correction) generated using the machine learning model. In this embodiment, the residual image is an image generated in a residual block including a jump connection, and is added to the input image of the residual block. Since the influence of unintentional luminance and color changes on the loss function is small, the absolute value of the residual is inevitably small. Therefore, by weakening the influence of pixels with small absolute values ​​in the first residual image, an estimated image (third image) in which unintentional luminance and color changes are suppressed can be obtained. Specific examples of this method will be described in Examples 1 and 2 below.

[0024] Example 1

[0025] An image processing system according to Example 1 will be described. The task performed by the machine learning model in Example 1 is upsampling (resolution improvement processing). Example 1 upsamples the entire image, but a portion of the image can be cropped and upsampled, just like digital zoom.

[0026] Similar effects can be obtained for tasks other than upsampling. Examples of other tasks include degradation correction, contrast improvement, demosaicing, dynamic range extension, grayscale increase, lighting change, changing the defocus blur shape, focal distance change, depth of field change, etc. Degradation here refers to blur, noise, smoke, disturbance (air fluctuations), etc. Blur includes aberrations, diffraction, defocus blur and jitter of the optical system. Noise includes noise generated by the image sensor and noise generated by data compression. Changing the defocus blur shape includes changing a blurred image with double-line blur to a blurred image with a flat light distribution in the defocused area of ​​the image, changing a blurred image with a defective shape due to vignetting to a circular blurred image, etc.

[0027] Figure 2 The configuration of the image processing system 100 according to Example 1 is illustrated. Figure 3 The figure shows the appearance of the image processing system 100. The image processing system 100 includes a training device 101 and an image pickup device 102.

[0028] The image pickup device 102 includes an optical system 121, an image sensor 122, a memory 123, a communication unit 124, an acquisition unit 125, a calculator 126, and a display unit 127. The optical system 121 condenses light incident from an object space to form an object image. The image sensor 122 receives the object image and performs photoelectric conversion to generate a captured image.

[0029] The calculator 126 performs predetermined image processing on the captured image generated by the image sensor 122. The memory 123 stores the captured image after the image processing.

[0030] The calculator 126 generates an output image (corresponding to a third image described later) by up-sampling a captured image (corresponding to a first image described later) generated by the image sensor 122 as an input image using a machine learning model.

[0031] The captured image to be upsampled may be an undeveloped original image or a developed image, but this example upsamples the luminance component of the developed image. Upsampling may be performed automatically when the image is captured, or may be performed at any timing specified by the user for a captured image that has been stored.

[0032] The model parameters of the machine learning model have been previously trained by the training device 101, previously acquired by the image pickup device 102 via the communication unit 112 of the training device 101 and the communication unit 124 of the image pickup device 102, and stored in the memory 123. Details on the training of the machine learning model and upsampling by the trained machine learning model will be described later. The output image generated by upsampling is displayed on the display unit 127 and stored in the memory 123.

[0033] refer to Figure 4 A description will be given of a method for training a machine learning model (a method for determining model parameters) performed by the training device 101 with reference to a flowchart. The training device 101 configured by a computer has a memory 111, a communication unit 112, an acquisition unit 113, a calculator 114, and an updater 115. The acquisition unit 113, the calculator 114, and the updater 115 perform the following processing (processing steps and updating steps) according to a (computer) program. The acquisition unit 113 and the calculator 114 constitute a processing unit, and the updater 115 corresponds to an updating unit.

[0034] In step S101, the acquisition unit 113 acquires one or more sets of training input data and true value images from the memory 111. The training input data is data to be input to the machine learning model during training. The training input data in Example 1 is a low-resolution image (input image for training). However, the training input data may include another image and a map in addition to the low-resolution image. For example, it may include a map indicating the intensity of degradation (noise, etc.) acting on the low-resolution image. The case of including a map will be explained in Example 2.

[0035] The low-resolution image has the number of pixels in the vertical, horizontal, and channel directions (h, w, c). Example 1 only upsamples the luminance component, so c = 1. In the case where there are multiple color components, for example, in the case where there are three color components RGB (red, green, blue), c = 3.

[0036] The true value image is a high-resolution image in which the same object as the low-resolution image exists. Example 1 quadruples the number of pixels by upsampling (twice vertically and twice horizontally), so the number of pixels in the high-resolution image is (2h, 2w, c=1). However, the upsampling ratio is not limited to this example. A pair of low-resolution and high-resolution images can be prepared by real imaging, imaging simulation, computer graphics (CG), etc.

[0037] In step S102, the calculator 114 inputs the training input data into the machine learning model and generates an upsampled estimated image. Example 1 Using Figure 5A and Figure 5B As a machine learning model, a convolutional neural network (CNN) shown in FIG. 1 is used. However, the type of machine learning model is not limited to this example, and a multilayer perceptron (MLP), a visual transformer (ViT), a generative adversarial network (GAN), a diffusion model, etc. may be used. In addition, model parameters refer to a plurality of variables determined by training in a machine learning model. More specifically, they include weights and biases of kernels (filters) of convolutional layers, and weights and biases of fully connected layers.

[0038] In Example 1, Figure 5AThe CNN shown in generates a first residual image 211 from the training input data 201. The training input data 201 is converted into a feature map via a convolutional layer and an activation function. Example 1 uses a rectified linear unit (ReLU) as an activation function, but Leaky can use a LeakyReLU or a sigmoid function. The convolutional layer includes 48 kernels and biases of size (3, 3, 1) in the vertical, horizontal, and channel directions. As a result, a feature map with a number of pixels of (h, w, 48) is generated. The feature map is the input of the residual block. The feature map refers to the output data of each layer generated during the calculation of the machine learning model. The size of the kernel and the number of kernels and biases are not limited to those described above.

[0039] The residual block is Figure 5B The residual block input 203 passes through the convolution layer twice, each time with 48 kernels of size (3, 3, 48) and bias (although ReLU is applied during the process), and the sum of the result and the residual block input 203 for each element is obtained. The residual block output 204 is generated by applying ReLU to the obtained sum.

[0040] There are multiple residual blocks, and 10 residual blocks are connected in Example 1. However, the number of residual blocks is not limited to this example. Each feature map generated in the residual block has the number of pixels of (h, w, 48). The last convolutional layer has four kernels of size (3, 3, 48) and a bias.

[0041] As a result of the calculation of the calculator 114, a residual image 202 having a number of pixels of (h, w, 4) is generated. The number of channels in the residual image 202 corresponds to the upsampling magnification. The calculator 114 generates a first residual image 211 having a number of pixels of (2h, 2w, 1) by rearranging the pixels of the residual image 202. The components of the first channel of the residual image 202 are set so that the indexes of the first residual image 211 in the vertical and horizontal directions are stored in the odd-numbered pixels. The components of the second channel are set so that the indexes in the vertical direction are stored in the odd-numbered pixels and the indexes in the horizontal direction are stored in the even-numbered pixels. The components of the third channel are set so that the indexes in the vertical direction are stored in the even-numbered pixels and the indexes in the horizontal direction are stored in the odd-numbered pixels. The components of the fourth channel are set so that the indexes in the vertical and horizontal directions are stored in the even-numbered pixels.

[0042] The calculator 114 generates an estimated image by calculating the sum of a bicubic enlarged image obtained by enlarging the low-resolution image to four times the number of pixels of (2h, 2w, 1) using bicubic interpolation and the first residual image 211. However, the low-resolution image may be enlarged by rule-based processing (such as bilinear interpolation or nearest neighbor interpolation) without using another machine learning model. The sum of A and B includes multiplying A or B by -1 and calculating the difference. The target to be estimated by the machine learning model may be directly the estimated image, rather than the residual.

[0043] In step S103, the updater 115 updates the model parameters of the machine learning model based on the difference between the estimated image and the true value image (high-resolution image). Example 1 uses the MSE of the estimated image and the high-resolution image as the loss function, but is not limited to this example, and the mean absolute error (MAE) or perceptual loss (PerceptualLoss) may be used. For updating, back propagation or the like is used.

[0044] In step S104, the updater 115 determines whether the training of the machine learning model has been completed. If not, the process returns to step S101 to obtain a new set or sets of training input data and true value images. If completed, the model parameters are stored in the memory 111.

[0045] Next, we will refer to the flowchart illustrating the process flow. Figure 1 and Figure 6 The flowchart in describes upsampling using a trained CNN (machine learning model) performed in the image pickup device 102. In the image pickup device 102, an acquisition unit 125 and a calculator 126 constituted by a computer (image processing device) perform the following processing (first and second steps) according to a (computer) program. The acquisition unit 125 and the calculator 126 constitute a first generator and a second generator.

[0046] In step S201, the acquisition unit 125 acquires the first image 221. Similar to training, the first image 221 is the luminance component of the developed captured image. The number of pixels in the vertical, horizontal, and channel directions in the first image 221 is expressed by (H, W, C), and C=1 in Example 1.

[0047] The first image 221 may be a partial area of ​​the captured image. By repeating a series of processes described below and by changing the position of the partial area and combining the results, the entire captured image may be upsampled. In the case where the partial area is set to the first image 221, only step S202 may be performed for each partial area, and the results may be combined before performing step S203 and subsequent steps. Multiple partial areas may have overlapping areas so that the dividing boundary line is not visible after upsampling. After upsampling, the overlapping areas may be averaged, or one of the results may be adopted.

[0048] In step S202, the calculator 126 generates a first residual image 231 based on the first image 221 using a machine learning model (CNN). Similar to training, the calculator 126 uses a machine learning model (CNN) having Figure 5A and Figure 5B A CNN of the configuration shown in , and obtains from the memory 123 and uses the model parameters trained by the training device 101. The input data input to the CNN is the first image 221. The output data output from the CNN is the first residual image 231. The number of pixels in the vertical, horizontal, and channel directions of the first residual image 231 is (2H, 2W, C), and C=1 in Example 1. The calculator 126 obtains a second image by calculating the sum of the fourth image 224 and the first residual image 231, the second image being a high-resolution upsampled image of the first image 221, the fourth image 224 being obtained by enlarging the first image 221 to (2H, 2W, C=1) using bicubic interpolation (the same process as performed on the low-resolution image during training).

[0049] Here, the second image may have an unintended brightness change that is not related to upsampling, and may also have a color change in the case where the first image 221 has multiple color components. As described before in the description of Example 1, components in the loss function during training that are sufficiently smaller in magnitude than components related to the task performed by the machine learning model (resolution increase processing in Example 1) may still exist even after training. These small components cause unintended brightness and color changes.

[0050] have Figure 5A and Figure 5BThe CNN of the structure shown in the figure rearranges the residual map 202 with the number of pixels being (H, W, 4) and generates the first residual image 211 with the number of pixels being (2H, 2W, 1). Therefore, the configuration in which the number of pixels changes in the vertical or horizontal direction is likely to cause unintentional brightness and color changes. Since the estimation error is likely to vary channel by channel, small errors δ1, δ2, δ3, and δ4 may appear in each channel of the residual map 202. When the residual map 202 with this error is rearranged and the first residual image 211 is generated, the pixels with errors δ1 to δ4 are arranged in adjacent 2×2 pixel units and a periodic pattern is generated. This pattern is particularly noticeable in flat areas where the signal value does not change and in gradient areas where the change is gradual. In order to suppress such unintentional brightness and color changes, the following processing is performed.

[0051] In step S203, the calculator 126 generates a second residual image 232 using nonlinear processing (nonlinear conversion) based on the absolute value of the first residual image 231. The nonlinear processing performed here is processing of pixels having non-zero signal values ​​among a plurality of pixels included in the first residual image 231 so that the residual attenuation rate becomes higher for pixels having smaller absolute values ​​of signal values.

[0052] Example 1 uses thresholding as nonlinear processing. Examples include Fig. 7A The hard thresholding shown in Figure 7B Soft thresholding as shown in . Fig. 7A and Figure 7B In , the horizontal axis represents the input signal value, the vertical axis represents the output signal value, the broken line represents the identity transformation, the solid line represents the hard or soft thresholding, and the long and short alternating dashed lines represent the thresholding threshold. Hard thresholding is expressed by the following equation (1):

[0053]

[0054] where s in is the input signal value, s out is the output signal value, and s thr is the thresholding threshold.

[0055] The presence or absence of an equal sign can be added to the bottom inequality instead of the top inequality.

[0056] Soft thresholding is expressed by the following equation (2):

[0057]

[0058] The signal value of each pixel of the third image 223, which is the sum of the second residual image 232 and the fourth image 224 generated by hard or soft thresholding, is a value between the signal values ​​of the pixels corresponding to each pixel of the third image 223 in the second image and the fourth image 224. Therefore, new problems other than unintended brightness and color changes suppressed by nonlinear processing are suppressed.

[0059] However, in hard thresholding, residuals whose absolute values ​​are greater than the threshold remain unchanged, so although the effect of the task (resolution increase processing in Example 1) is unlikely to decrease, discontinuous steps in signal values ​​may appear in the second residual image 232. Soft thresholding does not cause steps in the second residual image 232, but the residual approaches 0 by passing through the threshold, so the effect of the task is slightly reduced.

[0060] Therefore, when using soft thresholding, the residual can be further scaled as Figure 7C . Soft thresholding combined with scaling is expressed by the following equation (3):

[0061]

[0062] where f st is the soft thresholding in equation (2), and s0 is a value that controls a scaling factor applied to (multiplied by) a residual image obtained by performing soft thresholding on the first residual image 231 .

[0063] The scaling factor is based on the thresholding threshold s thr A coefficient greater than 1. For example, s0 may be set to a signal value width (a difference between a maximum signal value and a minimum signal value) that the first image 221 may have. For example, for an 8-bit developed image, the signal value width is 255, and for an original image having a maximum signal value of 32767 and an optical black level of 2048, the signal value width is 30719. In the case where the signal value is normalized to a range from 0 to 1, the signal value width is 1.

[0064] In the case where s0 is set as the signal value width, the effect of the second residual image 232 on the task is always equal to or less than the effect of the first residual image 231 on the task, so that the effect can be prevented from becoming excessive. Thus, the signal value of each pixel of the third image 223 becomes a value between the signal values ​​of the pixels in the second image and the fourth image 224 corresponding to each pixel of the third image 223.

[0065] Other processing may be performed as long as it is a nonlinear processing that increases the residual attenuation rate as the absolute value of the non-zero component of the first residual image 231 decreases. The other processing will be explained in Example 2. Such nonlinear processing performed on the first residual image 231 can generate a second residual image 232 in which unintended luminance and color changes are suppressed while suppressing a reduction in the effect of the task (resolution improvement processing).

[0066] In step S204, the calculator 126 generates a third image 223 by calculating the sum of the fourth image 224 and the second residual image 232. The third image 223 is a high-resolution image obtained by upsampling the first image 221, in which unintended luminance and color changes are suppressed. By using the second residual image 232, as the non-zero absolute value of the signal value of the pixel corresponding to each pixel of the third image in the first residual image 231 becomes smaller, the signal value of each pixel of the third image 223 becomes closer to the signal value of the pixel corresponding to each pixel of the third image in the fourth image 224 than the signal value of the pixel corresponding to each pixel of the third image in the second image. The number of pixels of the third image 223 is (2H, 2W, C). Example 1 upsamples only the luminance component, so C=1.

[0067] The chrominance components of the captured image are upscaled to four times the number of pixels by bicubic interpolation or the like, and are combined with the third image 223 to obtain an upsampled image having a plurality of color components.

[0068] Example 1 uses this configuration because human vision is not sensitive to color resolution, and processing can be accelerated by limiting the upsampling target of CNN to only luminance. However, images with multiple color components can also be upsampled by CNN. Example 1 increases the number of pixels in the vertical and horizontal directions within the machine learning model, but the first image input to the machine learning model can be an image enlarged by bicubic interpolation, etc.

[0069] Since unintended brightness and color changes are apparent in areas where changes in the signal value of an image are small, the effect of the nonlinear processing in step S203 may be limited to flat areas and gradient areas. This may be achieved by obtaining information about the edge of the fourth image 224 using spatial differentiation, edge extraction, etc. and by generating the third image 223 based on the information about the edge. The nonlinear processing in step S203 may be limited to non-edge areas to generate the second residual image 232. Alternatively, in step S204, the third image 223 may be generated by adding only the second residual image 232 to the non-edge area of ​​the fourth image 224 and adding the first residual image 231 to the edge area of ​​the fourth image 224.

[0070] Since unintentional brightness and color changes are noticeable in areas (dark areas) where the brightness value of an image is low, the effect of the nonlinear processing in step S203 can be limited to dark areas. This can be achieved by generating the third image 223 based on the magnitude of the signal value of each pixel of the fourth image 224. The nonlinear processing in step S203 can be limited to the dark areas of the fourth image 224 to generate the second residual image 232. Alternatively, in step S204, the third image 223 can be generated by adding the second residual image 232 only to the dark areas of the fourth image 224 and adding the first residual image 231 to other areas. In particular, in the case where the first image 221 is an undeveloped original image, unintentional brightness and color changes in dark areas are emphasized by gamma correction during development and are easily noticeable.

[0071] In the case where the number of bits of model parameters during image estimation using a trained machine learning model is less than the number of bits of model parameters during training of the machine learning model, unintended brightness and color changes may worsen due to quantization errors. Therefore, in the case where the number of bits of at least one or more model parameters of the machine learning model that generates the first residual image 231 is less than the number of bits during training, the effect of the processing is likely to be obtained from step S203 or subsequent steps. In the case where a simpler image estimation process using an application specific integrated circuit (ASIC) or the like is desired, the number of bits during image estimation is reduced.

[0072] The image estimation process can be simplified by reducing the number of bits of some of the model parameters of each layer of the machine learning model or some calculations such as convolution performed in each layer. In this case, unintentional brightness and color changes will also deteriorate. Therefore, in the case where the machine learning model includes multiple first layers with a first digit and multiple second layers with a second digit smaller than the first digit, the effect in the processing of step S203 and subsequent steps can be easily obtained.

[0073] The nonlinear processing based on the absolute value of the first residual image 231 in step S203 includes processing that can obtain a result similar to that of the nonlinear processing based on the absolute value even if the absolute value is not directly calculated.

[0074] For example, instead of calculating the absolute value, the positive and negative components of the first residual image 231 may be divided into two and nonlinear processing may be performed on each component. This will be described as an example of a case where soft thresholding is performed on the first residual image 231. The second residual image 232 is set to the sum of the result of subtracting a threshold (a positive value here) from the first residual image 231 and then performing ReLU and the result of subtracting a threshold from -1 times the first residual image 231 and then performing ReLU and multiplying by -1. This method can generate the second residual image 232 without directly calculating the absolute value of the first residual image 231.

[0075] As described above, this example can provide an image processing method and an image processing device (acquisition unit 125 and calculator 126 ), each of which can suppress unintended luminance and color changes in image estimation using a machine learning model.

[0076] Example 2

[0077] An image processing system according to Example 2 of the present disclosure will be described. The image estimation task performed by the machine learning model in Example 2 is blur correction. The targets to be corrected are aberrations generated in the optical system and blur due to diffraction. However, similar effects can be obtained for other defocus blur, jitter, etc. Similar effects can be obtained for tasks other than blur correction.

[0078] Figure 8 The configuration of an image processing system 300 according to Example 2 is illustrated. Fig. 9 The diagram shows an overview of an image processing system 300. The image processing system 300 includes a training device 301, a blur correction device 302, an image pickup device 303, and a lens device 304.

[0079] The image pickup device 303 is a lens-interchangeable type image pickup device, and a plurality of types of lens devices 304 can be connected. The object image formed by the lens device 304 is photoelectrically converted by the image sensor 331 to obtain a captured image. The captured image is affected by aberrations generated in the optical system 341 and image degradation due to diffraction. The calculator 334 performs necessary processing on the captured image, such as removing defective pixels. The memory 332 stores the captured image processed by the calculator 334.

[0080] The blur correction device 302 as an image processing device uses a machine learning model to correct the blur in the captured image to generate a blur-corrected third image. Example 2 performs blur correction on an undeveloped original image. However, blur correction can also be performed on a developed image. The model parameters of the machine learning model are trained by the training device 301, and are acquired by the blur correction device 302 in advance via the communication unit 312 of the training device 301 and the communication unit 322 of the blur correction device 302, and stored in the memory 321. The training of the machine learning model and the blur correction using the trained model parameters will be described later. The blur-corrected third image undergoes other necessary processing by the calculator 324, and is stored in the memory 321 of the blur correction device 302 or the memory 332 of the image pickup device 303, or is displayed on the display unit 325 of the blur correction device 302 or the display unit 335 of the image pickup device 303.

[0081] Now refer to the one also used in Example 1 Figure 4 , a description will be given of a method for training a machine learning model performed by a training device 301. The training device 301 includes a computer having a memory 311, a communication unit 312, an acquisition unit 313, a calculator 314, and an updater 315. The acquisition unit 313, the calculator 314, and the updater 315 perform the following processing (processing steps and updating steps) according to a program. The acquisition unit 313 and the calculator 314 configure a processing unit, and the updater 315 corresponds to an updating unit. In this example, CNN is used as a machine learning model, but other methods may be used.

[0082] In step S101, the acquisition unit 313 acquires one or more sets of training input data and true value images from the memory 311. The training input data includes a blurred image (input image for training) and a first image. The blurred image is an image degraded due to the blur to be corrected. The true value image is an image containing the same object as the corresponding blurred image and having less blur (or no blur) than the blurred image. The blurred image and the true value image are undeveloped original images. The blurred image and the true value image can be prepared by actual imaging, imaging simulation, CG, etc.

[0083] In Example 2, corrections for all blurs occurring in the optical system 341 are learned at once. Therefore, the memory 311 stores blurred images including various blurs occurring in the optical system 341 and true value images corresponding to them. The blur of the optical system 341 is determined by the state of the optical system 341 (focal length, F number (aperture value), focal distance, etc.) and image plane coordinates in the vertical and horizontal directions, etc. The focal distance is the distance between the object on which the optical system 341 focuses and the image sensor 331 or the optical system 341.

[0084] Since the image pickup device 303 according to Example 2 is an interchangeable lens type, the optical system 341 can be connected to multiple types of image pickup devices having image sensors of different sizes and different numbers of pixels. Therefore, the color acquired by the image sensor, the pixel pitch of the image sensor, and the effect of the optical low-pass filter can be changed. In Example 2, all image sensors of the image pickup device that can be connected to the optical system 341 have an RGB Bayer array, and the color is fixed to RGB. However, the color filter array is not limited to this example.

[0085] The multiple degraded images stored in the memory 311 include image degradations that are changed in various ways within the range of values ​​that the image plane coordinates, the pixel pitch of the image sensor 331, and the optical low-pass filter can take in addition to the focal length, aperture value, and focus distance of the optical system 341.

[0086] In Example 2, the first map is a map that specifies the blur that acts on the blurred image. The first map has six values ​​for each channel, namely, the focal length of the optical system 341, the aperture value, the focal distance, the pixel pitch of the image sensor 331, the vertical image plane coordinates, and the horizontal image plane coordinates, which correspond to the blurs respectively. The first map may have channels indicating information such as a separation method and a cutoff frequency of an optical low-pass filter and a lens manufacturing error. By inputting the first map together with the blurred image into the machine learning model, the machine learning model can identify the blur that has acted on the blurred image from the first map, and thus the blur correction effect can be improved. If only the blurred image is input, the machine learning model estimates the blur from the blurred image and corrects it, or averages the correction learned blurs of various shapes, and therefore a high correction effect cannot be obtained.

[0087] The first map may be a map indicating information other than blur as long as it is information about the task to be performed. For example, it may be a map indicating noise intensity in a blurred image, or a depth map or segmentation map corresponding to the blurred image.

[0088] In step S102 , the calculator 314 inputs the training input data to the machine learning model (CNN) and generates an estimated image by correcting blur of the blurred image. Fig.10 An example of the configuration of the CNN in Example 2 is illustrated. The number of pixels of the blurred image and the true value image is (h, w, c), c = 1, because both are original images of the Bayer array. Example 2 inputs the blurred image 401 rearranged as (h / 2, w / 2, 4) to the CNN. The R component of Bayer is the first channel, G adjacent to R in the horizontal direction is the second channel, G adjacent to R in the vertical direction is the third channel, and B is the fourth channel. However, the input method is not limited to this example.

[0089] The first image 411 is connected to the blurred image 401 in the channel direction, so the number of pixels is (h / 2, w / 2, 6). However, the blurred image 401 or the first image 411 may be input to a different layer and converted into a feature map, and then connected to the other (or a feature map generated therefrom). In this case, the number of pixels in the vertical and horizontal directions between the blurred image 401 and the first image 411 does not necessarily have to be consistent with each other. The CNN configuration is merely an example, and another structure may be used.

[0090] The calculator 314 inputs the concatenation result of the blurred image 401 and the first image 411 in the channel direction to the convolution layer 421. The convolution layer 421 has 64 kernels and biases. The kernel size is (3, 3, 10). The structures of the activation function 422 and the residual block 423 are similar to those of Example 1. The convolution layer of the residual block 423 has 64 kernels and biases, and the kernel size is (3, 3, 64). The first feature map 412 is generated by including five residual blocks 423. The number of pixels of the first feature map 412 is (h / 2, w / 2, 64).

[0091] The calculator 314 rearranges the first feature map 412 into (h / 4, w / 4, 256) and generates a first reduced feature map 413. The first reduced feature map 413 is converted into a second reduced feature map 414 through five residual blocks 424. The convolution layer of the residual block 424 has 256 kernels and biases, and the kernel size is (3, 3, 256). The number of pixels of the second reduced feature map 414 is (h / 4, w / 4, 256).

[0092] The calculator 314 rearranges the second reduced feature map 414 into (h / 2, w / 2, 64) and generates a second feature map 415. The calculator 314 cascades the first feature map 412 and the second feature map 415 in the channel direction and inputs them to the convolution layer 425. The convolution layer 425 has 64 kernels and biases, and the kernel size is (3, 3, 128). As a result of the activation function 426, the five residual blocks 427, and the convolution layer 428, the estimated image 402 is generated. The convolution layer of the residual block 427 has 64 kernels and biases, and the kernel size is (3, 3, 64). The convolution layer 428 has four kernels and biases, and the kernel size is (3, 3, 64). The estimated image 402, which is the output data of the machine learning model, is an image in which the blur of the blurred image 401 has been corrected, and has the number of pixels of (h / 2, w / 2, 4).

[0093] In step S103, the updater 315 updates the CNN model parameters based on the difference between the estimated image and the true value image. When calculating the difference, the estimated image or the true value image is rearranged so that the number of pixels in each direction is consistent.

[0094] In step S104, the updater 315 determines whether the training of the machine learning model has been completed. If the training has not been completed, the process returns to step S101 to obtain a new set or sets of training input data and true value images. If the training has been completed, the model parameters are stored in the memory 311.

[0095] Reference will now be made to the diagram illustrating the flow of the process. Fig.11 and Fig.12 The flowchart of FIG. 3 gives a description of blur correction using a trained CNN (machine learning model) performed by the blur correction device 302. The blur correction device 302 including a computer has a memory 321, a communication unit 322, an acquisition unit 323, a calculator 324, and a display unit 325. The acquisition unit 323 and the calculator 324 perform the following processing (first step and second step) according to a (computer) program. The acquisition unit 323 and the calculator 324 configure a first generator and a second generator.

[0096] In step S301 , the acquisition unit 323 acquires a captured image (first image) 451 which is an undeveloped original image. The first image 451 contains blur due to aberration and diffraction generated in the optical system 341 .

[0097] In step S302, the acquisition unit 323 acquires the model parameters of the CNN for blur correction of the first image 451. In Example 2, a plurality of CNN model parameters may be prepared for each type of lens apparatus. However, the model parameters may be prepared in common for a plurality of types of lens apparatuses, or a plurality of model parameters may be prepared for one type of lens apparatus. Example 2 uses a CNN having a common configuration for all lens apparatuses, but the configuration of the CNN may be changed depending on the type of lens apparatus.

[0098] For example, information about the lens apparatus 304 used to capture the first image 451 may be acquired from the metadata of the first image 451. Identification (ID) information of the lens apparatus 304 is stored in the memory 342. When the lens apparatus 304 and the image pickup apparatus 303 are connected or when the image pickup apparatus 303 performs imaging, the ID information is acquired by the image pickup apparatus 303 via the communication unit 333 of the image pickup apparatus 303 and the communication unit 343 of the lens apparatus 304, and is written into the metadata of the first image 451.

[0099] In step S303, the calculator 324 generates a first map 461, which is a map specifying blur acting on the first image 451. Similar to the training, each channel of the first map 461 has the values ​​of the focal length, aperture value, and focal distance of the optical system 341, the pixel pitch of the image sensor 331, and the vertical and horizontal image plane coordinates when capturing the first image 451. Information about each value is acquired from the metadata of the first image 451.

[0100] Similar to the training, the first image 451 input to the CNN is rearranged in pixels. The first image 451 has a Bayer array in which the number of pixels in the vertical, horizontal, and channel directions is (W, H, C=1). Due to the rearrangement, the number of pixels of the first image 451 becomes (W / 2, H / 2, 4). Similar to the training, the first graph 461 is cascaded with the first image 451 in the channel direction, so the number of pixels becomes (W / 2, H / 2, 6). Example 2 generates the first graph 461 based on the number of pixels of the first image 451.

[0101] In the first graph 461, the channels representing the focal length, aperture value, and focal distance of the optical system 341 and the pixel pitch of the image sensor 331 each have the same value in the vertical and horizontal directions. This is because this information does not change spatially. On the other hand, the image plane coordinates in the vertical and horizontal directions change for each pixel of the first image 451, so the channels representing them have non-uniform values ​​in the vertical or horizontal directions.

[0102] In Example 2, the values ​​of the vertical and horizontal image plane coordinates are the coordinates of the corresponding R pixels in the first image 451. However, the coordinates of pixels of other colors may be used. The order of steps S302 and S303 is not limited.

[0103] In step S304, the calculator 324 inputs the input data including the first image 451 and the first graph 461 to a CNN (machine learning model) and generates a second image 452 in which blur has been corrected. The CNN has Fig.10 The structure is the same as that of , and the model parameters acquired in step S302 are used.

[0104] The second image 452, which is the output data of the machine learning model, may have unintended brightness and color changes. Fig.10In the CNN shown in , because the first feature map 412 is rearranged as the first reduced feature map 413 and the second reduced feature map 414 is rearranged as the second feature map 415, the number of pixels in the vertical and horizontal directions changes. In image processing using a machine learning model that includes a change in the number of pixels in the vertical or horizontal direction, unintentional brightness or color changes may appear in the second image 452 due to changes in the value of each channel. A first image 461 having multiple channels or having non-uniform values ​​in the vertical or horizontal direction within a channel is input to the CNN together with the first image 451. In this configuration, the CNN performs different processing depending on the value of the first image 461, so the task performed by the CNN becomes complicated, and unintentional brightness and color changes may appear in the second image 452.

[0105] In step S305, the calculator 324 generates a first residual image 462 by calculating the difference between the second image 452 and the fourth image 454. In Example 2, the fourth image 454 is an image obtained by performing an identity transformation on the first image 451 (i.e., an image identical to the first image 451). However, the first image 451 may be used as the fourth image 454 as is without performing an identity transformation. The calculator 324 generates a first residual image 462 by subtracting the fourth image 454 from the second image 452.

[0106] In step S306, the calculator 324 generates a weight map 463 by nonlinear processing based on the absolute value of the first residual image 462. The weight map 463 represents the weight of the second image 452 in the weighted average of the second image 452 and the fourth image 454. The weight map 463 is, for example, Fig.13A and Fig. 13B The nonlinear processing shown in is used to generate the nonlinear processing. Fig.13A and Fig. 13B The horizontal axis represents the absolute value of the first residual image 462 , and the vertical axis represents the weight of the weight map 463 .

[0107] exist Fig.13A In the nonlinear processing of , the weight map 463 is generated as a binary value of 0 or 1. When calculating the weighted average, the boundary between 0 and 1 will appear discontinuous, so if you perform such as Fig.13A If the nonlinear processing shown in FIG. 4 is not performed, then spatial blurring can be performed on the weight map 463. On the other hand, Fig. 13B In the nonlinear processing of , the weights are continuously changed according to the absolute value of the residual, so such a problem is unlikely to occur. The weight of each pixel in the weight map 463 decreases as the absolute value of the corresponding pixel in the first residual image 462 decreases.

[0108] In step S307, the calculator 324 generates a third image 453 based on the second image 452, the fourth image 454, and the weight map 463. The third image 453 is an estimated image in which unintended brightness and color changes are suppressed and blur is corrected. The calculator 324 calculates a weighted average by calculating the sum (A+B) of the product (A) of the weight map 463 and the second image 452 and the product (B) of the fourth image 454 subtracted from 1, and generates the third image 453. Alternatively, the third image 453 may be generated by adding the product of the weight map 463 and the first residual image 462 to the fourth image 454.

[0109] Since the third image 453 is generated by weighted averaging of the second image 452 and the fourth image 454 , the signal value of each pixel of the third image 453 has a value between the signal values ​​of the pixels corresponding to each pixel of the third image 453 in the second image 452 and the fourth image 454 .

[0110] By using the weight map 463, as the non-zero absolute value of the signal value of the corresponding pixel in the first residual image 462 decreases, the signal value of each pixel of the third image 453 is closer to the signal value of the pixel corresponding to each pixel of the third image in the fourth image 454 than to the signal value of the pixel corresponding to each pixel of the third image in the second image 452.

[0111] This example can provide an image processing method and an image processing device (blur correction device 302 ), each of which can suppress unintended luminance and color changes in image estimation using a machine learning model. Other embodiments

[0112] The embodiments of the present disclosure may also be implemented by reading out and executing computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more completely referred to as a 'non-transitory computer-readable storage medium') to perform one or more functions of the above-mentioned embodiments and / or a computer of a system or device including one or more circuits (e.g., an application-specific integrated circuit (ASIC)) for performing one or more functions of the above-mentioned embodiments, and by a method performed by a computer of the system or device by, for example, reading out and executing computer executable instructions from a storage medium to perform one or more functions of the above-mentioned embodiments and / or controlling one or more circuits to perform one or more functions of the above-mentioned embodiments. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessing unit (MPU)), and may include a network of separate computers or separate processors to read out and execute computer executable instructions. Computer executable instructions may be provided to the computer, for example, from a network or a storage medium. The storage medium may include, for example, a hard disk, a random access memory (RAM), a read-only memory (ROM), a storage device of a distributed computing system, an optical disk (such as a compact disk (CD), a digital versatile disk (DVD), or a Blu-ray disk (BD) TM ), one or more of flash memory devices, memory cards, etc.

[0113] The embodiments of the present invention may also be implemented by providing software (program) for executing the functions of the above embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0114] Although the present disclosure has described example embodiments, it is to be understood that some embodiments are not limited to the disclosed embodiments.The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

[0115] Each example can suppress unintended brightness and color changes in image estimation using a machine learning model.

Claims

1. An image processing method, comprising: A first step of generating a first residual image based on the first image using a machine learning model; as well as a second step of generating a third image based on the absolute value of the first residual image and the fourth image, The fourth image is the first image or an image based on the first image. Wherein, in the case where the second image is the sum of the fourth image and the first residual image, as the non-zero absolute value of the signal value of the pixel in the first residual image corresponding to each pixel in the third image decreases, the second step sets the signal value of each pixel in the third image to a value that is closer to the signal value of the pixel in the fourth image corresponding to each pixel in the third image than the signal value of the pixel in the second image corresponding to each pixel in the third image.

2. The image processing method according to claim 1, characterized in that The second step sets the signal value of each pixel in the third image to a value between the signal value of the pixel in the fourth image corresponding to each pixel in the third image and the signal value of the pixel in the second image corresponding to each pixel in the third image.

3. The image processing method according to claim 1, characterized in that In the second step, the third image is generated using a result of nonlinear transformation based on an absolute value of the first residual image.

4. The image processing method according to claim 1, characterized in that In the second step, the third image is generated using a result of threshold processing based on an absolute value of the first residual image.

5. The image processing method according to claim 1, characterized in that In the second step, the third image is generated based on a second residual image generated from the first residual image using soft thresholding.

6. The image processing method according to claim 5, characterized in that In the second step, the second residual image is generated by applying a coefficient having a threshold value greater than 1 based on the soft thresholding to a residual image obtained by performing the soft thresholding on the first residual image.

7. The image processing method according to claim 1, characterized in that In the second step, the third image is generated based on: one of the first residual image and the second image, the fourth image, and a weight map based on an absolute value of the first residual image.

8. The image processing method according to claim 7, characterized in that The weight map represents the weight of the first residual image or the second image, In the weight map, for pixels in the first residual image with smaller non-zero absolute values, the weights are reduced.

9. The image processing method according to claim 1, characterized in that In the first step, the number of pixels in the vertical or horizontal direction in any one of the multiple feature maps generated within the machine learning model, the output data output from the machine learning model, and the input data input to the machine learning model are different from each other.

10. The image processing method according to claim 1, characterized in that In the first step, the input data input to the machine learning model includes the first image and the first graph, The first graph has a plurality of channels.

11. The image processing method according to claim 1, characterized in that In the first step, the input data input to the machine learning model includes the first image and the first graph, The first graph has non-uniform values ​​in the vertical or horizontal direction.

12. The image processing method according to claim 1, characterized in that The machine learning model has a plurality of model parameters previously determined through training, Wherein a number of bits of at least one of the plurality of model parameters is smaller during generation of the first residual image than during the training.

13. The image processing method according to claim 1, characterized in that In the second step, the third image is generated based on information about an edge of the fourth image or a magnitude of a signal value of each pixel in the fourth image.

14. The image processing method according to claim 1, characterized in that The second image is an image obtained by performing at least one of the following on the first image: degradation correction, resolution improvement, contrast improvement, demosaicing, dynamic range extension, grayscale increase, illumination change, change of defocus blur shape, focus distance change, and depth of field change.

15. The image processing method according to any one of claims 1 to 14, characterized in that The machine learning model has a plurality of first layers and a plurality of second layers, wherein the plurality of first layers have a first number of digits, and the plurality of second layers have a second number of digits that is smaller than the first number of digits. 16 . A non-transitory computer-readable storage medium storing a program for causing a computer to execute the image processing method according to claim 1 .

17. An image processing device, comprising: A first generator configured to generate a first residual image based on the first image using a machine learning model; as well as a second generator configured to generate a third image based on an absolute value of the first residual image and a fourth image, wherein the fourth image is the first image or an image based on the first image, and Wherein, in the case where the second image is the sum of the fourth image and the first residual image, as the non-zero absolute value of the signal value of the pixel in the first residual image corresponding to each pixel in the third image decreases, the second generator sets the signal value of each pixel in the third image to a value that is closer to the signal value of the pixel in the fourth image corresponding to each pixel in the third image than the signal value of the pixel in the second image corresponding to each pixel in the third image.

18. An image pickup device, comprising: The image processing device according to claim 17; as well as An image sensor is configured to acquire a captured image as a first image.

Images