Image processing methods, devices and systems, training methods and devices, storage media

Abstract

This invention relates to image processing methods, apparatus and systems, training methods and apparatus, and storage media. The image processing method includes: a first step for acquiring a captured image; and a second step for generating a first image based on the captured image using a machine learning model. The first image is a graph representing the magnitude and range of signal values ​​in a region where a subject in a brightness saturation region of the captured image is diffused due to blurring generated during the image capture step.

Description

Technical Field

[0001] This invention relates to an image processing method for performing recognition or regression tasks on blurred images using machine learning models. Background Technology

[0002] Li Xu et al., Deep Convolutional Neural Network for Image Deconvolution, Advances in Neural Information Processing Systems 27, NIPS 2014 (hereinafter referred to as "Xu et al."), disclosed a method for sharpening blur in captured images using a convolutional neural network (CNN), a type of machine learning model. This method generates a training dataset by blurring images with signal values ​​equal to or higher than the brightness saturation values ​​in the captured image, and sharpens the blur by training the CNN using the training dataset to suppress even adverse effects around brightness saturation regions.

[0003] However, the method disclosed by Xu et al. may produce artifacts (pseudo-structures) on the subject at locations unrelated to brightness saturation. An artifact is specifically a localized reduction or increase in signal values ​​that differs from the actual structure of the subject. Artifacts and how they are generated will be described in detail later. The accuracy of tasks involving blurred images (other than blur sharpening) is also degraded due to the effects of brightness saturation. Summary of the Invention

[0004] This invention provides an image processing method that can suppress the reduction in accuracy caused by brightness saturation in recognition or regression tasks using machine learning for blurred images.

[0005] An image processing method according to one aspect of the present invention includes: a first step of acquiring a captured image; and a second step of generating a first image based on the captured image using a machine learning model. The first image is a graph representing the magnitude and range of signal values ​​in a region where a subject in a brightness saturation region of the captured image is diffused due to blurring generated during the image capture step. An image processing apparatus corresponding to the above-described image processing method also constitutes another aspect of the present invention.

[0006] An image processing apparatus includes: an acquisition unit configured to acquire a captured image; and a generation unit configured to generate a first image based on the captured image using a machine learning model. The first image is a graph representing the magnitude and range of signal values ​​in a region where a subject in a brightness saturation region of the captured image is diffused due to blurring generated during the image capture process.

[0007] A storage medium storing a program that causes a computer to perform the image processing method described above.

[0008] According to another aspect of the present invention, a method for training a machine learning model includes the following steps: acquiring an original image; generating a blurred image by adding blur to the original image; setting a first region using a threshold based on image and signal values ​​of the original image; generating a first image having the signal values ​​of the original image in the first region; generating a first ground truth map by adding blur to the first image; and training the machine learning model using the blurred image and the first ground truth map. A training device corresponding to this training method also constitutes another aspect of the present invention.

[0009] A training apparatus for a machine learning model, the training apparatus comprising: an acquisition unit configured to acquire an original image; a blurring unit configured to generate a blurred image by adding blur to the original image; a setting unit configured to set a first region using a threshold based on image and signal values ​​of the original image; an image generation unit configured to generate a first image having the signal values ​​of the original image in the first region; a graph generation unit configured to generate a first ground truth map by adding blur to the first image; and a training unit configured to train the machine learning model using the blurred image and the first ground truth map.

[0010] A storage medium that stores a program that enables a computer to execute the training method of the machine learning model described above.

[0011] An image processing method includes the following steps: acquiring a captured image; and generating a model output based on the captured image using a machine learning model trained using the training method described above. The model output is a recognition label or a spatially arranged signal sequence corresponding to the captured image.

[0012] An image processing system includes a first device and a second device capable of communicating with each other. The first device includes a transmitter configured to send an captured image and a request for performing processing to the second device, wherein the second device includes: a receiver configured to receive the captured image and the request from the first device; and a generator configured to generate a first image based on the captured image using a machine learning model in response to the request, wherein the first image is a graph representing the magnitude and range of signal values ​​in a region where a subject in a saturated region of the captured image is diffused due to blur generated during the imaging step of the captured image.

[0013] Further features of the invention will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0014] Figure 1 This is a configuration diagram of the machine learning model according to the first embodiment.

[0015] Figure 2A and Figure 2B This is an explanatory diagram illustrating the relationship between the subject and the captured image according to the first to third embodiments, as well as the first figure.

[0016] Figure 3 This is a block diagram of an image processing system according to the first embodiment.

[0017] Figure 4 This is an external view of the image processing system according to the first embodiment.

[0018] Figures 5A to 5C This is an illustrative diagram of the artifacts in the first embodiment.

[0019] Figure 6 This is a flowchart of training a machine learning model according to the first to third embodiments.

[0020] Figure 7 This is a flowchart output based on the generative model of the first and second embodiments.

[0021] Figure 8 This is a block diagram of an image processing system according to the second embodiment.

[0022] Figure 9 This is an external view of the image processing system according to the second embodiment.

[0023] Figure 10 This is a configuration diagram of the machine learning model according to the second embodiment.

[0024] Figure 11 This is a block diagram of an image processing system according to a third embodiment.

[0025] Figure 12 This is an external view of the image processing system according to the third embodiment.

[0026] Figure 13 This is a configuration diagram of the machine learning model according to the third embodiment.

[0027] Figure 14 This is a flowchart output based on the generative model of the third embodiment. Detailed Implementation

[0028] A detailed description of embodiments of the invention will now be given with reference to the accompanying drawings. Corresponding elements in the various drawings will be designated by the same reference numerals, and repeated descriptions will be omitted.

[0029] Before detailing the various embodiments, the problem to be solved by the present invention will be described. In image recognition or regression tasks, theory-based methods may suffer from accuracy degradation due to (modeled) elements ignored by assumptions and approximations. On the other hand, since estimation can be achieved based on training data without assumptions and approximations, methods using machine learning models improve task accuracy by training the machine learning model using training data that includes these elements. That is, in image recognition or regression tasks, methods using machine learning models are more accurate than theory-based methods.

[0030] For example, techniques for sharpening blur in blurred images assume that the image's brightness saturation (also known as overexposure) is a factor mentioned above. Theoretical methods (such as the Wiener filter) assume no brightness saturation, thus failing to properly sharpen blur around saturated areas and leading to undesirable effects such as ringing. On the other hand, the machine learning method disclosed by Xu et al. can correct blur even in cases of brightness saturation. However, the method disclosed by Xu et al. may sometimes have lower accuracy due to artifacts contained in the corrected image.

[0031] The problem this invention addresses is the accuracy degradation caused by brightness saturation in recognition or regression tasks using machine learning models applied to blurred captured images. This blurring includes one or a combination of some or all of the following: blurring caused by aberrations, diffraction, or defocusing of the optical system used to capture the image; blurring caused by optical low-pass filters; blurring caused by pixel apertures in the image sensor; and blurring caused by camera shake or subject shake during image capture. Recognition tasks are tasks that determine the class corresponding to an input image. For example, recognition tasks are tasks that identify the nature or meaning of a subject, such as classifying a subject in an image as a person, dog, car, etc., and classifying facial images into various facial expressions (such as smiling and crying). Classes typically have discrete variables. Classes also have recognition labels as scalar values, or a signal sequence that spatially arranges the recognition labels, such as a segmentation map. Regression tasks are tasks that determine the signal sequence of continuous variables spatially arranged to correspond to an input image. For example, regression tasks can be used to estimate a blurred image with sharpened sharpening from a blurred image, or to estimate a depth map of the subject space from an image.

[0032] Now refer to Figure 2A Provide a description of the differences in properties between the periphery of brightness saturation and other areas in a blurred captured image. Figure 2A The brightness distribution relationship between the subject and the captured image is shown. Figure 2A In the diagram, the horizontal axis represents spatial coordinates, and the vertical axis represents brightness. Solid lines represent unblurred captured images, and dashed lines represent actually blurred captured images. Dotted lines represent the brightness distribution before brightness saturation cropping. Even when blurred during image processing, subject 251 has a brightness equal to or less than the brightness saturation value. Therefore, cropping using the brightness saturation value does not occur, and an unsaturated blurred image 261 is obtained. On the other hand, as a result of blurring during image processing, subject 252 has a brightness equal to or greater than the brightness saturation value, and is therefore cropped using the brightness saturation value, resulting in a saturated blurred image 262. In the unsaturated blurred image 261, the information of the subject is attenuated due to blurring. On the other hand, in the saturated blurred image 262, the information of the subject is attenuated not only due to blurring but also due to cropping using the brightness saturation signal value. Therefore, the way the subject information is attenuated varies depending on the brightness saturation. This is the first factor that distinguishes the nature of the periphery of brightness saturation from other areas.

[0033] The second factor causing the different properties is described below. That is, signal value cropping results in pseudo-edges in the subject that were not originally present at the edges of the brightness saturation region. The saturation blurred image 262 originally had a smooth brightness distribution represented by the dotted lines above the brightness saturation values, but the cropping of the brightness saturation values ​​creates discontinuous edges.

[0034] Furthermore, in the captured image, due to blur, signal values ​​leak from the subject 252 in the brightness saturation area to its periphery. The magnitude and range of the leaked signal values ​​increase with the brightness of the subject 252 in the brightness saturation area, but because the signal values ​​of the brightness saturation area are cropped, it is not easy to know the magnitude and range of the leaked signal values. Therefore, the third factor causing the different properties is that the signal values ​​of the subject and the signal values ​​leaked due to blur cannot be separated around the brightness saturation area (even if the shape of the blur is known).

[0035] Because of these three factors, the periphery of the saturated area has different properties than other areas, so high-precision tasks cannot be achieved unless different processing is performed on each of them.

[0036] Machine learning models can perform processing with different effects based on the features of the input image, rather than performing a uniform processing on the input image. Therefore, for example, in the example of sharpening a blur in a captured image, the machine learning model internally determines whether the target region contains a brightly saturated blur image (saturated blur image) or another type of blur image (unsaturated blur image) and performs different sharpening processes accordingly. Thus, both blurry images can be sharpened. However, the machine learning model's judgment may be incorrect. For example, in... Figure 2A In the saturated blurred image 262, when the target region is located around the brightness saturation region, the machine learning model can determine that the target region is affected by brightness saturation because the brightness saturation region is near the target region. However, when the target region is located at position 271 far from the brightness saturation region, it is not easy to determine whether position 271 is affected by brightness saturation, and ambiguity increases. As a result, the machine learning model may make incorrect judgments (misidentifications) at positions far from the brightness saturation region. Therefore, the task of sharpening the blur performs sharpening processing specifically for saturated blurred images on unsaturated blurred images. Then, artifacts appear in the image with sharpened blur, and the accuracy of the task deteriorates. This artifact will be described in detail in the first embodiment.

[0037] This discussion applies to tasks other than blur sharpening, and the accuracy of these tasks deteriorates due to misclassifications by machine learning models between regions affected by brightness saturation and other regions. For example, in recognition tasks, if an unsaturated blurred image is incorrectly identified as a saturated blurred image, it is judged that signal values ​​leaking from the brightness-saturated regions are added to the blurred image, resulting in the extraction of feature values ​​that differ from those of the actual unsaturated image, thus degrading the accuracy of the task.

[0038] The main idea of ​​this embodiment for solving the problem is described below. This embodiment uses a machine learning model to generate a first image from a blurred captured image. The first image is a graph (spatially arranged signal sequence) showing the magnitude and range of signal values ​​in the region where the subject in the saturated area of ​​the captured image has diffused due to blur generated during the image processing. In other words, the first image is a graph showing the diffusion of brightness values ​​in the high-brightness region that includes the saturated area of ​​the captured image (or a graph showing the distribution of the high-brightness subject that causes saturation due to blur generated during image processing).

[0039] Figure 2A An example of the first image in the captured image is provided by Figure 2B The dashed lines in the diagram illustrate this. By explicitly instructing the machine learning model to generate the first image, the model can accurately estimate the presence or absence of brightness saturation effects in the captured image and their magnitude. By generating the first image, the machine learning model can correctly perform processing specific to areas affected by brightness saturation, as well as processing specific to other areas, for any given region. Therefore, by instructing the machine learning model to generate the first image, the task's accuracy is significantly improved compared to not generating the first image (directly generating recognition labels and sharpened images from the captured image).

[0040] In the following description, the stage of determining the weights of a machine learning model based on a training dataset will be referred to as training, and the stage of performing a recognition or regression task on captured images using the machine learning model with the trained weights will be referred to as estimation. Machine learning models include, for example, neural networks, genetic programming, Bayesian networks, etc. Neural networks include CNNs (Convolutional Neural Networks), GANs (Generative Adversarial Networks), RNNs (Recurrent Neural Networks), etc.

[0041] First Embodiment

[0042] The image processing system according to a first embodiment of the present invention will now be described. As a task utilizing a machine learning model, this embodiment discusses sharpening blur in captured images, including those with saturated brightness. The blur to be sharpened includes blur caused by aberrations and diffraction generated in the optical system, as well as blur caused by optical low-pass filters. The effects of this embodiment can also be obtained when sharpening blur caused by pixel apertures, defocus, and jitter. This embodiment is also applicable to and effective in tasks other than blur sharpening.

[0043] Figure 3 This is a block diagram of the image processing system 100 according to this embodiment. Figure 4 This is an external view of the image processing system 100. The image processing system 100 includes a training device 101 and an image processing device 103 connected to each other via a wired or wireless network. The training device 101 includes a memory 101a, an acquisition unit 101b, a calculation unit 101c, and an update unit 101d. The image processing device 103 includes a memory 103a, an acquisition unit 103b, and a sharpening unit 103c. A camera device 102, a display device 104, a recording medium 105, and an output device 106 are connected to the image processing device 103 via a wired or wireless connection.

[0044] An image captured by camera device 102 of the subject space is input to image processing device 103. Due to aberrations and diffraction in the optical system 102a of camera device 102 and the optical low-pass filter in image sensor 102b, the captured image is blurred, and information about the subject is attenuated. Image processing device 103 uses a machine learning model to sharpen the blur in the captured image and generates a first image and a blurred (or deblurred) image (model output). The machine learning model is trained by training device 101. Image processing device 103 obtains information related to the machine learning model from training device 101 in advance and stores it in memory 103a. Image processing device 103 is used to adjust the blur sharpening intensity. The training and estimation of the machine learning model, as well as the adjustment of the blur sharpening intensity, will be described in detail later.

[0045] The user can adjust the blur sharpening intensity while inspecting the image displayed on the display device 104. The blurred and sharpened image with adjusted intensity is stored in the memory 103a or the recording medium 105 and output to an output device 106, such as a printer, as needed. The captured image can be a grayscale image or can have multiple color components. Undeveloped RAW images or developed images can be used.

[0046] Now refer to Figures 5A to 5C Describe the artifacts that appear when a machine learning model sharpens a blurred image. Artifacts are the reduction or increase of signal values ​​in local areas that differ from the actual structure of the subject. Figures 5A to 5C This is an illustration of artifacts, where the horizontal axis represents spatial coordinates and the vertical axis represents signal values. Figures 5A to 5C The spatial changes in the image's signal values ​​are shown, corresponding to the R, G, and B (red, green, and blue) color components, respectively. Since the image is displayed as an 8-bit image, the saturation value is 255.

[0047] exist Figures 5A to 5C In the diagram, the dashed line represents the captured image (blurred image), and the thin solid line represents the unblurred ground truth image. Since no pixels have reached their brightness saturation values, there is no effect from brightness saturation. The dotted line represents the blurred-sharpened image, where the blurred image is sharpened using a conventional machine learning model not applied in this embodiment. In the blurred-sharpened image represented by the dotted line, edge blur is sharpened, but a reduction in signal values ​​not present in the ground truth image appears near the center. This reduction does not occur adjacent to the edges but rather far from them, and due to the wide generated area, it exhibits a different undesirable effect than undershoot. This is an artifact that occurs when sharpening blur.

[0048] from Figures 5A to 5C The comparison shows that the degree of signal reduction varies depending on the color component. Figures 5A to 5C In the image, the signal value decreases in the order of G, R, and B. This trend is similar in the undeveloped RAW image. In a conventionally blurred and sharpened image represented by dotted lines, flat areas in the ground truth image are shown as artifacts in dark areas tinted green. Although Figures 5A to 5C An example is shown where the signal value is lower than that of the ground truth image, but the signal value may be higher.

[0049] As mentioned above, this artifact is generated due to the following: the machine learning model misclassifying regions affected by brightness saturation and other regions, and the incorrect application of blur sharpening techniques specifically designed for saturated blurred images to unsaturated blurred images. (See from...) Figure 2A Understandably, the higher the brightness of the subject, the larger the absolute value of the residual component of the blur sharpening (which is the difference between the blurred and unblurred images). If blur sharpening specifically designed for saturated blur images is applied to an unsaturated blurred image, the signal values ​​are excessively altered. As a result, such as... Figures 5A to 5C The dotted lines in the image show the regions where the generated signal value is lower than the signal value of the ground truth image (solid line).

[0050] Typically, optical systems used for visible light are designed to perform best in G within the RGB spectrum. Because the blur spread (PSF: point spread function) is wider in R and B than in G, the edges of saturated blurred images obtained by photographing high-intensity subjects are easily colored (purple stripes) in R and B. When correcting saturated blurred images, the residual components of blur sharpening in R and B become larger than in G. When a non-saturated blurred image is incorrectly identified as a saturated blurred image, the decrease in signal values ​​in R and B is greater than the decrease in signal value in G, and as shown in 5A to... Figure 5C As shown, artifacts appear in the dark areas tinted green.

[0051] on the other hand, Figures 5A to 5C The dashed lines shown represent the result of sharpening the blur using a machine learning model that estimates the first image according to this embodiment. It can be understood that blur sharpening is achieved by suppressing artifacts. This is because the machine learning model, which has been explicitly instructed to estimate the first image, is less likely to misjudge areas affected by brightness saturation compared to other areas. Figures 5A to 5C It should be understood that this embodiment suppresses the degradation of the task's accuracy.

[0052] Now refer to Figure 6 Describes the training of the machine learning model performed by training device 101. Figure 6 This is a flowchart of the training process for a machine learning model. Figure 6 Each step in the process is executed by the memory 101a, the acquisition unit 101b, the calculation unit 101c, or the update unit 101d in the training device 101.

[0053] First, in step S101, the acquisition unit (acquirer) 101b acquires one or more original images from the memory 101a. The original image is an image with a signal value higher than a second signal value, where the second signal value corresponds to the brightness saturation value of the captured image. Since the signal value can be standardized when inputting into the machine learning model, the second signal value and the brightness saturation value of the captured image do not necessarily need to be identical. Because the machine learning model is trained based on the original image, the original image can be an image with various frequency components (edges with different orientations and intensities, gradients, flat areas, etc.). The original image can be a real-world image or CG (computer graphics).

[0054] Next, in step S102, the calculation unit (blurring unit) 101c adds blur to the original image and generates a blurred image. The blurred image is the image input to the machine learning model during training and corresponds to the captured image during estimation. The added blur is the blur to be sharpened. In this embodiment, blur caused by aberrations and diffraction of the optical system 102a and blur caused by the optical low-pass filter in the image sensor 102b are added. The shape of the blur caused by aberrations and diffraction of the optical system 102a changes according to the image plane coordinates (image height and azimuth). It also changes according to the zoom, aperture (aperture stop), and focus state of the optical system 102a. When attempting to train the machine learning model comprehensively to sharpen all these blurs, multiple blurred images can be generated using multiple blurs generated by the optical system 102a. In the blurred image, signal values ​​exceeding a second signal value are cropped to reproduce the brightness saturation that occurs during the image processing of the captured image. If necessary, noise generated by the image sensor 102b can be added to the blurred image.

[0055] Next, in step S103, the calculation unit (setting unit) 101c sets a first region using a threshold based on the image and signal values ​​of the original image. This embodiment uses a blurred image as the image based on the original image, but the original image itself can be used. The first region is set by comparing the signal values ​​of the blurred image and a threshold value for the signal values. More specifically, the region where the signal value of the blurred image is equal to or higher than the threshold value is set as the first region. In this embodiment, the threshold value for the signal values ​​is set as a second signal value. Therefore, the first region represents a brightness saturation region in the blurred image. However, the threshold value for the signal values ​​and the second signal value do not necessarily have to be the same. The threshold value for the signal values ​​can be set to a value slightly smaller than the second signal value (such as 0.9 times).

[0056] Next, in step S104, the calculation unit (image generation unit) 101c generates a first image having the signal value of the original image in the first region. The first image has signal values ​​different from the signal values ​​of the original image in regions other than the first region. The first image may have a first signal value in regions other than the first region. In this embodiment, the first signal value is 0, but not limited to 0. In this embodiment, the first image has the signal value of the original image only in the brightness saturation region of the blurred image, and the signal value is 0 in other regions.

[0057] Next, in step S105, the calculation unit (image generation unit) 101c adds blur to the first image and generates a first ground truth map. The added blur is the same as the blur added to the blurred image. Thus, a first ground truth map is generated from the subject in the brightness saturation region of the blurred image. This first ground truth map is a graph (spatially arranged signal sequence) representing the magnitude and range of signal values ​​leaked to the surroundings due to blur. Similar to the blurred image, this embodiment uses a second signal value to crop the first ground truth map, but cropping may not be necessary.

[0058] Next, in step S106, the acquisition unit 101b acquires the ground truth model output. In this embodiment, since the task is to sharpen the blur, the ground truth model output is an image with less blur than the blurred image. This embodiment generates the ground truth model output by cropping the original image using the second signal value. If the original image lacks high-frequency components, the image created by reducing the size of the original image can be used as the ground truth model output. In this case, the reduction is similarly performed when the blurred image is generated in step S102. Step S106 can be executed at any time as long as it is after step S101 and before step S107.

[0059] Next, in step S107, the computing unit 101c uses a machine learning model to generate a first image and model output based on the blurred image. Figure 1 This is a block diagram of a machine learning model. This embodiment uses, but is not limited to, [the following]. Figure 1 The machine learning model shown. Figure 1 In this embodiment, a blurred image 201 and a brightness saturation map 202 are input into a machine learning model. The brightness saturation map 202 is a map (second map) representing the region of brightness saturation in the blurred image 201 (where the signal value is equal to or higher than a second signal value). For example, it can be generated by binarizing the blurred image 201 using the second signal value. However, it is not necessary to use the brightness saturation map 202. The blurred image 201 and the brightness saturation map 202 are concatenated along the channel direction and input into the machine learning model, but this embodiment is not limited to this example. For example, the blurred image 201 and the brightness saturation map 202 can be converted into feature maps, and these feature maps can be concatenated along the channel direction. Information other than the brightness saturation map 202 can be added to the input.

[0060] The machine learning model has multiple layers, in which a linear sum of the layer's input and weights is computed. The initial values ​​of the weights can be determined by random numbers, etc. This embodiment employs a CNN (Convolutional Neural Network) as the machine learning model, using convolutions of the input and filters as a linear sum (the values ​​of the individual elements of the filter correspond to the weights, and may include a sum with bias), but is not limited to this example. In each layer, nonlinear transformations are performed as needed using activation functions such as ReLU (Rectified Linear Unit) or the sigmoid function. If necessary, the machine learning model can have residual blocks or skip connections (also known as shortcut connections). The first image 203 is generated via multiple layers (16 convolutional layers in this embodiment). This embodiment generates the first image 203 by summing the output of layer 211 and the individual elements of the brightness saturation image 202, but the configuration is not limited to this example. The first image can be generated directly as the output of layer 211. Alternatively, the result of arbitrary processing of the output of layer 211 can be used as the first image 203.

[0061] Next, the first image 203 and the blurred image 201 are concatenated along the channel direction and input to subsequent layers, and a model output 204 is generated through multiple layers (16 convolutional layers in this embodiment). The model output 204 is also generated by summing the output of layer 212 and the individual elements of the blurred image 201, but this is not limited to this example. This embodiment uses 64 types of 3×3 filters in each layer (where the number of filter types in layers 211 and 212 is the same as the number of channels in the blurred image 201), but convolution is not limited to this example.

[0062] Next, in Figure 6 In step S108, the update unit (training unit) 101d updates the weights of the machine learning model based on the error function. In this embodiment, the error function is a weighted sum of the error between the first image 203 and the first ground truth image and the error between the model output 204 and the ground truth model output. MSE (mean squared error) is used to calculate the error. The weights for both are 1. The error function and weights are not limited to these examples. The weights can be updated using error backpropagation methods, etc. The error can be calculated using residual components. When using residual components, the error between the difference component between the first image 203 and the brightness saturation image 202 and the difference component between the first ground truth image and the brightness saturation image 202 is used. Similarly, the error between the difference component between the model output 204 and the blurred image 201 and the difference component between the ground truth model output and the blurred image 201 is used.

[0063] Next, in step S109, the update unit 101d determines whether the training of the machine learning model is complete. Training completion can be determined based on factors such as whether the number of weight update repetitions has reached a predetermined number, or whether the weight change during the update period is less than the default value. If it is determined in step S109 that training is not yet complete, the process returns to step S101, and the acquisition unit 101b acquires one or more new original images. Conversely, when it is determined that training is complete, the update unit 101d ends the training and stores the configuration and weight information of the machine learning model in the memory 101a.

[0064] The training method described above enables the machine learning model to estimate a first image, which represents the magnitude and range of signal values ​​of the subject diffused due to blurring in the saturated regions of the blurred image (the captured image under estimation). By explicitly estimating the first image, the machine learning model can sharpen the blur for each saturated and unsaturated blurred image within appropriate regions, thereby suppressing artifacts.

[0065] Now refer to Figure 7 A description is given of using a trained machine learning model executed by the image processing device 103 to sharpen blur in captured images. Figure 7 This is a flowchart of the generated model output. Figure 7 Each step in the process is executed by the memory 103a, the acquisition unit 103b, or the sharpening unit 103c in the image processing device 103.

[0066] First, in step S201, the acquisition unit (acquirer) 103b acquires the captured image and the machine learning model. Information related to the structure and weights of the machine learning model is retrieved from memory 103a.

[0067] Next, in step S202, the sharpening unit (generation unit) 103c uses a machine learning model to generate a first image from the captured image, and generates a blurred-sharpened image (model output) in which the blur in the captured image has been sharpened. Similar to the training process, the machine learning model has... Figure 1 The configuration shown is similar to that used in training. A brightness saturation map representing the brightness saturation regions in the captured image is generated and input, resulting in the first image and the model output.

[0068] Next, in step S203, the sharpening unit 103c synthesizes the captured image and the model output based on the first image. Unlike other areas, subject information is attenuated due to brightness saturation around brightness saturation areas in the captured image, making it difficult to sharpen the blur (estimate the attenuated subject information). Therefore, undesirable effects (ringing, undershoot, etc.) accompanying blur sharpening may occur around brightness saturation areas. To suppress this undesirable effect, the model output and the captured image are synthesized. At this time, the synthesis based on the first image can increase the weight of the captured image only around brightness saturation areas where undesirable effects may occur, while suppressing the reduction of blur sharpening effect in unsaturated blurred images. This embodiment provides synthesis in the following manner. The first image is normalized by a second signal value and used as a weight map for the captured image, and is weighted and averaged with the model output. For the model output, a weight map obtained by subtracting the weight map for the captured image from all 1 images is used. By changing the signal value that normalizes the first image, the balance between blur sharpening effect and undesirable effects can be adjusted. A synthesis method that replaces the model output with the captured image only in areas where the first image has a value equal to or higher than a predetermined signal value can be used.

[0069] The above configuration can provide an image processing system that can suppress the accuracy degradation caused by brightness saturation when using machine learning models to sharpen blur.

[0070] Second Embodiment

[0071] An image processing system according to a second embodiment of the present invention will now be described. In this embodiment, the task of the machine learning model is to transform a blurred effect (bokeh) into a captured image, including brightness saturation. The transformation of the blurred effect is the task of converting a defocus blur acting on the captured image into a blur with a shape different from the shape of the defocus blur itself. For example, when the defocus blur has double-line blur or vignetting, it is converted into a disk (shape with flat intensity) or a Gaussian blur. In the transformation of the blurred effect, the defocus blur is enlarged, and no blur sharpening (estimating the attenuated subject information) is performed. The method described in this embodiment can achieve the same effect in tasks other than transforming the blurred effect.

[0072] Figure 8 This is a block diagram of the image processing system 300 according to this embodiment. Figure 9This is an external view of the image processing system 300. The image processing system 300 includes a training device 301, a camera device 302, and an image processing device 303. The training device 301 and the image processing device 303, as well as the image processing device 303 and the camera device 302, are connected to each other via wired or wireless networks. The training device 301 includes a memory 311, an acquisition unit 312, a calculation unit 313, and an update unit 314. The camera device 302 includes an optical system 321, an image sensor 322, a memory 323, a communication unit 324, and a display unit 325. The image processing device 303 includes a memory 331, a communication unit 332, an acquisition unit 333, and a conversion unit 334.

[0073] The captured image by camera device 302 is affected by defocus blur of a shape corresponding to optical system 321. The captured image is transmitted to image processing device 303 via communication unit (transmitter) 324. Image processing device 303 receives the captured image via communication unit (receiver) 332 and transforms the blur effect using configuration and weight information of a machine learning model stored in memory 331. The configuration and weight information of the machine learning model is trained by training device 301, obtained in advance from training device 301, and stored in memory 331. The blurred transformed image (model output) of the transformed blurred image is sent to camera device 302, stored in memory 323, and displayed on display unit 325.

[0074] Now refer to Figure 6 The training of the machine learning model performed by training device 301 is described. Descriptions of matters common to the first embodiment will be omitted.

[0075] First, in step S101, the acquisition unit 312 acquires one or more original images from the memory 311. Next, in step S102, the calculation unit 313 sets a defocus amount for the original image and generates a blurred image that adds a defocus blur corresponding to the defocus amount to the original image. The shape of the defocus blur changes according to the zoom and aperture of the optical system 321. The defocus blur also changes according to the focal length of the optical system 321 and the current defocus amount of the subject. The defocus blur also changes according to the image height and azimuth angle. When attempting to comprehensively train a machine learning model capable of converting all these defocus blurs, multiple blurred images can be generated by using multiple defocus blurs generated in the optical system 321. In the conversion of the blur effect, the out-of-focus focused subject can be maintained before and after the conversion. Since the machine learning model needs to be trained to maintain the focused subject, a blurred image with a defocus amount of 0 is also generated. The blurred image with a defocus amount of 0 may not be blurred, or it may be blurred by aberrations or diffraction at the focal plane of the optical system 321.

[0076] Next, in step S103, the calculation unit 313 sets a first region based on a threshold of the blurred image and signal values. Next, in step S104, the calculation unit 313 generates a first image in the first region containing the signal values ​​of the original image. Next, in step S105, the calculation unit 313 adds a defocus blur identical to the defocus blur in the blurred image to the first image and generates a first ground truth map. Next, in step S106, the acquisition unit 312 acquires the ground truth model output. This embodiment trains a machine learning model to convert defocus blur into disc blur (a blur with a circular and flat intensity distribution). Therefore, disc blur is added to the original image to generate the ground truth model output. However, the shape of the blur to be added is not limited to this example. A disc blur with diffusion corresponding to the amount of defocus in the blurred image is added. The added disc blur is more blurred than the defocus blur added when generating the blurred image. In other words, the disc blur has a lower MTF (modulation transfer function) than the defocus blur added in the generation of the blurred image. When the defocus amount is 0, it is the same as the generation of the blurred image.

[0077] Next, in step S107, the computing unit 313 uses a machine learning model to generate a first image and model output from the blurred image. Figure 10 This is a block diagram of the machine learning model according to this embodiment. This embodiment uses a model with... Figure 10 The machine learning model shown is a configuration shown, but the invention is not limited to this embodiment. Figure 10 In this embodiment, a blurred image 401 and a brightness saturation map (second image) 402 representing the brightness saturation region in the blurred image 401 are connected and input along the channel direction, and a first feature map 411 is generated through multiple layers (nineteen convolutional layers). A first image 403 and a model output 404 are generated based on the first feature map. In this embodiment, the layers are branched in the middle of the machine learning model, and the first feature map 411 is input to each branch. The first image 403 is generated from the first feature map 411 through one layer (one convolutional layer), and the model output 404 is generated through multiple layers (twenty convolutional layers), but the number of layers is not limited to this embodiment. The layer may not be branched, and the first image 403 and the model output 404 may be generated from the first feature map 411 while they are connected to each other along the channel direction.

[0078] Figure 10 The configuration does not directly use the first image 403 to generate the model output 404. However, the first feature image 411, which serves as the source for generating the first image 403, contains information for separating regions affected by brightness saturation from other regions. By generating the model output 404 based on the first feature image 411, it is possible to obtain... Figure 1The configuration achieves the same effect. This embodiment performs convolution using 32 types of 3×3 filters in each layer (where the number of filter types in layers 421 and 422 is the same as the number of channels in the blurred image 401), but the configuration is not limited to this embodiment.

[0079] The number of linear sums (convolutions in this embodiment) performed up to the generation of the first feature map 403 from the blurred image 401 can be equal to or less than the number of linear sums performed up to the generation of the model output 404 from the blurred image 401. This is to enable the generation of the first feature map 411 in the middle of a model with information for separating regions affected by brightness saturation from other regions, and to perform the desired task (transforming the blur effect in this embodiment) in subsequent models. In this embodiment, the number of linear sums performed up to the generation of the first feature map 411 from the blurred image 401 is common, and the difference lies in the number of subsequent linear sums. Since the first image 403 and the model output 404 are generated from the first feature map 411 via one layer and twenty layers respectively, the number of linear sums performed up to the generation of the first image 403 is less. This is similar to estimation (the blurred image 401 can be replaced by a captured image).

[0080] Next, in step S108, the update unit 314 updates the weights of the machine learning model according to the error function. Next, in step S109, the update unit 314 determines whether the training of the machine learning model is complete. Information related to the configuration and weights of the trained machine learning model is stored in memory 311.

[0081] Now refer to Figure 7 A description is given of the process of transforming blur effects in captured images using a trained machine learning model, the transformation being performed by image processing device 303. Descriptions of matters common to the first embodiment will be omitted.

[0082] First, in step S201, the acquisition unit 333 acquires the captured image and the machine learning model. Next, in step S202, using the machine learning model, the conversion unit 334 generates a blurred converted image (model output) where the out-of-focus blur of the first image and the captured image are converted into blurry images with different shapes. Similar to training, the machine learning model has… Figure 10 The configuration shown is the same. Similar to training, a brightness saturation map representing the brightness saturation regions in the captured image is generated and input, and a first image and model output are generated. Next, in step S203, the conversion unit 334 synthesizes the captured image and model output based on the first image. If step S203 is not performed (if the model output of step S202 is used as the final blurred converted image), the first image is not needed. In this case, it is not necessary to perform... Figure 10The part enclosed by the dashed line. Therefore, the calculation of the part enclosed by the dashed line can be omitted, reducing the processing load.

[0083] The above configuration can provide an image processing system that can suppress the reduction in accuracy caused by brightness saturation when using machine learning models to transform blur effects.

[0084] Third Embodiment

[0085] An image processing system according to a third embodiment of the present invention will now be described. In this embodiment, the task using a machine learning model is to estimate a depth map for a captured image. Since the blur shape changes according to the amount of defocus in the optical system, the blur shape and depth (defocus) can be correlated with each other. The machine learning model can generate a depth map of the subject space by estimating the blur shape in various regions of the input captured image (explicitly or implicitly) in the model. The method described in this embodiment can achieve the same effect in tasks other than estimating depth maps.

[0086] Figure 11 This is a block diagram of the image processing system 500 in this embodiment. Figure 12 This is an external view of the image processing system 500. The image processing system 500 includes a training device 501 and a camera device 502 connected to each other via wired or wireless means. The training device 501 includes a memory 511, an acquisition unit 512, a calculation unit 513, and an update unit 514. The camera device 502 includes an optical system 521, an image sensor 522, an image processing unit 523, a memory 524, a communication unit 525, a display unit 526, and a system controller 527. The image processing unit 523 includes an acquisition unit 523a, an estimation unit 523b, and a blurring unit 523c.

[0087] The camera device 502 forms an image of the subject space via the optical system 521, and the image sensor 522 acquires the image as a captured image. The captured image is blurred due to aberrations and defocusing of the optical system 521. The image processing unit 523 generates a depth map of the subject space from the captured image using a machine learning model. The machine learning model is trained by the training device 501, and configuration and weight information are pre-acquired from the training device 501 via the communication unit 525 and stored in the memory 524. The captured image and the estimated depth map are stored in the memory 524 and displayed on the display unit 526 as needed. The depth map is used to add a blur effect to the captured image and to crop the subject. The system controller 527 performs a series of controls.

[0088] Now refer to Figure 6 A description of the training of the machine learning model performed by training device 501 is given. Descriptions of matters common to the first embodiment will be omitted.

[0089] First, in step S101, the acquisition unit 512 acquires one or more original images. Next, in step S102, the calculation unit 513 adds blur to the original images and generates a blurred image. A depth map (which may be a defocus map) corresponding to the original image and the focal length of the optical system 521 are set, and blur corresponding to the focal length of the optical system 521 and the amount of defocus from the optical system 521 is added. When the F-number (aperture value) is fixed, the larger the absolute value of the defocus amount, the larger the defocus blur becomes. Due to the influence of spherical aberration, the blur shape changes in front of and behind the focal plane. When spherical aberration is generated in the negative direction, it causes double-line blur in the subject space in the direction away from the focal plane from the optical system 521 (on the subject side), and the blur in the approach direction (on the image side) has a shape with a peak at the center. If spherical aberration is positive, the relationship becomes the opposite. Outside the optical axis, due to astigmatism and other effects, the blur shape further changes according to the amount of defocus.

[0090] Next, in step S103, the calculation unit 513 sets a first region based on the threshold of the blurred image and the signal. Next, in step S104, the calculation unit 513 generates a first image with the signal values ​​of the original image in the first region. Next, in step S105, the calculation unit 513 adds blur to the first image and generates a first ground truth map. In this embodiment, the first ground truth map is not cropped using the second signal values. This trains a machine learning model to estimate the brightness of the brightness saturation region before cropping when generating the first map. Next, in step S106, the acquisition unit 512 acquires the ground truth model output. The ground truth model output is the depth map set in step S102.

[0091] Next, in step S107, the computing unit 513 uses a machine learning model to generate a first ground truth map and model output. The machine learning model uses... Figure 13 Configuration. Figure 13 This is a block diagram of the machine learning model according to this embodiment. A first feature map 611 is generated from the blurred image 601 through multiple layers (ten convolutional layers in this embodiment), and a first image 603 and a model output 604 are generated based on the first feature map 611. The first image 603 is generated from the first feature map 611 through multiple layers (two convolutional layers), and the model output 604 is generated from the first feature map 611 through multiple layers (twenty convolutional layers). This embodiment performs convolution using 48 types of 5×5 filters in each layer (where the number of filter types in layer 621 is the same as the number of channels in the blurred image 601, and the number of filters in layer 622 is 1), but is not limited to this example.

[0092] Next, in step S108, the update unit 514 updates the weights of the machine learning model using the error function. Next, in step S109, the update unit 514 determines whether the training of the machine learning model is complete.

[0093] Now refer to Figure 14 The description includes the estimation of the depth map of the captured image using a machine learning model, performed by the image processing unit 523, and the addition of blur to the captured image. Figure 14 This is a flowchart output according to the generative model of this embodiment. Descriptions of items common to the first embodiment will be omitted.

[0094] First, in step S401, the acquisition unit 523a acquires the captured image and the machine learning model. Information related to the configuration and weights of the machine learning model is retrieved from memory 524. The machine learning model has… Figure 13 The configuration is shown. Next, in step S402, the estimation unit 523b uses a machine learning model to generate a model output (depth map) and a first map from the captured image.

[0095] Next, in step S403, blurring unit 523c adds blur to the captured image based on the model output and the first image, generating a blurred image (with shallow depth of field). Based on the depth map as the model output, blur is set for each region of the captured image corresponding to the amount of defocus. No blur is added to the focused areas, and a larger blur is added to areas with larger defocus. In the first image, the brightness before cropping in the brightness saturation areas of the captured image is estimated. After replacing the signal values ​​in the brightness saturation areas of the captured image with this brightness, blur is added. Therefore, an image with a natural blur effect can be generated, where sunlight, reflected light on water, and nighttime lights are not darkened by the added blur.

[0096] The above configuration can provide an image processing system that can suppress the accuracy reduction caused by brightness saturation when using machine learning models to estimate depth maps.

[0097] Other embodiments

[0098] The embodiments of the present invention can also be implemented by providing software (programs) that perform the functions of the above embodiments to a system or device via a network or various storage media, and the computer or central processing unit (CPU) or microprocessor unit (MPU) of the system or device reads out and executes the program.

[0099] Various embodiments may provide image processing methods and apparatus, methods and apparatus for training machine learning models, and storage media, each of which can suppress the reduction in accuracy caused by brightness saturation in recognition or regression tasks using machine learning models for blurred captured images.

[0100] While the invention has been described with reference to exemplary embodiments, it should be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims should be interpreted in the broadest possible sense to cover all such modifications and equivalent structures and functions.

[0101] For example, an image processing system may include an image processing device (first device) and a cloud device (second device) that are communicative with each other, according to various embodiments, wherein the second device performs actions based on a request from the first device. Figure 7 or Figure 14 The processing in this case involves a first device including a transmitter configured to send an captured image and a processing request to a second device. The second device includes a receiver configured to receive the captured image and the request from the first device, and a generator configured to generate a first image based on the captured image using a machine learning model according to the received request.

Claims

1. An image processing method, comprising: The first step is used to acquire the captured image; as well as The second step involves using a machine learning model to generate a first image based on the captured image. The first figure is characterized by being a graph representing the magnitude and range of signal values ​​in a region where the subject in the brightness saturation region of the captured image diffuses due to blur generated during the image capture process.

2. The image processing method according to claim 1, characterized in that, The machine learning model generates the first image based on the captured image and a second image representing the brightness saturation region of the captured image.

3. The image processing method according to claim 1 further includes a third step, wherein the third step is used to generate model output based on the captured image using the machine learning model. Its features are, The model output is a sequence of signals corresponding to the captured image, either a recognition tag or a spatial arrangement.

4. The image processing method according to claim 3, characterized in that, The model output includes a sharpened image of the captured image, an image of the captured image whose blur has been converted into a blur with different shapes, or a depth map of the subject space corresponding to the captured image.

5. The image processing method according to claim 3, characterized in that, The third step generates the model output based on the first graph.

6. The image processing method according to claim 3, characterized in that, The third step uses the machine learning model to generate a first feature map based on the captured image, and generates the first image and the model output based on the first feature map.

7. The image processing method according to claim 3, characterized in that, The number of linear sums performed up to the generation of the first image from the captured image is equal to or less than the number of linear sums performed up to the generation of the model output from the captured image.

8. The image processing method according to any one of claims 3 to 7, characterized in that, The model output is an image corresponding to the captured image. The image processing method further includes a fourth step, which is used to generate an image synthesized from the captured image and the model output based on the first image.

9. An image processing apparatus, comprising: An acquisition unit configured to acquire captured images; as well as The generation unit is configured to generate a first image based on the captured image using a machine learning model. The first figure is characterized by being a graph representing the magnitude and range of signal values ​​in a region where the subject in the brightness saturation region of the captured image diffuses due to blur generated during the image capture process.

10. A storage medium storing a program that causes a computer to perform the image processing method according to any one of claims 1 to 8.

11. A method for training a machine learning model, the training method comprising the following steps: Obtain the original image; A blurred image is generated by adding blur to the original image; The first region is set using a threshold based on the image and signal values ​​of the original image; Generate a first image in the first region having the signal values ​​of the original image; A first ground truth map is generated by adding blur to the first image; and The machine learning model is trained using the blurred image and the first ground truth map.

12. The training method for the machine learning model according to claim 11, characterized in that, The training process includes the following steps: The machine learning model is used to generate a first image based on the blurred image; and The machine learning model is trained using the error between the first map and the first ground truth map.

13. The training method for the machine learning model according to claim 11, characterized in that, The first image has signal values ​​in regions other than the first region that are different from the signal values ​​of the original image.

14. The training method for the machine learning model according to claim 11, characterized in that, The first image has a first signal value in regions other than the first region.

15. The training method for the machine learning model according to claim 11, characterized in that, The original image is an image with a signal value greater than the second signal value, and signal values ​​greater than the second signal value are cropped from the blurred image.

16. The training method for the machine learning model according to claim 15, characterized in that, The second signal value is equal to the threshold value of the signal value.

17. The training method for a machine learning model according to any one of claims 11 to 16, characterized in that, The training process includes the following steps: Obtain the ground truth model output corresponding to the blurred image; The machine learning model is used to generate model output based on the blurred image, and The characteristic feature is that the training step uses the error between the model output and the ground truth model output to train the machine learning model.

18. The training method for the machine learning model according to claim 17, characterized in that, The ground truth model output includes: an image with less blur compared to the blurred image, an image with a blur having a shape different from the blurred image added to the original image, or a depth map corresponding to the blurred image.

19. A training device for a machine learning model, the training device comprising: An acquisition unit configured to acquire the original image; A blurring unit, configured to generate a blurred image by adding blur to the original image; A setting unit is configured to set a first region using a threshold based on image and signal values ​​of the original image; An image generation unit is configured to generate a first image having the signal values ​​of the original image in the first region; The image generation unit is configured to generate a first ground truth map by adding blur to the first image; as well as A training unit is configured to train the machine learning model using the blurred image and the first ground truth map.

20. A storage medium storing a program that causes a computer to execute a training method for a machine learning model according to any one of claims 11 to 18.

21. An image processing method, comprising the following steps: Acquire captured images; and The machine learning model trained using the training method according to claim 17 or 18 is used to generate model output based on the captured images. The characteristic feature is that the model output is a signal sequence of identification tags or spatial arrangements corresponding to the captured image.

22. An image processing system comprising a first device and a second device capable of communicating with each other. Its features are, The first device includes a transmitter configured to send captured images and requests for processing to the second device. The second device includes: A receiver configured to receive the captured image and the request from the first device; and A generator, configured to generate a first image based on the captured image using a machine learning model in response to the request, and The first figure is a diagram showing the magnitude and range of signal values ​​in a region where the subject in the brightness saturation region of the captured image diffuses due to blur generated during the image capture process.

Images