Image processing method, image processing apparatus, image processing system, and storage medium

Abstract

An image processing method, an image processing apparatus, an image processing system, and a storage medium are disclosed. The image processing method includes acquiring a captured image obtained by imaging, generating a first image by correcting a blur component of the captured image, and generating a second image based on the captured image, the first image, and weight information. The weight information is generated based on (i) information about a luminance of the captured image or information about a scene of the captured image and (ii) information about a saturated region in the captured image.

Description

Technical Field

[0001] This invention relates to an image processing method for deblurring images. Background Technology

[0002] Japanese Patent Application Publication No. (“JP”) 2020-166628 discloses a method for sharpening blur in captured images using a convolutional neural network (CNN) as a machine learning model. By training the CNN using a training dataset (generated by blurring images containing signal values ​​greater than or equal to the brightness saturation value of the captured image), blur can be sharpened while minimizing side effects even around brightness saturation regions. JP 2020-166628 also discloses a method for adjusting the sharpening intensity by obtaining a weighted average of the captured image and the estimated image (deblurred image) based on the brightness saturation region.

[0003] JP 2018-201137 discloses a method for reducing ringed stripe pattern artifacts around the locations corresponding to saturated pixels by using deconvolution to correct blur in the captured image and combining the obtained corrected image and the captured image. In JP 2018-201137, as a weight in the combination, the combination ratio of the captured image is set to 1 for saturated pixels and to 0 for other pixels.

[0004] The method disclosed in JP 2020-166628 relies on an input image that may not suppress edge descent, ringing, etc., and these side effects may occur in the estimated image (deblurred image). Specifically, side effects may occur when the subject is severely blurred due to aberrations of the optical system. The severity of edge descent occurring around saturation regions varies depending on the image brightness. For example, in bright images captured outdoors during the day, edge descent around saturation regions is significant, but in dark images such as night scenes, edge descent is not significant.

[0005] Using the methods disclosed in JP 2020-166628 or JP 2018-201137, averaging is performed according to determined weights, regardless of the severity of edge descent, which varies depending on the brightness (or scene) of the image. In other words, if a weighted average of the input image and the estimated image is obtained, then the significant edge descent around the saturation region is reduced in bright images (bright scenes), but in dark images (dark scenes), the correction effect is reduced too much in the regions around the saturation region. Summary of the Invention

[0006] The present invention provides an image processing method, an image processing apparatus, an image processing system, and a storage medium, each of which can appropriately perform deblurring based on the brightness or scene of the image.

[0007] An image processing method according to one aspect of this disclosure includes: acquiring a captured image obtained through imaging; generating a first image by correcting blur components of the captured image; and generating a second image based on the captured image, the first image, and weight information. The weight information is generated based on (i) information about the brightness of the captured image or information about the scene of the captured image and (ii) information about saturated regions in the captured image.

[0008] An image processing apparatus according to one aspect of this disclosure includes: an acquisition unit configured to acquire a captured image obtained through imaging; a first generation unit configured to generate a first image by correcting blur components of the captured image; and a second generation unit configured to generate a second image based on the captured image, the first image, and weight information. The weight information is generated based on (i) information about the brightness of the captured image or information about the scene of the captured image and (ii) information about saturated regions in the captured image.

[0009] An image processing system according to one aspect of this disclosure includes a first device and a second device, the first device and the second device being capable of communicating with each other. The first device includes a transmitting unit configured to transmit a request to the second device related to the execution of processing of a captured image obtained through imaging. The second device includes: a receiving unit configured to receive the request; an acquiring unit configured to acquire the captured image; a first generating unit configured to generate a first image based on the request by correcting blur components of the captured image; and a second generating unit configured to generate a second image based on the captured image, the first image, and weight information. The weight information is generated based on (i) information about the brightness of the captured image or information about the scene of the captured image and (ii) information about saturated regions in the captured image.

[0010] The storage medium that enables the computer to execute the above image processing methods also constitutes another aspect of this disclosure.

[0011] Further features of the invention will become clear from the following description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0012] Figure 1 This is a configuration diagram of the machine learning model according to the first embodiment.

[0013] Figure 2 This is a block diagram illustrating an image processing system according to a first embodiment.

[0014] Figure 3 This is an external view of the image processing system according to the first embodiment.

[0015] Figures 4A to 4C This is an explanatory diagram illustrating the side effects caused by sharpening according to the first to fourth embodiments.

[0016] Figure 5 This is a flowchart of machine learning model training according to the first, third, and fourth embodiments.

[0017] Figure 6 It is a flowchart generated based on the model output of the first or third embodiment.

[0018] Figure 7 This is a flowchart of sharpening intensity adjustment according to the first to fourth embodiments.

[0019] Figure 8 This is an explanatory diagram illustrating the adjustment values ​​of a weighted graph relative to the average signal value according to the first to fourth embodiments.

[0020] Figure 9A and 9B This is an explanatory diagram illustrating the captured image and saturation effect map according to the first embodiment.

[0021] Figure 10A and 10B This is an explanatory diagram of the captured image and saturation effect map according to the first embodiment.

[0022] Figure 11 This is an explanatory diagram of the weighting graph according to the first embodiment.

[0023] Figure 12 This is an explanatory diagram of the weighting graph according to the first embodiment.

[0024] Figure 13 This is an explanatory diagram of the image restoration filter according to the second embodiment.

[0025] Figure 14 This is an explanatory diagram (cross-sectional view) of the image restoration filter according to the second embodiment.

[0026] Figure 15A and 15B This is an explanatory diagram of the point spread function PSF according to the second embodiment.

[0027] Figure 16A and16B This is an explanatory diagram of the amplitude component MTF and phase component PTF of the optical transfer function according to the second embodiment.

[0028] Figure 17 This is a block diagram of an image processing system according to the second embodiment.

[0029] Figure 18 This is a flowchart of the deblurred image generation according to the second embodiment.

[0030] Figure 19 This is a block diagram of an image processing system according to a third embodiment.

[0031] Figure 20 This is an external view of the image processing system according to the third embodiment.

[0032] Figure 21 This is a block diagram of an image processing system according to the fourth embodiment.

[0033] Figure 22 This is an external view of the image processing system according to the fourth embodiment.

[0034] Figure 23 This is a flowchart of the model output and sharpening intensity adjustment based on the fourth embodiment. Detailed Implementation

[0035] A description of embodiments of the invention will now be given with reference to the accompanying drawings. Corresponding elements in the corresponding figures will be labeled with the same reference numerals, and repeated descriptions will be omitted.

[0036] Before giving a detailed description of the embodiments, the main idea of ​​the invention is described. The present invention generates an estimated image (deblurred image, first image) from an image captured using an optical system, in which the blur caused by the optical system (image acquisition optical system) is sharpened. The estimated image is generated, for example, using a machine learning model. Then, the present invention generates a weight map (weight information) based on information about the brightness of the captured image or information about the scene of the captured image and information about saturated regions (luminance saturation regions) in the captured image, and obtains a weighted average of the captured image and the estimated image. Here, the blur caused by the optical system includes blur caused by aberrations, diffraction, defocus, the effect of optical low-pass filters, pixel aperture degradation of the image sensor, etc.

[0037] Machine learning models refer to, for example, neural networks, genetic programming, and Bayesian networks. Neural networks include convolutional neural networks (CNNs), generative adversarial networks (GANs), and recurrent neural networks (RNNs).

[0038] Blur sharpening refers to the process of recovering the frequency components of a subject that have been reduced or lost due to blurring. During blur sharpening, it may be impossible to suppress undershoot (edge ​​drop), ringing, etc., depending on the captured image, and these side effects may occur in the estimated image. Specifically, side effects occur when the subject is severely blurred due to aberrations of the optical system, or when there are luminance saturation regions in the image. Luminance saturation regions can occur in the image depending on the dynamic range of the image sensor, exposure during imaging, etc. In luminance saturation regions, it is impossible to obtain information about the spatial structure of the subject, so side effects are possible. The severity of undershoot generated around luminance saturation regions varies depending on the brightness of the image. For example, in bright images captured outdoors during the day, undershoot is significant around luminance saturation regions, but in dark images such as night scenes, undershoot is not significant.

[0039] Therefore, each embodiment obtains a weighted average of the captured and estimated images by using a weight map generated based on information about the brightness of the captured image or information about the scene of the captured image, and information about saturated regions in the captured image. This allows the deblurring effect (blur correction effect) to be maintained while suppressing side effects that occur around saturated regions and vary depending on the brightness or scene of the captured image.

[0040] In the following description, the phase of learning the weights of the machine learning model is referred to as the learning phase, and the phase of sharpening the blur using the machine learning model with the learned weights is referred to as the estimation phase.

[0041] First Embodiment

[0042] First, a description of an image processing system 100 according to a first embodiment of the present invention is given. In this embodiment, a machine learning model performs blur sharpening on an image including a brightness saturation capture image. The blur to be sharpened includes blur caused by aberrations and diffraction occurring in the optical system and blur caused by an optical low-pass filter. However, the effects of the present invention can also be obtained when blur caused by pixel apertures, defocus, or jitter is sharpened. In addition, the present invention can also be implemented and its effects obtained when performing tasks other than blur sharpening.

[0043] Figure 2 This is a block diagram of the image processing system 100. Figure 3This is an external view of the image processing system 100. The image processing system 100 includes a training device 101 and an image processing device 103 connected via a wired or wireless network. The image processing device 103 is connected via a wired or wireless network to each of an image pickup device 102, a display device 104, a recording medium 105, and an output device 106. A captured image acquired by imaging the subject space using the image pickup device 102 is input to the image processing device 103. The captured image is blurred due to aberrations and diffraction of the optical system (image pickup optics) 102a of the image pickup device 102 and the optical low-pass filter of the image sensor 102b of the image pickup device 102, and information about the subject is reduced.

[0044] Using a machine learning model, image processing device 103 performs blur sharpening on the captured image and generates a saturation effect map and a deblurred image (model output, first image). A detailed description of the saturation effect map will be given later. The machine learning model has been trained by training device 101, and image processing device 103 has pre-obtained information about the machine learning model from training device 101 and stored it in memory 103a. Image processing device 103 also has the function of adjusting the intensity of blur sharpening by obtaining a weighted average of the captured image and the deblurred image. A detailed description of the training and estimation of the machine learning model, and the adjustment of the intensity of blur sharpening, will be given later. The user can adjust the intensity of blur sharpening while examining the image displayed on display device 104. The intensity-adjusted deblurred image is stored in memory 103a or recording medium 105 and output to output device 106, such as a printer, as needed. The captured image can be grayscale or can have multiple color components. Alternatively, an undeveloped RAW image or a developed image can be used.

[0045] Next, refer to Figures 4A to 4C This provides a description of the reduction in estimation accuracy that occurs when performing blur sharpening using a machine learning model. Figures 4A to 4C This diagram illustrates the side effects of sharpening and shows the spatial changes in the image's signal values. Here, the image is an 8-bit developed image, therefore the saturation value is 255. Figures 4A to 4C In each of the diagrams, solid lines represent captured images (blurred images), and dotted lines represent deblurred images that have been sharpened using a machine learning model.

[0046] Figure 4A It is the result of sharpening a non-luminosity saturated subject that is severely blurred due to optical system aberrations. Figure 4B This is the result of sharpening a non-luminosity saturated subject that is slightly blurred due to optical system aberrations, and Figure 4CThis is the result of sharpening a luminance-saturated subject that is slightly blurred due to optical system aberrations. In cases where the image is severely blurred due to optical system aberrations, undershoot occurs on the darker sides of the edges. Furthermore, even in cases where the image is only slightly blurred due to optical system aberrations, if a luminance-saturated subject is sharpened, undershoot, which does not occur in non-luminance-saturated subjects, and side effects (such as a reduction in the value of originally saturated pixels) also occur. In luminance-saturated regions (hereinafter referred to as "luminance-saturated regions"), information about the spatial structure of the subject is lost, and false edges can appear at the boundaries of each region, making it impossible to extract the correct feature values ​​of the subject. As a result, the estimation accuracy of machine learning models decreases. These results indicate that the side effects caused by sharpening depend on the performance of the optical system and the luminance-saturated regions.

[0047] The aforementioned correction uses a machine learning model that has already been trained by incorporating the captured image and its corresponding luminance saturation map as input data to the model, as well as a method for generating a saturation influence map. That is, while these methods can reduce side effects, they are difficult to completely eliminate. A detailed description of the methods for using luminance saturation maps and generating saturation influence maps is given.

[0048] A description of a luminance saturation map is provided. A luminance saturation map is a map that indicates (represents, describes, or identifies) luminance-saturated regions in a captured image. In luminance-saturated regions (hereinafter also referred to as "luminance-saturated areas"), information about the spatial structure of the subject is lost, and false edges can appear at the boundaries of each region, making it impossible to extract the correct feature values ​​of the subject. By inputting a luminance saturation map, a neural network can identify the problematic regions as described above and prevent degradation in estimation accuracy.

[0049] Next, a description of the saturation effect map is given. Even when using a saturation map, machine learning models may not make correct determinations. For example, if the target region is near a saturation region, the machine learning model may determine that the target region is affected by saturation because a saturation region exists nearby. On the other hand, if the target region is located far from a saturation region, it becomes more difficult to determine whether the target region is affected by saturation, and ambiguity increases. As a result, the machine learning model may make incorrect determinations at locations far from saturation regions. Therefore, in cases where the task is blur sharpening, sharpening is performed on a non-saturated blurred image for a saturated blurred image. At this point, artifacts occur in the deblurred image, which reduces the accuracy of the task. Therefore, a machine learning model can generate a saturation effect map from a blurred captured image.

[0050] A saturation effect map is a map (spatial arrangement of signal sequences) that indicates (represents, describes, or identifies) the magnitude and range of signal values ​​of blurred and extended subjects within saturated regions of a captured image. In other words, a saturation effect map is based on information about saturated regions in a captured image. By generating a saturation effect map, a machine learning model can accurately estimate the presence or absence and degree of the effects of saturation in a captured image. By generating a saturation effect map, the machine learning model can perform the appropriate processing on the affected regions and on other regions. Therefore, when the machine learning model generates a saturation effect map, the accuracy of the task is improved compared to cases where the generation of a saturation effect map is not involved (i.e., simply generating identification labels and deblurred images directly from the captured image).

[0051] Although the two methods mentioned above are effective, they are difficult to completely eliminate the reference. Figures 4A to 4C The described side effects. Therefore, the side effects are reduced by obtaining a weighted average of the captured image and the deblurred image. Figure 4A The alternating long and short dashed lines in the diagram represent the signal values ​​obtained by weighted averaging of the captured and deblurred images. By obtaining a weighted average, undershoot in dark areas is reduced while maintaining the blur-sharpening effect. This embodiment generates a weight map (weight information) to be used to obtain the weighted average of the captured and deblurred images based on information about the brightness of the captured image or information about the scene of the captured image, as well as information about the saturation regions (information about the saturation regions). Thus, the correction effect is maintained in dark images where undershoot is not significant, and reduced in bright images where undershoot is significant, thereby reducing side effects. That is, the blur correction effect (deblurring effect) can be maintained while reducing side effects that occur around the saturation regions and vary depending on the scene.

[0052] Next, refer to Figure 5 This provides a description of the training of the machine learning model performed by the training device 101. Figure 5 This is a flowchart of training a machine learning model. The training device 101 includes a memory 101a, an acquisition unit 101b, a calculation unit 101c, and an update unit 101d, and these units perform the following steps.

[0053] First, in step S101, the acquisition unit 101b acquires one or more original images from the memory 101a. The original images are images that include signal values ​​higher than a second signal value. The second signal value is a signal value corresponding to the luminance saturation value of the captured image. The signal values ​​can be normalized when input into the machine learning model, so the second signal value and the luminance saturation value of the captured image do not necessarily have to match. The training of the machine learning model is performed based on the original images, therefore the original images can be images including various frequency components (edges with different orientations and intensities, gradients, flat parts, etc.). The original images can be real-world images or computer graphics (CG).

[0054] Subsequently, in step S102, the computing unit 101c generates a blurred image by blurring the original image. The blurred image is the image to be input into the machine learning model during training and corresponds to the captured image during estimation. The applied blur is the blur that is the target of sharpening. In this embodiment, the blur can refer to the blur caused by the aberrations and diffraction of the optical system 102a and the optical low-pass filter of the image sensor 102b. The shape of the blur caused by the aberrations and diffraction of the optical system 102a varies depending on the image plane coordinates (image height and azimuth). The shape also varies depending on the magnification change, aperture, and focus state of the optical system 102a. If training is to be performed once in a machine learning model that sharpens all these types of blur, multiple blurred images can be generated using multiple types of blur that occur in the optical system 102a. In the blurred image, signal values ​​exceeding a second signal value are pruned to achieve luminance saturation, which occurs during the imaging process of the captured image. If necessary, noise generated by the image sensor 102b can be applied to the blurred image.

[0055] Subsequently, in step S103, the calculation unit 101c sets a first region based on a threshold of the signal value and an image based on the original image. In this embodiment, a blurred image is used as the image based on the original image, but the original image itself can be used. The first region is set by comparing the signal value of the blurred image with a threshold of the signal value. More specifically, the first region is the region where the signal value of the blurred image is equal to or greater than the threshold of the signal value. In the first embodiment, the threshold of the signal value is a second signal value. Therefore, the first region indicates (represents, includes, or identifies) the brightness saturation region of the blurred image. However, the threshold of the signal value and the second signal value may not match. The threshold of the signal value may be set to a value slightly smaller than the second signal value (e.g., 0.9 times the second signal value).

[0056] Subsequently, in step S104, the calculation unit 101c generates a first region image, in which the signal value in the first region is the same as the signal value in the first region of the original image. In the first region image, the signal values ​​in regions outside the first region are different from the signal values ​​in regions outside the first region of the original image. The first region image may have a first signal value in regions outside the first region. In this embodiment, the first signal value is 0, but the invention is not limited thereto. In a first embodiment, in the first region image, the blurred image only has the signal value of the original image in the luminance saturation region, and the signal value is 0 in other regions.

[0057] Subsequently, in step S105, the calculation unit 101c generates a saturation effect ground truth map by blurring the first region image. The applied blur is the same as the blur applied to the blurred image. Thus, a saturation effect ground truth map is generated based on the subject in the luminance saturation region of the blurred image. The saturation effect ground truth map is a graph (spatially arranged signal sequence) indicating (representing, illustrating, or identifying) the magnitude and range of signal values ​​that have been expanded due to degradation during imaging (the relationship between the range of the area of ​​the subject in the saturation region that has been expanded due to the blur component and the signal value corresponding to that area). In the first embodiment, the saturation effect ground truth map is cropped at the second signal value in a manner similar to cropping the blurred image, but may not be cropped.

[0058] Subsequently, in step S106, the acquisition unit 101b acquires the ground truth model output. In this embodiment, since the task is blur sharpening, the ground truth model output is an image with less blur than the blurred image. In the first embodiment, the ground truth model output is generated by trimming the original image at the second signal value. If the original image does not include enough high-frequency components, then the image obtained by downscaling the original image can be used as the ground truth model output. In this case, downscaling is similarly performed in the generation of the blurred image in step S102. Step S106 can be executed at any time after step S101 and before step S107.

[0059] Subsequently, in step S107, by using a machine learning model, the computing unit 101c generates a saturation influence map and model output based on the blurred image. Figure 1 This is a configuration diagram of the machine learning model. In this embodiment, it uses... Figure 1The machine learning model shown is not limited to this. A blurred image 201 and a luminance saturation map 202 are input into the machine learning model. The luminance saturation map 202 is a map indicating (representing, describing, or identifying) the luminance saturation regions of the blurred image 201, where the luminance saturation regions are regions with signal values ​​equal to or greater than a second signal value. The luminance saturation map 202 can be generated, for example, by binarizing the blurred image 201 using the second signal value. However, the luminance saturation map 202 is not mandatory. The blurred image 201 and the luminance saturation map 202 are cascaded in the channel direction and input into the machine learning model. However, the invention is not limited to this. For example, each of the blurred image 201 and the luminance saturation map 202 can be converted into a feature map, and the feature maps can be cascaded in the channel direction. Information other than the luminance saturation map 202 can be added to the input.

[0060] The machine learning model comprises multiple layers and obtains a linear sum of the input to each layer and the weights in each layer. The initial values ​​of the weights can be determined by random numbers, etc. In a first embodiment, the machine learning model is a CNN that uses the input and the convolution of filters (each element of the filter corresponds to a weight, and the convolution can include a sum with the bias) as a linear sum, but the invention is not limited thereto. In each layer, activation functions such as the Corrected Linear Unit (ReLU) and the sigmoid function are used to perform nonlinear transformations as needed. Additionally, if necessary, the machine learning model can include residual blocks or skip connections (also known as shortcut connections). After the input passes through multiple layers (16 convolutional layers in this embodiment), a saturation effect map 203 is generated. In this embodiment, the saturation effect map 203 is obtained by obtaining the sum of the luminance saturation map 202 for each element and the output of layer 211, but the invention is not limited thereto. The saturation effect map can be directly generated as the output of layer 211. Alternatively, the result of any processing performed on the output of layer 211 can be used as the saturation effect map 203.

[0061] Subsequently, the saturation effect map 203 and the blurred image 201 are cascaded in the channel direction and input to subsequent layers and passed through multiple layers (16 convolutional layers in the first embodiment). As a result, a model output 204 is generated. The model output 204 is generated by obtaining the sum of the blurred image 201 for each element and the output of layer 212, but the configuration is not limited to this. In the first embodiment, convolution with 64 kinds of 3×3 filters is performed in each layer (however, the number of filter types in layers 211 and 212 is the same as the number of channels of the blurred image 201), but the invention is not limited to this.

[0062] Subsequently, in step S108, the updating unit 101d updates the weights of the machine learning model based on the error function. In the first embodiment, the error function is a weighted sum of the error between the saturation effect map 203 and the saturation effect ground truth map and the error between the model output 204 and the ground truth model output. The mean squared error (MSE) is used to calculate the error. Each weight is 1. However, the error function and weights are not limited to these. Backpropagation, etc., can be used to update the weights. The error can be obtained relative to the residual components. In the case of residual components, the error to be used is the error between the difference component between the saturation effect map 203 and the brightness saturation map 202 and the difference component between the saturation effect ground truth map and the brightness saturation map 202. Similarly, the error between the difference component between the model output 204 and the blurred image 201 and the difference component between the ground truth model output and the blurred image 201 is used.

[0063] Subsequently, in step S109, the update unit 101d determines whether the training of the machine learning model has been completed. This can be determined by whether the number of repetitions of weight updates has reached a predetermined number, or whether the change in weights during updates is less than a predetermined value. If it is determined in step S109 that training has not yet been completed, the process returns to step S101, and the acquisition unit 101b acquires one or more new original images. On the other hand, if it is determined that training has been completed, the update unit 101d ends the training and stores information about the configuration and weights of the machine learning model in the memory 101a.

[0064] Using the training methods described above, the machine learning model can estimate a saturation effect map that indicates (represents, describes, or identifies) the magnitude and range of signal values ​​of the blurred and extended subject within the luminance saturation regions of a blurred image (the captured image at the time of estimation). By explicitly estimating the saturation effect map, the machine learning model can perform blur sharpening on the appropriate regions for both saturated and unsaturated blurred images, which helps prevent artifacts.

[0065] Next, refer to Figure 6 The description of blur sharpening of the captured image using a trained machine learning model is given, and the blur sharpening is performed by the image processing device 103. Figure 6 This is a flowchart of model output generation. The image processing device 103 includes a memory 103a, an acquisition unit (acquisition task) 103b, and a sharpening unit (first generation unit, second generation unit, first generation task, second generation task) 103c, and these units perform the following steps.

[0066] First, in step S201, the acquisition unit 103b acquires the machine learning model and the captured image acquired through the optical system 102a. Information regarding the configuration and weights of the machine learning model is retrieved from memory 103a. Then, in step S202, using the machine learning model, the sharpening unit (first generation unit, first generation task) 103c generates a deblurred image (model output, first image) from the captured image, in which the blur of the captured image is sharpened. That is, the sharpening unit 103c generates the first image by correcting the blur components of the captured image. The machine learning model has the following characteristics as during training: Figure 1 The configuration shown is as follows. As during training, a saturation effect map and model output are generated by generating and inputting a saturation saturation map of the saturation saturation regions of the captured image.

[0067] Next, refer to Figure 7 The description of the combined captured image and model output (adjusting sharpening intensity) performed by the image processing device 103 is given. Figure 7 This is a flowchart for adjusting the sharpening intensity.

[0068] First, in step S211, the sharpening unit 103c generates a weight map based on the saturation influence map, which is used when combining the captured image and the model output. The generation of the weight map is described in detail. The weight map determines the proportion of each image when a weighted average of the captured image and the deblurred image is to be obtained, and has a continuous signal value from 0 to 1. For example, if the value of the weight map determines the proportion of the captured image, then if the value is 1, the weighted average image is the captured image. If the value of the weight map is 0.5, then the weighted average image is the sum of 50% of the pixel values ​​of the captured image and 50% of the pixel values ​​of the deblurred image region. In this embodiment, the value of the weight map represents the weight of the captured image. When determining the weight, the saturation influence map is normalized by a set signal value, and this is used as the weight map for the captured image. Changing the signal value through which the saturation influence map is normalized can adjust the balance between the blurring and sharpening effect and side effects.

[0069] In the captured image, the region surrounding the luminance saturation area includes a reduction in subject information due to luminance saturation. Therefore, blurring and sharpening that region (estimating the reduced subject information) is more difficult compared to blurring and sharpening other regions. Consequently, blurring and sharpening around luminance saturation areas can potentially cause side effects (ringing, undershoot, etc.). To suppress this side effect, the model output and the captured image are combined. Here, by combining them based on the saturation effect map, the side effect can be reduced by only increasing the weight of the captured image around the luminance saturation area where side effects may occur, while simultaneously reducing the reduction in blurring and sharpening effect in the unsaturated blurred image.

[0070] Subsequently, in step S212, the acquisition unit 103b acquires information about the brightness of the captured image or information about the scene of the captured image. Here, the information about the brightness of the captured image is a statistic related to the signal value of the captured image, and is based on at least one of the mean, median, variance, and histogram of the signal value of the captured image. The statistic related to the signal value of the captured image (which is information about the brightness of the captured image) can be related to the entire captured image, or it can use the statistics of each segmented region in the captured image (or the third image described below) (statistics related to each segmented region).

[0071] Information about the scene in which the image was captured includes information about the type of scene and the imaging mode used when imaging the captured image. Here, the information is about the type of scene, such as "daytime" or "night scene," or about the imaging mode, such as "night scene imaging mode," which allows for the determination of differences in brightness within the captured image. Information about the scene in the captured image can be obtained by determining the scene type (determining the scene type) or by retrieving information about the scene type or imaging mode written into the captured image.

[0072] In this embodiment, the average signal value of the captured image is obtained as information about the brightness of the captured image. However, when obtaining the average signal value of the captured image, if the captured image includes many saturated regions, then obtaining a large value as the average signal value (even when using a dark image such as a night scene) can lead to the image being identified as a bright image. Therefore, the average signal value can be obtained from a third image obtained by removing the saturation effect map from the captured image (using the information about the saturated regions and subtracting the captured image). By using the third image and obtaining the average signal value from the unsaturated regions in the captured image, it is possible to properly determine whether the captured image is a bright scene or a dark scene.

[0073] Subsequently, in step S213, the sharpening unit 103c adjusts the weight map of the captured image based on information about the brightness of the captured image or information about the scene of the captured image. That is, the weight map is generated based on information about the brightness of the captured image or information about the scene of the captured image, as well as information about saturated regions in the captured image.

[0074] When using the average signal value of the captured image (or third image) as information about the brightness of the captured image, the weight map is adjusted so that the larger the average signal value (i.e., the brighter the captured image (or third image)), the greater the weight of the captured image. Specifically, the relationship between the average signal value and the adjustment value of the weight map is stored as a linear function, the adjustment value corresponding to the average signal value of the captured image is obtained, and the obtained adjustment value is applied to the weight map (weight map multiplied by the adjustment value).

[0075] Figure 8 This is an interpretation diagram of the adjusted values ​​of the weighted graph relative to the average signal value. Figure 8 In this diagram, the horizontal axis represents the average signal value, and the vertical axis represents the adjustment values ​​of the weighted graph. For example, using values ​​corresponding to the average signal value and derived from... Figure 8 The relationship 121 shown is the adjustment value obtained. However, the relationship between the average signal value and the weight map adjustment value is not limited to this. In the case of obtaining the average signal value for each segmented region of the captured image, the distribution of the corresponding average signal value of the region can be converted into an average signal value map representing the average signal value corresponding to the pixels of the captured image, and the weight map can be adjusted by obtaining the adjustment value based on the corresponding average signal value in the pixels.

[0076] In step S203, if information about the scene of the captured image is obtained, the weight map is adjusted based on, for example, whether the captured image is a dark image. For example, if the captured image is a bright image, the weight map is used as is, and if the captured image is a dark image such as a night scene, the weight map is adjusted by halving the values.

[0077] When using the mean, median, variance, or histogram of the signal values ​​of the captured image as statistics related to the signal values, the weight map is adjusted so that the larger the statistic, the greater the weight of the captured image. For example, when using a histogram of the signal values ​​of the captured image, the weight map is adjusted so that the higher the centroid or peak of the histogram, the greater the weight of the captured image.

[0078] Figure 9A The illustration shows a captured image during the day as a bright scene, and Figure 9B The diagram illustrates the... Figure 9A The image shown is a saturation effect diagram of the captured image. Figure 10A The illustration shows a captured image of a night scene as a dark background, and Figure 10B The diagram illustrates the... Figure 10A The image shown is a saturation effect diagram of the captured image. Figure 11 It is based on daytime images captured as bright scenes. Figure 9BThe saturation effect map shown is a weighted map whose weights are adjusted based on the average signal value of the image, and the weighted map is significantly adjusted based on the average signal value of the image. By significantly adjusting the weighted map, noticeable side effects in bright scenes can be reduced. Figure 12 It is based on captured images of night scenes as dark environments. Figure 10B The saturation effect map shown is a weighted map whose weights are adjusted based on the average signal value of the image, and the weighted map is slightly adjusted based on the average signal value of the image. Since the side effects are not noticeable in dark scenes, the effect can be maintained by slightly adjusting the weighted map.

[0079] In this embodiment, since the weight map is generated based on the saturation influence map, it is applied to the saturated region and its surroundings. However, a second weight map, different from the first weight map (based on the saturation influence map), can be used for the unsaturated region. Additionally, a third weight map for the saturated region can be applied to the weight map (first weight map) calculated from the saturation influence map. By using the second and third weight maps, the intensity can be adjusted for each of the saturated and unsaturated regions. In that case, the weight map can be calculated as (1 - first weight map) × second weight map + first weight map × adjustment value × third weight map. Furthermore, the obtained weight map can be adjusted according to user instructions. For example, the intensity can be adjusted by multiplying the entire weight map by a coefficient.

[0080] Subsequently, in step S214, the sharpening unit (second generation unit, second generation task) 103c generates an intensity-adjusted image (second image) 205 by obtaining a weighted average of the captured image and the deblurred image (model output, first image) based on the weight map adjusted in step S213. That is, the sharpening unit 103c generates the second image based on the captured image, the first image, and the weight map. In this embodiment, the weight map obtained for the deblurred image is obtained by subtracting the weight map used for the captured image from all maps with values ​​of 1.

[0081] Using the above configuration, an image processing system can be provided that can generate an image with appropriate correction effect around the saturation area based on brightness or scene when performing blur sharpening using a machine learning model.

[0082] Second Embodiment

[0083] Next, a description of an image processing system according to a second embodiment of the present invention will be given. In this embodiment, a description of blur sharpening performed by image restoration processing as a method different from machine learning will be given.

[0084] First, an overview of image restoration processing is described. When assuming the captured image (degraded image) is g(x,y), the original image is f(x,y), and the point spread function PSF, which is the Fourier pair of the optical transfer function OTF, is h(x,y), the following equation (1) is established.

[0085] g(x,y)=h(x,y)*f(x,y) (1)

[0086] Here, * denotes convolution (convolution, multiplication and accumulation), and (x,y) denotes the coordinates on the captured image.

[0087] When equation (1) is Fourier transformed and converted into a display format on the frequency plane, equation (2) is obtained, which is represented by the product of each frequency.

[0088] G(u,v)=H(u,v)·F(u,v) (2)

[0089] Here, H represents the optical transfer function (OTF) obtained by performing a Fourier transform on the point spread function PSF(h), and G and F represent the functions obtained by performing Fourier transforms on the degraded image g and the original image f, respectively. (u,v) represents the coordinates on the two-dimensional frequency plane, i.e., the frequency.

[0090] To obtain the original image f from the captured degraded image g, both sides can be divided by the optical transfer function H as shown in the following equation (3).

[0091] G(u,v) / H(u,v)=F(u,v) (3)

[0092] Then, by performing an inverse Fourier transform on F(u,v) (i.e., G(u,v) / H(u,v)) and returning it to the real plane, the original image f(x,y) can be obtained as the restored image.

[0093] When it is assumed that R is obtained by performing an inverse Fourier transform on H-1, the original image f(x,y) can be obtained by performing a convolution process on the image on the real plane, as in the following equation (4).

[0094] g(x,y)*R(x,y)=f(x,y) (4)

[0095] Here, R(x,y) is called the image restoration filter. When the image is two-dimensional, the image restoration filter R generally becomes a two-dimensional filter with taps (cells) corresponding to the pixels of the image. Furthermore, generally speaking, the larger the number of taps (cells) in the image restoration filter R, the better the restoration accuracy. Therefore, the number of taps that can be implemented is set according to the required image quality, image processing capabilities, aberration characteristics, etc. Since the image restoration filter R needs to reflect at least the aberration characteristics, it differs from traditional edge enhancement filters that have approximately three taps for each of the horizontal and vertical directions. The image restoration filter R is set based on the optical transfer function (OTF), which allows for the correction of both amplitude and phase component degradation with high accuracy.

[0096] Real-world images contain noise components, so if an image restoration filter R, created by obtaining the reciprocal of the optical transfer function (OTF) as described above, is used, the noise components are significantly amplified as the degraded image is restored. This is because, with the noise amplitude added to the amplitude component of the image, the MTF (amplitude component) of the optical system increases, causing the MTF to recover to 1 at all frequencies. While the MTF recovers to 1 due to the amplitude degradation caused by the optical system, the power spectrum of the noise also increases simultaneously. As a result, the noise is amplified according to the degree of MTF increase (i.e., the restoration gain).

[0097] Therefore, if noise is included, a good image for viewing cannot be obtained. This is expressed by the following equations (5-1) and (5-2).

[0098] G(u,v)=H(u,v)·F(u,v)+N(u,v) (5-1)

[0099] G(u,v) / H=F(u,v)+N(u,v) / H(u,v) (5-2)

[0100] Here, N represents the noise component.

[0101] For images that include noise components, there are methods, such as in a Wiener filter represented by, for example, the method of controlling the degree of recovery based on the intensity ratio (SNR) of the image signal and the noise signal.

[0102]

[0103] Here, M(u,v) represents the frequency response of the Wiener filter, and |H(u,v)| represents the absolute value of the optical transfer function (OTF), i.e., the MTF. In this method, for each frequency, the smaller the MTF, the smaller the recovery gain (degree of recovery), and the larger the MTF, the greater the recovery gain. Generally, the MTF of image acquisition optics is high on the low-frequency side and low on the high-frequency side, so this method essentially reduces the recovery gain on the high-frequency side of the image.

[0104] Subsequently, reference Figure 13 and 14 Describe the image restoration filter. Figure 13 and 14 This is an explanatory diagram of an image restoration filter. The number of taps in the image restoration filter is determined based on the aberration characteristics of the image acquisition optical system and the required restoration accuracy. Figure 13 The image restoration filter in the example is a two-dimensional filter with 11×11 taps. Figure 13 In this text, the value (coefficient) of each tap is omitted. Figure 14 The diagram shows a cross-section of this image restoration filter. Ideally, the distribution of the corresponding values ​​(coefficient values) of the taps of the image restoration filter has the function of restoring the signal value (PSF) that has been spatially expanded due to aberrations to the original point.

[0105] The taps of the image restoration filter undergo a convolution process (convolution, multiplication, and accumulation) during image restoration processing, while being associated with individual pixels of the image. During convolution, to improve the signal value of a predetermined pixel, that pixel is aligned with the center of the image restoration filter. Then, for each corresponding (associated) pixel of the image and the image restoration filter, the signal value of the image and the coefficient value of the filter are multiplied, and the signal value of the center pixel is replaced by the sum of the products.

[0106] Subsequently, reference Figures 15A to 16B It provides a description of the characteristics of image restoration in real space and frequency space. Figure 15A and 15B This is an explanatory graph of the point spread function PSF. Figure 15A The diagram illustrates the Point Spread Function (PSF) before image restoration, and... Figure 15B The diagram illustrates the point spread function (PSF) after image restoration. Figure 16A and 16B It is the amplitude component MTF((M) of the optical transfer function OTF). Figure 16A ) and phase component PTF((P), Figure 16B The diagram explains the meaning of the image. Figure 16A In (M), the broken line (a) represents the MTF before image restoration, and the alternating long and short dashed lines (b) represent the MTF after image restoration. Figure 16BIn (P), the broken line (a) represents the PTF before image restoration, and the alternating long and short dashed lines (b) represent the PTF after image restoration. Figure 15A As shown, the point spread function (PSF) before image restoration has an asymmetric spread, which causes the phase component (PTF) to have a non-linear value with respect to frequency. Because the image restoration process performs corrections that amplify the amplitude component (MTF) and reduce the phase component (PTF) to zero, the PSF after image restoration has a symmetrical and sharp shape.

[0107] As described above, an image restoration filter can be obtained by performing an inverse Fourier transform on a function designed based on the inverse function of the optical transfer function (OTF) of an image-picking optical system. The image restoration filter used in this embodiment can be appropriately changed; for example, a Wiener filter as described above can be used. When using a Wiener filter, if an inverse Fourier transform is performed on equation (6), a real-space image restoration filter that is to be convolved with the actual image can be created.

[0108] Since the optical transfer function (OTF) generated by aberrations changes depending on the image height (image position) of the image-picking optics even in an imaging state, the image restoration filter to be used needs to be changed according to the image height. On the other hand, when the vignetting effect of the optical system is small, the optical transfer function (OTF) generated by diffraction, whose influence becomes dominant with increasing F-number, can be regarded as an OTF uniform with respect to the image height.

[0109] When the correction target of image restoration processing does not include aberrations but diffraction (diffraction blur), the image restoration filter depends only on the aperture value and the wavelength of light, and is independent of the image height (image position). Therefore, a uniform (identical) image restoration filter can be used for a single image. That is, the image restoration filter used to correct diffraction blur is generated based on the optical transfer function resulting from the diffraction blur generated according to the aperture value. The optical transfer function for each color component can be generated by calculating the optical transfer function at multiple wavelengths and weighting each wavelength based on the spectrum of a hypothetical light source or information about the light-receiving sensitivity of the image sensor. Alternatively, calculations can be performed using predetermined representative wavelengths for each color component. The image restoration filter can then be generated based on the optical transfer function for each color component.

[0110] Therefore, when the correction target is only diffraction, processing can be performed by applying a uniform (identical) image restoration filter to the image based on the imaging conditions regarding the aperture value, by pre-storing multiple image restoration filters that depend on the aperture value. The aperture degradation component caused by the shape of the pixel aperture and the characteristics of the optical low-pass filter can also be considered.

[0111] Next, refer to Figure 17 The image processing system 200 according to this embodiment is described. Figure 17 This is a block diagram of the image processing system 200. The external view of the image processing system 200 is as described in the first embodiment. Figure 3 As shown in the figure. The image processing system 200 includes an image processing device 203 connected via a wired or wireless network. The image processing device 203 is connected via a wired or wireless network to each of the image pickup device 202, the display device 204, the recording medium 205, and the output device 206.

[0112] A captured image, obtained by imaging the subject space using image pickup device 202, is input to image processing device 203. The captured image is blurred due to aberrations and diffraction of the optical system 202a in image pickup device 202 and due to the optical low-pass filter of image sensor 202b in image pickup device 202, resulting in reduced information about the subject. Image processing device 203 generates a deblurred image by performing blur sharpening on the captured image using image restoration processing. Additionally, image processing device 203 acquires a saturation effect map. Details of the saturation effect map will be described later. Image processing device 203 also has the function of adjusting the intensity of blur sharpening by obtaining a weighted average of the captured image and the deblurred image. The user can adjust the intensity of blur sharpening while inspecting the image displayed on display device 204. The intensity-adjusted deblurred image is stored in memory 203a or recording medium 205 and output to an output device 206, such as a printer, as needed. The captured image can be grayscale or may include multiple color components. The captured image can be an undeveloped RAW image or a developed image.

[0113] Next, refer to Figure 18 This provides a description of the blur sharpening of the captured image performed by the image processing device 203. Figure 18 This is a flowchart of the deblurred image generation process. The image processing apparatus 203 includes a memory 203a, an acquisition unit 203b, and a sharpening unit 203c, and these components perform the following steps.

[0114] First, in step S301, the acquisition unit 203b acquires the captured image. Then, in step S302, the image processing device 203 acquires an image restoration filter to be used for the image restoration processing described later. In this embodiment, an example of acquiring aberration information (optical information) based on imaging conditions and acquiring an image restoration filter based on the aberration information is given.

[0115] First, the image processing device acquires the imaging conditions (imaging condition information) used by the image pickup device when generating a captured image through imaging. In addition to the focal length, aperture value (F-number), and imaging distance of the image pickup optics, the imaging conditions also include identification information (camera ID) about the image pickup device. Furthermore, in the case of an image pickup device with a replaceable image pickup optics system, the imaging conditions may include identification information (lens ID) about the image pickup optics system (replaceable lens). As described above, the imaging condition information can be acquired as information attached to the captured image, or it can be acquired via wired or wireless communication or a storage medium.

[0116] Subsequently, the image processing device 203 acquires aberration information suitable for the imaging conditions. In this embodiment, the aberration information is an optical transfer function (OTF). The image processing device selects and acquires a suitable OTF from a plurality of pre-stored OTFs based on the imaging conditions. Alternatively, when the imaging conditions, such as aperture value, imaging distance, and focal length of the zoom lens, are specific imaging conditions, the OTF corresponding to that imaging condition can be generated from pre-stored OTFs corresponding to other imaging conditions through an interpolation process. In this case, the amount of OTF data to be stored can be reduced. As an interpolation process, for example, bilinear interpolation (linear interpolation), bicubic interpolation, etc., can be used, but the interpolation process is not limited to these.

[0117] In this embodiment, the image processing apparatus 203 acquires the optical transfer function (OTF) as aberration information, but the aberration information is not limited to this. Instead of the OTF, aberration information such as the point spread function (PSF) can be acquired. Alternatively, in this embodiment, the image processing apparatus can acquire coefficient data by fitting and approximating the aberration information to a predetermined function, and can reconstruct the OTF or PSF from the coefficient data. For example, the OTF can be acquired by fitting using a Legendre polynomial. Alternatively, other functions, such as the Chebusheev polynomial, can be used in the fitting. In this embodiment, the OTF is discretely deployed at multiple locations in the captured image.

[0118] Subsequently, the image processing apparatus 203 converts the optical transfer function (OTF) into an image restoration filter; that is, it generates the image restoration filter by using the OTF deployed at multiple locations. The image restoration filter is generated by creating restoration filter characteristics in the frequency space based on the OTF and converting it into a real-space filter (image restoration filter) through inverse Fourier transform.

[0119] When the correction objective of image restoration processing is to eliminate aberrations and blur (such as diffraction (diffraction blur)) that is independent of image height (image position), a uniform (identical) optical transfer function (OTF) or a uniform (identical) image restoration filter can be used for an image.

[0120] The generation and acquisition of image restoration filters have been described above, but the present invention is not limited thereto, and image restoration filters can be generated and stored in advance and acquired based on imaging conditions.

[0121] Subsequently, in step S303, the sharpening unit 203c generates a deblurred image (first image) by performing image restoration processing on the captured image, in which the blur of the captured image is sharpened. The image restoration processing is performed based on the image restoration filter obtained in step S302.

[0122] In the convolution of an image restoration filter, pixels at locations other than where the image restoration filter is placed can be generated by interpolation using multiple filters placed in neighboring locations. In this case, the image restoration filter includes a first image restoration filter at a first location in the captured image and a second image restoration filter at a second location in the captured image. The first image restoration filter is generated using an unfolded optical transfer function. The second image restoration filter is generated by interpolating the first image restoration filter. By performing such an interpolation process, the image restoration filter can be changed for, for example, each pixel.

[0123] Subsequently, in step S304, the sharpening unit 203c estimates the saturation effect map. In this embodiment, the saturation effect map is generated based on the luminance saturation regions in the captured image and the aberration information of the image pickup optics. That is, the saturation effect map is estimated by convolving the PSF representing the blur of the image pickup optics with a luminance saturation map that serves as an indication (representation, description, or identification) of the luminance saturation regions in the captured image. Information about the structure of the subject space is lost in the luminance saturation regions (luminance saturation areas), so the luminance saturation map obtained by estimating the original signal values ​​in the luminance saturation regions can be used.

[0124] Next, the image processing device 203 combines the captured image and the deblurred image. The combination of the captured image and the deblurred image is as follows: Figure 7 As shown in the flowchart, its detailed description is omitted. In this embodiment, image restoration processing is used as a method for blur sharpening, but the method is not limited to this and can be various sharpening methods such as sharpening and unsharpening masks.

[0125] Using the above configuration, an image processing system can be provided that can generate an image with appropriate correction effect around the saturation area based on brightness or scene during blur sharpening.

[0126] Third Embodiment

[0127] Next, a description of an image processing system 300 according to a third embodiment of the present invention will be given. Figure 19 This is a block diagram of the image processing system 300. Figure 20 This is an external view of the image processing system 300. The image processing system 300 includes a training device 301, an image acquisition device 302, and an image processing device 303. The training device 301 and the image processing device 303, as well as the image processing device 303 and the image acquisition device 302, are connected via a wired or wireless network. The image acquisition device 302 includes an optical system 321, an image sensor 322, a memory 323, a communication unit 324, and a display unit 325. Captured images are transmitted to the image processing device 303 via the communication unit 324.

[0128] Image processing device 303 receives a captured image via communication unit 332 and performs blur sharpening using information about the configuration and weights of a machine learning model stored in memory 331. The information about the machine learning model's configuration and weights has been acquired beforehand from training device 301 through training performed by training device 301 and stored in memory 331. Image processing device 303 has the function of adjusting the intensity of blur sharpening. The blurred and sharpened deblurred image (model output) of the captured image and a weighted average image including the adjusted intensity are sent to image picking device 302, stored in memory 323, and displayed on display unit 325.

[0129] The generation of learning data and learning weights performed by training device 301 (learning phase), the blurring and sharpening of the captured image using the trained machine learning model performed by image processing device 303 (estimation phase), and the combination of the captured image and model output are similar to those in the first embodiment. Therefore, their description is omitted.

[0130] Using the above configuration, an image processing system can be provided that can generate an image with appropriate correction effect around the saturation area based on brightness or scene when performing blur sharpening using a machine learning model.

[0131] Fourth embodiment

[0132] Next, a description of an image processing system 400 according to a fourth embodiment of the present invention will be given. Figure 21 This is a block diagram of the image processing system 400. Figure 22This is an external view of an image processing system 400. The image processing system 400 includes a learning device 401, a lens device 402, an image acquisition device 403, a control device (first device) 404, an image estimation device (second device) 405, and networks 406 and 407. The learning device 401 and the image estimation device 405 are, for example, servers. The control device 404 is a user-operated device such as a personal computer or a mobile terminal. The learning device 401 and the image estimation device 405 can communicate with each other, and the control device 404 and the image estimation device 405 can also communicate with each other.

[0133] The learning device 401 includes a memory 401a, an acquisition unit 401b, a calculation unit 401c, and an update unit 401d, and learns the weights of a machine learning model that performs blur-sharpening on a captured image formed using the lens device 402 and the image pickup device 403. The learning method is similar to that of the first embodiment, and therefore its description is omitted. The image pickup device 403 includes an image sensor 403a, which performs photoelectric conversion on the optical image formed by the lens device 402 to acquire the captured image. The lens device 402 and the image pickup device 403 are detachably attached to each other and can be combined with each other in multiple types.

[0134] The control device 404 includes a communication unit 404a, a display unit 404b, a memory 404c, and an acquisition unit 404d, and controls the processing to be performed on the captured image acquired from the wired or wirelessly connected image acquisition device 403 according to the user's operation. Alternatively, the captured image imaged by the image acquisition device 403 can be pre-stored in the memory 404c, and the control device 404 can read the captured image.

[0135] Image estimation device 405 includes a communication unit 405a, an acquisition unit 405b, a memory 405c, and a sharpening unit 405d. Image estimation device 405 performs blur sharpening processing on a captured image according to a request from a control device 404 connected via a network 406. Image estimation device 405 acquires information about learned weights from a learning device 401 connected via the network 406 during or before the blur sharpening estimation and uses it to estimate the blur sharpening of the captured image. The estimated image after blur sharpening estimation, after the sharpening intensity is adjusted, is again sent to the control device 404, stored in the memory 404c, and displayed on the display unit 404b. The generation of learning data and learning weights (learning phase) performed by the learning device 401 is as in the first embodiment, and therefore its description is omitted.

[0136] Next, refer to Figure 23This provides a description of the blur sharpening of the captured image performed by the control device 404 and the image estimation device 405. Figure 23 This is a flowchart of the model output and sharpening intensity adjustment.

[0137] First, in step S401, the acquisition unit 404d acquires the captured image and the sharpening intensity specified by the user. Then, in step S402, the communication unit 404a sends the captured image and a request related to the execution of the blur sharpening estimation process to the image estimation device 405.

[0138] Subsequently, in step S403, communication unit 405a receives and acquires the transmitted captured image and a processing-related request. Then, in step S404, acquisition unit 405b retrieves information from memory 405c regarding the weights corresponding to the learned weights (suitable for) the captured image. The weight information is pre-read from memory 401a and stored in memory 405c. Then, in step S405, sharpening unit 405d generates a deblurred image (model output, first image) from the captured image using a machine learning model, in which the blur of the captured image is sharpened. The machine learning model has the characteristics of training. Figure 1 The configuration shown is as follows. As during training, a saturation effect map and model output are generated by generating and inputting a saturation saturation map of the saturation saturation regions of the captured image.

[0139] Subsequently, in step S406, the sharpening unit 405d generates a weight map. The weight map is generated and the captured image and the deblurred image (model output) are combined using the method described in the first embodiment. The intensity can be adjusted by adjusting the weight map according to the sharpening intensity specified by the user. For example, it can be adjusted by changing... Figure 8 The relationship between the average signal value and the adjustment value shown is used to adjust the intensity in the saturated region. Alternatively, when using a second weighted graph related to the intensity in the unsaturated region and a third graph related to the intensity in the saturated region, the intensity in both the unsaturated and saturated regions can be adjusted by adjusting the second and third graphs. Alternatively, the entire weighted graph can be adjusted.

[0140] Subsequently, in step S407, the sharpening unit 405d combines the captured image and the deblurred image (model output) based on the weight map. Then, in step S408, the communication unit 405a sends the combined image to the control device 404. Finally, in step S409, the communication unit 404a acquires the transmitted combined image.

[0141] Using the above configuration, an image processing system can be provided that can generate an image with appropriate correction effect around the saturation area based on brightness or scene when performing blur sharpening using a machine learning model.

[0142] Other embodiments

[0143] Embodiments of the present invention can also be implemented by a computer of a system or apparatus that reads and executes computer-executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more fully referred to as a 'non-transitory computer-readable storage medium') to perform one or more functions of the above embodiments and / or includes one or more circuits (e.g., application-specific integrated circuits (ASICs)) for performing one or more functions of the above embodiments, and by a method performed by a computer of the system or apparatus by, for example, reading and executing computer-executable instructions from the storage medium to perform one or more functions of the above embodiments and / or controlling one or more circuits to perform one or more functions of the above embodiments. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessor unit (MPU)) and may include separate computers or networks of separate processors to read and execute computer-executable instructions. The computer-executable instructions may be provided to the computer, for example, from a network or storage medium. The storage medium may include, for example, a hard disk, random access memory (RAM), read-only memory (ROM), storage devices for distributed computing systems, optical discs (such as CDs, DVDs, or Blu-ray discs). TM One or more of the following: flash memory devices, memory cards, etc.

[0144] The embodiments of the present invention can also be implemented by providing software (programs) that perform the functions of the above embodiments to a system or device via a network or various storage media, and the computer or central processing unit (CPU) or microprocessor unit (MPU) of the system or device reads out and executes the program.

[0145] According to each embodiment, an image processing method, an image processing apparatus, an image processing system, and a storage medium may be provided, each of which may appropriately perform deblurring based on the brightness or scene of the image.

[0146] While the invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be given the broadest interpretation in order to cover all such modifications and equivalent structures and functions.

Claims

1. An image processing method, comprising: Acquire captured images obtained through imaging; A first image is generated by correcting the blur components of the captured image; as well as A second image is generated based on the captured image, the first image, and the weight information. The weighting information is characterized by being generated based on (i) information about the brightness of the captured image or information about the scene of the captured image and (ii) information about the saturation regions in the captured image. The information regarding the saturation region includes (i) the extent of the area in the saturation region where the subject has been expanded due to the blurring component and (ii) the relationship between the signal value corresponding to the region.

2. The image processing method according to claim 1, wherein when generating the first image, the blurring component is corrected by inputting the captured image into a machine learning model.

3. The image processing method according to claim 1, wherein the information regarding the brightness of the captured image is a statistic related to the signal value of the captured image.

4. The image processing method according to claim 3, wherein the statistic is at least one of the mean, median, variance, and histogram of the signal values.

5. The image processing method according to claim 1, wherein the information about the scene of the captured image is information about the type of scene of the captured image or information about the imaging mode used in the imaging.

6. The image processing method of claim 1, wherein information about the saturated region is obtained by inputting the captured image into a machine learning model.

7. The image processing method of claim 1, wherein information about the saturation region is obtained based on the saturation region and optical information about the optical system used in the imaging.

8. The image processing method according to claim 1, wherein the weight information is generated based on a third image, the third image being obtained by subtracting the captured image from information about the saturated region.

9. The image processing method according to claim 8, wherein the weight information is generated based on the signal values ​​of pixels among a plurality of pixels in the third image, the pixels corresponding to pixels in the unsaturated region of the captured image.

10. The image processing method of claim 8, wherein the weight information is generated based on statistics related to the signal values ​​of each segmented region in the third image.

11. The image processing method according to claim 8, wherein, As the brightness of the third image increases, the weight of the captured image, as indicated by the weight information, increases.

12. The image processing method according to claim 11, wherein the brightness of the third image is determined based on the average signal value of the third image.

13. The image processing method according to claim 1, wherein, As the brightness of the captured image increases, the weight of the captured image, as indicated by the weight information, increases.

14. The image processing method of claim 1, wherein the blur component is based on optical information about the optical system used in the imaging.

15. The image processing method according to any one of claims 1 to 14, wherein when generating the second image, the second image is generated by obtaining a weighted average of the captured image and the first image based on the weight information.

16. An image processing apparatus, comprising: An acquisition unit, configured to acquire a captured image obtained through imaging; A first generation unit is configured to generate a first image by correcting the blur components of the captured image; as well as The second generation unit is configured to generate a second image based on the captured image, the first image, and weight information. The weighting information is characterized by being generated based on (i) information about the brightness of the captured image or information about the scene of the captured image and (ii) information about the saturation regions in the captured image. The information regarding the saturation region includes (i) the extent of the area in the saturation region where the subject has been expanded due to the blurring component and (ii) the relationship between the signal value corresponding to the region.

17. An image processing system, the image processing system comprising a first device and a second device, the first device and the second device being capable of communicating with each other. Its features The first device includes a transmitting unit configured to send a request to the second device relating to the execution of processing of a captured image obtained through imaging. The second device includes: A receiving unit, configured to receive the request; Acquisition unit, the acquisition unit being configured to acquire the captured image; A first generation unit, configured to generate a first image based on the request by correcting the blur components of the captured image; and The second generation unit is configured to generate a second image based on the captured image, the first image, and weight information. The weighting information is generated based on (i) information about the brightness of the captured image or information about the scene of the captured image and (ii) information about the saturation regions in the captured image. The information regarding the saturation region includes (i) the extent of the area in the saturation region where the subject has been expanded due to the blurring component and (ii) the relationship between the signal value corresponding to the region.

18. A non-transitory computer-readable storage medium storing a computer program that causes a computer to perform the image processing method according to any one of claims 1 to 15.

Images