Image processing method, device and system, storage medium and learning model manufacturing method

Abstract

The present invention provides an image processing method, an apparatus and a system, a storage medium and a learning model manufacturing method. The image processing method is used to generate an estimated image, in which the defocused blur shape in the captured image is corrected, and the image processing method comprises: a first step of acquiring input data, the input data comprising the captured image and shape specifying information, the shape specifying information specifying the defocused blur shape in the estimated image; and a second step of inputting the input data into a machine learning model and generating an estimated image.

Description

Technical Field

[0001] The present invention relates to an image processing method for obtaining a good image by shaping blur caused by defocus in a captured image. Background Art

[0002] Japanese Patent Laid-Open (“JP”) No. 2016-220016 discloses a method for correcting a blur shape (defocus blur or defocus bokeh) caused by defocus, the method comprising: dividing a pupil in an optical system into a plurality of parts; capturing a plurality of parallax images obtained by observing a subject space from each of the divided pupils; and adjusting a weight for combining the plurality of parallax images.

[0003] However, the method disclosed in JP 2016-220016 cannot reproduce the defocus blur corresponding to a pupil larger than the pupil in the optical system because the method adjusts the weights of the respective divided pupils and combines a plurality of parallax images. In other words, the method cannot compensate for the defocus blur defect caused by vignetting. When the weights for combining a plurality of parallax images become uneven, the noise increases. Since double-line blur and the like have a fine structure, it is necessary to divide the pupil in the optical system into finer parts to reduce its influence. Then, the spatial resolution of the respective parallax images decreases or the noise increases.

[0004] Defocus blur shapes can be corrected using machine learning models such as CNN (Convolutional Neural Network). Due to users' personal preferences for defocus blur shapes, it is necessary to learn defocus blurs with various shapes as ground truth images. However, when there are multiple correct images corresponding to a single training image for certain situations, it is difficult to learn the learning data when only the weights in one machine learning model are used. Therefore, even if the captured image is input to the learned CNN, the average value of various defocus blur shapes as correct images can be output as an estimated image. This is because CNN learns multiple correct images on average. If different weights are learned for each defocus blur shape as a correct image, the learning load and the amount of data stored will increase. Summary of the invention

[0005] The present invention provides image processing methods, etc., each of which is capable of correcting the defocused blur shape in a captured image to various shapes while suppressing the increase in the learning load of a machine learning model and the amount of stored data.

[0006] According to an image processing method on one aspect of the present invention, a method for generating an estimated image in which a defocused blur shape in a captured image is corrected is provided. The method comprises: a first step of acquiring input data, wherein the input data comprises the captured image and shape specifying information, wherein the shape specifying information specifies a defocused blur shape in the estimated image; and a second step of inputting the input data into a machine learning model and generating the estimated image.

[0007] A non-transitory computer-readable storage medium storing a program for causing a computer to execute the above image processing method also constitutes another aspect of the present invention. An image processing apparatus corresponding to the above image processing method also constitutes another aspect of the present invention.

[0008] According to another aspect of the present invention, a learning model manufacturing method includes the following steps: obtaining a training image and a correct image with different blur shapes for the same subject; obtaining shape designation information for designating the defocus blur shape of the correct image; and performing learning at a machine learning model based on the training image, the correct image, and the shape designation information. An image processing device corresponding to the above learning model manufacturing method also constitutes another aspect of the present invention.

[0009] An image processing system according to another aspect of the present invention includes a first device and a second device that communicate with each other. The first device includes a transmitter, which is configured to send a processing execution request for a captured image to the second device. The second device includes: a receiver, which is configured to receive the request; an acquirer, which is configured to acquire input data, the input data including the captured image and shape designation information, the shape designation information being used to specify a defocus blur shape of an estimated image; and a generator, which is configured to input the input data into a machine learning model according to the request, and generate the estimated image, in which the defocus blur shape of the captured image is corrected.

[0010] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 The construction of the machine learning model according to the first embodiment is shown.

[0012] Figure 2 is a block diagram of an image processing system according to the first embodiment.

[0013] Figure 3 is an appearance of the image processing system according to the first embodiment.

[0014] Figure 4is a flowchart of the learning data generating method according to the first to fourth embodiments.

[0015] Figure 5 is a flowchart for weight learning according to the first, third and fourth embodiments.

[0016] Figure 6 is a flowchart for generating an estimation image according to the first embodiment.

[0017] Figure 7 is a flowchart for editing a captured image according to the first embodiment.

[0018] Figure 8 A user interface according to a first embodiment is shown.

[0019] Fig. 9 The construction of the machine learning model according to the second embodiment is shown.

[0020] Fig.10 is a block diagram of an image processing system according to a second embodiment.

[0021] Fig.11 is an appearance of an image processing system according to the second embodiment.

[0022] Fig.12 is a flowchart for weight learning according to the second embodiment.

[0023] Fig.13 is a flowchart for generating an estimation image according to the second embodiment.

[0024] Fig.14 A user interface according to a second embodiment is shown.

[0025] Fig.15 The construction of the machine learning model according to the third embodiment is shown.

[0026] Fig.16 is a block diagram of an image processing system according to a third embodiment.

[0027] Fig.17 is an appearance of an image processing system according to the third embodiment.

[0028] Fig.18 is a flowchart for generating an estimation image according to the third embodiment.

[0029] Fig.19 A user interface according to a third embodiment is shown.

[0030] Fig. 20 The construction of a machine learning model according to a fourth embodiment is shown.

[0031] Fig.21 is a block diagram of an image processing system according to a fourth embodiment.

[0032] Fig. 22 is an appearance of an image processing system according to a fourth embodiment.

[0033] Fig.23 is a flowchart for generating an estimation image according to the fourth embodiment.

[0034] Fig.24 A user interface according to a fourth embodiment is shown.

[0035] FIG. 25A to FIG. 25C Point spread functions (PSFs) at defocus distances are shown in accordance with various embodiments. DETAILED DESCRIPTION

[0036] Now, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Corresponding elements in the various drawings are designated by the same reference numerals, and their repeated description will be omitted.

[0037] Before describing the embodiments in detail, the main points of the present invention will be described. The present invention uses a machine learning model to reshape the defocus blur in the captured image. Machine learning models include, for example, neural networks, genetic programming, Bayesian networks, etc. Neural networks include CNN, GAN (Generative Adversary Network), RNN (Recurrent Neural Network), etc. Defocus blur shaping refers to reshaping the defocus blur form or shape without restoring the frequency component. For example, it includes reshaping from two-line defocus blur to Gaussian defocus blur or circular defocus blur, etc. The details of various defocus blur shapes will be described later. Other defocus blurs to be corrected include, for example, defocus blur defects caused by vignetting, and annular defocus blur caused by pupil occlusion such as a reflective refractive lens. The present invention does not limit the target defocus blur shape or the corrected defocus blur shape.

[0038] Next, the difference from the defocus blur addition performed on a smartphone is described. It is difficult for a smartphone with a wide-angle lens and a small sensor size to create defocus blur. Therefore, defocus blur is added to create the desired defocus blur. On the other hand, the defocus blur correction according to the present invention corrects an already defocused subject to the desired defocus blur. In other words, the present invention requires the application of a defocus blur that satisfies the difference between the pre-existing defocus blur and the desired defocus blur, and requires more complex processing.

[0039] The input data input to the machine learning model includes a captured image and information specifying the corrected defocus blur shape (shape specifying information). The shape refers to at least one of the type and size of the defocus blur. In the learning of the machine learning model, by inputting the shape specifying information together with the training image, the machine learning model can learn multiple correct images with different defocus blur shapes for a single training image. In other words, even if the correct image includes defocus blurs with various shapes, weights for creating different shapes for each defocus blur shape can be learned instead of weights for creating an average shape of the defocus blur. Therefore, learning data including defocus blurs with various shapes can be intensively learned with high accuracy. As a result, the present invention can correct defocus blurs with various shapes according to the user's preferences while suppressing the learning load and the amount of stored data.

[0040] Hereinafter, the stage of learning the weights of the machine learning model is referred to as the learning stage, and the stage of correcting the defocus blur shape using the machine learning model and the learned weights is referred to as the estimation stage.

[0041] First embodiment

[0042] Now refer to Figure 2 and Figure 3 , an image processing system according to a first embodiment of the present invention will be described. Figure 2 is a block diagram of the image processing system 100 . Figure 3 2 is an external view of the image processing system 100 .

[0043] The image processing system 100 includes a learning device 101, a camera 102, an image estimation device 103, a display device 104, a recording medium 105, an output device 106, and a network 107. The learning device 101 includes a memory 101a, an acquirer 101b, a generator 101c, and an updater 101d, and learns the weights of a machine learning model for correcting defocus blur (to make a learning model). The details of the weight learning and the defocus blur correction process using the weights will be described later.

[0044] The imaging device 102 has an optical system 102a and an image sensor 102b, and captures an image of a subject space to obtain a captured image. The optical system 102a collects light incident from the subject space and forms an optical image (subject image). The image sensor 102b acquires a captured image by photoelectrically converting the optical image. The image sensor 102b is, for example, a CCD (charge coupled device) sensor and a CMOS (complementary metal oxide semiconductor) sensor.

[0045] The image estimation device (image processing device) 103 includes a memory 103a, an acquirer 103b, a blur shaping unit (blur shape corrector or generator) 103c and a generator 103d. The image estimation device 103 generates an estimated image in which the defocus blur shape is corrected for (at least a portion of) the captured image captured by the camera 102. For defocus blur correction, a machine learning model is used, which uses weights learned by the learning device 101. The learning device 101 and the image estimation device 103 are connected via a network 107, and the image estimation device 103 reads out the learned weight information from the learning device 101 before or during defocus blur correction. The estimated image is output to at least one of the display device 104, the recording medium 105, and the output device 106.

[0046] The display device 104 is, for example, a liquid crystal display or a projector. The user can perform editing, etc. while checking the image being processed via the display device 104. The details of the user interface during editing will be described later. The recording medium 105 is, for example, a semiconductor memory, a hard disk drive, a server on a network, etc., and stores the estimated image. The recording device 106 is a printer, etc.

[0047] Now refer to Figure 4 , the generation of learning data performed by the learning device 101 will be described, Figure 4 It is a flowchart of the learning data generation method. Figure 4 The various steps in are mainly performed by various units in the learning device 101. This embodiment uses CNN as a machine learning model, but other models can be similarly applied.

[0048] First, in step S101, the acquirer 101b acquires an original image. The number of original images may be one or more. The original image is an actually shot image, a CG (computer graphics) image, etc. In a subsequent step, defocus blur is applied to the original image to create a training image and a correct image. Therefore, the original image may include an image having edges, textures, gradients, flat parts, etc. with various intensities and directions, so as to correctly perform defocus blur shape conversion on various subjects.

[0049] The original image may have a signal value higher than the brightness saturation value of the image sensor 102b. This is because even when the camera 102 captures an image of an actual subject under a specific exposure condition, there are some subjects that do not fall within the brightness saturation value. The correct image and the training image receive defocus blur and are then clipped by the brightness saturation value of the image sensor 102b. Thus, subjects that do not fall within the actual brightness saturation value can be reproduced.

[0050] Next, in step S102, the generator (second acquirer) 101c generates (acquires) shape designation information (which specifies the defocus blur shape of the correct solution image) and stores it in the memory 101a. The shape designation information specifies at least one of the size and type of the defocus shape after correction. The type indicates the difference in Gaussian blur, circular blur, double line blur, etc. caused by the difference in PSF (point spread function) intensity distribution.

[0051] Now refer to FIG. 25A to FIG. 25C , double-line blur, circular blur, and Gaussian blur will be described. Fig.25A The point spread function (PSF) of the two-line blur is shown. Fig.25A In the example, the horizontal axis represents the spatial coordinate (position), while the vertical axis represents the intensity. This also applies to the later described Fig.25B and Fig.25C .like Fig.25A As shown in , the double line blur has a PSF with separated peaks. When the PSF at the defocus distance has Fig.25A When the shape is shown, the subject that originally had a single line appears to be double blurred due to the defocus. Fig.25B A PSF with a circular blur is shown. A circular blur has a PSF with a flat intensity. Fig.25C PSF with Gaussian blur is shown. Gaussian blur has a PSF with a Gaussian distribution. Size refers to the range of intensities that the PSF has.

[0052] Specifying a size corresponds to virtually changing the F number (aperture value) of the optical system 102a. When the F number changes, the size of the pupil of the optical system 102a changes, so that the size of the defocus blur changes. Image processing to correct the defocus blur can change the F number to an F number that the optical system 102a cannot physically obtain from the captured image. Specifying a type corresponds to virtually changing the optical system 102a to a different lens configuration. The type of defocus blur (such as double-line blur, circular blur, and Gaussian blur) depends on the pupil function determined by the lens configuration of the optical system 102a. In other words, specifying the size or type of the corrected defocus corresponds to specifying a virtual lens parameter. More specifically, specifying the F number corresponds to changing the extension of the pupil function. In addition, specifying types such as double-line blur and circular blur corresponds to changing the amplitude or phase of the pupil function.

[0053] The shape designation information may be a scalar or a map, but in the present embodiment, it is a map. When the shape designation information is a scalar, the defocus blur shape cannot be specified for each region in the image. On the other hand, when the shape designation information is a map, the defocus blur shape can be specified for each region in the image. Therefore, the shape designation information may be a map. When the shape designation information is a map, the number of channels is one or more, the type and size of the defocus blur are indicated by numerical values, and the numerical values ​​are normalized. For example, when the blur type is specified in a map with one channel, 0 may be set to Gaussian blur, and 1 may be set to circular blur. When specifying using a map with multiple channels, 1 may be set to the area of ​​Gaussian blur on the first channel, and 1 may be set to the area of ​​circular blur on the second channel. When specifying the blur size by a map with one channel, 0 may be set to the size corresponding to F1.0, and 1 may be set to the size corresponding to F2.0. When specifying using a map with multiple channels, 1 may be set to the area of ​​the size corresponding to F2.0 on the first channel, and 1 may be set to the area of ​​the size corresponding to F1.0 on the second channel.

[0054] A smoothing filter or the like may be applied to the shape designation information. Thus, the boundaries of the designated different shapes can be accurately corrected. The shape designation information generates a plurality of pieces of information that designate various defocused blur shapes. In this embodiment, the various defocused blur shapes are displayed in the user interface. Figure 8 The defocus blur shape in specifies the shapes that can be selected on the label 104a.

[0055] Next, in Figure 4 In step S103, the generator 101c generates a training image and stores it in the memory 101a. The training image is an image obtained by applying a target defocus blur to an original image and by performing an imaging simulation. In order to process all captured images, defocus blur corresponding to various defocus amounts can be applied. Defocus blur can be applied by convolving the PSF with the original image or by taking the product of the frequency characteristics of the original image and the OTF (optical transfer function). Before and after correcting the defocus blur, the image on the focal plane may remain unchanged, so that a training image and a correct solution image to which defocus blur is not applied are also generated.

[0056] Next, in step S104, the generator 101c generates a plurality of correct images corresponding to a plurality of shape designation information for a single training image, and stores them in the memory 101a. The correct image is an image obtained by applying the corrected defocus blur shape to the original image and by performing an imaging simulation. For the corrected defocus blur shape, various patterns of different sizes and types corresponding to the shape designation information are prepared, such as a circular blur corresponding to F2.0 and a Gaussian blur corresponding to F1.0. The correct image and the training image may be an undeveloped RAW image or a developed image. The order of generating the training image, the correct image, and the shape designation information may be interchanged.

[0057] The actual captured image captured by the optical system 102a and the image sensor 102b can be used to generate learning data, for example, a training image is obtained using the optical system 102a. The correct image is obtained by capturing the same subject as the subject of the training image using another imaging optical system having a defocus blur shape different from the defocus blur shape of the optical system 102a. In other words, the training image and the correct image may have the same subject and have different defocus blur shapes. In this case, a plurality of shape designation information is generated for different imaging optical systems having a defocus blur shape different from the defocus blur shape of the optical system 102a, and the plurality of shape designation information has different numerical values. A partial area having a predetermined number of pixels can be extracted from the training image and the correct image generated by the above two methods and used for learning. For the shape designation information, a partial area having a predetermined number of pixels can be extracted at the same timing as each of the training image and the correct image. Alternatively, after a partial area having a predetermined number of pixels can be extracted from the training image and the correct image, a partial area for shape designation information can be generated.

[0058] Now refer to Figure 5 , weight learning (learning phase) will be described. Figure 5 This is a flowchart for weight learning (learning model creation method). Figure 5 The various steps in are mainly performed by the various units in the learning device 101.

[0059] First, in step S111, the acquirer 101b acquires one or more groups of correct solution images and training input data from the memory 101a. The training input data is the input data of the CNN learning stage. The training input data includes training images and shape designation information. A correct solution image with defocus blur corresponding to the shape designation information is selected and acquired from a plurality of correct solution images.

[0060] Next, in step S112, the generator 101c inputs the training input data into the CNN and generates an output image. Figure 1 , generation of an output image in this embodiment will be described. Figure 1 The construction of the machine learning model is shown. The training input data includes a training image 201 and shape designation information 202. The training image 201 may be expressed by grayscale or may have multiple channel components.

[0061] In this embodiment, the CNN (convolutional neural network) 206 has one or more convolutional layers or fully connected layers. At the beginning of learning, the weights (individual filter elements and bias values) of the CNN 206 are generated by random numbers. The connection layer 205 connects the training image 201 and the shape designation information 202 in a predetermined order in the channel direction. Other data can be connected between the training image 201 and the shape designation information 202. The CNN 206 inputs the training input data 203 that connects the training image 201 and the shape designation information 202, and generates an output image 204. When multiple groups of training input data are acquired in step S111, an output image 204 is generated for each group. Even for the same training image 201, the output image 204 will be different according to the shape designation information 202. The position where the training image 201 or the shape designation information 202 is input to the network is not limited. The shape designation information 202 can be input after it is converted into a feature map by the subnet. For example, the shape specification information 202 may be converted into a feature map by the subnet and input from the middle layer of the CNN 206. If the shape specification information is a scalar, the scalar is converted into a feature map by the subnet and then compared with Figure 1 The same processing as the processing of the shape designation information 202 in FIG.

[0062] Next, in Figure 5 In step S113, the updater 101d updates the weights of the CNN according to the error or difference between the output image and the correct image. In this embodiment, the Euclidean norm of the difference between the signal value of the output image and the signal value of the correct image is set as the loss function. However, the loss function is not limited to this embodiment. When multiple groups of training input data and correct images are obtained in step S111, the value of the loss function is calculated for each group. The weight is updated according to the calculated value of the loss function by a back propagation method or the like.

[0063] Next, in step S114, the updater 101d determines whether the weight learning is completed. Completion can be determined based on whether the number of learning (weight update) iterations has reached a specified number or whether the weight change during the update is less than a specified value. If it is determined that the weight learning has not been completed, the process returns to step S111 to obtain one or more sets of new training input data and correct images. When it is determined that the weight learning has been completed, the learning ends and the weight information is stored in the memory 101a.

[0064] Next, the defocus blur shape correction (estimation phase) of the captured image performed by the image estimation device 103 and the user interface are described. In this embodiment, the user determines a replacement area, which is a defocus blur correction area in the captured image. Then, the generator 103d replaces the replacement area with the estimated image, thereby generating an image in which the defocus blur in the arbitrary area specified by the user is corrected. Figure 8 As shown, the display device 104 displays a defocused blur shape designation label 104a, a brush 104b, and a captured image 104c. The user can correct the defocused blur while confirming the display on the display device 104.

[0065] Now refer to Figure 6 , generating an estimated image will be described. Figure 6 is a flow chart for generating the estimated image. Figure 6 The various steps in are mainly performed by various units in the image estimation device 103.

[0066] First, in step S201, the acquirer 103b acquires a captured image and weight information. The captured image to be acquired may be a part of the entire captured image. The weight information is read out from the memory 101a in advance and stored in the memory 103a.

[0067] Next, in step S202, the acquirer 103b acquires a defocus map of the captured image. The defocus map is a map showing information related to the defocus blur applied to the subject in the captured image, and indicates the defocus amount of the subject by a numerical value. The defocus map can be acquired by capturing a parallax image or using DFD (Depth from Defocus). The defocus map has two main functions. The first function is to extract the subject area mixed in the captured image by utilizing the fact that the defocus amount of each subject is different, and use the subject area to determine the replacement area. The second function is to input into the machine learning model to perform shape correction that distinguishes between the subject at the in-focus position and the defocus blur with high precision. This embodiment uses the former function or extracts the subject area mixed in the captured image. The details of the latter will be described later.

[0068] Next, in step S203, the generator 103d generates shape specifying information corresponding to the shape selectable on the defocused blur shape specifying tag 104a. In the present embodiment, since the user specifies the replacement area, the shape specifying information does not have a spatial distribution and can have the same value throughout the entire figure.

[0069] Next, in step S204, the blur shaping unit 103c inputs the input data to the CNN and generates an estimated image. The input data includes the captured image and shape designation information that specifies the defocus blur shape in the estimated image. The estimated image is an image in which the defocus blur shape in the captured image is corrected, and the shape of the image is determined by the shape designation information. Similar to learning, the image is processed using the shape designation information. Figure 1 The CNN 206 shown in FIG. 2 generates an estimated image. In the CNN 206, the acquired learned weights are used. Figure 8 An estimated image is generated for each of the plurality of shape designation information corresponding to the defocus blur selectable on the defocus blur shape designation label 104a shown. The CNN uses the same weight for the first shape designation information for designating at least one of the size and type of the defocus blur and the second shape designation information different from the first shape designation information for designating at least one of the size and type of the defocus blur. The generated plurality of estimated images are stored in the memory 103a.

[0070] Now refer to Figure 7 and Figure 8 , editing of a captured image by a user will be described. Figure 7 is a flowchart for editing captured images. Figure 7 The various steps in are mainly performed by various units in the image estimation device 103.

[0071] First, in step S211, the acquirer 103b acquires the Figure 8 The shape designation tag 104a shown is information on the defocused blur shape after correction.

[0072] Next, in step S212, the acquirer 103b acquires the area information (replacement area) in the captured image 104c selected by the user using the brush 104b. The area is determined by the user pixel by pixel or based on the signal value distribution or defocus map of the captured image from the area roughly selected by the user using the brush 104b. The signal value distribution or defocus map of the captured image enables the target subject area to be extracted. The signal value distribution of the captured image refers to: specifying an area where the pixel value of the captured image changes discontinuously and extracting the subject area.

[0073] Next, in step S213, the generator 103d replaces the replacement area selected by the user with the estimated image having the specified defocus blur shape. As described above, the present embodiment acquires information about the replacement area of ​​the captured image and replaces the replacement area of ​​the captured image with the estimated image. Therefore, compared with the case where the estimated image is generated each time the replacement area is selected, the number of times the estimated image is generated is smaller, and the defocus blur correction preferred by the user can be realized in a short time. The estimated image can use an image stored in the memory 103a.

[0074] The above configuration can correct the defocused blur shape while suppressing the learning load and the amount of stored data. The configuration may be modified so that a map of the shape designation information is generated based on the defocused blur shape and the replacement area selected by the user using the shape designation tag 104a and the brush 104b, and step S204 is performed. The shape designation information having a spatially different distribution will be described in detail in the second and subsequent embodiments.

[0075] Next, conditions for enhancing the effects of this embodiment are described. The input data may also include a brightness saturation map. The brightness saturation map indicates a brightness saturated pixel area in an image and has the same size as the image. The learning stage generates a brightness saturation map based on the training image. The estimation stage generates a brightness saturation map based on the captured image. Since there are false edges in the brightness saturated part that are different from the subject structure caused by the brightness saturation, it is difficult for the machine learning model to distinguish the brightness saturated part from the part with an edge (such as defocus blur and focus position with high-frequency components). The brightness saturation map enables the machine learning model to distinguish defocus blur and focus position with high-frequency components from the brightness saturated part, and enables high-precision shape correction. When a PSF with a peak (such as a double-line blur) is applied, defocus blur with a high-frequency component may occur.

[0076] The input data may also include a defocus map. The defocus map is a map showing information about the defocus blur applied to the subject in the captured image. The learning stage converts the defocus amount corresponding to the defocus blur applied to the training image into a numerical value. For example, the focal plane can be set to 0, the direction away from the camera device can be set to negative, and the approaching direction can be set to positive. The estimation stage obtains a defocus map of the captured image by capturing a parallax image, using DFD (defocus ranging), etc. The numerical value of the defocus map at this time corresponds to the numerical value of the learning stage. The defocus map can correct the shape and defocus blur of the subject at the focus position with high precision. If there is no defocus map, the subject at the focus position and the defocus blur with a high-frequency component cannot be distinguished from each other, and the subject at the focus position will be blurred.

[0077] The input data may also include a state diagram, which is information that the state of the optical system 102a during imaging is represented by (Z, F, D). In (Z, F, D), Z corresponds to the zoom position, F corresponds to the F number, and D corresponds to the focal length.

[0078] The input data may also include a position map. A position map is a map showing the image plane coordinates of each pixel on the image. The position map may be a polar coordinate system (corresponding to image height and azimuth).

[0079] Defocus blur varies depending on the lens state, image height, and orientation. Since the CNN is learned so as to correct all defocus blur shapes included in the learning data on average, the correction accuracy for each different defocus blur shape is reduced. Therefore, the machine learning model can specify the PSF applied to the captured image by inputting the state map and the position map into the machine learning model. Thus, in the learning stage, even if the training image used for learning does not contain various defocus blur shapes, the machine learning model learns weights for different corrections for each blur shape, rather than weights for correcting the defocus blur shape on average. Thus, in the estimation stage, each defocus blur shape can be corrected with high accuracy. Therefore, the present embodiment can concentrate on learning the learning data for correcting various defocus blur shapes while suppressing the reduction in shape correction accuracy.

[0080] Second embodiment

[0081] Now refer to Fig.10 and Fig.11 , an image processing system according to a second embodiment of the present invention will be described. Fig.10 is a block diagram of an image processing system 300 according to this embodiment. Fig.11 FIG. 3 is an external view of the image processing system 300 .

[0082] The image processing system 300 has a learning device 301, a camera device 302, an image estimation device 303, and networks 304 and 305. The learning device 301 has a memory 301a, an acquirer 301b, a generator 301c, and an updater 301d, and learns the weights of a machine learning model for correcting a defocus blur shape. The details of weight learning and defocus blur correction using weights will be described later.

[0083] The imaging device 302 has an optical system 302a, an image sensor 302b, an acquirer 302c, a recording medium 302d, a display unit 302e, and a system controller 302f. The optical system 302a collects light incident from the subject space and forms an optical image (subject image). The image sensor 302b converts the optical image into an electrical signal through photoelectric conversion and generates a captured image.

[0084] The image estimation device (image processing device) 303 includes a memory 303a, a blur shaping unit (blur shape corrector or generator) 303b, an acquirer 303c, and a generator 303d. The image estimation device 303 generates an estimated image in which the defocus blur shape is corrected for (at least a part of) the captured image captured by the camera 302. The information of the learned weight learned by the learning device 301 is used to generate the estimated image. The weight information is stored in the memory 303a. The acquirer 302c acquires the estimated image, and the recording medium 302d stores the estimated image. The system controller 302f controls a series of operations of the camera 302.

[0085] Now refer to Fig.12 , the weight learning (learning phase) performed by the learning device 301 will be described. Fig.12 It is a flowchart for weight learning (learning model making method). Fig.12 The steps in are mainly performed by the various units in the learning device 301. This embodiment uses GAN for the machine learning model, but other models can be similarly applied. GAN is an adversarially generated network that includes a generator that generates an image and a discriminator that recognizes the generated image. In this embodiment, the description of the same elements as the first embodiment will be omitted.

[0086] First, in step S301, the acquirer 301b acquires one or more sets of correct answer images and training input data from the memory 301a. The generation of the correct answer images and the training images is the same as in the first embodiment. Fig. 9 , the generation of training input data according to this embodiment will be described. Fig. 9 The structure of GAN as a machine learning model is shown. The concatenation layer 406 connects the training image 401 and the shape designation information 402 in a predetermined order in the channel direction to generate training input data 403.

[0087] Next, in Fig.12 In step S302, the generator 301c inputs the training input data 403 into the generator 407 and generates an output image 404. The generator 407 is, for example, a CNN. Next, in step S303, the updater 301d updates the weights of the generator 407 according to the error or difference between the output image 404 and the correct image 405. The Euclidean norm of the difference at each pixel is used for the loss function. Next, in step S304, the updater 301d determines whether the first learning has been completed. If the first learning has not been completed, the process returns to step S301. On the other hand, when the first learning has been completed, the process proceeds to step S305 and the second learning is performed.

[0088] Next, in step S305, as in step S301, the acquirer 301b acquires one or more sets of correct solution images 405 and training input data 403 from the memory 301a. Next, in step S306, as in step S302, the generator 301c inputs the training input data 403 into the generator 407 and generates an output image 404.

[0089] In step S307, the updater 301d updates the weight of the discriminator 408 according to the output image 404 and the correct image 405. The discriminator 408 discriminates whether the input image is a false image generated by the generator 407 or a true image as the correct image 405. The output image 404 or the correct image 405 is input to the discriminator 408 to generate an identification label (false or true). The weight of the discriminator 408 is updated based on the error or difference between the identification label and the correct label (the output image 404 is false, and the correct image 405 is true). Although the sigmoid cross entropy is used for the loss function, other loss functions can be used.

[0090] Next, in step S308, the updater 301d updates the weights of the generator 407 according to the output image 404 and the correct image 405. The loss function is the weighted sum of the Euclidean norm in step S303 and the following two terms. The first term is a term called content loss, which is the Euclidean norm of the difference between each element of the output image 404 and the correct image 405 after the output image 404 and the correct image 405 are converted into feature maps. By adding the difference of the feature map to the loss function, the more abstract properties of the output image 404 can be made closer to the correct image 405. The second term is the S-shaped cross entropy of the identification label obtained by inputting the output image 404 into the discriminator 408, which is called adversarial loss. Learning is performed so that the discriminator 408 can distinguish between false and true, and the generator 407 is trained to output an output image 404 that looks more objectively like the correct image 405.

[0091] Next, in step S309, the updater 301d determines whether the second learning has been completed. Similar to step S304, if the second learning has not been completed, the flow returns to step S305. On the other hand, when the second learning has been completed, the learned weight information of the generator 407 is stored in the memory 301a.

[0092] Now refer to Fig.13 and Fig.14 , the defocus blur shape correction (estimation phase) and the user interface performed by the image estimation device 303 will be described. Fig.13 This is a flowchart for generating an estimated image in this embodiment. Fig.14A user interface according to this embodiment is shown. In this embodiment, the user can correct the defocus blur shape by specifying a threshold value, by dividing areas in a captured image, and by specifying a defocus blur shape for each divided area.

[0093] like Fig.14 As shown, the display unit 302e displays a captured image 302e-1, a brightness value histogram 302e-2, an area division image 302e-3, and a defocused blur shape designation label 302e-4. The captured image 302e-1 in this embodiment has a person in focus, and a high brightness subject and other subjects in the background are out of focus.

[0094] First, in Fig.13 In step S401, the acquirer 303c acquires (at least a portion of) the captured image. Next, in step S402, the acquirer 303c acquires a threshold value specified by the user by viewing the histogram 302e-2 in the captured image 302e-1. Then, in step S403, the generator 303d generates a region division image 302e-3 based on the acquired threshold value. For example, for a captured image with a brightness value of 0 to 255, it is assumed that the brightness value of region A is 250 to 255, and the brightness value of region B is 0 to 250. Thus, high-brightness subjects and other subjects in the background can be divided. When the defocus blur shape is uniformly corrected in the entire image, no region division is required.

[0095] Next, in step S404, the acquirer 303c acquires the corrected defocus blur shape specified by the user on the defocus blur shape designation tag 302e-4. The shape is specified for each divided area in step S403. The specified information includes size and type. The size is specified as F1.0 or F2.0. For the type, various defocus blur shapes such as Gaussian blur and circular blur are specified. For example, area A is specified as F1.0 circular blur, and area B is specified as F2.0 Gaussian blur.

[0096] Next, in step S405, the generator 303d generates shape designation information based on the acquired area division information and the information designating the corrected defocus blur shape. Next, in step S406, the acquirer 303c acquires input data and learned weight information. The input data includes the captured image and the shape designation information. The weight information is read out from the memory 301a in advance and stored in the memory 303a.

[0097] Next, in step S407, the blur shaping unit 303b inputs the input data to the CNN and generates an estimated image. In step S404, when the area A is specified as F1.0 circular blur and the area B is specified as F2.0 Gaussian blur, an estimated image is generated in which the defocus blur shape in the captured image 302e-1 is corrected to the specified shape.

[0098] The above configuration can correct the defocus blur shape to various shapes while suppressing the learning load and the amount of stored data.

[0099] Third embodiment

[0100] Now refer to Fig.16 and Fig.17 , an image processing system according to a third embodiment of the present invention will be described. Fig.16 is a block diagram of an image processing system 500 according to this embodiment. Fig.17 2 is an external view of the image processing system 500 .

[0101] The image processing system 500 includes a learning device 501, a camera 502, a lens device 503, a control device (first device) 504, an image estimation device (second device) 505, and networks 506 and 507. The control device 504 and the image estimation device 505 can communicate with each other via the network 506. The learning device 501 and the image estimation device 505 can communicate with each other via the network 507. The learning device 501 and the image estimation device 505 include, for example, a server. The control device 504 includes a user terminal (smart phone, tablet, etc.). The learning device 501 includes a memory 501a, an acquirer 501b, a generator 501c, and an updater 501d, and learns the weights of a machine learning model for correcting a defocused blur shape in a captured image captured using the camera 502. Details about the learning will be described later.

[0102] The camera 502 has an image sensor 502a, and the image sensor 502a photoelectrically converts an optical image formed by the lens device 503 to obtain a captured image. The lens device 503 can be attached to and detached from the camera 502, and the lens device 503 and the camera 502 can be combined by various types. The control device 504 includes a communicator 504a, a display unit 504b, a memory 504c, and an acquirer 504d, and controls the processing of the captured image acquired from the camera 502 connected by wire or wirelessly according to the operation of the user. Alternatively, the captured image captured by the camera 502 can be stored in the memory 504c in advance, and the stored captured image can be read out.

[0103] The image estimation device 505 includes a communicator 505a, an acquirer (acquirer) 505b, a memory 505c, and a shaping unit (shape corrector or generator) 505d. The image estimation device 505 performs defocus blur correction processing on the captured image in response to a request from the control device 504 connected via the network 506. The image estimation device 505 acquires learned weight information from the learning device 501 connected via the network 507 before or during the defocus blur correction, and uses it for the defocus blur correction of the captured image. The estimated image after the defocus blur shape correction is sent to the control device 504 again, stored in the memory 504c, and displayed on the display unit 504b.

[0104] Next, the weight learning (learning phase) performed by the learning device 501 is described. The input method of the shape designation information in this embodiment is different from that in the first embodiment. The steps of the flowchart for weight learning are the same as those in the first embodiment. Figure 5 same.

[0105] Will refer to Fig.15 Describes the method of inputting shape specification information. Fig.15 The construction of the machine learning model according to the present embodiment is shown. The training input data is a training image 601 and shape designation information 602. The training image 601 and the shape designation information 602 are converted into feature maps 603 and 604 using subnets 607 and 608, respectively. The concatenation layer 609 connects the feature maps 603 and 604 in a predetermined order in the channel direction. The CNN 610 inputs the concatenated feature maps 603 and 604 and generates an output image 605.

[0106] Now refer to Fig.18 and Fig.19 , defocus blur correction (estimation phase) of a captured image and a user interface performed by the control device 504 and the image estimation device 505 in this embodiment will be described. Fig.18 is a flowchart for generating an estimation image according to the present embodiment. Fig.18 The various steps in are mainly performed by various units in the image estimation device 505. Fig.19 A user interface according to this embodiment is shown.

[0107] In this embodiment, the user can correct the defocused blur shape in the captured image by selecting a preset mode. Fig.19 As shown, the display unit 504b displays a mode selection label 504b-1 and a captured image 504b-2. The captured image 504b-2 in this embodiment has three persons at the focused position, and the background is out of focus.

[0108] First, in Fig.18In step S501, the acquirer 504d acquires the information selected by the user on the mode selection label 504b-1. For example, the following three optional modes are pre-set, including (1) all Gaussian blur (F number is 1.0); (2) all circular blur (F number is 2.0); (3) the brightness saturated part is circular blur and the other part is Gaussian blur (F number is 1.0). The mode selection corresponds to the selection of the shape designation information. Other modes can be set in which the areas are divided according to the defocus amount of the defocus map, and the F number is lower in the area with a larger defocus amount. This structure can create an image that emphasizes the sense of distance.

[0109] Next, in step S502, the communicator (transmitter) 504a sends a request (the request is a request for executing processing for the captured image) to generate a captured image, a defocus map, selection mode information, and an estimated image to the image estimation device 505. The defocus map can be generated by the method described in the first embodiment.

[0110] Next, in step S503, the communicator 505a receives and acquires the sent captured image, defocus map, mode information, and processing request. Next, in step S504, the acquirer 505b acquires the learned weight information from the memory 505c. The weight information is read out from the memory 501a in advance and stored in the memory 505c. Next, in step S505, the acquirer 505b acquires the brightness saturation map from the captured image.

[0111] Next, in step S506, the shaping unit 505d generates shape designation information based on the selected pattern information. A brightness saturation map or a defocus map can be used to generate shape designation information. When a brightness saturation map or a defocus map is used, the subject area can be extracted, and the shape designation information has a spatial distribution. However, if the subject area is extracted only by the brightness saturation map or the defocus map, defects may occur in the subject area. For example, when trying to extract a light source area such as a street lamp using a brightness saturation map, part of the brightness may be unsaturated due to the uneven brightness of the light source or the applied defocus blur distribution. In this case, opening processing can be used to compensate for the defects in the area. Therefore, opening processing can be used to extract the subject area.

[0112] Next, in step S507, the shaping unit 505d inputs the input data to the CNN and generates an estimated image in which the defocused blur shape in the captured image is corrected. The input data includes the captured image and shape designation information. In step S501, when (1) all is Gaussian blur (F number is 1.0) is selected, the defocused blur shape in the captured image 504b-2 can be corrected to an estimated image with a corrected F1.0 Gaussian blur. Next, in step S508, the communicator 505a sends the estimated image to the control device 504.

[0113] Next, in step S509, the communicator 504a acquires the transmitted estimated image and stores it in the memory 504c. Alternatively, in step S502, a request to generate all captured images and estimated images in a preset selectable mode may be sent, and a plurality of estimated images generated in subsequent steps may be stored in the memory 505c. When the user selects a mode, the corresponding estimated image is sent to the control device 504.

[0114] The above configuration can correct the defocus blur shape to various shapes while suppressing the learning load and the amount of stored data.

[0115] Fourth embodiment

[0116] Now refer to Fig.21 and Fig. 22 , an image processing system according to a fourth embodiment of the present invention will be described. Fig.21 is a block diagram of an image processing system 700 . Fig. 22 FIG. 7 is an external view of the image processing system 700 .

[0117] Image processing system 700 includes learning device 701, camera 702, lens device 703, control device (first device) 704, image estimation device (second device) 705, and networks 706 and 707. Learning device 701 includes memory 701a, acquirer 701b, generator 701c and updater 701d. Camera 702 has image sensor 702a. Control device 704 has communicator 704a, display unit 704b, memory 704c and acquirer 704d. Image estimation device 705 includes communicator 705a, acquirer 705b, memory 705c and shaping unit (shape corrector or generator) 705d. Since the construction and operation of each component in image processing system 700 are the same as those of image processing system 500 according to the third embodiment, description thereof will be omitted. In this embodiment, a personal computer is used for user terminal.

[0118] Next, the weight learning (learning phase) performed by the learning device 701 is described. This embodiment is different from the first embodiment in the method of inputting shape designation information. The other steps are common.

[0119] Will refer to Fig. 20 Describes the method of inputting shape specification information. Fig. 20 The construction of the machine learning model according to the present embodiment is shown. The training input data includes a training image 801 and shape designation information 802. In this embodiment, the shape designation information 802 is converted into a feature map 803 using a subnet 806. The concatenation layer 807 concatenates the training image 801 and the feature map 803 in a predetermined order in the channel direction. The CNN 808 inputs the concatenated training image 801 and the feature map 803, and generates an output image 804.

[0120] Now refer to Fig.23 and Fig.24 , defocus blur correction (estimation phase) of a captured image performed by the control device 704 and the image estimation device 705 and the user interface will be described. Fig.23 is a flowchart for generating an estimation image according to the present embodiment. Fig.23 The various steps in are mainly performed by various units in the image estimation device 705. Fig.24 A user interface according to this embodiment is shown.

[0121] In this embodiment, the user can correct the defocus blur shape by dividing the captured image into a plurality of areas using a cursor or the like, and by specifying the defocus blur shape for each of the divided areas. Fig.24 As shown, the display unit 704b displays a captured image 704b-1, a region division image 704b-2, and a defocus blur shape designation label 704b-3. The captured image 704b-1 according to the present embodiment has a person in focus, and a high-brightness subject and other subjects in the background are out of focus.

[0122] First, in Fig.23In step S601, the acquirer 704d acquires the area division information specified by the user, and divides the captured image into a plurality of areas. The user specifies the area division information of the captured image while viewing the display unit 704b. For example, when the user wants to extract area A, the user uses a cursor or the like to roughly select an area around area A. Next, the user acquires area A from the roughly selected area by using a brightness saturation map or a defocus map. When the area to be acquired (extracted) is a brightness saturated part, a brightness saturation map is used. If the area to be acquired is a specific subject area, a defocus map is used. When acquiring a brightness saturated area, a brightness saturation map is used because the defocus map can include areas other than the brightness saturated part. This kind of map is used because it is difficult for the user to specify an area with high precision using a cursor or the like.

[0123] Next, in step S602, the acquirer 704d acquires information about the corrected defocus blur shape specified by the user. This is specified by the user for each divided area using the blur shape designation tag 704b-3. For example, area A is specified as F1.4 circular blur, and area B is specified as F2.8 Gaussian blur. Next, in step S603, the communicator (transmitter) 704a sends the captured image, a request to generate an estimated image (a request to execute processing for the captured image), area division information, and information about the corrected defocus blur shape to the image estimation device 705.

[0124] Next, in step S604, the communicator 705a receives and acquires the sent captured image, processing request, region division information, and information about the corrected defocus blur shape. Next, in step S605, the acquirer 705b acquires the learned weight information from the memory 705c. The weight information is read from the memory 701a in advance and stored in the memory 705c. Next, in step S606, the shaping unit (shape corrector or generator) 705d generates shape designation information based on the region division information and the information about the corrected defocus blur shape. Next, in step S607, the shaping unit 705d inputs the input data into the CNN and generates an estimated image in which the defocus blur shape is corrected. In step S602, when area A is designated as F1.4 circular blur and area B is designated as F2.8 Gaussian blur, the following estimated image can be generated in which the defocus blur shape in the captured image 704b-1 has been corrected to the specified shape.

[0125] Next, in step S608, the communicator 705a transmits the estimated image to the control device 704. Then, in step S609, the communicator 704a acquires the transmitted estimated image and stores it in the memory 704c.

[0126] The above configuration can correct the defocus blur shape to various shapes while suppressing the learning load and the amount of stored data.

[0127] Other embodiments

[0128] The embodiments of the present invention may also be implemented by reading and executing computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more completely referred to as a "non-transitory computer-readable storage medium") to perform one or more functions in the above-mentioned embodiments, and / or a computer of a system or device including one or more circuits (e.g., an application-specific integrated circuit (ASIC)) for performing one or more functions in the above-mentioned embodiments, and the embodiments of the present invention may be implemented by, for example, reading and executing the computer executable instructions from the storage medium by the computer of the system or device to perform one or more functions in the above-mentioned embodiments, and / or controlling the one or more circuits to perform one or more functions in the above-mentioned embodiments. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessing unit (MPU)), and may include a network of separate computers or separate processors to read and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, a hard disk, a random access memory (RAM), a read-only memory (ROM), a memory of a distributed computing system, an optical disk (such as a compact disc (CD), a digital versatile disc (DVD), or a Blu-ray disc (BD) TM ), flash memory devices, memory cards, etc.

[0129] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0130] Each embodiment can provide an image processing method capable of correcting the defocus blur shape in a captured image to various shapes (according to the user's preference) while suppressing the increase in the learning load of a machine learning model and the amount of stored data. Each embodiment can provide a non-transitory computer-readable storage medium, an image processing device, an image processing system, and a learning model manufacturing method for obtaining the same effect.

[0131] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments.The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. An image processing method for generating an estimated image in which a defocused blur shape in a captured image is corrected, characterized in that: The image processing method comprises: The first step is to obtain input data, wherein the input data includes the captured image and shape specifying information, wherein the shape specifying information specifies a defocus blur shape in the estimated image; The second step is to input the input data into a machine learning model and generate the estimated image; and The third step is to obtain information related to the replacement area of ​​the captured image, and to replace the replacement area of ​​the captured image with the estimated image. wherein the shape specifying information includes information specifying at least one of a size and a type of defocus blur, and The machine learning model uses the same weight for first shape designation information for specifying one of a size and a type of defocus blur, and second shape designation information different from the first shape designation information for specifying one of a size and a type of defocus blur.

2. The image processing method according to claim 1, characterized in that: The size and type correspond to the virtual lens parameters.

3. The image processing method according to claim 1, characterized in that: The shape specifying information is determined based on a signal value of the captured image.

4. The image processing method according to claim 3, characterized in that: The shape specifying information is determined based on a distribution of signal values ​​of the captured image.

5. The image processing method according to claim 1, characterized in that: The shape specifying information is determined based on a defocus map of the captured image.

6. The image processing method according to claim 1, characterized in that: The shape specifying information has a spatial distribution.

7. The image processing method according to claim 1, characterized in that: The shape designation information is a map of at least one channel and indicates at least one of a size and a type of defocus blur using a numerical value, and Therein, the values ​​are normalized.

8. The image processing method according to claim 1, characterized in that: The replacement area is determined based on a signal value or a defocus map of the captured image.

9. The image processing method according to claim 1, characterized in that: The input data further includes a defocus map or a brightness saturation map of the captured image. 10 . A non-transitory computer-readable storage medium storing a program for causing a computer to execute the image processing method according to claim 1 .

11. An image processing device for generating an estimated image in which a defocused blur shape in a captured image is corrected, the image processing device comprising: an acquirer configured to acquire input data including the captured image and shape specifying information specifying a defocus blur shape in the estimated image; a generator configured to input the input data into a machine learning model and generate the estimated image; as well as a replacer that acquires information related to a replacement area of ​​the captured image and replaces the replacement area of ​​the captured image with the estimated image, wherein the shape specifying information includes information specifying at least one of a size and a type of defocus blur, and The machine learning model uses the same weight for first shape designation information for specifying one of a size and a type of defocus blur, and second shape designation information different from the first shape designation information for specifying one of a size and a type of defocus blur.

12. A method for making a machine learning model, comprising the following steps: Obtaining training images and correct images with different blur shapes for the same subject; acquiring shape specifying information for specifying a defocused blur shape of the correct solution image; and performing learning at a machine learning model based on the training image, the correct image, and the shape specifying information, in, The shape designation information includes information designating at least one of a size and a type of defocus blur, and The machine learning model uses the same weight for first shape designation information for specifying one of a size and a type of defocus blur, and second shape designation information different from the first shape designation information for specifying one of a size and a type of defocus blur.

13. An image processing device, comprising: A first acquirer is configured to acquire a training image and a correct image having different blur shapes for the same object; A second acquirer configured to acquire shape specifying information for specifying a defocused blur shape of the correct solution image; as well as a learner configured to perform learning based on the training image, the correct answer image, and the shape specifying information at a machine learning model, wherein the shape specifying information includes information specifying at least one of a size and a type of defocus blur, and The machine learning model uses the same weight for first shape designation information for specifying one of a size and a type of defocus blur, and second shape designation information different from the first shape designation information for specifying one of a size and a type of defocus blur.

14. An image processing system comprising a first device and a second device communicating with each other, It is characterized in that The first device includes a transmitter configured to transmit a processing execution request for a captured image to the second device, Wherein, the second device comprises: a receiver configured to receive the request; an acquirer configured to acquire input data, the input data including the captured image and shape specifying information for specifying a defocus blur shape of an estimated image; a generator configured to input the input data into a machine learning model according to the request and generate the estimated image in which a defocused blur shape of the captured image is corrected; and a replacer that acquires information related to a replacement area of ​​the captured image and replaces the replacement area of ​​the captured image with the estimated image, wherein the shape specifying information includes information specifying at least one of a size and a type of defocus blur, and The machine learning model uses the same weight for first shape designation information for specifying one of a size and a type of defocus blur, and second shape designation information different from the first shape designation information for specifying one of a size and a type of defocus blur.

Images