Image processing method, image processing device, image processing system, learnt weight generating method and program

Abstract

To provide an image processing method which suppresses a learning load on a machine-learning model and an amount of possessed data thereof, and highly precisely sharpens or shapes blur of a picked-up image.SOLUTION: An image processing method includes: a first step of obtaining input data that includes a picked-up image and information on a state of an optical system used for imaging the picked-up image; and a second step of inputting the input data in a machine-learning model, and generating an estimated image obtained by sharpening the picked-up image or an estimated image obtained by shaping blur included in the picked-up image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing method for sharpening or shaping blur caused by an optical system in an image captured using an optical system. [Background technology]

[0002] Patent Document 1 discloses a method for correcting blur caused by aberration in a captured image by processing based on a Wiener filter to obtain a sharpened image. Patent Document 2 discloses a method for correcting blur caused by defocus in a captured image by using a convolutional neural network (CNN). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-123589 [Patent Document 2] Japanese Patent Application Publication No. 2017-199235 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the method disclosed in Patent Document 1 uses processing (linear processing) based on a Wiener filter, and therefore is unable to perform high-precision blur sharpening. For example, it is unable to restore information about a subject whose spatial frequency spectrum has been reduced to zero or to the same intensity as noise due to blur. Furthermore, because different Wiener filters must be used for different aberrations, the amount of data stored for sharpening (the amount of data representing multiple Wiener filters) becomes large in an optical system that generates various aberrations.

[0005] On the other hand, the CNN disclosed in Patent Document 2 uses nonlinear processing, making it possible to estimate the spatial frequency spectrum of a subject that has been reduced to near zero. However, when sharpening an image captured using an optical system that generates various aberrations, the sharpening accuracy decreases, or the learning load and amount of stored data increase. When blur sharpening is performed using a CNN, blur that has not been learned is not sharpened correctly. Because the blur generated by an optical system changes depending on factors such as zoom, aperture, and focus distance, the following two methods can be considered to be able to sharpen all of this blur.

[0006] The first method is to train a CNN using training data that includes all blurs that can occur in an optical system. However, in this case, the CNN is trained to sharpen all blurs contained in the training data on an average basis, resulting in reduced sharpening accuracy for each blur of different shapes. The second method is to divide the blurs that can occur in an optical system into multiple similar groups and train a CNN separately using the training data for each group. However, in this case, for optical systems that generate various aberrations, such as high-magnification zoom lenses, the number of groups becomes enormous, increasing the training load and the amount of retained data (the amount of data indicating the weights of the trained CNN). This makes it difficult to achieve both blur sharpening accuracy and the training load and amount of retained data.

[0007] Therefore, the present invention aims to provide an image processing method, an image processing device, an image processing system, a method for manufacturing trained weights, and a program that reduce the learning load and amount of data stored on a machine learning model and sharpen or reshape blur in a captured image with high precision. [Means for solving the problem]

[0008] An image processing method according to one aspect of the present invention includes a first step of acquiring input data including a captured image and first information relating to the state of an optical system used to capture the captured image at the time of capture, and a second step of inputting the input data into a machine learning model and generating an estimated image by sharpening the captured image or correcting blur contained in the captured image.

[0009] Another aspect of the present invention is an image processing device that includes an acquisition means for acquiring input data including a captured image and first information regarding the state of an optical system used to capture the captured image at the time of capture, and a generation means for inputting the input data into a machine learning model and generating an estimated image by sharpening the captured image or shaping blur contained in the captured image.

[0010] Another aspect of the present invention is an image processing system comprising an image processing device characterized by having an acquisition means for acquiring input data including a captured image and first information regarding the state of the optical system used to capture the captured image at the time of capture, and a generation means for inputting the input data into a machine learning model and generating an estimated image by sharpening the captured image or shaping blur contained in the captured image, and a control device capable of communicating with the image processing device, wherein the control device has a transmission means for transmitting a request to the image processing device regarding the execution of processing on the captured image, and the second device has a receiving means for receiving the request and generates the estimated image in response to the request.

[0011] Another aspect of the present invention is an image processing method comprising a first step of acquiring input data including a training image and first information relating to the state of the optical system used to capture the training image at the time of capture; a second step of inputting the input data into a machine learning model and generating an output image by sharpening the training image or correcting blur contained in the training image; and a third step of updating weights of the machine learning model based on the output image and a ground truth image.

[0012] Another aspect of the present invention is a method for generating learned weights, which includes a first step of acquiring input data including a training image and first information regarding the state of the optical system used to capture the training image at the time of capture; a second step of inputting the input data into a machine learning model and generating an output image by sharpening the training image or correcting blur contained in the training image; and a third step of updating the weights of the machine learning model based on the output image and a ground truth image.

[0013] Another aspect of the present invention is an image processing device that includes an acquisition means for acquiring input data including a training image and first information regarding the state of the optical system used to capture the training image at the time of capture, a generation means for inputting the input data into a machine learning model and generating an output image by sharpening the training image or shaping blur contained in the training image, and an update means for updating the weights of the machine learning model based on the output image and a correct image.

[0014] Another aspect of the present invention is a program for causing a computer to execute the image processing method.

[0015] Other objects and features of the present invention are illustrated in the following examples. [Effects of the Invention]

[0016] According to the present invention, it is possible to provide an image processing method, an image processing device, an image processing system, a method for manufacturing trained weights, and a program that reduce the learning load of a machine learning model and the amount of data stored, and sharpen or reshape blur in a captured image with high precision. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a configuration diagram of a machine learning model in a first embodiment. [Figure 2] 1 is a block diagram of an image processing system according to a first embodiment. [Figure 3] 1 is an external view of an image processing system according to a first embodiment. [Figure 4] 10 is a flowchart relating to weight learning in the first and second embodiments. [Figure 5] 4 is a flowchart relating to generation of an estimated image in the first embodiment. [Figure 6] FIG. 10 is a block diagram of an image processing system according to a second embodiment. [Figure 7] FIG. 10 is an external view of an image processing system according to a second embodiment. [Figure 8] FIG. 10 is a configuration diagram of a machine learning model in a second embodiment. [Figure 9] 10 is a diagram showing the relationship between the image sensor and the image circle of the optical system in Example 2. FIG. [Figure 10] FIG. 10 is a diagram illustrating an example of a position map in the second embodiment. [Figure 11] 10 is a flowchart relating to generation of an estimated image in the second embodiment. [Figure 12] FIG. 10 is a diagram showing the effect of sharpening in Example 2. [Figure 13] FIG. 10 is a diagram illustrating an example of a manufacturing variation map in the second embodiment. [Figure 14] FIG. 10 is a block diagram of an image processing system according to a third embodiment. [Figure 15] FIG. 11 is an external view of an image processing system according to a third embodiment. [Figure 16] 11 is a flowchart relating to weight learning in the third embodiment. [Figure 17] FIG. 10 is a configuration diagram of a machine learning model in a third embodiment. [Figure 18] 11 is a flowchart relating to generation of an estimated image in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same components are designated by the same reference numerals, and redundant explanations will be omitted.

[0019] Before describing each embodiment in detail, the gist of the present invention will be explained. The present invention uses a machine learning model to sharpen or reshape blur caused by (occurring in) an optical system in an image captured using the optical system. The term "optical system" as used here refers to anything that exerts an optical effect on imaging. That is, the term "optical system" includes not only an imaging optical system but also, for example, an optical low-pass filter and a microlens array of an image sensor. Therefore, blur caused by the optical system includes blur due to aberration, diffraction, and defocus, the effect of an optical low-pass filter, and pixel aperture degradation of an image sensor.

[0020] Machine learning models include, for example, neural networks, genetic programming, Bayesian networks, etc. Neural networks include convolutional neural networks (CNNs), generative adversarial networks (GANs), recurrent neural networks (RNNs), etc.

[0021] Blur sharpening refers to the process of restoring frequency components of a subject that have been reduced or lost due to blur. Blur shaping refers to the transformation of the blur shape without restoring frequency components. Examples include the transformation of bilinear blur to Gaussian or disk (flat circular distribution), and the transformation of defocus blur missing due to vignetting to circular defocus blur.

[0022] The input data input to the machine learning model includes a captured image and information about the state of the optical system when the captured image was captured. The state of the optical system refers to the state of the device that can affect the optical function related to imaging. The state of the optical system includes, for example, the zoom, aperture, and focus distance of the optical system. In addition, the information about the state of the optical system may include information about the presence or absence (or type) of an optical low-pass filter and the presence (or type) of an accessory (e.g., a converter lens) attached to the optical system.

[0023] By inputting information about the state of the optical system into the learning of the machine learning model and the estimation after learning, the machine learning model can identify the state of the optical system that caused the blur affecting the captured image. As a result, even if the learning includes blur of various shapes, the machine learning model learns weights that perform different sharpening (or shaping) for each state of the optical system, rather than weights that sharpen (or shape) the blur on an average.

[0024] This enables highly accurate sharpening (or shaping) of each blur. Therefore, it is possible to collectively learn training data containing blur of various shapes while suppressing a decrease in the accuracy of sharpening (or shaping). As a result, it is possible to suppress the learning load and the amount of stored data, and to sharpen or shape blur caused by the optical system of a captured image with high accuracy. The effects of the present invention will be quantitatively demonstrated in Example 2. In the following description, the stage of learning the weights of the machine learning model will be referred to as the learning phase, and the stage of sharpening or shaping blur using the machine learning model using the learned weights will be referred to as the estimation phase. [Example]

[0025] First, an image processing system according to a first embodiment of the present invention will be described. This embodiment performs blur sharpening, but can also be applied to shaping blur. In this embodiment, the object to be sharpened is blur caused by aberration and diffraction, but it can also be applied to blur caused by defocus.

[0026] FIG. 2 is a block diagram of an image processing system 100 according to this embodiment. FIG. 3 is an external view of the image processing system 100. The image processing system 100 includes a learning device (image processing device) 101, an imaging device 102, and a network 103. The learning device 101 and the imaging device 102 are connected via the network 103, which may be wired or wireless. The learning device 101 includes a storage unit 111, an acquisition unit (acquisition means) 112, a calculation unit (generation means) 113, and an update unit (update means) 114, and learns weights for sharpening blur using a machine learning model. The imaging device 102 captures an object space to acquire a captured image, and sharpens blur in the captured image using weight information read after capturing or in advance. Details of the weight learning performed by the learning device 101 and the blur sharpening performed by the imaging device 102 will be described later.

[0027] The imaging device 102 has an optical system (imaging optical system) 121 and an imaging element 122. The optical system 121 collects light incident on the imaging device 102 from the subject space. The imaging element 122 receives (photoelectrically converts) an optical image (subject image) formed via the optical system 121 to generate a captured image. The imaging element 122 is, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal-Oxide Semiconductor) sensor.

[0028] The image processing unit (image processing device) 123 has an acquisition unit (acquisition means) 123a and a sharpening unit (generation means) 123b, and generates an estimated image (sharpened image) in which blur is sharpened from the captured image. The estimated image is generated using learned weight information learned by the learning device 101. The weight information is read in advance from the learning device 101 via a wired or wireless network 103 and stored in the memory unit 124. The stored weight information may be the weight's numerical value itself or in an encoded format. The recording medium 125 stores the estimated image. Alternatively, the captured image may be stored in the recording medium 125, and the image processing unit 123 may read the captured image to generate the estimated image. The display unit 126 displays the estimated image stored in the recording medium 125 in accordance with a user's instruction. The system controller 127 controls the series of operations described above.

[0029] Next, weight learning (learning phase, method for manufacturing a trained model) performed by the learning device 101 in this embodiment will be described with reference to Fig. 4. Fig. 4 is a flowchart related to weight learning. Each step in Fig. 4 is mainly executed by the acquisition unit 112, calculation unit 113, or update unit 114 of the learning device 101. Note that in this embodiment, CNN is used as the machine learning model, but other models can also be similarly applied.

[0030] First, in step S101, the acquisition unit 112 acquires one or more pairs of a reference image and training input data from the storage unit 111. The training input data is input data in the learning phase of CNN. The training input data includes a training image and information about the state of the optical system corresponding to the training image. The training image and the reference image are a pair of images containing the same subject, but differing in the presence or absence of blur. The reference image is an image without blur, and the training image is an image with blur. The blur is a combination of aberration and diffraction occurring in the optical system 121 and pixel aperture degradation of the image sensor 122. A single training image is affected by the combination of aberration and diffraction occurring in the optical system 121 at a specific zoom, aperture, and focus distance, and pixel aperture degradation. The information about the state of the optical system corresponding to the training image is information indicating at least one of the specific zoom, aperture, and focus distance. In other words, the information about the state of the optical system is information identifying the blur affecting the training image. In this embodiment, the information about the state of the optical system includes all of the zoom, aperture, and focus distance. The training images are not limited to captured images, but may be images generated by computer graphics or the like.

[0031] Examples of methods for generating the correct image and training input data stored in the storage unit 111 are shown below. The first generation method is a method of performing an imaging simulation using an original image as the subject. The original image is a real-life image or a CG (Computer Graphics) image, etc. In order to perform correct sharpening on various subjects, it is desirable that the original image be an image that has edges with various strengths and directions, textures, gradations, flat areas, etc. The original image may be one or more. The correct image is an image obtained by performing an imaging simulation on the original image without applying blur. The training image is an image obtained by performing an imaging simulation on the original image by applying blur to be sharpened.

[0032] In this embodiment, blurring due to aberration and diffraction occurring in the state (Z, F, D) of the optical system 121 and pixel aperture degradation is applied. Here, Z represents the zoom state, F represents the aperture, and D represents the focus distance state. When the image sensor 122 acquires multiple color components, blurring of each color component is applied to the original image. Bluring can be performed by convolving the original image with a point spread function (PSF) or by multiplying the frequency characteristics of the original image by an optical transfer function (OTF). Information regarding the state of the optical system corresponding to the training image to which blurring specified by (Z, F, D) has been applied is information that identifies (Z, F, D). The reference image and training images may be unprocessed RAW images or processed images. Multiple different (Z, F, D) blurrings are applied to one or more original images to generate multiple pairs of reference image and training image.

[0033] In this embodiment, correction for all blurs occurring in the optical system 121 is learned all at once. Therefore, (Z, F, D) is changed within the range that can be taken by the optical system 121, and multiple pairs of a correct image and training image are generated. Furthermore, even for the same (Z, F, D), multiple blurs exist depending on the image height and azimuth, so pairs of a correct image and training image are generated for each different image height and azimuth.

[0034] Preferably, the original image has a signal value higher than the brightness saturation value of the image sensor 122. This is because even in real life, there are some subjects that do not fall within the brightness saturation value when captured by the image capture device 102 under specific exposure conditions. The ground truth image is generated by clipping the signal of the original image at the brightness saturation value of the image sensor 122. The training images are generated by applying blurring and then clipping at the brightness saturation value.

[0035] Furthermore, the original image may be downsized when generating the reference image and training images. When using a real-life image as the reference image, blurring due to aberration and diffraction already occurs, so downsizing reduces the effect of blurring and allows for the generation of a high-resolution reference image. In this case, the training images are also downsized to match the scale with the reference image. The order of downsizing and blurring does not matter. If blurring is performed first, the blur sampling rate must be finer to take downsizing into account. For PSF, this can be done by finer spatial sampling points, and for OTF, this can be done by increasing the maximum frequency. Note that if the original image contains sufficient high-frequency components, downsizing is not necessary, as high-precision sharpening is possible.

[0036] Furthermore, distortion is not included in the blur used to generate the training images. This is because large distortion changes the position of the subject, and the subject captured in the target image and the training images may differ. For this reason, the CNN used for training in this embodiment does not correct distortion. In the estimation phase, distortion is corrected after blur sharpening using bilinear interpolation or bicubic interpolation, for example. Similarly, lateral chromatic aberration is not included in the blur used to generate the training images. In the estimation phase, lateral chromatic aberration is corrected before blur sharpening using a shift of each color component, for example.

[0037] The second method for generating the reference image and training input data is to use images captured by the optical system 121 and the image sensor 122. The optical system 121 captures an image in a state (Z, F, D) to obtain a training image. Information about the state of the optical system corresponding to the training image is information that specifies (Z, F, D). The reference image can be obtained, for example, by capturing an image of the same subject as the training image using an optical system with higher performance than the optical system 121. Note that a partial region with a predetermined number of pixels may be extracted from the training image and the reference image generated by the two methods described above and used for learning.

[0038] Next, in step S102 of Fig. 4, the calculation unit 113 inputs the training input data to the CNN and generates an output image. Generation of the output image in this embodiment will be described with reference to Fig. 1. Fig. 1 is a configuration diagram of the machine learning model in this embodiment.

[0039] The training input data includes a training image 201 and information (z, f, d) 202 about the state of the optical system. The training image 201 may be grayscale or may have multiple channel components. The same applies to the ground truth image. (z, f, d) is a normalized (Z, F, D). The normalization is performed based on the ranges that the optical system 121 can take for the zoom, aperture, and focus distance. For example, let Z be the focal length, F be the aperture value, and D be the reciprocal of the absolute value of the distance from the image capture device 102 to the focus subject. Let Z be the minimum and maximum values ​​of the focal length of the optical system 121. min and Z max , the minimum and maximum aperture values ​​are F min and F max , the minimum and maximum values ​​of the reciprocal of the absolute value of the focusable distance are D min and D max Here, if the focusable distance is infinity, D min = 1 / |∞| = 0. The normalized (z, f, d) is calculated using the following equation (1).

[0040]

number

[0041] In equation (1), x is a dummy variable that indicates one of (z, f, d), and X is a dummy variable that indicates one of (Z, F, D). min =X max In this case, x is a constant. Alternatively, since x has no degree of freedom, it is excluded from the information regarding the state of the optical system. Here, since the change in performance of the optical system 121 generally increases as the focus distance decreases, D is set to the reciprocal of the distance.

[0042] In this embodiment, the CNN 211 includes a first subnetwork 221 and a second subnetwork 223. The first subnetwork 221 includes one or more convolutional layers or full-connection layers. The second subnetwork 223 includes one or more convolutional layers. During the first learning cycle, the weights (values ​​of each filter element and bias) of the CNN 211 are generated using random numbers. The first subnetwork 221 receives information (z, f, d) 202 about the state of the optical system and generates a state map 203 converted into a feature map. The state map 203 is a map indicating the state of the optical system and has the same number of elements (pixels) as one channel component of the training image 201. The concatenation layer 222 concatenates the training image 201 and the state map 203 in a specified order in the channel direction. Note that other data may be concatenated between the training image 201 and the state map 203. The second sub-network 223 receives the concatenated training images 201 and state map 203 as input and generates an output image 204. If multiple sets of training input data are acquired in step S101 of Fig. 4, an output image 204 is generated for each set. Alternatively, the training images 201 may be converted into a feature map by a third sub-network, and the feature map and state map 203 may be connected by a connection layer 222.

[0043] Next, in step S103 of FIG. 4, the update unit 114 updates the weights of the CNN based on the error between the output image and the correct image. In this embodiment, the Euclidean norm of the difference in signal values ​​between the output image and the correct image is used as the loss function. However, the loss function is not limited to this. If multiple pairs of training input data and correct images are acquired in step S101, the value of the loss function is calculated for each pair. The weights are updated using backpropagation or the like based on the calculated loss function value.

[0044] Next, in step S104, the update unit 114 determines whether weight learning is complete. Completion can be determined by, for example, whether the number of iterations of learning (weight update) has reached a specified number, or whether the amount of change in weight during update is smaller than a specified value. If it is determined in step S104 that weight learning is not complete, the process returns to step S101, and the acquisition unit 112 acquires one or more pairs of new training input data and a correct image. On the other hand, if it is determined that weight learning is complete, the update unit 114 ends learning and saves weight information in the storage unit 111.

[0045] Next, blur sharpening (estimation phase) of a captured image executed by the image processing unit 123 in this embodiment will be described with reference to Fig. 5. Fig. 5 is a flowchart related to blur sharpening (generation of an estimated image) in this embodiment. Each step in Fig. 5 is mainly executed by the acquisition unit 123a or sharpening unit 123b of the image processing unit 123.

[0046] First, in step S201, the acquisition unit 123a acquires input data and weight information. The input data includes a captured image and information about the state of the optical system 121 when the captured image was captured. The captured image to be acquired may be a portion of the entire captured image. The information about the optical system is (z, f, d) that indicates the state of the zoom, aperture, and focus distance of the optical system 121. The weight information is acquired by reading it from the storage unit 124.

[0047] Next, in step S202, the sharpening unit 123b inputs the input data to the CNN to generate an estimated image. The estimated image is an image in which blur caused by aberration and diffraction of the optical system 121 and pixel aperture deterioration of the image sensor 122 has been sharpened in the captured image. As in the learning process, the estimated image is generated using the CNN shown in FIG. 1. The CNN uses the acquired trained weights. In this embodiment, blur sharpening weights are learned collectively for all possible values ​​(z, f, d) of the optical system 121. Therefore, blur sharpening is performed by the CNN using the same weights for all captured images of (z, f, d).

[0048] In this embodiment, instead of the sharpening unit 123b that sharpens the captured image, a shaping unit that rectifies blur contained in the captured image may be provided. This also applies to Example 2 described later. In this embodiment, the image processing device (image processing unit 123) has an acquisition means (acquisition unit 123a) and a generation means (sharpening unit 123b or shaping unit). The acquisition means acquires input data including a captured image and information about the state of the optical system used to capture the captured image. The generation means inputs the input data to a machine learning model and generates an estimated image in which the captured image is sharpened or an estimated image in which blur contained in the captured image is rectified. In this embodiment, the image processing device (learning device 101) also has an acquisition means (acquisition unit 112), a generation means (computation unit 113), and an update means (update unit 114). The acquisition means acquires input data including training images and information about the state of the optical system corresponding to the training images. The generating means inputs input data to the machine learning model and generates an output image by sharpening the training image or correcting blur contained in the training image. The updating means updates the weights of the machine learning model based on the output image and the correct image.

[0049] According to this embodiment, it is possible to realize an image processing device and an image processing system that can reduce the learning load of the machine learning model and the amount of data stored, and sharpen blur caused by the optical system of a captured image with high accuracy. [Example]

[0050] Next, an image processing system according to a second embodiment of the present invention will be described. This embodiment executes blur sharpening processing, but the present invention can also be applied to blur shaping processing.

[0051] FIG. 6 is a block diagram of an image processing system 300 in this embodiment. FIG. 7 is an external view of the image processing system 300. The image processing system 300 includes a learning device (image processing device) 301, a lens device 302, an imaging device 303, an image estimation device 304, a display device 305, a recording medium 306, an output device 307, and a network 308. The learning device 301 includes a storage unit 301a, an acquisition unit (acquisition means) 301b, a calculation unit (generation means) 301c, and an update unit (update means) 301d, and learns weights of a machine learning model used for blur sharpening. Note that details regarding weight learning and blur sharpening processing using the weights will be described later.

[0052] The lens device 302 and the image capture device 303 are detachable and can be connected to different types of lens devices 302 or image capture devices 303. The focal length, aperture, and focus distance that can be taken by the lens device 302 vary depending on the type. Furthermore, the lens configuration varies depending on the type, and therefore the shape of blur due to aberration and diffraction also varies. The image capture device 303 has an image sensor 303a, and the presence or absence and type of optical low-pass filter (separation method, cutoff frequency, etc.), pixel pitch (including pixel aperture), color filter array, etc. vary depending on the type.

[0053] The image estimation device 304 includes a storage unit 304a, an acquisition unit (acquisition means) 304b, and a sharpening unit (generation means) 304c. The image estimation device 304 generates an estimated image by sharpening blur caused by the optical system of an image (or at least a part thereof) captured by the imaging device 303. A variety of combinations of lens devices 302 and imaging devices 303 can be connected to the image estimation device 304. A machine learning model using weights trained by the learning device 301 is used for blur sharpening. The learning device 301 and the image estimation device 304 are connected via a network 308, and the image estimation device 304 reads information about the trained weights from the learning device 301 during or before blur sharpening. The estimated image is output to at least one of the display device 305, the recording medium 306, and the output device 307. The display device 305 is, for example, a liquid crystal display or a projector. A user can perform editing work while checking images being processed via a display device 305. The recording medium 306 is, for example, a semiconductor memory, a hard disk, or a server on a network. The output device 307 is, for example, a printer.

[0054] Next, weight learning (learning phase) performed by the learning device 301 will be described with reference to Fig. 4. In this embodiment, CNN is used as the machine learning model, but other models can also be used in the same way. Note that the same description as in the first embodiment will be omitted.

[0055] First, in step S101, the acquisition unit 301b acquires one or more pairs of correct images and training input data from the storage unit 301a. The storage unit 301a stores training images for multiple combinations of lens devices 302 and image capture devices 303. In this embodiment, learning of blur sharpening weights is performed collectively for each type of lens device 302. For this reason, the type of lens device 302 for which the weights are to be learned is first determined, and training images are acquired from the collection of training images corresponding to that type. The collection of training images corresponding to a certain type of lens device 302 is a collection of images to which blurring has been applied that differs in zoom, aperture, focus distance, image height and azimuth, optical low-pass filter, pixel pitch, color filter array, etc.

[0056] In this embodiment, learning is performed using the CNN configuration shown in FIG. 8. FIG. 8 is a configuration diagram of a machine learning model in this embodiment. Training input data 404 includes training images 401, a state map 402, and a position map 403. The state map 402 and the position map 403 are generated in this step. The state map 402 and the position map 403 are maps that indicate (Z, F, D) and (X, Y), respectively, which correspond to the blur acting on the acquired training images. (X, Y) are the coordinates (horizontal and vertical directions) of the image plane shown in FIG. 9, and correspond to the image height and azimuth in polar coordinate display.

[0057] In this embodiment, the coordinates (X, Y) have the optical axis of the lens device 302 as their origin. FIG. 9 shows the relationship between the image circle 501 of the lens device (optical system) 302, the first effective pixel area 502 and the second effective pixel area 503 of the image sensor 303a, and the coordinates (X, Y). Since the image sensor 303a has different sizes depending on the type of image sensor 303, there are types of image sensor 303 that have the first effective pixel area 502 and types that have the second effective pixel area 503. Of the image sensor 303 that can be connected to the lens device 302, the image sensor 303 that has the largest size image sensor 303a has the first effective pixel area 502.

[0058] The position map 403 is generated based on (x, y) obtained by normalizing the coordinates (X, Y). Normalization is performed by dividing (X, Y) by a length (radius of the image circle) 511 based on the image circle 501 of the lens device 302. Alternatively, normalization may be performed by dividing X by the horizontal length 512 of the first effective pixel area from the origin and Y by the vertical length 513 of the first effective pixel area from the origin. If (X, Y) were normalized so that the edge of the captured image always equals 1, images captured with different sizes of image sensors 303a would have different positions (X, Y) even for the same (x, y) value, and the correspondence between (x, y) and blur would not be uniquely determined. This would result in a decrease in the accuracy of blur sharpening. The position map 403 is a two-channel map having (x, y) values ​​as channel components. Note that polar coordinates may be used for the position map 403, and the method of determining the origin is not limited to that shown in FIG. 9.

[0059] The state map 402 is a three-channel map having normalized (z, f, d) values ​​as channel components. The number of elements (number of pixels) per channel in each of the training image 401, state map 402, and position map 403 is equal. Note that the configurations of the position map 403 and state map 402 are not limited to this. For example, as shown in FIG. 10 illustrating an example of a position map, the first effective pixel region 502 may be divided into multiple partial regions and a numerical value may be assigned to each partial region, thereby representing the position map in one channel. Note that the number of partial region divisions and the method of assigning numerical values ​​are not limited to those shown in FIG. 10. Similarly, (Z, F, D) may also be divided into multiple partial regions in a three-dimensional space with each axis as an axis, and numerical values ​​may be assigned to represent the state map in one channel. The training image 401, state map 402, and position map 403 are concatenated in a specified order in the channel direction by the concatenation layer 411 in FIG. 8 to generate training input data 404.

[0060] 4, the calculation unit 301c inputs the training input data 404 to the CNN 412 to generate an output image 405. Then, in step S103, the update unit 301d updates the weights of the CNN based on the error between the output image and the correct image. Then, in step S104, the update unit 301d determines whether learning is complete. Information about the learned weights is stored in the storage unit 301a.

[0061] Next, blur sharpening (estimation phase) of a captured image executed by the image estimation device 304 will be described with reference to Fig. 11. Fig. 11 is a flowchart related to blur sharpening (generation of an estimated image). Each step in Fig. 11 is mainly executed by the acquisition unit 304b or the sharpening unit 304c of the image estimation device 304.

[0062] First, in step S301, the acquisition unit 304b acquires a captured image (or at least a portion thereof). Subsequently, in step S302, the acquisition unit 304b acquires weight information corresponding to the captured image. In the second embodiment, weight information for each type of lens device 302 is read out in advance from the storage unit 301a and stored in the storage unit 304a. Therefore, the acquisition unit 304b acquires weight information corresponding to the type of lens device 302 used to capture the captured image from the storage unit 304a. The type of lens device 302 used for capturing is identified, for example, from metadata in the file of the captured image.

[0063] Next, in step S303, the acquisition unit 304b generates a state map and a position map corresponding to the captured image to generate input data. The state map is generated based on the number of pixels in the captured image and information about the state (Z, F, D) of the lens device 302 when the captured image was captured. The number of elements (number of pixels) per channel in the captured image and the state map are equal. (Z, F, D) are identified, for example, from metadata of the captured image. The position map is generated based on the number of pixels in the captured image and information about the position of each pixel in the captured image. The number of elements (number of pixels) per channel in the captured image and the position map are equal. The size of the effective pixel area of ​​the image sensor 303a used to capture the captured image is identified from metadata of the captured image, and a normalized position map is generated using, for example, the length of the image circle of the lens device 302 identified in the same way. As in FIG. 8, the input data is generated by concatenating the captured image, state map, and position map in a specified order in the channel direction. Note that in this embodiment, the order of steps S302 and S303 does not matter. Furthermore, a state map and a position map may be generated when a captured image is captured and saved together with the captured image.

[0064] Next, in step S304, the sharpening unit 304c inputs the input data to the CNN, as in FIG. 8, to generate an estimated image. FIG. 12 illustrates the sharpening effect at 90% image height for a specific zoom lens at (Z, F, D). In FIG. 12, the horizontal axis represents spatial frequency, and the vertical axis represents the measured SFR (Spatial Frequency Response). The SFR corresponds to the MTF (Modulation Transfer Function) at a certain cross section. The Nyquist frequency of the image sensor used for capturing the image is 76 lp / mm. The solid line 601 represents the captured image, while the dashed line 602, dashed line 603, and two-dot dashed line 604 represent the results of blur sharpening of the captured image using the CNN. The dashed line 602, dashed line 603, and two-dot dashed line 604 represent the results of blur sharpening of the captured image using training data that combines all of the aberrations and diffractive blurs that occur in the zoom lens.

[0065] The input data of the dashed line 602 in the estimation phase (or learning phase) is only the captured image (or training image). The input data of the dashed-dotted line 603 is the captured image (or training image) and a position map. The input data of the two-dot-dash line 604 is the captured image (or training image), a position map, and a state map, which corresponds to the configuration of this embodiment. The CNNs used for the dashed line 602, the dashed-dotted line 603, and the two-dot-dash line 604 differ only in the number of channels of the first layer filter (due to the different number of channels of the input data), but are common in other aspects such as the filter size, number of filters, and number of layers. Therefore, the learning load and stored data volume (data capacity of CNN weight information) of the dashed line 602, the dashed-dotted line 603, and the two-dot-dash line 604 are approximately the same. On the other hand, the two-dot-dotted line 604, which employs the configuration of this embodiment, has a high sharpening effect, as shown in FIG. 12.

[0066] According to this embodiment, it is possible to realize an image processing device and an image processing system that can reduce the learning load of the machine learning model and the amount of data stored, and sharpen blur caused by the optical system of a captured image with high accuracy.

[0067] Next, preferable conditions for enhancing the effect of this embodiment will be described.

[0068] The input data preferably further includes information indicating the presence or absence and type of optical low-pass filter of the image capture device 303 used to capture the captured image. This improves the blur sharpening effect. The type refers to the separation method (vertical two-point separation, horizontal two-point separation, four-point separation, etc.) and the cutoff frequency. A map containing numerical values ​​that can identify the presence or absence and type of filter as elements may be generated based on the number of pixels in the captured image and included in the input data.

[0069] The input data preferably further includes information regarding manufacturing variations of the lens device 302 used to capture the captured image. This enables highly accurate blur sharpening that takes manufacturing variations into account. In the learning phase, training images are generated by applying blur, including manufacturing variations, to the original image, and information indicating manufacturing variations is included in the training input data to train the machine learning model. The information indicating manufacturing variations may, for example, be a numerical value indicating the degree of actual performance, including manufacturing variations, relative to design performance. For example, when the actual performance and design performance are equal, the numerical value is set to 0, and as the actual performance becomes inferior to the design performance, the numerical value moves in the negative direction, and as the actual performance becomes superior to the design performance, the numerical value moves in the positive direction. In the estimation phase, as shown in FIG. 13, the input data includes a map containing numerical values ​​indicating the degree of actual performance relative to design performance for multiple partial regions of the captured image (or for each pixel). FIG. 13 is a diagram illustrating an example of a manufacturing variation map. This map is generated based on the number of pixels in the captured image. The map can be obtained by measuring the actual performance of the lens device 302, including manufacturing errors, during manufacturing etc. Furthermore, manufacturing variations may be divided into several categories, such as deterioration in performance of the entire image (worsening spherical aberration) and performance variations due to azimuth (partial blur), and the manufacturing variations may be indicated by a numerical value indicating the category.

[0070] The input data should also preferably include information about the distribution of distance in object space at the time of capture. This enables highly accurate blur sharpening, taking into account performance changes due to defocusing. Due to axial chromatic aberration and field curvature, optical performance at a defocused object plane can be improved compared to that at a focused plane. Ignoring this and performing blur sharpening using a machine learning model trained solely on blur at the focused plane results in excessive resolution and unnatural images. To address this issue, training is first performed in the learning phase using training images in which defocused blur is applied to the original image. The training input data should also include a numerical value indicating the amount of defocus (equivalent to distance in object space). For example, the focused plane can be set to 0, with a direction away from the imaging device being negative and a direction toward the imaging device being positive. In the estimation phase, a defocus map (information about the distribution of distance in object space) of the captured image is obtained using parallax imaging or Depth from Defocus (DFD), and included in the input data. The defocus map is generated based on the number of pixels in the captured image.

[0071] The input data preferably also includes information regarding the pixel pitch or color filter array of the image sensor 303a used to capture the captured image. This enables highly accurate blur sharpening regardless of the type of image sensor 303a. The degree of pixel aperture degradation and the magnitude of blur for each pixel vary depending on the pixel pitch. The shape of the blur also varies depending on the color components constituting the color filter array. Color components include, for example, RGB (Red, Green, Blue) or complementary colors such as CMY (Cyan, Magenta, Yellow). Furthermore, when training images or captured images are undeveloped Bayer images, the shape of the blur varies even for pixels at the same position depending on the arrangement order of the color filter array. During the learning phase, the training input data includes information identifying the pixel pitch or color filter array corresponding to the training image. For example, the training input data may include a map whose elements are the numerical values ​​of normalized pixel pitches. For normalization, it is preferable to use the largest pixel pitch among multiple types of image sensors 303 as a divisor. A map whose elements are numerical values ​​indicating the color components of the color filter array may also be included. In the estimation phase, the accuracy of sharpening can be improved by including a similar map in the input data. The map is generated based on the number of pixels in the captured image.

[0072] The input data desirably further includes information indicating the presence and type of accessories of the lens device 302. Accessories include wide-angle converters, teleconverters, close-up lenses, wavelength cutoff filters, etc. Because the shape of the blur changes depending on the type of accessory, inputting information regarding the presence and type of accessory allows for sharpening that takes their influence into account. In the learning phase, the influence of the accessory is included in the blur applied to the training images, and information identifying the accessory is included in the training input data. For example, a map is used whose elements are numerical values ​​indicating the presence and type of accessory. Similar information (map) can also be included in the input data in the estimation phase. This map is generated based on the number of pixels in the captured image. [Example]

[0073] Next, an image processing system according to a third embodiment of the present invention will be described. This embodiment executes blur shaping processing, but the present invention is also applicable to blur sharpening processing.

[0074] FIG. 14 is a block diagram of an image processing system 700. FIG. 15 is an external view of the image processing system 700. The image processing system 700 includes a learning device 701, a lens device (optical system) 702, an imaging device 703, a control device (first device) 704, an image estimation device (second device) 705, and networks 706 and 707. The learning device 701 and the image estimation device 705 are, for example, servers. The control device 704 is a device operated by a user, such as a personal computer or a mobile terminal. The learning device 701 includes a storage unit 701a, an acquisition unit (acquisition means) 701b, a calculation unit (generation means) 701c, and an update unit (update means) 701d, and learns weights of a machine learning model that corrects blur in an image captured using the lens device 702 and the imaging device 703. Details regarding learning will be described later. The blur to be shaped in this embodiment is blur caused by defocus, but the same can be applied to aberration, diffraction, and the like.

[0075] The imaging device 703 has an imaging element 703a. The imaging element 703a photoelectrically converts an optical image formed by the lens device 702 to acquire a captured image. The lens device 702 and the imaging device 703 are detachable and can be combined with a plurality of types of each other. The control device 704 has a communication unit 704a, a storage unit 704b, and a display unit 704c, and controls the processing to be executed on the captured image acquired from the imaging device 703 connected by wire or wirelessly in accordance with a user's operation. Alternatively, the control device 704 may store the captured image captured by the imaging device 703 in advance in the storage unit 704b and read out the captured image.

[0076] The image estimation device 705 has a communication unit 705a, a storage unit 705b, an acquisition unit (acquisition means) 705c, and a shaping unit (generation means) 705d. The image estimation device 705 executes blur shaping processing on a captured image in response to a request from a control device 704 connected via a network 707. The image estimation device 705 acquires information on learned weights from a learning device 701 connected via a network 706 during blur shaping or in advance, and uses the information for blur shaping of the captured image. The estimated image after blur shaping is transmitted again to the control device 704, stored in the storage unit 704b, and displayed on a display unit 704c.

[0077] Next, weight learning (learning phase, method for manufacturing a trained model) performed by the learning device 701 in this embodiment will be described with reference to FIG. 16. FIG. 16 is a flowchart related to weight learning. Each step in FIG. 16 is mainly executed by the acquisition unit 701b, the calculation unit 701c, or the update unit 701d of the learning device 701. Note that in this embodiment, a GAN is used as the machine learning model, but other models can be similarly applied. A GAN has a generator that generates an output image in which defocus blur has been shaped, and a classifier that distinguishes between a correct image and an output image generated by the generator. In learning, first, a first learning using only the generator is performed as in the first embodiment, and when the weights of the generator have converged to a certain extent, a second learning using the generator and the classifier is performed. Hereinafter, a description of the same parts as in the first embodiment will be omitted.

[0078] First, in step S401, the acquisition unit 701b acquires one or more pairs of a reference image and training input data from the storage unit 701a. In this embodiment, the reference image and the training image are a pair of images with different defocus blur shapes. The training image is an image to which the defocus blur of the target to be shaped has been applied. Examples of targets for shaping include double-line blur, blur loss due to vignetting, ring-shaped blur due to pupil obstruction in a catadioptric lens, and annular blur patterns caused by uneven cutting of the mold for an aspherical lens. The reference image is an image to which the defocus blur after shaping has been applied. The shape of the defocus blur after shaping may be determined according to the user's preference, such as Gaussian or disk (circular distribution with flat intensity). In Example 3, multiple training images and reference images generated by applying blur to an original image are stored in the storage unit 701a. When generating multiple training images and a ground truth image, blurring corresponding to various defocus amounts is applied so that shaping accuracy can be ensured for various defocus amounts. Also, since it is desirable that the image does not change before and after blur shaping on the focus plane, training images and ground truth images with a defocus amount of zero are also generated.

[0079] In this embodiment, weights for converting blur shaping are learned collectively for multiple types of lens devices 702. For this reason, the information about the state of the optical system includes information for identifying the type of lens device 702. Training images of blur corresponding to the type of lens device 702 to be learned collectively are obtained from the storage unit 701a. The information about the state of the optical system further includes information for identifying the zoom, aperture, and focus distance of the lens device 702 corresponding to the blur acting on the training images.

[0080] 17 is a diagram showing the configuration of the machine learning model (GAN). A lens state map 802, whose channel components are numerical values ​​(L, z, f, d) that specify the type of lens device 702, zoom, aperture, and focus distance, is generated based on the number of pixels in the training image 801. A concatenation layer 811 concatenates the training image 801 and the lens state map 802 in a predetermined order in the channel direction to generate training input data 803.

[0081] 16, the calculation unit 701c inputs the training input data 803 to the generator 812 to generate an output image 804. The generator 812 is, for example, a CNN. Then, in step S403, the update unit 701d updates the weights of the generator 812 based on the error between the output image 804 and the correct image 805. The loss function uses the Euclidean norm of the difference at each pixel.

[0082] Next, in step S404, the update unit 701d determines whether the first learning has been completed. If the first learning has not been completed, the process returns to step S401. On the other hand, if the first learning has been completed, the process proceeds to step S405, where the update unit 701d executes the second learning.

[0083] In step S405, the acquisition unit 701b acquires one or more pairs of a ground truth image 805 and training input data 803 from the storage unit 701a, as in step S401. Subsequently, in step S406, the calculation unit 701c inputs the training input data 803 to the generator 812, as in step S402, to generate an output image 804. Subsequently, in step S407, the update unit 701d updates the weights of the classifier 813 from the output image 804 and the ground truth image 805. The classifier 813 classifies whether the input image is a fake image generated by the generator 812 or a real image, which is the ground truth image 805. The output image 804 or the ground truth image 805 is input to the classifier 813, and an identification label (fake or real) is generated. Based on the error between the classified label and the correct label (the output image 804 is fake, and the correct image 805 is real), the update unit 701d updates the weight of the classifier 813. Although sigmoid cross entropy is used as the loss function, other loss functions may also be used.

[0084] Next, in step S408, the update unit 701d updates the weights of the generator 812 based on the output image 804 and the reference image 805. The loss function is a weighted sum of the Euclidean norm of step S403 and the following two terms. The first term is called Content Loss, which is a term obtained by converting the output image 804 and the reference image 805 into feature maps and taking the Euclidean norm of the difference for each element. By adding the difference in the feature map to the loss function, the more abstract properties of the output image 804 can be made closer to the reference image 805. The second term is called Adversarial Loss, which is the sigmoid cross entropy of the classification label obtained by inputting the output image 804 to the classifier 813. By having the classifier 813 learn to classify images as real, an output image 804 that subjectively looks more like the reference image 805 can be obtained.

[0085] Next, in step S409, the update unit 701d determines whether the second learning has been completed. As in step S404, if the second learning has not been completed, the process returns to step S405. On the other hand, if the second learning has been completed, the update unit 701d stores information about the weights of the trained generator 812 in the storage unit 701a.

[0086] Next, blur shaping (estimation phase) executed by the control device 704 and the image estimation device 705 will be described with reference to Fig. 18. Fig. 18 is a flowchart related to blur shaping (generation of an estimated image). Each step in Fig. 18 is mainly executed by each unit of the control device 704 or the image estimation device 705.

[0087] First, in step S501, the communication unit 704a of the control device 704 transmits a captured image and a request for execution of blur correction processing to the image estimation device 705. Next, in step S601, the communication unit 705a of the image estimation device 705 receives and acquires the captured image and processing request transmitted from the control device 704. Next, in step S602, the acquisition unit 705c of the image estimation device 705 acquires learned weight information corresponding to the captured image from the storage unit 705b. The weight information is read out in advance from the storage unit 701a and stored in the storage unit 705b.

[0088] Next, in step S603, the acquisition unit 705c acquires information about the state of the optical system corresponding to the captured image and generates input data. From the metadata of the captured image, information specifying the type of lens device (optical system) 702 at the time the captured image was captured, the zoom, aperture, and focus distance is acquired, and a lens state map is generated as in FIG. 17. The input data is generated by concatenating the captured image and the lens state map in a predetermined order in the channel direction. In this embodiment, the captured image or a feature map based on the captured image and the state map can be concatenated in the channel direction before or during input to the machine learning model.

[0089] Next, in step S604, the shaping unit 705d inputs the input data to the generator and generates a blur-shaped estimated image. The generator uses weight information. Next, in step S605, the communication unit 705a transmits the estimated image to the control device 704. Next, in step S502, the communication unit 704a of the control device 704 acquires the estimated image transmitted from the image estimation device 705.

[0090] In this embodiment, a sharpening unit that sharpens the captured image may be provided instead of the shaping unit 705d that rectifies blur contained in the captured image. In this embodiment, the image processing system 700 includes a first device (control device 704) and a second device (image estimation device 705) that can communicate with each other. The first device includes a transmitting unit (communication unit 704a) that transmits a request for executing processing on the captured image to the second device. The second device includes a receiving unit (communication unit 705a), an acquiring unit (acquiring unit 705c), and a generating unit (shaping unit 705d or sharpening unit). The receiving unit receives the request. The acquiring unit acquires input data including the captured image and information about the state of the optical system used to capture the captured image. The generating unit inputs the input data to a machine learning model based on the request, and generates an estimated image in which the captured image has been sharpened, or an estimated image in which blur contained in the captured image has been rectified.

[0091] According to this embodiment, it is possible to realize an image processing device and an image processing system that can reduce the learning load of the machine learning model and the amount of data stored, and can accurately correct blur caused by the optical system of a captured image.

[0092] In this embodiment, an example has been described in which the communication unit 704a of the control device 704 transmits the captured image together with a request for processing the captured image to the image estimation device 705 in step S501, but transmission of the captured image by the control device 704 is not essential. For example, the control device may be configured to transmit only a request for processing the captured image to the image estimation device 705, and the image estimation device 705, which has received the request, may acquire the captured image corresponding to the request from another image storage server or the like.

[0093] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0094] According to each embodiment, it is possible to provide an image processing method, an image processing device, an image processing system, a method for manufacturing learned weights, and a program that reduce the learning load of a machine learning model and the amount of data stored, and sharpen or reshape blur in a captured image with high precision.

[0095] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0096] 123 Image processing unit (image processing device) 123a Acquisition unit (acquisition means) 123b Sharpening section (generation means)

Claims

1. a first step of acquiring input data including a captured image and first information relating to a state of an optical system used to capture the captured image at the time of capturing the image; and a second step of inputting the input data into a machine learning model and generating an estimated image by sharpening the captured image or correcting blur contained in the captured image.

2. 2. The image processing method according to claim 1, wherein the state includes at least one of the zoom, aperture, and focus distance of the optical system.

3. The image processing method according to claim 1 or 2, characterized in that in the second step, the machine learning model with the same weight is used for a first captured image obtained by capturing an image in a first state of the optical system and a second captured image obtained by capturing an image in a second state of the optical system different from the first state.

4. the first information includes a numerical value indicating at least one of a zoom, an aperture, and a focus distance of the optical system, 4. The image processing method according to claim 1, wherein the numerical values ​​are normalized based on a range that can be taken by the optical system with respect to at least one of zoom, aperture, and focus distance.

5. the input data includes a state map indicating a state of the optical system; 5. The image processing method according to claim 1, wherein the state map is generated based on the number of pixels of the captured image and the first information.

6. 6. The image processing method according to claim 5, wherein the state map has, as elements, numerical values ​​indicating at least one of the zoom, aperture, and focus distance of the optical system.

7. 7. The image processing method according to claim 5, wherein the state map has numerical values ​​indicating at least two of the zoom, aperture, and focus distance of the optical system as elements of different channels.

8. 8. The image processing method according to claim 5, wherein elements included in the same channel in the state map have the same numerical value.

9. 9. The image processing method according to claim 5, wherein the captured image or a feature map based on the captured image is linked to the state map in a channel direction.

10. 10. The image processing method according to claim 1, wherein the input data includes second information relating to the position of each pixel of the captured image.

11. 11. The image processing method according to claim 10, wherein the second information includes a numerical value normalized by a length based on an image circle of the optical system.

12. 12. The image processing method according to claim 1, wherein the first information includes information about the type of the optical system.

13. 13. The image processing method according to claim 1, wherein the first information includes information on the presence or absence of an optical low-pass filter or information on the type of the optical low-pass filter.

14. 14. The image processing method according to claim 1, wherein the state includes information about the presence or absence of an accessory for the optical system, or the type of the accessory.

15. 15. The image processing method according to claim 1, wherein the first information includes information relating to manufacturing variations of the optical system.

16. 16. The image processing method according to claim 1, wherein the input data includes information about a distribution of distances in a subject space at the time of capturing the image.

17. 17. The image processing method according to claim 1, wherein the input data includes information about a pixel pitch or a color filter array of an image sensor used for capturing the image.

18. 18. The image processing method according to claim 1, wherein the second step is performed by generating the estimated image by processing that is executed by inputting the input data into the machine learning model.

19. The image processing method according to claim 1 , wherein the machine learning model is trained in advance to output the estimated image based on the input data.

20. 20. The image processing method according to claim 1, wherein in the first step, the input data and weight information of the machine learning model are acquired, and in the second step, the input data is input to the machine learning model based on the weight information.

21. The image processing method according to claim 20, characterized in that the weight information is learned in advance using training input data including training images and information regarding the state of the optical system corresponding to the training images so as to perform different processing depending on the first information.

22. A program causing a computer to execute the image processing method according to any one of claims 1 to 21.

23. an acquisition means for acquiring input data including a captured image and first information relating to a state of an optical system used to capture the captured image at the time of capturing the image; and a generation means for inputting the input data into a machine learning model and generating an estimated image by sharpening the captured image or correcting blur contained in the captured image.

24. An image processing system comprising the image processing device according to claim 23 and a control device capable of communicating with the image processing device, the control device has a transmission means for transmitting a request for execution of processing on the captured image to the image processing device, The image processing device includes: an image processing system comprising a receiving means for receiving the request, and generating the estimated image in response to the request;

25. a first step of acquiring input data including training images and first information relating to the state of an optical system used to capture the training images at the time of the images being captured; a second step of inputting the input data into a machine learning model and generating an output image by sharpening the training image or correcting blur contained in the training image; and a third step of updating the weights of the machine learning model based on the output image and a correct image.

26. A program causing a computer to execute the image processing method according to claim 25.

27. a first step of acquiring input data including a training image and first information relating to a state of an optical system used to capture the training image at the time of the image capture; a second step of inputting the input data into a machine learning model and generating an output image by sharpening the training image or correcting blur contained in the training image; A method for generating learned weights, characterized by having a third step of updating the weights of the machine learning model based on the output image and a correct image.

28. an acquisition means for acquiring input data including a training image and first information relating to a state of an optical system used to capture the training image at the time of the image capture; a generation means for inputting the input data into a machine learning model and generating an output image by sharpening the training image or correcting blur contained in the training image; and updating means for updating weights of the machine learning model based on the output image and a correct image.

Images