Image processing method, program image processing device, method of producing trained model, and image processing system

Abstract

To provide an image processing method which allows for highly accurately estimating distance information from defocus blur of a captured image, while suppressing the learning load and the amount of retained data of a machine learning model.SOLUTION: An image processing method is provided, comprising acquiring input data including a captured image and information on the state of an optical system used to acquire the captured image, and inputting the input data to a machine learning model to estimate distance information of the captured image, where the information on the state of the optical system includes at least one of a focal length, an aperture value, and a focus distance.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing method for estimating distance information from an image captured using an optical system. [Background technology]

[0002] Non-Patent Document 1 discloses a method for estimating distance information using a machine learning model from defocus blur in an image captured using a single optical system. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Physical Cue based Depth-Sensing by Color Coding with Deaberration Network,https: / / arxiv.org / abs / 1908.00329 Summary of the Invention [Problem to be solved by the invention]

[0004] The method disclosed in Non-Patent Document 1 leads to a decrease in estimation accuracy or an increase in the learning load and the amount of stored data when estimating distance information from an image captured by an optical system in which various aberrations occur. In an optical system, defocus blur changes depending on the focal length, aperture value, focus distance, etc. For this reason, the following two methods are considered for estimating distance information from defocus blur.

[0005] The first method is to train a machine learning model with training data that includes all defocus blur that can occur in an optical system. However, if the training data includes multiple defocus blurs with similar shapes, the estimation accuracy of distance information for each defocus blur decreases. The second method is to divide the defocus blurs that can occur in an optical system into multiple similar groups, and train a machine learning model individually with the training data of each group. However, in this case, in an optical system that generates various aberrations, such as a high-magnification zoom lens, the number of groups becomes enormous, and the learning load and the amount of stored data (the capacity of data that indicates the weights of the trained machine learning model) increase. For this reason, it is difficult to balance the estimation accuracy of distance information with the learning load and the amount of stored data.

[0006] Therefore, an object of the present invention is to provide an image processing method capable of estimating distance information with high accuracy from defocus blur in a captured image by reducing the learning load of a machine learning model and the amount of stored data. [Means for solving the problem]

[0007] An image processing method as one aspect of the present invention includes a step of acquiring input data including an captured image and information regarding the state of an optical system used to capture the captured image, and a step of inputting the input data into a machine learning model and estimating distance information of the captured image, wherein the information regarding the state of the optical system includes at least one of a focal length, an aperture value, or a focus distance.

[0008] Other objects and features of the present invention are illustrated in the following examples. Effect of the Invention

[0009] According to the present invention, it is possible to provide an image processing method capable of estimating distance information with high accuracy from defocus blur in a captured image by reducing the learning load of a machine learning model and the amount of stored data. [Brief description of the drawings]

[0010] [Figure 1] FIG. 2 is a diagram illustrating a configuration of a machine learning model in the first embodiment. [Diagram 2] 1 is a block diagram of an image processing system according to a first embodiment. [Diagram 3] 1 is an external view of an image processing system according to a first embodiment. [Figure 4] 5 is a diagram showing the relationship between the magnitude of defocus blur and the subject distance in the first embodiment. FIG. [Diagram 5] FIG. 4 is a diagram showing a point image intensity distribution at a defocus position in the first embodiment. [Figure 6] 6 is a diagram showing the relationship between the magnitude of defocus blur and the subject distance when the lens state is changed in the first embodiment. FIG. [Figure 7] 11 is a flowchart relating to weight learning in the first to third embodiments. [Figure 8] 4 is a flowchart related to generation of an estimated image in the first embodiment. [Figure 9] FIG. 11 is a diagram illustrating a configuration of a machine learning model in a second embodiment. [Figure 10] FIG. 11 is a block diagram of an image processing system according to a second embodiment. [Figure 11] FIG. 11 is an external view of an image processing system according to a second embodiment. [Figure 12] 13 is a diagram showing the relationship between an image sensor and an image circle of an optical system in Example 2. FIG. [Figure 13] 13 is a flowchart related to generation of an estimated image in the second embodiment. [Figure 14] FIG. 11 is a block diagram of an image processing system according to a third embodiment. [Figure 15] FIG. 11 is an external view of an image processing system according to a third embodiment. [Figure 16] 13 is a flowchart related to generation of an estimated image in the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same components are designated by the same reference numerals, and duplicated descriptions will be omitted.

[0012] Before describing the specific embodiment, the gist of the present invention will be described. In the present invention, distance information is estimated using a machine learning model from the defocus blur of an image captured using a single optical system. Since the shape of the defocus blur changes depending on the distance from the in-focus position, distance information can be estimated by utilizing this property. The machine learning model includes, for example, a neural network, genetic programming, or a Bayesian network. The neural network includes a CNN (Convolutional Neural Network), etc. Input data input to the machine learning model includes a captured image and information about the state of the optical system when the captured image is captured. The state of the optical system is, for example, the focal length, aperture value, or focus distance of the optical system, but is not limited to these.

[0013] In the learning of the machine learning model and the estimation after learning, by inputting information on the state of the optical system, the machine learning model can identify in which state of the optical system the defocus blur acting on the captured image occurred. As a result, even if the learning includes defocus blur of various shapes, the machine learning model learns weights for estimating different distance information for each state of the optical system. Therefore, it is possible to estimate highly accurate distance information for each defocus blur. Therefore, it is possible to suppress a decrease in the estimation accuracy of the distance information and to learn learning data including defocus blur of various shapes all at once. As a result, it is possible to suppress the learning load and the amount of stored data and estimate the distance information with high accuracy from the defocus blur of the captured image.

[0014] In the following, the stage of learning the weights of the machine learning model will be referred to as the learning phase, and the stage of estimating distance information using the machine learning model using the learned weights will be referred to as the estimation phase. EXAMPLES

[0015] First, an image processing system according to a first embodiment of the present invention will be described with reference to Fig. 2 and Fig. 3. Fig. 2 is a block diagram of an image processing system 100. Fig. 3 is an external view of the image processing system 100.

[0016] The image processing system 100 includes a learning device 101, an imaging device (image processing device) 102, and a network 103. The learning device 101 and the imaging device 102 are connected via the network 103, which may be wired or wireless. The learning device 101 includes a storage unit 111, an acquisition unit 112, a calculation unit 113, and an update unit 114, and learns weights for estimating distance information using a machine learning model (manufactures a trained model). The imaging device 102 captures an image of a subject space and acquires a captured image, and estimates distance information from the captured image using weight information read after capturing or in advance. Details of the weight learning performed by the learning device 101 and the distance information estimation performed by the imaging device 102 will be described later.

[0017] The imaging device 102 has an imaging optical system (optical system) 121 and an imaging element 122. The imaging optical system 121 collects light incident from a subject space and forms an optical image (subject image). The imaging element 122 converts the optical image into an electrical signal by photoelectric conversion to generate a captured image. The imaging element 122 is, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal-Oxide Semiconductor) sensor.

[0018] The image processing unit 123 has an acquisition unit (acquisition means) 123a and a distance estimation unit (estimation means) 123b, and generates an estimated image (distance information image) by estimating distance information from a captured image. Learned weight information learned by the learning device 101 is used to generate the estimated image. The weight information is stored in the storage unit 124. The recording medium 125 stores the estimated image. Alternatively, the captured image may be stored in the recording medium 125, and the image processing unit 123 may read the captured image to generate the estimated image. The display unit 126 displays the estimated image stored in the recording medium 125 according to a user's instruction. The system controller 127 controls the above series of operations.

[0019] Next, the shape of defocus blur and subject distance will be described with reference to Fig. 4. Fig. 4 is a diagram showing the relationship between the magnitude of defocus blur and subject distance, and shows the results of a geometrical optical calculation of the relationship between the magnitude (pixels) of defocus blur on the axis and the subject distance (mm). In Fig. 4, the horizontal axis shows the subject distance (mm), and the vertical axis shows the magnitude of defocus blur (px). The calculation conditions are a focus position of 2500 mm, an F-number of 1.4, a focal length of 50 mm, and a pixel pitch of 5.5 μm.

[0020] The size of the defocus blur increases as the object moves away from the focal position. For example, when the object distance is 5000 mm, the size is about 65 pixels, and when the object distance is 6000 mm, the size is about 75 pixels. On the other hand, when the object distance is 1700 mm, the size is also about 65 pixels, and the size of the defocus blur is the same as when the object distance is 5000 mm. However, in an actual optical system, even if the size of the PSF (Point Spread Function) is the same, the intensity distribution is different due to the influence of aberration. In this embodiment, the size of the PSF corresponds to the range in which the PSF has intensity, and the shape of the PSF corresponds to the intensity distribution of the PSF. Therefore, it is possible to estimate distance information by distinguishing between the defocus blur at 5000 mm and 1700 mm. Specifically, differences in intensity distribution cause differences in Gaussian blur, circular blur, double line blur, etc.

[0021] Here, double-line blur, ball blur, and Gaussian blur will be described with reference to FIG. 5. FIG. 5(A) is a diagram showing the point spread function (PSF) of double-line blur. In FIG. 5(A), the horizontal axis indicates spatial coordinates (position), and the vertical axis indicates intensity. This point is the same for FIG. 5(B) and (C) described later. As shown in FIG. 5(A), double-line blur has a PSF with a separated peak. When the PSF at the defocus distance has a shape as shown in FIG. 5(A), a subject that is originally a single line appears to be doubly blurred when defocused. FIG. 5(B) is a diagram showing the PSF of ball blur. Ball blur has a PSF with flat intensity. FIG. 5(C) is a diagram showing the PSF of Gaussian blur. Gaussian blur has a PSF with a Gaussian distribution. As described above, there is a correlation between the shape of defocus blur and the subject distance, and distance information can be estimated from the shape of defocus blur.

[0022] Next, the shape of defocus blur and the lens state (focal length, aperture value, focus distance) will be described with reference to FIG. 6. The shape of defocus blur changes depending on the lens state. FIG. 6 is a diagram showing the relationship between the magnitude of defocus blur and the subject distance when the lens state is changed. FIG. 6 shows the result of a geometrical optical calculation of the relationship between the magnitude (pixels) of defocus blur on the axis and the subject distance (mm) when the focal length, aperture value, and focus distance are changed from the lens state of FIG. 4. The result of FIG. 6 is a case where the focal length is changed from the lens state of FIG. 4 to 80 mm (two-dot chain line 1001), the aperture value to F2.8 (one-dot chain line 1002), and the focus distance to 5000 mm (dotted line 1003).

[0023] As shown in FIG. 6, the relationship between the magnitude of defocus blur and the subject distance changes depending on the lens state. That is, when the lens state changes, there are many subject distances corresponding to the magnitude of a specific defocus blur. As described above, since the number of defocuses in a specific lens state is small, it is possible to estimate distance information from the intensity distribution of the PSF. However, as the number of defocus blurs to be learned increases, it is difficult to estimate distance information only from the shape of the defocus blur, and the estimation accuracy decreases. Therefore, in this embodiment, by inputting information on the state of the optical system together with the captured image into the machine learning model, weights for estimating different distance information for each state of the optical system are learned. This makes it possible to estimate distance information with high accuracy for each defocus blur.

[0024] Next, weight learning (learning phase) executed by the learning device 101 will be described with reference to Fig. 7. Fig. 7 is a flowchart related to weight learning (a method for manufacturing a trained model). Each step in Fig. 7 is mainly executed by each unit of the learning device 101. Note that in this embodiment, CNN is used as the machine learning model, but other models can be similarly applied.

[0025] First, in step S101, the acquisition unit 112 acquires one or more pairs of a correct answer image and training input data from the storage unit 111. The training input data is input data in the learning phase of the CNN. The training input data includes a training image and information on the state of the optical system corresponding to the training image. The training image and the correct answer image are a pair of an image affected by defocus blur and a distance information image corresponding to the defocus blur. The training image is an image affected by defocus blur, and the correct answer image is a distance information image corresponding to the defocus blur. The distance information image has the same number of elements (number of pixels) as one channel component of the training image. As an example, a case will be shown in which the distance information image has a numerical value normalized based on the possible range of the subject distance. Let L be the subject distance, and the minimum and maximum values ​​of the subject distance be L and L, respectively. min , L maxIn this case, the normalized l is calculated by the following formula (1).

[0026]

number

[0027] There is no restriction on how the numerical values ​​are taken, and the closest distance may be 1, and the farthest distance from the imaging device may be 0. Also, the distance information image may be a numerical value normalized based on the magnitude of the possible defocus blur, rather than a numerical value normalized based on the possible range of the subject distance. In this case, the same magnitude of defocus blur exists before and after the focus distance. Therefore, it is desirable to have information that can distinguish between the front blur and the rear blur. For example, the first channel of the distance information image may be a numerical value normalized based on the magnitude of the defocus blur, and the second channel may be a numerical value indicating the positional relationship between the front and rear with respect to the focus distance. A training image is affected by defocus blur at a specific focal length, aperture value, and focus distance.

[0028] The information on the state of the optical system corresponding to the training image is information indicating at least one of a specific focal length, an aperture value, or a focus distance. In other words, the information on the state of the optical system is information specifying the defocus blur acting on the training image. In this embodiment, the information on the state of the optical system includes all of the focal length, the aperture value, and the focus distance. However, this embodiment is not limited to this, and the information on the state of the optical system may include only a part of the focal length, the aperture value, and the focus distance, or may include other information.

[0029] Below, examples of methods for generating the correct image and training input data stored in the storage unit 111 are shown. The first example is a method for performing an imaging simulation using an original image as a subject. The original image is a real-life image or a CG (Computer Graphics) image. In order to correctly estimate distance information for various subjects, it is desirable for the original image to be an image having edges with various intensities and directions, textures, gradations, flat parts, etc. The original image may be one or more. The training image is an image obtained by performing an imaging simulation by applying defocus blur to the original image.

[0030] In this embodiment, defocus blur occurring in the state (Z, F, D) of the imaging optical system 121 is applied. Here, Z indicates the focal length, F indicates the aperture value, and D indicates the focus distance state. When the image sensor 122 acquires multiple color components, the defocus blur of each color component is applied to the original image. The defocus blur can be applied by convolving the original image with a PSF (Point Spread Function), or by multiplying the frequency characteristic of the original image with an OTF (Optical Transfer Function). Information on the state of the optical system corresponding to the training image to which the defocus blur specified by (Z, F, D) has been applied is information that specifies (Z, F, D).

[0031] The correct image is a distance information image corresponding to the defocus blur. The correct image and the training image may be an undeveloped RAW image or a developed image. A plurality of different (Z, F, D) defocus blurs are applied to one or more original images to generate a plurality of pairs of correct images and training images. In this embodiment, the estimation of distance information for all defocus blurs generated in the imaging optical system 121 is learned at once. Therefore, (Z, F, D) is changed within the range that the imaging optical system 121 can take, and a plurality of pairs of correct images and training images are generated. In addition, since a plurality of defocus blurs exist depending on the image height and azimuth even for the same (Z, F, D), a pair of correct images and training images is generated for each different image height and azimuth.

[0032] Preferably, the original image has a signal value higher than the brightness saturation value of the image sensor 122. This is because even in real subjects, there are subjects that do not fall within the brightness saturation value when imaged by the image capture device 102 under specific exposure conditions. The ground truth image is generated by clipping the signal of the original image at the brightness saturation value of the image sensor 122. The training images are generated by applying blur and then clipping at the brightness saturation value.

[0033] The second example of the method for generating the correct image and the training input data is a method using an image captured by the imaging optical system 121 and the image sensor 122. The imaging optical system 121 captures an image in a state of (Z, F, D) to obtain a training image. Information on the state of the optical system corresponding to the training image is information that specifies (Z, F, D). The correct image is obtained by acquiring distance information when capturing the training image. The distance information can be acquired by using a ToF (Time Of Flight) sensor or the like, or by using a measuring tool such as a tape measure if the captured subject is at the same distance over the entire angle of view. Note that a partial area with a predetermined number of pixels may be extracted from the training image and the correct image generated by the above two methods and used for learning.

[0034] Next, in step S102 of FIG. 7, the calculation unit 113 inputs the training input data to the CNN to generate an output image. Here, the generation of the output image in this embodiment will be described with reference to FIG. 1. FIG. 1 is a diagram showing the configuration of a machine learning model. The training input data includes a training image 201 and information (z, f, d) 202 on the state of the optical system. The training image 201 may be grayscale or may have multiple channel components. The same applies to the ground truth image. (z, f, d) 200 is a normalized (Z, F, D). The normalization is performed based on the possible ranges of the imaging optical system 121 for each of the focal length, the aperture value, and the focus distance.

[0035] For example, let Z be the focal length, F be the aperture value, and D be the reciprocal of the absolute value of the distance from the image capture device 102 to the focus subject. Let Z be the minimum and maximum values ​​of the focal length Z of the imaging optical system 121, respectively. min , Zmax , the minimum and maximum aperture F are F min , F max , the minimum and maximum values ​​of the inverse of the absolute value of the focusable distance D are D min , D max Here, if the focusable distance is infinite, D min = 1 / |∞| = 0. The normalized (z, f, d) is given by the following equation (2).

[0036]

number

[0037] x is a dummy variable that indicates either (z, f, d), and X is a dummy variable that indicates either (Z, F, D). min =X max In this case, x is a constant. Alternatively, since x has no degree of freedom, it is excluded from the information regarding the state of the optical system. Here, since the closer the focus distance, the greater the change in performance of the imaging optical system 121, D is set to the reciprocal of the distance.

[0038] In this embodiment, the CNN 211 has a first sub-network 221 and a second sub-network 223. The first sub-network 221 has one or more convolution layers or full connection layers. The second sub-network 223 has one or more convolution layers. The range of influence of the convolution layer (filter) is determined by the number of layers and the size of the filter. For example, if the number of layers of the filter is 20 and the size is 3×3 pixels, the influence extends to pixels up to 20 pixels away from the pixel of interest. It is preferable to determine the number of layers and the size of the filter according to the size of the defocus blur to be learned. That is, if the size of the defocus blur is 40 pixels, the filter is applied to the entire defocus blur by setting the number of layers of the filter to 20 and the size to 3×3 pixels.

[0039] In the first learning, the weights (values ​​of each filter element and bias) of the CNN 211 are generated by random numbers. The first sub-network 221 receives information (z, f, d) 202 relating to the state of the optical system as input, and generates a state map 203 converted into a feature map. The state map 203 is a map showing the state of the optical system, and has the same number of elements (number of pixels) as one channel component of the training image 201. In this embodiment, the state map 203 is generated based on the number of pixels of the captured image and information relating to the state of the optical system. Also, in this embodiment, elements of the same channel in the state map 203 have the same numerical value.

[0040] The concatenation layer 222 concatenates the training image 201 and the state map 203 in a specified order in the channel direction. Other data may be concatenated between the training image 201 and the state map 203. The second sub-network 223 receives the concatenated training image 201 and state map 203 as input, and generates an output image 204. If multiple sets of training input data are acquired in step S101, an output image 204 is generated for each of them. Alternatively, the training image 201 may be converted into a feature map by a third sub-network, and the feature map and the state map 203 may be concatenated by the concatenation layer 222.

[0041] Next, in step S103 of FIG. 7, the update unit 114 updates the weights of the CNN based on the error between the output image and the correct image. In this embodiment, the Euclidean norm of the difference in signal values ​​between the output image and the correct image is used as the loss function. However, the loss function is not limited to this. When multiple sets of training input data and correct images are acquired in step S101, the value of the loss function is calculated for each set. The update unit 114 updates the weights based on the calculated loss function value by backpropagation or the like.

[0042] Next, in step S104, the update unit 114 determines whether the weight learning is complete. Completion can be determined by whether the number of iterations of learning (weight update) has reached a specified number, whether the amount of change in weight at the time of update is smaller than a specified value, etc. If it is determined that the learning is incomplete, the process returns to step S101, and one or more pairs of new training input data and a correct answer image are obtained. On the other hand, if it is determined that the learning is complete, the learning is terminated, and the weight information is stored in the storage unit 111.

[0043] Next, the estimation of distance information of a captured image (estimation phase) executed by the image processing unit 123 will be described with reference to Fig. 8. Fig. 8 is a flowchart related to generation of an estimated image. Each step in Fig. 8 is mainly executed by each unit of the image processing unit 123.

[0044] First, in step S201, the acquisition unit 123a acquires input data and weight information. The input data includes a captured image and information about the state of the optical system when the captured image is captured. The captured image to be acquired may be a part of the entire captured image. The information about the optical system is (z, f, d) that indicates the focal length, aperture value, and focus distance state of the imaging optical system 121. The weight information can be acquired by reading it from the storage unit 124.

[0045] Next, in step S202, the distance estimation unit 123b inputs the input data to the CNN to generate an estimated image. The estimated image is an image in which distance information is estimated from the defocus blur caused by the imaging optical system 121 for the captured image. As in the learning process, the estimated image is generated using the CNN shown in FIG. 1. The acquired learned weights are used for the CNN. There is no limit to the size (number of pixels) of the input data, and the size may be larger than the range affected by the convolution layer of the CNN. When the captured image is divided so as to fit within the range affected by the convolution layer and input to the CNN, the distance information is estimated for each divided image, so that the processing time increases. For this reason, it is preferable that the structure of the machine learning model is such that the input data may be larger than the range affected by the convolution layer. In other words, it is preferable that the area of ​​the captured image used by the machine learning model to obtain (estimate) a part of the area of ​​the distance information is smaller than the entire captured image input to the machine learning model. In this embodiment, the weights for distance information estimation are learned collectively for all (z, f, d) that the imaging optical system can take. Therefore, for all captured images (z, f, d), distance information is estimated using CNN with the same weights.

[0046] With the above configuration, according to this embodiment, it is possible to realize an image processing system that can reduce the learning load of the machine learning model and the amount of stored data, and estimate distance information with high accuracy from the defocus blur of a captured image. EXAMPLES

[0047] Next, an image processing system according to a second embodiment of the present invention will be described with reference to Fig. 10 and Fig. 11. Fig. 10 is a block diagram of an image processing system 300 according to this embodiment. Fig. 11 is an external view of the image processing system 300.

[0048] The image processing system 300 includes a learning device 301, an imaging device 302, an image estimation device (image processing device) 303, and networks 304 and 305. The learning device 301 and the image estimation device 303 can communicate with each other via the network 304. The imaging device 302 and the image estimation device 303 can communicate with each other via the network 305. The learning device 301 includes a storage unit 301a, an acquisition unit 301b, a generation unit 301c, and an update unit 301d, and learns weights of a machine learning model used to estimate distance information. Details regarding learning the weights and estimating distance information using the weights will be described later.

[0049] The imaging device 302 includes an optical system 302a, an imaging element 302b, an acquisition unit 302c, a recording medium 302d, and a system controller 302e. The optical system 302a collects light incident from a subject space and forms an optical image (subject image). The imaging element 302b converts the optical image into an electrical signal by photoelectric conversion to generate a captured image.

[0050] The image estimation device (image processing device) 303 has a storage unit 303a, a distance estimation unit (estimation means) 303b, and an acquisition unit (acquisition means) 303c. The image estimation device 303 generates an estimated image by estimating distance information for a captured image (or at least a part thereof) captured by the imaging device 302. The estimated image is generated using learned weight information learned by the learning device 301. The weight information is stored in the storage unit 303a. The acquisition unit 302c acquires the estimated image, and the recording medium 302d stores the estimated image. The system controller 302e controls a series of operations of the imaging device 302.

[0051] Next, weight learning (learning phase) executed by the learning device 301 will be described with reference to Fig. 7. Each step in Fig. 7 is mainly executed by each unit of the learning device 301. Note that in this embodiment, CNN is used as the machine learning model, but other models can be similarly applied. Also, descriptions similar to those in the first embodiment will be omitted.

[0052] First, in step S101, the acquisition unit 301b acquires one or more pairs of correct images and training input data from the storage unit 301a. The storage unit 301a stores training images for multiple combinations of the optical system 302a and the image sensor 302b. In the second embodiment, learning of weights for distance information estimation is performed collectively for each type of optical system 302a. For this reason, the type of optical system 302a for learning the weights is first determined, and training images are acquired from a collection of training images corresponding to the type. A collection of training images corresponding to a certain type of optical system 302a is a collection of images that have been subjected to defocus blur with different focal lengths, aperture values, focus distances, image heights, azimuths, etc.

[0053] In this embodiment, learning is performed with the configuration of CNN shown in FIG. 9. FIG. 9 is a diagram showing the configuration of a machine learning model in this embodiment. Training input data 404 includes a training image 401, a state map 402, and a position map 403. The state map 402 and the position map 403 are generated in this step. The position map is information about the position of each pixel of the captured image. The state map 402 and the position map 403 are maps indicating (Z, F, D) and (X, Y), respectively, which correspond to the defocus blur acting on the acquired training image. (X, Y) are the coordinates (horizontal and vertical directions) of the image plane shown in FIG. 12, and correspond to the image height and azimuth in polar coordinate display. In this embodiment, the coordinates (X, Y) have the optical axis of the optical system 302a as the origin.

[0054] 12 is a diagram showing the relationship between the image circle 501 of the optical system 302a, the first effective pixel area 502 and the second effective pixel area 503 of the image sensor 302b, and the coordinates (X, Y). The size of the image sensor 302b varies depending on the type of the image sensor 302. For this reason, there are types of image sensor 302 that have the first effective pixel area 502 and types that have the second effective pixel area 503. Of the image sensors 302 that can be connected to the optical system 302a, the image sensor 302 that has the largest size image sensor 302b has the first effective pixel area 502.

[0055] The position map 403 in FIG. 9 is generated based on (x, y) obtained by normalizing the coordinates (X, Y). The normalization is performed by dividing (X, Y) by a length (radius of the image circle) 511 based on the image circle 501 of the optical system 302a. Alternatively, normalization may be performed by dividing X by a horizontal length 512 from the origin to the first effective pixel area, and Y by a vertical length 513 from the origin to the first effective pixel area. If (X, Y) is normalized so that the edge of the captured image is always 1, the position (X, Y) indicated by the same value of (x, y) differs depending on the image captured by the image sensor 302b of a different size, and the correspondence between (x, y) and blur cannot be uniquely determined. This leads to a decrease in the accuracy of distance information estimation. The position map 403 is a two-channel map having the values ​​of (x, y) in the channel components. Note that polar coordinates may be used for the position map 403, and the method of taking the origin is not limited to that shown in FIG. 12.

[0056] The state map 402 is a three-channel map having normalized (z, f, d) values ​​as channel components. That is, in this embodiment, the state map 402 has values ​​indicating at least two of the focal length, aperture value, and focus distance of the optical system as elements of different channels. The number of elements (number of pixels) per channel of each of the training image 401, state map 402, and position map 403 is equal. Note that the configuration of the position map 403 and state map 402 is not limited to this. The position map may be expressed in one channel by dividing the first effective pixel region 502 into multiple partial regions and assigning a numerical value to each partial region. Similarly, (Z, F, D) may also be divided into multiple partial regions in a three-dimensional space with each axis as an axis, and a numerical value may be assigned to each partial region, and the state map may be expressed in one channel. The training image 401, state map 402, and position map 403 are linked in a specified order in the channel direction by the link layer 411 of FIG. 9, and training input data 404 is generated.

[0057] 7, the generation unit 301c inputs the training input data 404 to the CNN 412 to generate an output image 405. Then, in step S103, the update unit 301d updates the weights of the CNN based on the error between the output image and the correct image. Then, in step S104, the update unit 301d determines whether learning is complete. Information on the learned weights is stored in the storage unit 301a.

[0058] 13, which is a flowchart related to generation of an estimated image in relation to estimation of distance information of a captured image (estimation phase) executed by the image estimation device 303. Each step in FIG. 13 is mainly executed by each unit of the image estimation device 303.

[0059] First, in step S301, the acquisition unit 303c acquires a captured image (or at least a part thereof). Then, in step S302, the acquisition unit 303c acquires weight information corresponding to the captured image. In this embodiment, weight information for each type of optical system 302a is read out in advance from the storage unit 301a and stored in the storage unit 303a. Therefore, weight information corresponding to the type of optical system 302a used to capture the captured image is acquired from the storage unit 303a. The type of optical system 302a used to capture the captured image is identified, for example, from metadata in the file of the captured image.

[0060] Next, in step S303, the acquisition unit 303c generates a state map and a position map corresponding to the captured image, and generates input data. The state map is generated based on the number of pixels of the captured image and information on the state (Z, F, D) of the optical system 302a when the captured image is captured. The number of elements (number of pixels) per channel of the captured image and the state map are equal. (Z, F, D) are specified, for example, from the metadata of the captured image. The position map is generated based on the number of pixels of the captured image and information on the position of each pixel of the captured image. The number of elements (number of pixels) per channel of the captured image and the position map are equal. The size of the effective pixel area of ​​the image sensor 302b used to capture the captured image is specified from the metadata of the captured image, and a normalized position map is generated using, for example, the length of the image circle of the optical system 302a specified in the same manner. The input data is generated by concatenating the captured image, the state map, and the position map in a specified order in the channel direction, as in FIG. 9. Note that the order of steps S302 and S303 does not matter. Furthermore, a state map and a position map may be generated when a captured image is captured, and may be stored together with the captured image.

[0061] Next, in step S304, the distance estimation unit 303b inputs the input data to the CNN to generate an estimated image, similarly to FIG.

[0062] With the above configuration, according to this embodiment, it is possible to realize an image processing system that can reduce the learning load of the machine learning model and the amount of stored data, and estimate distance information with high accuracy from the defocus blur of a captured image.

[0063] Next, a preferred condition for enhancing the effect of this embodiment will be described. The input data preferably includes information on the pixel pitch of the image sensor 302b used to capture the captured image. This allows highly accurate estimation of distance information regardless of the type of image sensor 302b. The strength of pixel aperture deterioration and the magnitude of defocus blur for pixels vary depending on the pixel pitch. In the learning phase, information specifying the pixel pitch corresponding to the training image is included in the training input data. For example, a map with a normalized pixel pitch value as an element is included. For normalization, the maximum pixel pitch among the multiple types of image sensors 302 may be used as a divisor. In the estimation phase, the accuracy of distance information estimation can be improved by including a similar map in the input data. Such a map is generated based on the number of pixels in the captured image. EXAMPLES

[0064] Next, an image processing system according to a third embodiment of the present invention will be described with reference to Fig. 14 and Fig. 15. Fig. 14 is a block diagram of an image processing system 600 according to the third embodiment. Fig. 15 is an external view of the image processing system 600.

[0065] The image processing system 600 includes a learning device 601, a lens device 602, an imaging device 603, a control device (first device) 604, an image estimation device (second device) 605, and networks 606 and 607. The learning device 601 and the image estimation device 605 can communicate with each other via the network 606. The control device 604 and the image estimation device 605 can communicate with each other via the network 607. The learning device 601 and the image estimation device 605 are, for example, servers. The control device 604 is a device operated by a user, such as a personal computer or a mobile terminal. The learning device 601 includes a storage unit 601a, an acquisition unit 601b, a calculation unit 601c, and an update unit 601d, and learns weights of a machine learning model that estimates distance information from an image captured using the lens device 602 and the imaging device 603. Note that the learning method of this embodiment is similar to that of the first embodiment, and therefore the description thereof will be omitted.

[0066] The imaging device 603 has an imaging element 603a, and the imaging element 603a photoelectrically converts an optical image formed by the lens device 602 to obtain a captured image. The lens device 602 and the imaging device 603 are detachable, and can be combined with each other in a plurality of types. The control device 604 has a communication unit 604a, a storage unit 604b, and a display unit 604c, and controls the processing to be executed on the captured image obtained from the imaging device 603 connected by wire or wirelessly according to a user's operation. Alternatively, the captured image captured by the imaging device 603 may be stored in advance in the storage unit 604b, and the captured image may be read out.

[0067] The image estimation device 605 includes a communication unit 605a, a storage unit 605b, an acquisition unit 605c, and a distance estimation unit 605d. The image estimation device 605 executes distance information estimation processing of a captured image in response to a request from a control device 604 connected via a network 607. The image estimation device 605 acquires learned weight information from a learning device 601 connected via a network 606 when estimating distance information or in advance, and uses the learned weight information to estimate distance information of a captured image. The estimated image after the estimation of distance information is transmitted again to the control device 604, stored in the storage unit 604b, and displayed on the display unit 604c. Note that the generation of learning data and learning of weights (learning phase) performed by the learning device 601 are the same as those in the first embodiment, and therefore their description will be omitted.

[0068] Next, the estimation of distance information (estimation phase) executed by the control device 604 and the image estimation device 605 will be described with reference to Fig. 16. Fig. 16 is a flowchart relating to the generation of an estimated image in this embodiment.

[0069] First, in step S401, the communication unit 604a transmits to the image estimation device 605 a captured image and a request for execution of distance information estimation processing.

[0070] Next, in step S501, the communication unit 605a receives and acquires the captured image and the processing request transmitted from the control device 604. Next, in step S502, the acquisition unit 605c acquires information on learned weights corresponding to the captured image from the storage unit 605b. The weight information is read out in advance from the storage unit 601a and stored in the storage unit 605b.

[0071] Next, in step S503, the acquisition unit 605c acquires information on the state of the optical system corresponding to the captured image to generate input data. Information that specifies the type, focal length, aperture value, and focus distance of the imaging optical system 602 when the captured image was captured is acquired from the metadata of the captured image, and a state map (lens state map) is generated as in Fig. 1. The input data is generated by linking the captured image and the state map in a predetermined order in the channel direction.

[0072] Next, in step S504, the distance estimation unit 605d inputs the input data to the generator and generates an estimated image by estimating the distance information. The generator uses the weight information. Next, in step S505, the communication unit 605a transmits the estimated image to the control device 604.

[0073] Next, in step S402, the communication unit 604a acquires the estimated image transmitted from the image estimation device 605.

[0074] With the above configuration, according to this embodiment, it is possible to realize an image processing system that can reduce the learning load of the machine learning model and the amount of stored data, and estimate distance information with high accuracy from the defocus blur of a captured image.

[0075] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0076] According to each embodiment, it is possible to provide an image processing method, a program, an image processing device, a method for manufacturing a trained model, and an image processing system that can reduce the learning load and the amount of retained data of a machine learning model and estimate distance information with high accuracy from the defocus blur of a captured image.

[0077] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0078] 102 Imaging device (image processing device) 123a Acquisition unit (acquisition means) 123b Distance estimation unit (generation means)

Claims

1. acquiring input data including a captured image and information about a state of an optical system used to capture the captured image; inputting the input data into a machine learning model and estimating distance information of the captured image; 2. An image processing method, comprising: the information on the state of the optical system including at least one of a focal length, an aperture value, and a focus distance.

2. The image processing method according to claim 1 , wherein the area of ​​the captured image used by the machine learning model to estimate a portion of the distance information is smaller than the entire captured image input to the machine learning model.

3. the information about the state of the optical system includes a numerical value indicating at least one of a focal length, an aperture value, or a focus distance of the optical system; 3. The image processing method according to claim 1, wherein the numerical values ​​are normalized.

4. the input data includes a state map indicating a state of the optical system; 4. The image processing method according to claim 1, wherein the state map is generated based on the number of pixels of the captured image and information on a state of the optical system.

5. 5. The image processing method according to claim 4, wherein the state map has numerical values ​​indicating at least two of the focal length, the aperture value, and the focus distance of the optical system as elements of different channels.

6. 6. The image processing method according to claim 4, wherein elements of the same channel in the state map have the same numerical value.

7. 7. The image processing method according to claim 1, wherein the input data further includes information relating to a position of each pixel of the captured image.

8. 8. The image processing method according to claim 7, wherein the information about the position has a normalized value based on a length based on an image circle of the optical system.

9. 9. The image processing method according to claim 1, wherein the input data further includes information on a pixel pitch of an image pickup device used to capture the captured image.

10. A program for causing a computer to execute the image processing method according to any one of claims 1 to 9.

11. An acquisition means for acquiring input data including a captured image and information regarding a state of an optical system used to capture the captured image; an estimation means for inputting the input data into a machine learning model and estimating distance information of the captured image, 13. An image processing device, wherein the information on the state of the optical system includes at least one of a focal length, an aperture value, and a focus distance.

12. acquiring a training image, a correct answer image having distance information corresponding to the training image, and information regarding a state of an optical system; and learning a machine learning model based on the training images, the ground truth images, and information about the state of the optical system; A learning method characterized in that the information regarding the state of the optical system includes at least one of a focal length, an aperture value, or a focus distance.

13. A program for causing a computer to execute the learning method according to claim 12.

14. acquiring a training image, a correct answer image having distance information corresponding to the training image, and information regarding a state of an optical system; and learning a machine learning model based on the training images, the ground truth images, and information about the state of the optical system; A method for manufacturing a trained model, characterized in that the information regarding the state of the optical system includes at least one of a focal length, an aperture value, or a focus distance.

15. An acquisition means for acquiring a training image, a correct image having distance information corresponding to the training image, and information regarding the state of the optical system; A learning means for learning a machine learning model based on the training image, the correct image, and information on the state of the optical system, 13. An image processing device, wherein the information on the state of the optical system includes at least one of a focal length, an aperture value, and a focus distance.

16. An image processing system having a first device and a second device capable of communicating with each other, the first device has a transmission means for transmitting a request for execution of processing on a captured image to the second device; The second device comprises: A receiving means for receiving the request; an acquisition means for acquiring input data including the captured image and information regarding a state of an optical system used to capture the captured image; and an estimation means for inputting the input data into a machine learning model based on the request, and estimating distance information of the captured image, 11. An image processing system, wherein the information on the state of the optical system includes at least one of a focal length, an aperture value, and a focus distance.

Images