Image processing method, program, image processing device, method for manufacturing trained model, and image processing system

Abstract

To provide an image processing method which allows for highly accurately estimating distance information from defocus blur of a captured image, while suppressing the learning load and the amount of retained data of a machine learning model.SOLUTION: An image processing method is provided, comprising acquiring input data including a captured image and information on the state of an optical system used to acquire the captured image, and inputting the input data to a machine learning model to estimate distance information of the captured image, where the information on the state of the optical system includes at least one of a focal length, an aperture value, and a focus distance.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing method for estimating distance information from an image captured using an optical system. [Background technology]

[0002] Non-Patent Document 1 discloses a method for estimating distance information using a machine learning model from defocus blur in an image captured using a single optical system. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Physical Cue based Depth-Sensing by Color Coding with Deaberration Network,https: / / arxiv.org / abs / 1908.00329 Summary of the Invention [Problem to be solved by the invention]

[0004] The method disclosed in Non-Patent Document 1, when estimating distance information from an image captured using an optical system that generates various aberrations, results in a decrease in estimation accuracy or an increase in the learning load and amount of stored data. In an optical system, defocus blur changes depending on the focal length, aperture value, focus distance, etc. For this reason, the following two methods are conceivable for estimating distance information from defocus blur.

[0005] The first method is to train a machine learning model using training data that includes all defocus blur that can occur in an optical system. However, if the training data contains multiple defocus blurs with similar shapes, the accuracy of estimating distance information for each defocus blur will decrease. The second method is to divide the defocus blur that can occur in an optical system into multiple similar groups and train a machine learning model individually using the training data for each group. However, in this case, for optical systems that generate various aberrations, such as high-magnification zoom lenses, the number of groups will become enormous, increasing the training load and the amount of data stored (the amount of data indicating the weights of the trained machine learning model). For this reason, it is difficult to balance the accuracy of distance information estimation with the training load and amount of data stored.

[0006] Therefore, an object of the present invention is to provide an image processing method and the like that can reduce the learning load of a machine learning model and the amount of data stored, and estimate distance information with high accuracy from defocus blur in a captured image. [Means for solving the problem]

[0007] An image processing method according to one aspect of the present invention includes: No. 1 A captured image obtained by imaging using an optical system, No. 1 and a step of estimating information about a subject distance in the captured image by inputting the input data into a machine learning model, wherein the machine learning model estimates a training image, a correct answer image having information about the subject distance in the training image, and a correct answer image corresponding to the training image. No. 2 and second information about the state of the optical system, wherein the first information and the second information include at least one of a focal length, an aperture value, or a focus distance. the first information is a map showing the state of the first optical system generated based on the number of pixels of the captured image, and the second information is a map showing the state of the second optical system generated based on the number of pixels of the training image. .

[0008] Other objects and features of the present invention are illustrated in the following examples. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide an image processing method that can reduce the learning load of a machine learning model and the amount of data stored, and estimate distance information with high accuracy from defocus blur in a captured image. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating a configuration of a machine learning model according to a first embodiment. [Figure 2] 1 is a block diagram of an image processing system according to a first embodiment. [Figure 3] 1 is an external view of an image processing system according to a first embodiment. [Figure 4] FIG. 4 is a diagram showing the relationship between the magnitude of defocus blur and the subject distance in the first embodiment. [Figure 5] FIG. 4 is a diagram showing a point image intensity distribution at a defocus position in Example 1. [Figure 6] 10 is a diagram showing the relationship between the magnitude of defocus blur and the subject distance when the lens state is changed in Example 1. FIG. [Figure 7] 10 is a flowchart relating to weight learning in the first to third embodiments. [Figure 8] 4 is a flowchart relating to generation of an estimated image in the first embodiment. [Figure 9] FIG. 10 is a diagram illustrating a configuration of a machine learning model in a second embodiment. [Figure 10] FIG. 10 is a block diagram of an image processing system according to a second embodiment. [Figure 11] FIG. 10 is an external view of an image processing system according to a second embodiment. [Figure 12] 10 is a diagram showing the relationship between the image sensor and the image circle of the optical system in Example 2. FIG. [Figure 13] 10 is a flowchart relating to generation of an estimated image in the second embodiment. [Figure 14] FIG. 10 is a block diagram of an image processing system according to a third embodiment. [Figure 15]FIG. 10 is an external view of an image processing system according to a third embodiment. [Figure 16] 11 is a flowchart relating to generation of an estimated image in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same components are designated by the same reference numerals, and redundant explanations will be omitted.

[0012] Before describing this embodiment in detail, the gist of the present invention will be described. The present invention estimates distance information using a machine learning model from defocus blur in an image captured using a single optical system. Because the shape of defocus blur changes depending on the distance from the in-focus position, this property can be utilized to estimate distance information. Examples of machine learning models include neural networks, genetic programming, and Bayesian networks. Examples of neural networks include convolutional neural networks (CNNs). Input data input to the machine learning model include the captured image and information about the state of the optical system when the image was captured. Examples of the state of the optical system include, but are not limited to, the focal length, aperture value, and focus distance of the optical system.

[0013] By inputting information about the state of the optical system into the learning of the machine learning model and the estimation after learning, the machine learning model can identify the state of the optical system in which the defocus blur affecting the captured image occurred. As a result, even if the learning includes defocus blur of various shapes, the machine learning model learns weights that estimate different distance information for each state of the optical system. This makes it possible to estimate distance information with high accuracy for each defocus blur. Therefore, it is possible to suppress a decrease in the accuracy of distance information estimation and to learn learning data including defocus blur of various shapes all at once. As a result, it is possible to reduce the learning load and the amount of data stored and estimate distance information with high accuracy from the defocus blur of a captured image.

[0014] In the following, the stage of learning the weights of the machine learning model will be referred to as the learning phase, and the stage of estimating distance information using the machine learning model using the learned weights will be referred to as the estimation phase. [Example]

[0015] First, an image processing system according to a first embodiment of the present invention will be described with reference to Fig. 2 and Fig. 3. Fig. 2 is a block diagram of the image processing system 100. Fig. 3 is an external view of the image processing system 100.

[0016] The image processing system 100 includes a learning device 101, an imaging device (image processing device) 102, and a network 103. The learning device 101 and the imaging device 102 are connected via the network 103, which may be wired or wireless. The learning device 101 includes a storage unit 111, an acquisition unit 112, a calculation unit 113, and an update unit 114, and learns weights for estimating distance information using a machine learning model (produces a trained model). The imaging device 102 captures an image of a subject space to acquire the captured image, and estimates distance information from the captured image using weight information read after capturing or in advance. Details regarding the weight learning performed by the learning device 101 and the distance information estimation performed by the imaging device 102 will be described later.

[0017] The imaging device 102 has an imaging optical system (optical system) 121 and an imaging element 122. The imaging optical system 121 collects light incident from a subject space and forms an optical image (subject image). The imaging element 122 converts the optical image into an electrical signal by photoelectric conversion to generate a captured image. The imaging element 122 is, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal-Oxide Semiconductor) sensor.

[0018] The image processing unit 123 has an acquisition unit (acquisition means) 123a and a distance estimation unit (estimation means) 123b, and generates an estimated image (distance information image) by estimating distance information from a captured image. The estimated image is generated using learned weight information learned by the learning device 101. The weight information is stored in the memory unit 124. The recording medium 125 stores the estimated image. Alternatively, the captured image may be stored in the recording medium 125, and the image processing unit 123 may read the captured image to generate the estimated image. The display unit 126 displays the estimated image stored in the recording medium 125 in accordance with a user's instruction. The system controller 127 controls the above series of operations.

[0019] Next, the shape of defocus blur and subject distance will be described with reference to Fig. 4. Fig. 4 is a diagram showing the relationship between the magnitude of defocus blur and subject distance, and shows the results of a geometrical optical calculation of the relationship between the magnitude (pixels) of defocus blur on the axis and the subject distance (mm). In Fig. 4, the horizontal axis represents the subject distance (mm), and the vertical axis represents the magnitude of defocus blur (px). The calculation conditions are a focus position of 2500 mm, an F-number of 1.4, a focal length of 50 mm, and a pixel pitch of 5.5 μm.

[0020] The size of defocus blur increases as the object moves away from the in-focus position. For example, when the object distance is 5000 mm, the size is approximately 65 pixels, and when the object distance is 6000 mm, the size is approximately 75 pixels. On the other hand, when the object distance is 1700 mm, the size of defocus blur is also approximately 65 pixels, the same as when the object distance is 5000 mm. However, in an actual optical system, even if the size of the PSF (Point Spread Function) is the same, the intensity distribution differs due to the influence of aberration. Note that in this embodiment, the size of the PSF corresponds to the range in which the PSF has intensity, and the shape of the PSF corresponds to the intensity distribution of the PSF. Therefore, it is possible to estimate distance information by distinguishing between defocus blur at 5000 mm and 1700 mm. Specifically, differences in intensity distribution result in differences such as Gaussian blur, circular blur, and two-line blur.

[0021] Here, double-line blur, circular blur, and Gaussian blur will be described with reference to FIG. 5. FIG. 5(A) is a diagram showing the point spread function (PSF) of double-line blur. In FIG. 5(A), the horizontal axis represents spatial coordinates (position), and the vertical axis represents intensity. This also applies to FIGS. 5(B) and 5(C), which will be described later. As shown in FIG. 5(A), double-line blur has a PSF with separate peaks. When the PSF at the defocus distance has a shape like that of FIG. 5(A), an object that is actually a single line appears doubly blurred when defocused. FIG. 5(B) is a diagram showing the PSF of circular blur. Circle blur has a PSF with flat intensity. FIG. 5(C) is a diagram showing the PSF of Gaussian blur. Gaussian blur has a PSF with a Gaussian distribution. As described above, there is a correlation between the shape of defocus blur and object distance, and distance information can be estimated from the shape of defocus blur.

[0022] Next, the shape of defocus blur and the lens state (focal length, aperture value, and focus distance) will be described with reference to FIG. 6. The shape of defocus blur changes depending on the lens state. FIG. 6 is a diagram showing the relationship between the magnitude of defocus blur and the subject distance when the lens state is changed. FIG. 6 shows the results of a geometrical optical calculation of the relationship between the magnitude (pixels) of on-axis defocus blur and the subject distance (mm) when the focal length, aperture value, and focus distance are changed from the lens state of FIG. 4. The results in FIG. 6 are obtained when the focal length is changed to 80 mm (two-dot chain line 1001), the aperture value to F2.8 (one-dot chain line 1002), and the focus distance to 5000 mm (dotted line 1003) from the lens state of FIG. 4.

[0023] As shown in Figure 6, the relationship between the magnitude of defocus blur and subject distance changes depending on the lens state. In other words, when the lens state changes, there are many subject distances corresponding to specific defocus blur magnitudes. As described above, because the number of defocus blurs in a specific lens state is small, it is possible to estimate distance information from the intensity distribution of the PSF. However, as the number of defocus blurs to be learned increases, it becomes difficult to estimate distance information from the shape of the defocus blur alone, and the estimation accuracy decreases. Therefore, in this embodiment, by inputting information about the state of the optical system along with the captured image into a machine learning model, weights for estimating different distance information for each state of the optical system are learned. This enables highly accurate estimation of distance information for each defocus blur.

[0024] Next, weight learning (learning phase) executed by the learning device 101 will be described with reference to Fig. 7. Fig. 7 is a flowchart related to weight learning (method of manufacturing a trained model). Each step in Fig. 7 is mainly executed by each unit of the learning device 101. Note that in this embodiment, CNN is used as the machine learning model, but other models can also be similarly applied.

[0025] First, in step S101, the acquisition unit 112 acquires one or more pairs of a correct image and training input data from the storage unit 111. The training input data is input data in the learning phase of the CNN. The training input data includes a training image and information on the state of the optical system corresponding to the training image. The training image and the correct image are a pair of an image affected by defocus blur and a distance information image corresponding to the defocus blur. The training image is an image affected by defocus blur, and the correct image is a distance information image corresponding to the defocus blur. The distance information image has the same number of elements (number of pixels) as one channel component of the training image. As an example, a case will be shown in which the distance information image has numerical values ​​normalized based on the possible range of the subject distance. Let L be the subject distance, and the minimum and maximum values ​​of the subject distance be L and L, respectively. min , L max In this case, the normalized l is calculated using the following equation (1).

[0026]

number

[0027] There are no restrictions on how the values ​​are calculated; the closest distance may be 1 and the farthest distance from the imaging device may be 0. Furthermore, the distance information image may be a value normalized based on the magnitude of possible defocus blur, rather than a value normalized based on the range of possible subject distances. In this case, the same magnitude of defocus blur exists before and after the focus distance. Therefore, it is desirable to have information that can distinguish between foreground and background blur. For example, the first channel of the distance information image may be a value normalized based on the magnitude of defocus blur, and the second channel may be a value indicating the positional relationship between the front and rear of the focus distance. Each training image is affected by defocus blur at a specific focal length, aperture value, and focus distance.

[0028] The information about the state of the optical system corresponding to the training image is information indicating at least one of a specific focal length, aperture value, or focus distance. In other words, the information about the state of the optical system is information that identifies the defocus blur affecting the training image. In this embodiment, the information about the state of the optical system includes all of the focal length, aperture value, and focus distance. However, this embodiment is not limited to this, and the information about the state of the optical system may include only some of the focal length, aperture value, and focus distance, or may include other information.

[0029] Below, examples of methods for generating the correct image and training input data stored in the storage unit 111 are shown. The first example is a method for performing an imaging simulation using an original image as the subject. The original image may be a real-life image or a CG (Computer Graphics) image. To enable accurate estimation of distance information for various subjects, it is desirable that the original image be an image that has edges with various intensities and directions, textures, gradations, flat areas, etc. The original image may be one or more. The training image is an image obtained by performing an imaging simulation by applying defocus blur to the original image.

[0030] In this embodiment, defocus blur that occurs in the state (Z, F, D) of the imaging optical system 121 is applied. Here, Z indicates the focal length, F indicates the aperture value, and D indicates the focus distance state. When the image sensor 122 acquires multiple color components, the defocus blur of each color component is applied to the original image. The defocus blur can be applied by convolving the original image with a PSF (Point Spread Function), or by multiplying the frequency characteristics of the original image by an OTF (Optical Transfer Function). Information regarding the state of the optical system corresponding to the training image to which the defocus blur specified by (Z, F, D) has been applied is information that identifies (Z, F, D).

[0031] The reference image is a distance information image corresponding to defocus blur. The reference image and training images may be undeveloped RAW images or developed images. Multiple different defocus blurs (Z, F, D) are applied to one or more original images to generate multiple pairs of reference images and training images. In this embodiment, distance information estimation for all defocus blurs generated in the imaging optical system 121 is learned all at once. Therefore, (Z, F, D) is changed within the range that the imaging optical system 121 can take, and multiple pairs of reference images and training images are generated. Furthermore, since multiple defocus blurs exist depending on the image height and azimuth even for the same (Z, F, D), pairs of reference images and training images are generated for different image heights and azimuths.

[0032] Preferably, the original image has a signal value higher than the brightness saturation value of the image sensor 122. This is because even in real subjects, there are subjects that do not fall within the brightness saturation value when captured by the image capture device 102 under specific exposure conditions. The ground truth image is generated by clipping the signal of the original image at the brightness saturation value of the image sensor 122. The training images are generated by applying blur and then clipping at the brightness saturation value.

[0033] A second example of a method for generating a ground truth image and training input data is a method using real images captured by the imaging optical system 121 and the image sensor 122. The imaging optical system 121 captures an image in a state (Z, F, D) to obtain a training image. Information about the state of the optical system corresponding to the training image is information that specifies (Z, F, D). The ground truth image is obtained by acquiring distance information when capturing the training image. Distance information can be obtained using a ToF (Time Of Flight) sensor or the like, or, if the captured subject is at the same distance over the entire field of view, by using a measuring tool such as a tape measure. Note that a partial region with a predetermined number of pixels may be extracted from the training image and ground truth image generated by the two methods described above and used for learning.

[0034] Next, in step S102 of FIG. 7, the calculation unit 113 inputs the training input data to the CNN to generate an output image. Here, the generation of the output image in this embodiment will be described with reference to FIG. 1. FIG. 1 is a diagram showing the configuration of a machine learning model. The training input data includes a training image 201 and information (z, f, d) 202 about the state of the optical system. The training image 201 may be grayscale or may have multiple channel components. The same applies to the ground truth image. (z, f, d) 200 is a normalized (Z, F, D). The normalization is performed based on the possible ranges of the imaging optical system 121 for the focal length, aperture value, and focus distance.

[0035] For example, let Z be the focal length, F be the aperture value, and D be the reciprocal of the absolute value of the distance from the image capture device 102 to the focus subject. The minimum and maximum values ​​of the focal length Z of the imaging optical system 121 are respectively Z min , Z max , the minimum and maximum values ​​of the aperture value F are F min , F max , the minimum and maximum values ​​of the inverse of the absolute value of the focusable distance D are D, respectively. min , D max Here, if the focusable distance is infinity, D min = 1 / |∞| = 0. The normalized (z, f, d) is calculated using the following equation (2).

[0036]

number

[0037] x is a dummy variable that indicates one of (z, f, d), and X is a dummy variable that indicates one of (Z, F, D). min =X max In this case, x is a constant. Alternatively, since x has no degree of freedom, it is excluded from the information regarding the state of the optical system. Here, since the change in performance of the imaging optical system 121 generally increases as the focus distance decreases, D is set to the reciprocal of the distance.

[0038] In this embodiment, the CNN 211 has a first subnetwork 221 and a second subnetwork 223. The first subnetwork 221 has one or more convolutional layers or full-connection layers. The second subnetwork 223 has one or more convolutional layers. The range of influence of a convolutional layer (filter) is determined by the number of layers and size of the filter. For example, if the number of filter layers is 20 and the size is 3 × 3 pixels, the influence will extend to pixels up to 20 pixels away from the pixel of interest. The number of layers and size of the filter are preferably determined according to the size of the defocus blur to be learned. That is, if the size of the defocus blur is 40 pixels, the filter will be applied to the entire defocus blur by setting the number of filter layers to 20 and the size to 3 × 3 pixels.

[0039] In the first learning, the weights of the CNN 211 (values ​​of each filter element and bias) are generated by random numbers. The first sub-network 221 receives information (z, f, d) 202 relating to the state of the optical system as input and generates a state map 203 converted into a feature map. The state map 203 is a map indicating the state of the optical system, and has the same number of elements (number of pixels) as one channel component of the training image 201. In this embodiment, the state map 203 is generated based on the number of pixels of the captured image and information relating to the state of the optical system. Furthermore, in this embodiment, elements of the same channel in the state map 203 have the same numerical value.

[0040] The concatenation layer 222 concatenates the training images 201 and the state map 203 in a specified order in the channel direction. Note that other data may be concatenated between the training images 201 and the state map 203. The second sub-network 223 receives the concatenated training images 201 and state map 203 as input and generates an output image 204. If multiple sets of training input data are acquired in step S101, an output image 204 is generated for each set. Alternatively, the training images 201 may be converted into feature maps by a third sub-network, and the feature maps and the state map 203 may be concatenated by the concatenation layer 222.

[0041] Next, in step S103 of FIG. 7, the update unit 114 updates the weights of the CNN based on the error between the output image and the correct image. In this embodiment, the Euclidean norm of the difference in signal values ​​between the output image and the correct image is used as the loss function. However, the loss function is not limited to this. If multiple pairs of training input data and correct images are acquired in step S101, the value of the loss function is calculated for each pair. The update unit 114 updates the weights based on the calculated loss function value using backpropagation or the like.

[0042] Next, in step S104, the update unit 114 determines whether weight learning is complete. Completion can be determined by, for example, whether the number of iterations of learning (weight update) has reached a specified number, or whether the amount of change in weight during update is smaller than a specified value. If it is determined to be incomplete, the process returns to step S101, and one or more pairs of new training input data and correct image are acquired. On the other hand, if it is determined to be complete, the learning is terminated, and weight information is saved in the storage unit 111.

[0043] Next, the estimation of distance information of a captured image (estimation phase) executed by the image processing unit 123 will be described with reference to Fig. 8. Fig. 8 is a flowchart related to generation of an estimated image. Each step in Fig. 8 is mainly executed by each unit of the image processing unit 123.

[0044] First, in step S201, the acquisition unit 123a acquires input data and weight information. The input data includes a captured image and information about the state of the optical system when the captured image was captured. The captured image to be acquired may be a portion of the entire captured image. The information about the optical system is (z, f, d) that indicates the focal length, aperture value, and focus distance state of the imaging optical system 121. The weight information can be acquired by reading it from the storage unit 124.

[0045] Next, in step S202, the distance estimation unit 123b inputs the input data to the CNN to generate an estimated image. The estimated image is an image in which distance information is estimated from the defocus blur caused by the imaging optical system 121 for the captured image. As in the training process, the estimated image is generated using the CNN shown in FIG. 1. The CNN uses the acquired trained weights. Note that there is no limit to the size (number of pixels) of the input data, and it may be larger than the range affected by the convolutional layer of the CNN. If the captured image is divided so that it fits within the range affected by the convolutional layer and input to the CNN, distance information is estimated for each divided image, which increases processing time. For this reason, it is preferable that the structure of the machine learning model be such that input data larger than the range affected by the convolutional layer is acceptable. In other words, it is preferable that the area of ​​the captured image used by the machine learning model to obtain (estimate) a partial area of ​​distance information is smaller than the entire captured image input to the machine learning model. In this embodiment, the weights for distance information estimation are learned collectively for all possible values ​​(z, f, d) of the imaging optical system. Therefore, distance information is estimated using CNN with the same weights for all captured images (z, f, d).

[0046] With the above configuration, according to this embodiment, it is possible to realize an image processing system that can reduce the learning load of the machine learning model and the amount of data stored, and estimate distance information with high accuracy from the defocus blur of a captured image. [Example]

[0047] Next, an image processing system according to a second embodiment of the present invention will be described with reference to Fig. 10 and Fig. 11. Fig. 10 is a block diagram of an image processing system 300 according to this embodiment. Fig. 11 is an external view of the image processing system 300.

[0048] Image processing system 300 includes a learning device 301, an imaging device 302, an image estimation device (image processing device) 303, and networks 304 and 305. Learning device 301 and image estimation device 303 can communicate with each other via network 304. Imaging device 302 and image estimation device 303 can communicate with each other via network 305. Learning device 301 includes a storage unit 301a, an acquisition unit 301b, a generation unit 301c, and an update unit 301d, and learns weights of a machine learning model used to estimate distance information. Details regarding learning the weights and estimating distance information using the weights will be described later.

[0049] The imaging device 302 includes an optical system 302a, an imaging element 302b, an acquisition unit 302c, a recording medium 302d, and a system controller 302e. The optical system 302a collects light incident from the subject space and forms an optical image (subject image). The imaging element 302b converts the optical image into an electrical signal by photoelectric conversion, generating a captured image.

[0050] The image estimation device (image processing device) 303 has a storage unit 303a, a distance estimation unit (estimation means) 303b, and an acquisition unit (acquisition means) 303c. The image estimation device 303 generates an estimated image by estimating distance information for an image (or at least a part thereof) captured by the imaging device 302. The estimated image is generated using learned weight information learned by the learning device 301. The weight information is stored in the storage unit 303a. The acquisition unit 302c acquires the estimated image, and the recording medium 302d stores the estimated image. The system controller 302e controls a series of operations of the imaging device 302.

[0051] Next, weight learning (learning phase) executed by the learning device 301 will be described with reference to Fig. 7. Each step in Fig. 7 is mainly executed by each unit of the learning device 301. Note that in this embodiment, CNN is used as the machine learning model, but other models can also be similarly applied. Also, descriptions similar to those in the first embodiment will be omitted.

[0052] First, in step S101, the acquisition unit 301b acquires one or more pairs of a ground truth image and training input data from the storage unit 301a. The storage unit 301a stores training images for multiple combinations of optical systems 302a and image sensors 302b. In the second embodiment, learning of weights for distance information estimation is performed collectively for each type of optical system 302a. For this reason, the type of optical system 302a for which the weights are to be learned is first determined, and training images are acquired from a collection of training images corresponding to that type. A collection of training images corresponding to a certain type of optical system 302a is a collection of images affected by defocus blur with different focal lengths, aperture values, focus distances, image heights, azimuths, etc.

[0053] In this embodiment, learning is performed using the CNN configuration shown in FIG. 9. FIG. 9 is a diagram showing the configuration of a machine learning model in this embodiment. Training input data 404 includes training images 401, a state map 402, and a position map 403. The state map 402 and the position map 403 are generated in this step. The position map is information about the position of each pixel in the captured image. The state map 402 and the position map 403 are maps that indicate (Z, F, D) and (X, Y), respectively, which correspond to the defocus blur acting on the acquired training image. (X, Y) are the coordinates (horizontal and vertical directions) of the image plane shown in FIG. 12, and correspond to the image height and azimuth in polar coordinate representation. In this embodiment, the coordinates (X, Y) have the optical axis of the optical system 302a as their origin.

[0054] 12 is a diagram showing the relationship between the image circle 501 of the optical system 302a, the first effective pixel area 502 and the second effective pixel area 503 of the image sensor 302b, and the coordinates (X, Y). The size of the image sensor 302b varies depending on the type of image sensor 302. As a result, there are types of image sensor 302 that have the first effective pixel area 502 and types that have the second effective pixel area 503. Of the image sensors 302 that can be connected to the optical system 302a, the image sensor 302 that has the largest size image sensor 302b has the first effective pixel area 502.

[0055] The position map 403 in FIG. 9 is generated based on (x, y) obtained by normalizing the coordinates (X, Y). Normalization is performed by dividing (X, Y) by a length (radius of the image circle) 511 based on the image circle 501 of the optical system 302a. Alternatively, normalization may be performed by dividing X by a horizontal length 512 from the origin to the first effective pixel area and Y by a vertical length 513 from the origin to the first effective pixel area. If (X, Y) were normalized so that the edge of the captured image always equals 1, images captured with different sizes of image sensor 302b would indicate different positions (X, Y) even if the values ​​of (x, y) are the same, and the correspondence between (x, y) and blur cannot be uniquely determined. This results in a decrease in the accuracy of distance information estimation. The position map 403 is a two-channel map having (x, y) values ​​as channel components. Note that polar coordinates may be used for the position map 403, and the method of determining the origin is not limited to that shown in FIG. 12.

[0056] The state map 402 is a three-channel map having normalized (z, f, d) values ​​as channel components. In other words, in this embodiment, the state map 402 has numerical values ​​indicating at least two of the optical system's focal length, aperture value, or focus distance as elements of different channels. The number of elements (number of pixels) per channel of the training image 401, state map 402, and position map 403 is equal. Note that the configuration of the position map 403 and state map 402 is not limited to this. The position map may be expressed in one channel by dividing the first effective pixel region 502 into multiple subregions and assigning a numerical value to each subregion. Similarly, (Z, F, D) may also be divided into multiple subregions in a three-dimensional space with each axis as an axis, and assigned a numerical value, and the state map may be expressed in one channel. The training image 401, state map 402, and position map 403 are concatenated in a specified order in the channel direction by the concatenation layer 411 in FIG. 9 to generate training input data 404.

[0057] 7, the generation unit 301c inputs the training input data 404 to the CNN 412 to generate an output image 405. Then, in step S103, the update unit 301d updates the weights of the CNN based on the error between the output image and the correct image. Then, in step S104, the update unit 301d determines whether learning is complete. Information about the learned weights is stored in the storage unit 301a.

[0058] 13, which is a flowchart related to generation of an estimated image in relation to estimation of distance information of a captured image (estimation phase) executed by the image estimation device 303. The steps in FIG. 13 are mainly executed by the respective units of the image estimation device 303.

[0059] First, in step S301, the acquisition unit 303c acquires a captured image (or at least a portion thereof). Subsequently, in step S302, the acquisition unit 303c acquires weight information corresponding to the captured image. In this embodiment, weight information for each type of optical system 302a is read out in advance from the storage unit 301a and stored in the storage unit 303a. Therefore, weight information corresponding to the type of optical system 302a used to capture the captured image is acquired from the storage unit 303a. The type of optical system 302a used for capturing is identified, for example, from metadata in the file of the captured image.

[0060] Next, in step S303, the acquisition unit 303c generates a state map and a position map corresponding to the captured image to generate input data. The state map is generated based on the number of pixels in the captured image and information on the state (Z, F, D) of the optical system 302a when the captured image was captured. The number of elements (number of pixels) per channel in the captured image and the state map are equal. (Z, F, D) are identified, for example, from metadata of the captured image. The position map is generated based on the number of pixels in the captured image and information on the position of each pixel in the captured image. The number of elements (number of pixels) per channel in the captured image and the position map are equal. The size of the effective pixel area of ​​the image sensor 302b used to capture the captured image is identified from metadata of the captured image, and a normalized position map is generated using, for example, the length of the image circle of the optical system 302a identified in the same way. As in FIG. 9, the input data is generated by concatenating the captured image, state map, and position map in a specified order in the channel direction. Note that the order of steps S302 and S303 does not matter. Furthermore, a state map and a position map may be generated when a captured image is captured and saved together with the captured image.

[0061] Next, in step S304, the distance estimation unit 303b inputs the input data to the CNN, as in FIG. 9, and generates an estimated image.

[0062] With the above configuration, according to this embodiment, it is possible to realize an image processing system that can reduce the learning load of the machine learning model and the amount of data stored, and estimate distance information with high accuracy from the defocus blur of a captured image.

[0063] Next, preferred conditions for enhancing the effects of this embodiment will be described. The input data preferably also includes information regarding the pixel pitch of the image sensor 302b used to capture the captured image. This enables highly accurate estimation of distance information regardless of the type of image sensor 302b. The degree of pixel aperture degradation and the magnitude of defocus blur for pixels vary depending on the pixel pitch. In the learning phase, information specifying the pixel pitch corresponding to the training image is included in the training input data. For example, a map containing elements of normalized pixel pitch values ​​is included. For normalization, it is preferable to use the largest pixel pitch among multiple types of image sensors 302 as the divisor. In the estimation phase, the accuracy of distance information estimation can be improved by including a similar map in the input data. Such a map is generated based on the number of pixels in the captured image. [Example]

[0064] Next, an image processing system according to a third embodiment of the present invention will be described with reference to Fig. 14 and Fig. 15. Fig. 14 is a block diagram of an image processing system 600 according to this embodiment. Fig. 15 is an external view of the image processing system 600.

[0065] The image processing system 600 includes a learning device 601, a lens device 602, an imaging device 603, a control device (first device) 604, an image estimation device (second device) 605, and networks 606 and 607. The learning device 601 and the image estimation device 605 can communicate with each other via the network 606. The control device 604 and the image estimation device 605 can communicate with each other via the network 607. The learning device 601 and the image estimation device 605 are each, for example, a server. The control device 604 is a device operated by a user, such as a personal computer or a mobile terminal. The learning device 601 includes a storage unit 601a, an acquisition unit 601b, a calculation unit 601c, and an update unit 601d, and learns weights of a machine learning model that estimates distance information from an image captured using the lens device 602 and the imaging device 603. Note that the learning method of this embodiment is the same as that of the first embodiment, and therefore its description will be omitted.

[0066] The imaging device 603 has an imaging element 603a, which photoelectrically converts an optical image formed by the lens device 602 to obtain a captured image. The lens device 602 and the imaging device 603 are detachable and can be combined with each other in a variety of ways. The control device 604 has a communication unit 604a, a storage unit 604b, and a display unit 604c, and controls the processing to be performed on the captured image obtained from the imaging device 603 connected by wire or wirelessly in accordance with a user's operation. Alternatively, the captured image captured by the imaging device 603 may be stored in advance in the storage unit 604b, and the captured image may be read out.

[0067] The image estimation device 605 includes a communication unit 605a, a storage unit 605b, an acquisition unit 605c, and a distance estimation unit 605d. The image estimation device 605 executes distance information estimation processing for a captured image in response to a request from a control device 604 connected via a network 607. The image estimation device 605 acquires learned weight information from a learning device 601 connected via a network 606 when estimating distance information or in advance, and uses the learned weight information to estimate distance information for a captured image. After distance information estimation, the estimated image is transmitted again to the control device 604, stored in the storage unit 604b, and displayed on the display unit 604c. Note that the generation of learning data and weight learning (learning phase) performed by the learning device 601 are the same as those in the first embodiment, and therefore description thereof will be omitted.

[0068] Next, the estimation of distance information (estimation phase) executed by the control device 604 and the image estimation device 605 will be described with reference to Fig. 16. Fig. 16 is a flowchart relating to the generation of an estimated image in this embodiment.

[0069] First, in step S401, the communication unit 604a transmits to the image estimation device 605 a captured image and a request for execution of distance information estimation processing.

[0070] Next, in step S501, the communication unit 605a receives and acquires the captured image and processing request transmitted from the control device 604. Next, in step S502, the acquisition unit 605c acquires information on learned weights corresponding to the captured image from the storage unit 605b. The weight information has been read out in advance from the storage unit 601a and stored in the storage unit 605b.

[0071] Next, in step S503, the acquisition unit 605c acquires information about the state of the optical system corresponding to the captured image and generates input data. Information specifying the type, focal length, aperture value, and focus distance of the imaging optical system 602 when the captured image was captured is acquired from the metadata of the captured image, and a state map (lens state map) is generated as in Fig. 1. The input data is generated by concatenating the captured image and the state map in a predetermined order in the channel direction.

[0072] Next, in step S504, the distance estimation unit 605d inputs the input data to the generator and generates an estimated image by estimating distance information. The generator uses weight information. Next, in step S505, the communication unit 605a transmits the estimated image to the control device 604.

[0073] Next, in step S402, the communication unit 604a acquires the estimated image transmitted from the image estimation device 605.

[0074] With the above configuration, according to this embodiment, it is possible to realize an image processing system that can reduce the learning load of the machine learning model and the amount of data stored, and estimate distance information with high accuracy from the defocus blur of a captured image.

[0075] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0076] According to each embodiment, it is possible to provide an image processing method, program, image processing device, method for manufacturing a trained model, and image processing system that can reduce the learning load and amount of data stored in a machine learning model and estimate distance information with high accuracy from defocus blur in a captured image.

[0077] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0078] 102 Imaging device (image processing device) 123a Acquisition unit (acquisition means) 123b Distance estimation unit (generation means)

Claims

1. A step of acquiring input data including an image obtained by imaging using a first optical system and first information regarding the state of the first optical system; and estimating information about a subject distance in the captured image by inputting the input data into a machine learning model, the machine learning model is a trained model obtained by training using training images, a correct image having information on the subject distance in the training images, and second information on the state of the second optical system corresponding to the training images; the first information and the second information include at least one of a focal length, an aperture value, or a focus distance, the first information is a map that indicates a state of the first optical system and is generated based on the number of pixels of the captured image, An image processing method characterized in that the second information is a map indicating the state of the second optical system generated based on the number of pixels of the training image.

2. 2. The image processing method according to claim 1, wherein the map has numerical values ​​indicating at least one of a focal length, an aperture value, and a focus distance.

3. the map includes a plurality of channels; 2. The image processing method according to claim 1, wherein each of the plurality of channels has a numerical value indicating either the focal length, the aperture value, or the focus distance of the first optical system as a channel element.

4. 4. The image processing method according to claim 3, wherein each element included in one of said plurality of channels has the same numerical value.

5. 5. The image processing method according to claim 1, wherein the input data includes information about the position of each pixel of the captured image.

6. the captured image is acquired by imaging using an imaging element, 6. The image processing method according to claim 1, wherein the input data includes information about a pixel pitch of the image sensor.

7. A program causing a computer to execute the image processing method according to any one of claims 1 to 6.

8. An image processing device that performs learning of a machine learning model that estimates information about a subject distance in an input image, an acquisition means for acquiring input data including a captured image obtained by imaging using a first optical system and first information relating to a state of the first optical system; an estimation means for estimating information about a subject distance in the captured image by inputting the input data into a machine learning model, the machine learning model is a trained model obtained by training using training images, a correct image having information on the subject distance in the training images, and second information on the state of the second optical system corresponding to the training images; the first information and the second information include at least one of a focal length, an aperture value, or a focus distance; the first information is a map that indicates a state of the first optical system and is generated based on the number of pixels of the captured image, The image processing device is characterized in that the second information is a map indicating the state of the second optical system generated based on the number of pixels of the training image.

9. 9. An image processing system comprising the image processing device according to claim 8 and a control device capable of communicating with the image processing device, the control device has a transmission means for transmitting a request for execution of processing on the captured image to the image processing device, The image processing system is characterized in that the image processing device has a means for executing processing on the captured image based on the request.

10. A method for learning a trained model that estimates information about a subject distance in an input image, acquiring training images, a ground truth image having information on the subject distance in the training images, and information on the state of the optical system; and training a machine learning model based on the training images, the ground truth image, and information about the state of the optical system, A learning method characterized in that the information regarding the state of the optical system includes at least one of focal length, aperture value, or focus distance, and is a map showing the state of the optical system generated based on the number of pixels in the training image and the information regarding the state of the optical system.

11. A program causing a computer to execute the learning method according to claim 10.

12. A method for generating a trained model that trains a machine learning model that estimates information about a subject distance in an input image, acquiring training images, a ground truth image having information on the subject distance in the training images, and information on the state of the optical system; and training a machine learning model based on the training images, the ground truth image, and information about the state of the optical system, The information regarding the state of the optical system includes at least one of focal length, aperture value, or focus distance, and is a map showing the state of the optical system generated based on the number of pixels in the training image and the information regarding the state of the optical system.

Images