Learning method, image processing method and image processing device

Abstract

To learn suitable optical characteristics for both of distance estimation and image processing different from the same.SOLUTION: A learning method includes the steps of: generating a first image on the basis of an original image including an object, correct answer distance information and optical characteristics of an optical system; using a first neural network to generate first distance information based on the first image; using a second neural network to generate output data based on the first image, and second distance information relating to the object included in the original image; and implementing learning on the basis of the first distance information, the correct answer distance information, the output data and correct answer data based on the original image. The learning updates a first parameter to determine the optical characteristics, a second parameter to determine the first neural network, and a third parameter to determine the second neural network.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a method for learning optical properties and multiple image processing including distance estimation. [Background technology]

[0002] One method for acquiring distance information to an object is to estimate the distance from a single image using deep learning, but there is a demand for further improvement in the accuracy of this method. Patent Document 1 discloses a method for estimating distance with higher accuracy by simultaneously learning a neural network that estimates the distance based on an image obtained by capturing an image of an object using a monocular camera incorporating a phase mask for controlling optical characteristics, and the shape of the phase mask. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Patent Publication No. 2020 / 0349729 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the method disclosed in Patent Document 1, the phase mask is trained to a shape suitable for distance estimation, so that when an image processing other than distance estimation is applied to a captured image, the phase mask may be trained to a shape that is not suitable for the image processing. For example, when performing blur correction, it is preferable that a high-frequency structure remains in an image to which aberration is imparted by a phase mask. On the other hand, when performing distance estimation, it is preferable that the optical characteristics change more greatly depending on the distance. As such, since the optical characteristics suitable for each process are different, there is no guarantee that a high-frequency structure remains in learning that takes into account only distance estimation, and blur correction may become difficult.

[0005] The present invention provides a learning method, an image processing method, and an image processing device that are capable of learning optical characteristics that are preferable for both distance estimation and image processing different from the distance estimation. [Means for solving the problem]

[0006] A learning method according to one aspect of the present invention includes the steps of: generating a first image based on an original image including an object, ground-truth distance information, and optical characteristics of an optical system; generating first distance information based on the first image using a first neural network; generating output data based on the first image and second distance information on the object included in the original image using a second neural network; and learning based on the first distance information, ground-truth distance information, the output data, and ground-truth data based on the original image. In the learning, a first parameter that determines the optical characteristics, a second parameter that determines the first neural network, and a third parameter that determines the second neural network are updated.

[0007] Furthermore, an image processing method as another aspect of the present invention is characterized by having the steps of acquiring an captured image based on first parameters updated by the above-mentioned learning method, generating third distance information based on the captured image and second parameters updated by the above-mentioned learning method, and generating estimated data based on the captured image, the third distance information, and the third parameters updated by the above-mentioned learning method.

[0008] A learning device according to another aspect of the present invention includes a first generating means for generating a first image based on an original image including an object, ground-truth distance information, and optical characteristics of an optical system, a second generating means for generating first distance information based on the first image using a first neural network, a third generating means for generating output data based on the first image and second distance information regarding an object included in the original image using a second neural network, and a learning means for performing learning based on the first distance information, ground-truth distance information, the output data, and ground-truth data based on the original image. The learning means is characterized in that, in learning, it updates a first parameter that determines the optical characteristics, a second parameter that determines the first neural network, and a third parameter that determines the second neural network.

[0009] Another aspect of the present invention is an image processing device characterized by having an acquisition means for acquiring an captured image based on first parameters updated by the above-mentioned learning device, a distance generation means for generating third distance information based on the captured image and the second parameters updated by the learning device, and an estimation means for generating estimated data based on the captured image, the third distance information, and the third parameters updated by the learning device. Effect of the Invention

[0010] According to the present invention, it is possible to learn optical characteristics that are preferable for both distance estimation and image processing different from the distance estimation. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing a configuration of an image processing system according to a first embodiment. [Diagram 2] FIG. 1 is an external view of an image processing system according to a first embodiment. [Diagram 3] 5 is a flowchart showing a learning process of optical characteristics, distance estimation, and blur correction in the first embodiment. [Figure 4] FIG. 4 is a diagram showing a learning flow of the distance estimation neural network in the first embodiment. [Diagram 5] FIG. 4 is a diagram showing a learning flow of the image processing neural network in the first embodiment. [Figure 6] 6 is a flowchart showing a process of generating estimated distance information and estimated data in the first embodiment. [Figure 7] 10 is a flowchart showing a learning process of optical characteristics, distance estimation, and blur correction in the second embodiment. [Figure 8] FIG. 11 is a block diagram showing the configuration of an image processing system according to a third embodiment. [Figure 9] FIG. 11 is an external view of an image processing system according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0013] First, before describing specific examples, an overview of the examples will be described. The learning method of the examples learns distance estimation for an object, image processing different from distance estimation (blur correction, object detection, blur shape conversion, etc.), and optical characteristics of an optical system that achieves a predetermined balance between the accuracy of these.

[0014] In the learning method, in the first step, an original image (an image obtained by imaging or the like) including an object and distance information (second distance information such as ground truth distance information) related to the object are obtained. Then, an object space is defined in which the object included in the original image is located at a distance represented by the distance information. Furthermore, an imaging simulation, which will be described later, is performed on the object space to generate a blurred image (first image) corresponding to a captured image obtained by imaging the object space using an optical system. The optical characteristics of the optical system at this time are determined based on the optical parameters (first parameters).

[0015] Next, in the second step, the generated blurred image is input to a neural network that performs distance estimation (a first neural network: hereinafter, referred to as a distance estimation neural network). The distance estimation neural network generates output distance information (first distance information) based on the blurred image.

[0016] Next, in the third step, the blurred image and the input distance information are input to a neural network (second neural network: hereinafter referred to as image processing neural network) that performs image processing different from distance estimation. The input distance information here is input to perform image processing based on distance, and may be the same information as the output distance information or correct distance information described above, or may be information obtained by performing processing such as noise removal on the output distance information or correct distance information. The image processing neural network generates output data based on the blurred image and the input distance information.

[0017] Next, in the fourth step, the optical parameters, the parameters determining the distance estimation neural network (second parameters), and the parameters determining the image processing neural network (third parameters) are updated based on the output distance information, the correct distance information, the output data, and the correct data (correct image, etc.) based on the original image.

[0018] According to the above learning method, it is possible to learn optical characteristics that are suitable for both distance estimation and image processing other than the distance estimation. For example, when the image processing other than the distance estimation is blur correction, it is possible to learn optical characteristics that change the shape of the PSF (Point Spread Function) according to the distance while leaving the high frequency components of the MTF (Modulation Transfer Function).

[0019] If the optical characteristics are learned to improve only the accuracy of distance estimation without using this learning method, the optical characteristics are learned to change more significantly depending on the distance. This is because the blur of the blurred image has a different shape depending on the distance, and the distance can be estimated with higher accuracy using the shape of the blur as a clue. On the other hand, assuming that a good ornamental image is generated by performing blur correction or image processing such as object recognition, it is preferable that a high-frequency structure remains in the image. As described above, if the optical characteristics are learned to improve only the accuracy of distance estimation, the optical characteristics are learned to increase the change in blur, so that the image quality of the image obtained by imaging is reduced and blur correction is likely to be difficult.

[0020] In the learning method of the embodiment, the optical parameters that determine the optical characteristics of the optical system are updated using the outputs of the distance estimation neural network and the image processing neural network. Therefore, when the image processing different from the distance estimation is blur correction, it is possible to learn optical characteristics that are favorable for both distance estimation and blur correction, such that the MTF has a value equal to or greater than a predetermined value up to high frequencies while the optical characteristics change significantly according to the distance.

[0021] Furthermore, according to the learning method of the embodiment, it is possible to learn optical characteristics that can control the balance between the accuracy of distance estimation and the accuracy of image processing other than distance estimation. Note that, in the embodiment, distance information is input to the image processing neural network in order to perform image processing with higher accuracy based on the input distance information, since the optical characteristics change depending on the distance.

[0022] Here, definitions of terms used in the embodiments will be explained. The optical characteristics of the optical system are any of the above-mentioned point spread function (PSF), optical transfer function (OTF), the above-mentioned MTF which is the amplitude component of the OTF, and PTF (Phase Transfer Function) which is the phase component of the OTF, and indicate the blur of the image due to the aberration or diffraction of the optical system. The optical characteristics may be characteristics including the blur of the image according to the characteristics of the imaging element.

[0023] Distance estimation is a process of generating (estimating) distance information about an object contained in a captured image from the captured image. This distance information may be any information that indicates the distance about an object, such as information that indicates the distance from an optical system to the object, information that indicates the relative distance between multiple objects, or information that indicates the distance of an image conjugate to the object.

[0024] Blur correction is a process for correcting blur caused by aberration or diffraction of the optical system contained in a captured image, and generating a sharpened image by reducing the effect of blur.

[0025] [Example 1] The first embodiment aims to learn optical characteristics that are favorable for both distance estimation and blur correction, and to generate estimated distance information for an image and a blur-corrected image using the learned parameters.

[0026] Fig. 1 shows the configuration of an image processing system in Example 1. Fig. 2 shows the external appearance of an image processing system 100. The image processing system 100 includes a learning device (image processing device) 101, an imaging device 102, an image estimation device 103, a display device 104, a recording medium 105, an output device 106, and a network 107. The learning device 101 and the image estimation device 103 are connected to each other via the wired or wireless network 107.

[0027] The learning device 101 includes a storage unit 101a, an acquisition unit 101b, a generation unit (first to third generation means) 101c, and an update unit (learning means) 101d. The learning device 101 performs learning of optical characteristics, distance estimation, and blur correction. The learning process performed by the learning device 101 will be described later in detail.

[0028] The imaging device 102 has an optical system 102a and an imaging element 102b. The optical system 102a is designed based on the optical characteristics learned by the learning device 101, and collects light incident from the object space to the imaging device 102. The imaging element 102b receives (photoelectrically converts) an object image as an optical image formed by the optical system 102a to generate a captured image. The imaging element 102b is a photoelectric conversion element such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal-Oxide Semiconductor) sensor. The captured image generated by the imaging device 102 contains blur due to aberration and diffraction of the optical system 102a and noise due to the imaging element 102b.

[0029] The image estimation device 103 includes a storage unit 103a, an acquisition unit (acquisition means) 103b, a distance estimation unit (distance generation means) 103c, and a blur correction unit (estimation means) 103d. The image estimation device 103 acquires a captured image from the imaging device 102, and generates estimated distance information and a blur-corrected image (estimated data).

[0030] The distance estimation unit 103c performs processing using a distance estimation neural network. Information on the parameters of the distance estimation neural network is read from the storage unit 103a. The parameters of the distance estimation neural network are learned by the learning device 101. The image estimation device 103 reads information on the parameters of the distance estimation neural network from the storage unit 101a via the network 107 in advance and stores the information in the storage unit 103a. The stored information may be the numerical values ​​of the parameters of the distance estimation neural network themselves, or may be information in a format in which the parameters are encoded. Details of learning the parameters of the distance estimation neural network and the distance estimation process using the parameters of the distance estimation neural network will be described later.

[0031] The blur correction unit 103d performs processing using an image processing (blur correction) neural network. Parameters of the image processing neural network are read from the storage unit 103a. The parameters of the image processing neural network are learned by the learning device 101. The image estimation device 103 reads information on the parameters of the image processing neural network from the storage unit 101a via the network 107 in advance and stores it in the storage unit 103a. The stored information may be the numerical values ​​of the parameters of the image processing neural network themselves, or may be information in an encoded format. The learning of the parameters of the image processing neural network and the blur correction processing using the parameters of the image processing neural network will be described in detail later.

[0032] The estimated distance information and the blur-corrected image are output to at least one of a display device 104, a recording medium 105, and an output device 106. The display device 104 is a liquid crystal display, a projector, or the like. A user can perform editing work and the like while checking an image being processed via the display device 104. The recording medium 105 is a semiconductor memory, a hard disk, a server on a network, or the like. The output device 106 is a printer, or the like. The image estimation device 103 has a function of performing development processing and other image processing as necessary.

[0033] Next, the learning process (learning method) of optical characteristics, distance estimation, and blur correction executed by the learning device 101 in this embodiment will be described with reference to Figures 3, 4, and 5. Figure 3 is a flowchart showing the learning process, and each step (process) is executed according to a program by the acquisition unit 101b, generation unit 101c, or update unit 101d of the learning device 101 configured by a computer. Figure 4 shows the learning flow of the distance estimation neural network, and Figure 5 shows the learning flow of the image processing neural network.

[0034] In step S101 of FIG. 3, the acquisition unit 101b acquires an original image, correct distance information, and optical parameters from the storage unit 101a.

[0035] The original image is a plurality of images, and may be a real-life image obtained by actual shooting or an image generated by CG (Computer Graphics). The original image is a high-resolution (high-quality) image with little blurring due to aberration or diffraction. When using a real-life image as the original image, it is preferable that the defocus blur according to the distance is small. In addition, since the real-life image already has blurring due to aberration or diffraction, it can be used as a high-resolution (high-quality) original image by reducing the effect of blurring. Note that if the real-life image contains sufficient high-frequency components, it is not necessary to reduce the image. In addition, in order to improve the robustness of the neural network to be trained, it is preferable that the original image is an image containing various objects. For example, it is preferable that the original image has edges, textures, gradations, flat parts, etc. with various intensities and directions.

[0036] The correct distance information represents distance information related to an object included in the original image, and in this embodiment is a two-dimensional map having distance information for each pixel. When a CG image is used as the original image, the correct distance information is calculated based on the CG image. When a real image is used as the original image, the correct distance information may be generated by distance measurement using LiDAR (Light Detection and Ranging) or a stereo camera. The original image and the correct distance information may each be an image obtained by trimming one image to a predetermined number of pixels (for example, 64 x 64 pixels).

[0037] In this embodiment, the optical parameters are parameters that determine the wavefront aberration generated by the wavefront modulation element included in the optical system. For example, a transparent plate having a different thickness for each position where a light ray passes through is incorporated into the optical system as a wavefront modulation element, and the thickness for each position is set as the optical parameter. However, the optical parameters may be other parameters that determine the optical characteristics of the optical system. For example, the wavefront modulation element may be a transparent plate having a different refractive index for each position, and in this case, the optical parameter may be the refractive index for each position of the wavefront modulation element. The wavefront modulation element may also be an aspheric lens, and the design parameters of the aspheric lens (such as thickness and curvature, or coefficients when expressing the aspheric surface as a polynomial) may be set as the optical parameters.

[0038] The optical parameters are information that are updated by learning, and in the first learning, they are represented by an arbitrary array or a random number array, etc. In this embodiment, for learning the optical parameters, mini-batch learning is used, in which an image group is divided into a plurality of groups and the optical parameters are updated for each group. For this reason, in step S101, a plurality of sets of original images and ground-truth distance information are obtained. However, online learning, in which an image is input and updated one by one, or batch learning, in which parameters are updated using all images at once, may also be used.

[0039] Next, in step S102, the generating unit 101c generates a blurred image by imaging simulation. The imaging simulation is performed by the following method. First, the generating unit 101c calculates a PSF by an optical system having optical characteristics determined by optical parameters. At this time, since the optical characteristics differ depending on the distance of the subject, a plurality of PSFs that differ depending on the distance are obtained. Next, the generating unit 101c adds blur to each pixel of the original image based on the PSF corresponding to the correct distance information, thereby generating a blurred image corresponding to the captured image obtained by capturing an object space using the optical system.

[0040] At this time, it is desirable that the shape of the wavefront aberration generated by the optical system is a shape that is non-rotationally symmetrical with respect to the optical axis. As a result, the shape of the blur added to the image can be changed not only in the radial direction but also in the azimuth direction according to the distance of the object. As a result, it is possible to change the size of the blur for each distance of the object while also changing the shape of the blur. Therefore, it is possible to learn optical characteristics with a higher degree of freedom, and a blur that is more preferable for both distance estimation and blur correction can be added to the blurred image. As a result, distance estimation and blur correction can be performed with higher accuracy. In addition, noise may be added to the blurred image. Since the blurred image corresponds to the captured image at the time of estimation, learning can be performed with higher accuracy by adding noise expected as the captured image.

[0041] Next, in step S103, the acquisition unit 101b acquires parameters of the distance estimation neural network from the storage unit 101a. Here, the parameters of the distance estimation neural network include the components and biases of the filters of each layer. The parameters of the distance estimation neural network are information that is updated by learning, and are represented by an arbitrary sequence or a random number sequence at the first learning.

[0042] Next, in step S104, the generating unit 101c inputs the blurred image to the distance estimation neural network to generate output distance information. In this embodiment, the output distance information is generated by the distance estimation neural network based on an image of a single viewpoint. The output distance information may be generated based on images of multiple viewpoints, but in that case, the blurred images must be multiple images corresponding to images captured from different viewpoints. In this embodiment, the distance information can be estimated with high accuracy by learning optical characteristics, so that it is more preferable to use an image of a single viewpoint because the imaging device 102 can be made smaller and the processing load can be reduced.

[0043] In this embodiment, a CNN (Convolutional Neural Network) having multiple convolutional layers and deconvolutional layers is used as the distance estimation neural network. Fig. 4 shows a flow from inputting a blurred image 301 to a neural network to calculating an error between the input image and an output value 302 generated. In Fig. 4, CN represents a convolutional layer, DC represents a deconvolutional layer, and PL represents a pooling layer. In the pooling layer PL, a process of locally extracting a representative value (such as a maximum value or an average value) is performed to reduce the resolution of the image.

[0044] In this embodiment, Max Pooling is used to extract the pixel with the maximum value for each predetermined region. In both the convolution layer CN and the deconvolution layer DC, the input is convolved with a filter, the sum of the convolution result and the bias is calculated, and the calculation result is nonlinearly transformed by the activation function. The initial values ​​of each filter component and bias are arbitrary, and are determined by random numbers in this embodiment. As the activation function, ReLU (Rectified Linear Unit), sigmoid function, etc. can be used.

[0045] The output of each layer except the final layer is called a feature map. Skip connections 311, 312, 313, and 314 combine feature maps output from discontinuous layers. The combination of feature maps may be done by summing elements or by concatenation in the channel direction. In this embodiment, concatenation in the channel direction is adopted. However, the distance estimation neural network is not limited to CNN, and other network structures such as Transformer may be used.

[0046] Next, in step S105, the acquisition unit 101b acquires parameters of the image processing neural network and input distance information to be input to the image processing neural network. The parameters of the image processing neural network include components and biases of filters of each layer. The parameters of the image processing neural network are information that is updated by learning, and are expressed by an arbitrary array or a random number array or the like in the first learning. The input distance information indicates the distance of an object included in the blurred image, and may be the same information as the correct distance information or the output distance information generated in step S104 as described above, or may be information obtained by performing image processing such as noise removal on the correct distance information or the output distance information. By learning using the output distance information that has been subjected to noise removal, it is possible to suppress the influence of noise due to erroneous estimation and perform blur correction with high accuracy.

[0047] More preferably, the distance information input to the image processing neural network is output distance information generated by a distance estimation neural network. When the correct distance information is input to train the image processing neural network, the blur correction is learned assuming that the distance information input to the image processing neural network is free of error. However, at the time of estimation described below, the estimated distance information generated by the distance estimation neural network is input to the image processing neural network, so that distance information including an estimated error is input. Therefore, at the time of estimation, a correction according to a distance different from the actual distance is performed, and the image quality may deteriorate due to the error of the estimated distance information. On the other hand, when the image processing neural network is trained by inputting the output distance information, the distance information including an error is input to the image processing neural network even at the time of training. Therefore, even if the distance information includes an error, the image processing neural network can be trained to suppress deterioration of the image quality due to the error of the estimated distance information.

[0048] Next, in step S106, the generating unit 101c inputs the blurred image and the input distance information to the image processing neural network to generate an output image (output data). The image processing neural network has a concatenated layer (not shown) in the first layer. The concatenated layer concatenates the blurred image and the input distance information in the channel direction. At this time, other information may be concatenated. For example, information on the focal length, F value, focal distance, and sampling pitch of the image sensor 102b of the optical system 102a may be concatenated. In addition, when the blurred image contains noise added in the process of generating the learning data, information representing the strength of the added noise may be concatenated. The information concatenated in this step is also input to the neural network during estimation, which will be described later.

[0049] In this embodiment, the image processing neural network uses a CNN having a plurality of convolution layers after a concatenation layer. FIG. 5 shows a flow from inputting a blurred image 401 to a neural network to calculating an error with a generated output value 402. CN in FIG. 5 represents a convolution layer. In the convolution layer CN, a filter is convolved with respect to the input, the sum of the convolution result and the bias is calculated, and the calculation result is nonlinearly transformed by an activation function. The initial values ​​of each component of the filter and the bias are arbitrary, and are determined by random numbers in this embodiment. Here again, ReLU, sigmoid function, etc. can be used as the activation function.

[0050] As with the distance estimation neural network, the output of each layer except the final layer is called a feature map. Skip connections 411, 412, 413, 414, and 415 combine feature maps output from discontinuous layers. The combination of feature maps may be done by summing elements or by concatenation in the channel direction. In this embodiment, the summing of elements is used. However, the image processing neural network is not limited to the CNN used in this embodiment, and other network structures such as Transformer may be used.

[0051] In this embodiment, in order to perform blur correction, the output image generated by the image processing neural network is an image in which the blurred image has been sharpened.

[0052] Next, in step S107, the updating unit 101d updates the optical parameters and the parameters of each neural network based on the output distance information and the output image from the distance estimation and image processing neural network and the correct value (correct distance information and a correct image that is identical to the original image). In this embodiment, the backpropagation method is used to update the parameters. However, other methods may be used. In this embodiment, the optical parameters, the parameters of the distance estimation neural network, and the parameters of the image processing neural network are updated as follows.

[0053] The optical parameters are updated so as to reduce both the error between the output distance information and the correct distance information and the error between the output image and the original image.

[0054] The parameters of the distance estimation neural network are updated so as to reduce at least one of the error between the output distance information and the correct distance information and the error between the output image and the original image.

[0055] The parameters of the image processing neural network are updated to reduce the error between the output image and the original image.

[0056] In this embodiment, the loss function is defined as the mean square error L RMSE and the mean square error of the gradient along each of the vertical and horizontal directions of the pixel L GRAD It is defined using L RMSE and L GRAD are expressed by the following equations (1) and (2), respectively.

[0057]

number

[0058]

number

[0059] In equations (1) and (2), t is the correct value, t^ is the output value of the neural network, N is the number of pixels, ∂ / ∂x is the gradient along the horizontal direction of the pixel, and ∂ / ∂y is the gradient along the vertical direction of the pixel. JPEG2025063745000004.jpg1418

[0060] is the L2 norm.

[0061] In this embodiment, L shown in formula (3) RMSE and L GRAD The error function is the weighted sum L of the above. The weights α1 and β1 are set to arbitrary values ​​based on the user's selection.

[0062]

number

[0063] The optical parameters are calculated by applying the error function shown in Eq. (3) to the distance estimation neural network. DEPTH and L applied to image processing neural networks RGB The weights α2 and β2 are set to arbitrary values ​​based on the user's selection.

[0064]

number

[0065] In this way, by controlling the weights of the error function based on the output of distance estimation and the error function based on the output of blur correction and performing learning, it becomes possible to learn optical characteristics that control the balance between the accuracy of distance estimation and blur correction. For example, by making α2 larger than β2, learning is performed with an emphasis on the accuracy of distance estimation.

[0066] However, the method of updating each parameter is not limited to the method of this embodiment. The error function is also not limited to the definition of this embodiment, and a known method such as the L1 norm may be used as the error function.

[0067] Next, in step S108, the update unit 101d determines whether or not learning of all parameters is complete. Completion of learning can be determined by, for example, whether the number of iterations of learning (updating of parameters) reaches a preset value or whether the amount of change in parameters at the time of updating is smaller than a preset value. If it is determined that learning is not complete, the process returns to step S101, and a new original image, correct distance information, and updated optical parameters are obtained. If it is determined that learning is complete, the learning device 101 (update unit 101d) ends learning, and stores information on the learned parameters (optical parameters, parameters of the distance estimation neural network, and parameters of the image processing neural network) in the storage unit 101a. This ends the learning process.

[0068] The above learning process makes it possible to learn optical characteristics, distance estimation, and blur correction that are favorable for both distance estimation accuracy and blur correction accuracy, thereby making it possible to obtain an image in which blur is favorably corrected while achieving high distance estimation accuracy.

[0069] Next, a generation process (estimation method) of estimated distance information and a blur-corrected image (estimated data) executed by the imaging device 102 and the image estimation device 103 will be described with reference to the flowchart in Fig. 6. Each step in Fig. 6 is executed according to a program by the acquisition unit 103b, the distance estimation unit 103c, or the blur correction unit 103d of the image estimation device 103, which is a computer.

[0070] First, in step S201, the acquisition unit 103b acquires a captured image, trained parameters of a distance estimation neural network, and trained parameters of an image processing neural network from the storage unit 103a. The captured image is an image generated by imaging performed by the imaging device 102 having an imaging optical system (optical system 102a) designed based on optical characteristics learned by the learning device 101. The captured image may be expressed in grayscale or may have multiple channel components. In addition, only a part of the image generated by imaging by the imaging device 102 may be used as the captured image to be acquired. Note that the parameters of the distance estimation neural network and the parameters of the image processing neural network are read out from the storage unit 101a in advance and stored in the storage unit 103a.

[0071] Next, in step S202, the distance estimation unit 103c inputs the captured image to a distance estimation neural network to generate estimated distance information (third distance information). The distance estimation neural network here is determined by parameters of a trained distance estimation neural network.

[0072] Next, in step S203, the blur correction unit 103d inputs the captured image and the estimated distance information to an image processing neural network to generate a blur-corrected image. The blur-corrected image is an image in which blur caused by optical characteristics in the captured image has been corrected. The image processing neural network here is determined by the parameters of a trained image processing neural network. This completes the generation process.

[0073] In this embodiment, the case where the image processing different from the distance estimation is blur correction has been described, but other image processing may be used. For example, it may be object detection, or it may be blur shape conversion, which is a process of converting blur due to optical characteristics in the same way as blur correction.

[0074] Object detection is image processing that detects what is in an image and outputs estimated data including information indicating the position of the detected object and information indicating what the object is. In object detection, a neural network that performs image processing (object detection) different from distance estimation is used as a second neural network. Unlike distance estimation, learning is performed using information indicating the position of the detected object and information indicating what the object is. The output data is an estimate of this information, and the correct answer data indicates the correct answer to this information. Furthermore, the estimated data at the time of estimation is not an image, but information indicating the position of the detected object and information indicating what the object is.

[0075] The blur shape conversion is an image processing to convert the shape of the defocus blur caused by the optical system into a different shape. For example, it refers to an image processing to convert a PSF shape having a peak at the outer edge like a double line blur into a uniform PSF shape like a circular blur. In this embodiment, in order to learn optical parameters preferable for distance estimation, the blur shape conversion refers to an image processing to convert the blur shape caused by the optical system based on the optical parameters into a shape desired by the user. If necessary, image processing to correct the blur well near the focal plane where the defocus blur is small may be performed at the same time. In the blur shape conversion, an image blurred to a predetermined shape is generated by simulation, and the generated image is used as the correct image (correct data).

[0076] In addition, a neural network that performs image processing other than distance estimation outputs an image (output data) in which the blur shape of the input image is converted, and learns based on the error between this image and a correct image that has been generated in advance by simulation.

[0077] When the image processing using the neural network is object detection, it is preferable that not only the low frequency features of the subject but also the high frequency structure remain in the image to be processed, as in the case of blur correction. In the blur shape conversion, the correct image is an image blurred to a shape desired by the user. Since the frequency components contained in this correct image differ depending on the amount of blur, it is preferable that the frequency components contained in the correct image remain in the image to be processed.

[0078] In addition, when the blur amount of the correct image changes depending on the distance of the subject, the preferred optical characteristics change depending on the distance of the subject. In this way, the optical characteristics preferred for performing both object detection and blur shape conversion are different from the optical characteristics preferred for performing distance estimation. Therefore, by using a learning method similar to that of this embodiment, it is possible to learn optical characteristics preferred for both distance estimation and object detection or blur shape conversion, and it becomes possible to perform each process with higher accuracy.

[0079] In addition, the learning process in this embodiment is performed in the order of generating output distance information and then generating an output image, but the order is not limited to this. For example, if the input distance information input to the image processing neural network is the same as the correct distance information, it is also possible to generate the output distance information after generating the output image. In addition, in the update process during learning in this embodiment, after the generation process of each neural network is completed, the optical parameters, the parameters of the distance estimation neural network, and the parameters of the image processing neural network are updated simultaneously, but the order is not limited to this. For example, learning may be performed by alternately repeating the learning of the distance estimation neural network and the learning of the image processing neural network a predetermined number of times.

[0080] In this embodiment, learning may be performed in advance for each pixel pitch of the image sensor, and information on the pixel pitch of the image sensor may be input during estimation to switch the learned parameters to be used. The information on the pixel pitch may be information indicating the pixel pitch itself, or may be information that can be converted into the pixel pitch.

[0081] The degree of blur in a captured image varies depending on the pixel pitch of the image sensor. Therefore, by switching the learned parameters for each pixel pitch, it is possible to use the optical characteristics and distance estimation neural network learned according to a specific pixel pitch, and it is possible to estimate distance information with higher accuracy.

[0082] [Example 2] Next, a second embodiment will be described. In this embodiment, in learning optical characteristics that are favorable for both distance estimation and blur correction, learning is performed by adjusting (controlling) the strength of sharpening for blur correction based on distance information. The strength of sharpening is an index that indicates the degree of sharpness of the blur after sharpening that is not dependent on the blur before sharpening, and may be determined based on, for example, the value of the MTF of a predetermined frequency of the image after correction, or based on the target MTF after correction. It may also be determined based on the MTF of the correct image used for learning. Furthermore, as described later, when blur is further added to the correct image used for learning, it may be determined based on the MTF of the blur to be added.

[0083] In this embodiment, the sharpening strength is adjusted to further improve, for example, the accuracy of correction of the focal plane and the accuracy of distance estimation compared to embodiment 1. The sharpening strength is adjusted by varying the accuracy target after correction for each distance.

[0084] The configuration of the image processing system in this embodiment is the same as that of the image processing system 100 in the first embodiment shown in FIGS. 1 and 2, and the same components as those in the first embodiment are denoted by the same reference numerals as those in the first embodiment.

[0085] The flowchart in Fig. 7 shows a learning process (learning method) of blur correction in which optical parameters, parameters of a distance estimation neural network, and sharpening strength are adjusted, which is executed by the learning device 101 in this embodiment. Each step in Fig. 7 is executed by the acquisition unit 101b, the generation unit 101c, or the update unit 101d of the learning device 101. Moreover, the processes from step S301 to step S306 are the same as the processes from step S101 to step S106 in the first embodiment, and therefore the description thereof will be omitted here.

[0086] In step S307, which is the next step after step S306, the generating unit 101c generates a correct image (correct data). In the first embodiment, the correct image is the same as the original image that represents the object before imaging. This corresponds to a correction aimed at a deep focus image in which blurring due to aberration and diffraction is corrected regardless of correct distance information in all areas of the captured image and there is no defocus blur.

[0087] In contrast, in this embodiment, the correct image is an image to which defocus blur due to an ideal optical system is added when a predetermined object distance is set as the focal distance. In other words, the correct image is an image to which defocus blur, the magnitude of which changes according to the distance from the focal plane, is added to the original image. This makes it possible to vary (control) the strength of sharpening based on the correct distance information. As a result, a good blur-corrected image in which blur due to aberration and diffraction is corrected for all object distances including the focal plane is obtained, while the blur due to defocus is corrected according to the distance. At this time, for an object at a distance away from the focal plane, the high-frequency components contained in the correct image become smaller. Therefore, the need for high-frequency components is reduced for blur at a distance away from the focal plane.

[0088] In this way, by changing the sharpening strength according to the object distance, the degree of freedom of the optical characteristics can be increased compared to Example 1, and it becomes easier to achieve both distance estimation and blur correction. As a result, the accuracy of distance estimation and the accuracy of correction of blur due to aberration can be further improved, and a sharper image can be obtained than in Example 1, especially on the focal plane.

[0089] In this embodiment, the correct image is an image to which defocus blur due to an ideal optical system is added, but the correct image is not limited to this. As a method for varying the sharpening strength depending on the distance, for example, a correct image may be generated by adding a uniform blur to an area that is a predetermined distance or more away from the focal plane.

[0090] The process in step S308 is the same as step S107 in the first embodiment, and therefore a description thereof will be omitted here.

[0091] In this embodiment, each parameter is updated in the same manner as in the first embodiment, but the correct value of the blur-corrected image is different from that in the first embodiment. In the first embodiment, the original image is used as the correct value of the blur-corrected image, but in this embodiment, the above-mentioned correct image is used. Since the correct image is an image that is more degraded than the original image, the strength of sharpening is weakened.

[0092] The generation process (estimation method) of the estimated distance information and the blur-corrected image, which is executed by the imaging device 102 and the image estimation device 103 in this embodiment, is the same as in the first embodiment.

[0093] In this embodiment, as the process during learning, a correct image is generated based on distance information, and the correct value of the image processing neural network is changed to adjust the sharpening strength, but the process during learning is not limited to this. For example, for an area that is a predetermined distance or more away from the focus position during estimation, the sharpening strength may be adjusted by taking a weighted average of images before and after correction and using it as a corrected image. In this case, since the process during learning is the same as in Example 1, the optical characteristics and each neural network to be learned are also the same as in Example 1. In addition, blur correction may not be performed on parts of the captured image where distance estimation is difficult, such as low-contrast areas.

[0094] According to this embodiment, the strength of the sharpening of the blur correction can be adjusted according to the distance of the object. For example, by weakening the strength of the sharpening in the region other than the in-focus region in the output image compared to the strength of the sharpening in the in-focus region near the in-focus plane, it is possible to learn optical characteristics that emphasize blur correction and distance estimation in the in-focus plane. As a result, compared to the first embodiment, it is possible to learn optical characteristics that further improve the blur correction accuracy and distance estimation accuracy near the in-focus plane.

[0095] [Example 3] Next, a description will be given of a third embodiment. Fig. 8 shows the configuration of an image processing system 200 in this embodiment, and Fig. 9 shows the external appearance of the image processing system 200.

[0096] In the first embodiment, the case has been described where the image estimation device 103, the display device 104, and the recording medium 105 are located outside the imaging device 102 as shown in Fig. 1. In contrast, in the present embodiment, an image estimation unit 223, a display unit 226, and a recording medium 225, which correspond to the image estimation device 103, the display device 104, and the recording medium 105, respectively, may be located inside the imaging device 202 as shown in Figs.

[0097] The learning device (image processing device) 201 has a storage unit 211, an acquisition unit 212, a generation unit 213, and an update unit 214, and learns optical characteristics, distance estimation, and blur correction. The learning process performed by the learning device 201 as a computer according to a program is the same as that shown in FIG. 3 of the first embodiment.

[0098] The imaging device 202 includes an optical system 221, an imaging element 222, an image estimation unit 223, a storage unit 224, a recording medium 225, a display unit 226, and a system controller 227. The optical system 221 is designed based on the optical characteristics learned by the learning device 201, similar to the first embodiment.

[0099] The image estimation unit 223 has an acquisition unit 223a, a distance estimation unit 223b, and a blur correction unit 223c, and generates estimated distance information and a blur-corrected image (estimated data). A method for generating the estimated distance information and the blur-corrected image will be described later. The estimated distance information and the blur-corrected image are output to at least one of the display unit 226 and the recording medium 225. The operation of the imaging device 202 is controlled by a system controller 227. The learning device 201 and the imaging device 202 are connected to each other via a wired or wireless network 203.

[0100] Using the flowchart in Figure 6, we will explain the generation process (estimation method) of estimated distance information and blur-corrected image executed according to a program by the imaging device 202 in this embodiment (the acquisition unit 223a, distance estimation unit 223b or blur correction unit 223c of the image estimation unit 223, which is a computer).

[0101] First, in step S201, the acquisition unit 223a acquires a captured image, trained parameters of a distance estimation neural network, and trained parameters of an image processing neural network from the storage unit 224. The captured image is an image generated by capturing an optical image formed by an optical system 221 designed based on optical characteristics learned by the learning device 201, using an image sensor 222, and stored in the storage unit 224. The captured image may be expressed in grayscale or may have multiple channel components. The captured image to be acquired may be a part of an image generated by capturing. The parameters of the distance estimation neural network and the parameters of the image processing neural network are read out from the storage unit 211 in advance and stored in the storage unit 224.

[0102] Next, in step S202, the distance estimation unit 223b inputs the captured image to a distance estimation neural network to generate estimated distance information. The distance estimation neural network here is determined by parameters of a trained distance estimation neural network.

[0103] Next, in step S203, the blur correction unit 223c inputs the captured image and the estimated distance information to an image processing neural network to generate a blur-corrected image. The image processing neural network here is determined by the parameters of a trained image processing neural network. This ends the generation process.

[0104] In this embodiment, as in the first embodiment, it is possible to learn optical characteristics, distance estimation, and blur correction that are favorable for both distance estimation accuracy and blur correction accuracy, thereby making it possible to obtain an image in which blur is well corrected while achieving high distance estimation accuracy.

[0105] The above embodiments include the following methods.

[0106] (Configuration 1) generating a first image based on an original image including an object, distance information, and optical characteristics of an optical system; generating first distance information based on the first image using a first neural network; generating output data using a second neural network based on the first image and second distance information about the object in the original image; A step of performing learning based on the first distance information, the correct distance information, the output data, and correct data based on the original image, A learning method characterized in that, in the learning, a first parameter that determines the optical characteristics, a second parameter that determines the first neural network, and a third parameter that determines the second neural network are updated. (Configuration 2) 2. The learning method according to claim 1, wherein the first distance information is generated based on an image from a single viewpoint. (Configuration 3) 3. The learning method according to configuration 1 or 2, wherein the optical characteristics are characteristics related to blur in the optical system. (Configuration 4) The learning method according to any one of configurations 1 to 3, characterized in that the first parameter determines a wavefront aberration generated by a wavefront modulation element included in the optical system. (Configuration 5) The learning method according to configuration 4, wherein the wavefront aberration is an aberration having a rotationally asymmetric shape. (Configuration 6) 6. The learning method according to any one of configurations 1 to 5, wherein the second distance information is the same information as the correct distance information or the first distance information. (Configuration 7) 7. The learning method according to any one of configurations 1 to 6, wherein the correct answer data is an image identical to the original image or an image degraded from the original image. (Configuration 8) 8. The learning method according to any one of configurations 1 to 7, wherein the correct answer data is generated using the correct answer distance information and the original image. (Configuration 9) The learning method according to any one of configurations 1 to 8, wherein the output data is an image in which the first image has been sharpened. (Configuration 10) The learning method according to configuration 9, further comprising controlling the strength of the sharpening in the output data based on the ground-truth distance information. (Configuration 11) The learning method described in configuration 10, characterized in that in controlling the strength of the sharpening, the strength of the sharpening in areas other than the in-focus area in the output data is weakened compared to the strength of the sharpening in the in-focus area. (Configuration 12) 12. The learning method according to any one of configurations 1 to 11, characterized in that the first parameter is updated based on the first distance information, the correct distance information, the output data, and the correct data. (Configuration 13) The learning method described in any one of configurations 1 to 12, characterized in that the second parameter is updated based on the first distance information and the correct distance information, or based on the output data and the correct data. (Configuration 14) 14. The learning method according to any one of configurations 1 to 13, further comprising updating the third parameter based on the output data and the correct answer data. (Configuration 15) Acquiring a captured image based on the first parameter updated by the learning method according to any one of configurations 1 to 13; generating third distance information based on the captured image and the second parameters updated by the learning method according to any one of configurations 11 to 13; and generating estimated data based on the captured image, the third distance information, and the third parameter updated by the learning method according to any one of configurations 11 to 13. (Configuration 16) Obtaining information about pixel pitch; 16. The image processing method according to configuration 15, further comprising the step of changing the second parameter and the third parameter based on information relating to the pixel pitch. (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0107] The embodiments described above are merely representative examples, and various modifications and alterations are possible for each embodiment when implementing the present invention. [Explanation of symbols]

[0108] 101 Learning device (image processing device) 101c generator 101d Update Department 201 Learning device (image processing device) 213 Generation part 214 Update Department

Claims

1. generating a first image based on an original image including an object, distance information, and optical characteristics of an optical system; generating first distance information based on the first image using a first neural network; generating output data using a second neural network based on the first image and second distance information for the object in the original image; A step of performing learning based on the first distance information, the correct distance information, the output data, and correct data based on the original image, A learning method characterized in that, in the learning, a first parameter that determines the optical characteristics, a second parameter that determines the first neural network, and a third parameter that determines the second neural network are updated.

2. The learning method according to claim 1 , wherein the first distance information is generated based on a single-viewpoint image.

3. 2. The learning method according to claim 1, wherein the optical characteristic is a characteristic related to blur in the optical system.

4. 2. The learning method according to claim 1, wherein the first parameter determines a wavefront aberration caused by a wavefront modulation element included in the optical system.

5. 5. The learning method according to claim 4, wherein the wavefront aberration is an aberration having a rotationally asymmetric shape.

6. 2. The learning method according to claim 1, wherein the second distance information is the same as the correct distance information or the first distance information.

7. 2. The learning method according to claim 1, wherein the correct answer data is an image identical to the original image or an image degraded more than the original image.

8. The learning method according to claim 1 , wherein the correct answer data is generated using the correct answer distance information and the original image.

9. The learning method according to claim 1 , wherein the output data is an image in which the first image has been sharpened.

10. The learning method according to claim 9 , further comprising controlling the strength of the sharpening in the output data based on the ground-truth distance information.

11. The learning method according to claim 10 , wherein in controlling the strength of the sharpening, the strength of the sharpening in the region other than the in-focus region in the output data is made weaker than the strength of the sharpening in the in-focus region.

12. 2. The learning method according to claim 1, further comprising updating the first parameter based on the first distance information, the correct distance information, the output data, and the correct data.

13. 2. The learning method according to claim 1, wherein the second parameter is updated based on the first distance information and the correct distance information, or based on the output data and the correct data.

14. 2. The learning method according to claim 1, further comprising updating the third parameter based on the output data and the correct answer data.

15. acquiring a captured image based on the first parameters updated by the learning method according to claim 1; generating third distance information based on the captured image and the second parameters updated by the learning method according to claim 1; An image processing method comprising a step of generating estimated data based on the captured image, the third distance information, and the third parameter updated by the learning method described in claim 1.

16. Obtaining information about pixel pitch; 16. The image processing method according to claim 15, further comprising the step of changing the second parameter and the third parameter based on information relating to the pixel pitch.

17. a first generating means for generating a first image based on an original image including an object, distance information, and optical characteristics of an optical system; a second generating means for generating first distance information based on the first image by using a first neural network; a third generating means for generating output data based on the first image and second distance information related to the object included in the original image using a second neural network; a learning means for performing learning based on the first distance information, the correct distance information, the output data, and correct data based on the original image, A learning device characterized in that the learning means updates a first parameter that determines the optical characteristics, a second parameter that determines the first neural network, and a third parameter that determines the second neural network during learning.

18. An acquisition means for acquiring a captured image based on the first parameters updated by the learning device according to claim 17; a distance generating means for generating third distance information based on the captured image and the second parameters updated by the learning device according to claim 17; 20. An image processing device comprising: an estimation means for generating estimated data based on the captured image, the third distance information, and the third parameter updated by the learning device according to claim 17.

19. A program for causing a computer to execute the learning method according to any one of claims 1 to 13.

20. A program for causing a computer to execute the image processing method according to claim 15 or 16.

Images