Imaging device and program

Abstract

To provide an imaging apparatus and the like which can extend a range of a depth of field.SOLUTION: A plurality of image pick-up devices are provided such that ranges of a depth of field are different from each other and capture images. A spatial frequency component map generation unit generates a spatial frequency component map expressing a spatial frequency component of the image obtained by each of the plurality of image pick-up devices. A learning object region priority decision unit decides the priority expressing which image has the high spatial frequency component among the plurality of images on the basis of the corresponding spatial frequency component map of the image based on the images obtained by the plurality of image pick-up devices. An image generation unit generates an output image on the basis of the image obtained by the image pick-up device. The image generation unit is configured to have a model that can perform machine-learning. The model updates an internal parameter of the model by performing machine-learning processing by using a teacher image and a learning object image based on the priority.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an imaging device and a program. [Background technology]

[0002] Imaging devices are becoming increasingly miniaturized to make it easier to produce high-definition images such as 4K and 8K.

[0003] In order to reduce the size of an imaging device, it is important not to increase the size of the optical system. However, Non-Patent Document 1 states that, for the same optical format, as the resolution of an imaging device increases, the pixels of the imaging element become smaller. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Takayuki Yamashita, Tsuyoshi Sakiyama, Masatsugu Mitsuhashi, "On the Optical System of 4K / 8K Cameras", Journal of the Institute of Image Information and Television Engineers, VOL.71, No.6, pp.819-824(2017). Summary of the Invention [Problem to be solved by the invention]

[0005] As the pixels of the image sensor in an image capture device become smaller, it becomes necessary to open the lens aperture more (increase the effective aperture of the lens) during image capture in order to obtain a sufficient amount of light. In particular, in an image capture device using a small, ultra-high definition image sensor, it becomes necessary to open the lens aperture more during image capture in order to avoid the effects of diffraction that occur as pixel size becomes smaller.

[0006] When shooting with the lens aperture open, there is a problem that the depth of field becomes shallow, which means that it is difficult to obtain a high-definition image of a subject that is shifted forward or backward from the focal point.

[0007] The present invention has been made based on the recognition of the above-mentioned problems, and aims to provide an imaging device and a program that can deepen the depth of field even when, for example, the lens aperture is open. [Means for solving the problem]

[0008] [1] In order to solve the above problem, an imaging device according to one aspect of the present invention includes a plurality of imaging elements that are arranged to capture images with different depth of field ranges, a spatial frequency component map generation unit that generates a spatial frequency component map representing the spatial frequency components of an image obtained by each of the plurality of imaging elements, a learning target area priority determination unit that determines a priority representing which image among the plurality of images has a high spatial frequency component based on the spatial frequency component maps corresponding to each of images based on the images obtained by the plurality of imaging elements, and an image generation unit that generates an output image based on the images obtained by the imaging elements, wherein the image generation unit is configured with a model capable of machine learning, and the model updates internal parameters of the model by performing machine learning processing using the teacher image and the learned image based on the priority. The imaging device may include imaging elements other than "a plurality of imaging elements provided so as to have mutually different depth of field ranges."

[0009] [2] Furthermore, in one aspect of the present invention, in the above-mentioned imaging device, the image generation unit uses an image obtained by synthesizing the multiple images obtained from the multiple imaging elements based on the priority as the teacher image, and uses any one of the multiple images obtained from the multiple imaging elements as the learning image.

[0010] [3] Also, in one aspect of the present invention, in the above-mentioned imaging device, the image generation unit generates the output image by inputting an image obtained by synthesizing a plurality of the images obtained from a plurality of the imaging elements based on the priority into an internal model.

[0011] [4] Moreover, an imaging device according to one aspect of the present invention includes a plurality of imaging elements for capturing images, and an image generation unit for generating an output image based on the images obtained by the imaging elements, the image generation unit having a model for inputting the images and outputting the output image, the model having internal parameters that are the result of learning by the imaging device.

[0012] [5] Also, one aspect of the present invention is an imaging device as described above, wherein the plurality of imaging elements include a red component image imaging element for capturing a red component image and a blue component image imaging element for capturing a blue component image, the ranges of depth of field being the same for each other, and a green component image imaging element for capturing a green component image, the ranges of depth of field being different from those of the red component image imaging element and the blue component image imaging element.

[0013] [6] Also, one aspect of the present invention is an imaging device as described above, wherein the multiple imaging elements include a first green component image imaging element for capturing a first green component image and a second green component image imaging element for capturing a second green component image, the imaging elements being arranged so as to have different depth of field ranges from each other.

[0014] [7] Furthermore, one aspect of the present invention is that in the above-mentioned imaging device, when the imaging element having the shorter flange back distance of any two of the plurality of imaging elements is designated as a first imaging element and the imaging element having the longer flange back distance is designated as a second imaging element, the sum of the rear focal depth length of the first imaging element and the front focal depth length of the second imaging element is equal to or greater than the distance obtained by subtracting the flange back distance of the first imaging element from the flange back distance of the second imaging element.

[0015] [8] Moreover, one aspect of the present invention is a program for causing a computer to function as a signal processing unit, the program comprising: a spatial frequency component map generation unit that generates a spatial frequency component map representing the spatial frequency components of an image obtained by each of a plurality of image sensors arranged so that the ranges of depth of field are different from each other; a learning target area priority determination unit that determines a priority representing which image among a plurality of images has a high spatial frequency component based on the spatial frequency component maps corresponding to each of the images based on the images obtained by the plurality of image sensors; and an image generation unit that generates an output image based on the images obtained by the image sensors, the image generation unit being configured with a model capable of machine learning, and the model updating internal parameters of the model by performing machine learning processing using the teacher image and the learned image based on the priority. Effect of the Invention

[0016] According to the present invention, the imaging device can learn a model of the image generator based on a training image and a teacher image, and can extend the range of the depth of field. [Brief description of the drawings]

[0017] [Figure 1] 1 is a block diagram showing a schematic functional configuration of an imaging apparatus according to an embodiment of the present invention. [Diagram 2] 2 is a cross-sectional view showing a first configuration example of an optical system and an image sensor included in the image pickup device according to the embodiment. FIG. [Diagram 3] 4 is a cross-sectional view showing a second example of the configuration of the optical system and the image sensor included in the image pickup device according to the embodiment. FIG. [Figure 4] 11 is a cross-sectional view showing a third example of the configuration of the optical system and the image sensor included in the image pickup device according to the embodiment. FIG. [Diagram 5] 10 is a flowchart showing a processing procedure performed by a signal processing unit of the imaging device according to the embodiment. [Figure 6] 10 is a side view showing an example of the arrangement of an optical system and an image sensor for controlling the position of the image sensor according to the embodiment. FIG. [Figure 7] 7 is a side view corresponding to FIG. 6 and showing the range of depth of field of image sensor 103f, the range of depth of field of image sensor 103n, and the extended range of depth of field. [Figure 8] 4 is a schematic diagram for explaining a movable range of an imaging element. FIG. [Figure 9] 2 is a functional block diagram showing an example of the configuration of a computer included in the imaging device according to the embodiment. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Next, an embodiment of the present invention will be described.

[0019] 1 is a block diagram showing a schematic functional configuration of an image pickup apparatus according to the present embodiment. As shown in the figure, the image pickup apparatus 1 includes an optical system 102, an image pickup element 103, and a signal processing unit 104.

[0020] The optical system 102 is a system for collecting light and forming an image of the light on the light receiving surface of the image sensor 103. The optical system 102 includes a lens and a light splitting means such as a prism or a beam splitter.

[0021] The image sensor 103 has a large number of pixels, and converts the light received by each pixel into an electrical signal and outputs it. The pixels are usually arranged vertically and horizontally in a grid pattern. However, the arrangement of the pixels is not limited to this. The image sensor 103 outputs an electrical signal of an image formed on the light receiving surface. The image sensor 103 passes the output electrical signal of the image to the signal processing unit 104.

[0022] The imaging device 1 of this embodiment is configured to include a plurality of imaging elements 103. The optical system 102 and the imaging elements 103 are arranged so that light from the optical system 102 is split by a beam splitter or the like and reaches each of the imaging elements 103. The plurality of imaging elements may be provided so that the distances (flange back distances) from the flange surface of the lens in the optical system 102 are different from each other. At least one pair of two imaging elements 103 among the plurality of imaging elements 103 included in the imaging device 1 are provided so that the distances from the flange surface of the lens are different from each other. The pair of imaging elements 103 output images of the same angle of view (or almost the same angle of view) in the same direction as seen from the imaging device 1. In other words, the imaging elements 103 are a plurality of imaging elements that capture images provided so that the ranges of the depth of field are different from each other.

[0023] Examples of the arrangement of the optical system 102 and the image sensor 103 will be described in further detail later with reference to FIGS.

[0024] The signal processing unit 104 generates an image and outputs data of the generated image based on the electrical signal received from the imaging element 103. The internal functional configuration of the signal processing unit 104 will be described next.

[0025] As shown in FIG. 1, the signal processing unit 104 includes an image acquisition unit 105, a spatial frequency component map generation unit 106, a comparison unit 107, a learning area priority determination unit 108, an image generation unit 109, and an image output unit 111. Each functional unit constituting the signal processing unit 104 can be realized, for example, by a computer and a program. Each functional unit has a storage unit as necessary. The storage unit is, for example, a variable in the program or a memory allocated by the execution of the program. Non-volatile storage units such as a magnetic hard disk drive or a solid state drive (SSD) may be used as necessary. At least a part of the functions of each functional unit may be realized as a dedicated electronic circuit rather than a program. The functions of each unit are as follows.

[0026] The image acquisition unit 105 acquires an image output by the imaging element 103. As described above, the imaging device 1 is provided with a plurality of imaging elements 103. The image acquisition unit 105 acquires an image from each of the imaging elements 103. The plurality of imaging elements 103 output at least an image Yf and an image Yn. The images Yf and Yn are images output from imaging elements that are at different distances on the optical path from the flange surface of the lens in the optical system 102. The image Yf is an image output from the imaging element 103 on the side with the shorter distance. The image Yf is focused on an object located relatively far from the lens. The image Yn is an image output from the imaging element 103 on the side with the longer distance. The image Yn is focused on an object located relatively close to the lens. That is, the image acquisition unit 105 acquires a plurality of images including at least the above images Yf and Yn from the plurality of imaging elements 103. The image acquisition unit 105 passes the acquired image to the spatial frequency component map generation unit 106 .

[0027] The spatial frequency component map generating unit 106 generates a spatial frequency component map for each of the images Yf and Yn. The spatial frequency component map is data representing the spatial frequency components for each region in the image. For example, for each of the images Yf and Yn, the spatial frequency component map generating unit 106 generates the spatial frequency component map by calculating the intensity of the spatial frequency components for each pixel in the image based on the surrounding pixel patches including the pixel. The spatial frequency component map generating unit 106 does not necessarily need to calculate the spatial frequency components for all pixels, and may, for example, calculate the intensity of the spatial frequency components for sample pixels at predetermined intervals. In addition, the spatial frequency component map generating unit 106 does not necessarily need to calculate the components for all frequency bands, and may, for example, calculate only the components of a specific frequency band. As an example, the spatial frequency component map generating unit 106 may generate the spatial frequency component map by calculating only the intensity of the components of a band higher than a predetermined frequency for each region in the image. That is, the spatial frequency component map generating unit 106 of this embodiment may generate maps FH(Yf) and FH(Yn) indicating positions having high spatial frequency components in each of the images Yf and Yn. The spatial frequency component map generating unit 106 passes the generated spatial frequency component maps to the comparing unit 107. That is, the spatial frequency component map generating unit 106 generates spatial frequency component maps that indicate the spatial frequency components of the images obtained by each of the multiple image pickup elements 103.

[0028] The comparison unit 107 compares the spatial frequency component maps of the images Yf and Yn generated by the spatial frequency component map generation unit 106. The comparison unit 107 passes the comparison result to the learning region priority determination unit 108.

[0029] The learning target area priority determination unit 108 determines, for each area in the image, which of the images Yf and Yn has a clearer image based on, for example, the spatial frequency component maps FH(Yf) and FH(Yn). Specifically, the learning target area priority determination unit 108 determines, for each area, which of the images Yf and Yn has a stronger high-frequency component of the spatial frequency and to what extent based on the spatial frequency component maps FH(Yf) and FH(Yn).

[0030] For example, when the spatial frequency component maps FH(Yf) and FH(Yn) each have a value (real number) representing the degree of strength of the high-frequency component of the spatial frequency for each pixel in the image, the priority P is determined for each area according to the number of pixels where FH(Yf) < FH(Yn). For the same pixel, when FH(Yf) < FH(Yn), the image Yn has a stronger high-frequency component at that pixel position than the image Yf. Similarly, when FH(Yf) > FH(Yn), the image Yf has a stronger high-frequency component at that pixel position than the image Yn. And when FH(Yf) = FH(Yn), the strengths of the high-frequency components of the images Yf and Yn at that pixel position are the same. In this example, the learning target area priority determination unit 108 determines the value of the priority P for a certain area according to the number of pixels where FH(Yf) < FH(Yn) or according to the number of pixels where FH(Yf) > FH(Yn). P may be, for example, a real number in the range of 0 or more and 1 or less. The learning target area priority determination unit 108 passes the obtained priority P data to the image generation unit 109. This P is used as a numerical value representing how much the teacher image should be emphasized when the image generation unit 109 performs machine learning.

[0031] That is, the learning target area priority determination unit 108 determines a priority P indicating which image of which image sensor is to be prioritized for each area in an image based on a result of comparing spatial frequency component maps corresponding to the images obtained by the multiple image sensors 103. That is, the learning target area priority determination unit 108 determines a priority indicating which image of the multiple images has a high spatial frequency component based on the spatial frequency component maps corresponding to each of the images based on the images obtained by the multiple image sensors 103.

[0032] The priority P determined by the learning target area priority determination unit 108 has the effect of giving greater weight to the image having the high spatial frequency components when only one of the images Yf and Yn has high spatial frequency components.

[0033] The image generating unit 109 generates an output image Yp based on the image Yf and the image Yn. The image generating unit 109 has an internal model capable of machine learning. The image generating unit 109 may have, for example, a neural network as the model. In this case, the image generating unit 109 is also provided with a convolution neural network for calculating image features to be input to the neural network, and a neural network for performing the inverse operation.

[0034] When learning this model, the image generating unit 109 performs a learning process of the neural network, for example, with the image Yf as the learning image and the image Yn as the teacher image. The image generating unit 109 inputs the value of P (a value corresponding to the area in the image) passed from the learning area priority determining unit 108 to the neural network. The image generating unit 109 uses these data to learn the neural network. As a learning method, the back propagation method can be used. As a result of the learning, the image generating unit 109 updates the values ​​of the internal parameters of the neural network. The internal parameters are vectors of weight values ​​for the calculation of the weighted sum in each node constituting the neural network. The image generating unit 109 includes a storage unit that stores these weight values. As a result of the learning, the image generating unit 109 updates the weight values ​​stored in the storage unit.

[0035] The model held by image generation unit 109 has the function of inputting image Yf and outputting image Yp. A model that has completed sufficient and satisfactory learning replaces an area in image Yf having a predetermined pixel arrangement pattern with another pixel arrangement pattern. This "another pixel arrangement pattern" is based on the priority value P acquired by learning and the pixel arrangement pattern of teacher image Yn. The internal parameters held by a model that has completed sufficient and satisfactory learning are parameters for realizing such a function.

[0036] When generating an image Yp using this model, the image generating unit 109 reads out the values ​​of the internal parameters (the weight values ​​described above) from the storage unit and uses them for calculations at each node of the neural network. The image Yf is input to the trained neural network to obtain an output image Yp. Assuming that the training is sufficient, the output image Yp is expected to include a pixel pattern of an area containing high-frequency components of the image Yn captured at a position (a different position in depth) different from that of the image Yf. In other words, the output image Yp is expected to have both an image with high spatial frequency components in the range of the depth of field at the position of the image sensor that outputs the image Yf and the range of the depth of field at the position of the image sensor that outputs the image Yn. In other words, the output image Yp can be expected to be an image with an expanded range of the depth of field in the optical system 102.

[0037] That is, the image generation unit 109 generates an output image based on an image obtained by the imaging element 103. The image generation unit 109 is also configured to include a model capable of machine learning, and the model updates internal parameters of the model by performing machine learning processing using a teacher image based on the priority P and a trained image.

[0038] The signal processing unit 104 having the above configuration can operate in one of different modes: a mode in which machine learning of a model is performed and an output image Yp is generated (for convenience, referred to as "mode A"), and a mode in which the output image Yp is only generated without machine learning of a model (for convenience, referred to as "mode B").

[0039] When processing in mode A, the imaging device 1 can update the weight values ​​in the neural network of the image generation unit 109 while capturing an image. In other words, the more the user uses the imaging device in mode A, the more learning progresses, resulting in an effect of expanding the range of the depth of field of the imaging device 1. When processing in mode B, the imaging device 1 can generate image Yp using a trained neural network (with fixed weight values) without updating the weight values ​​in the neural network.

[0040] The imaging device 1 according to the present embodiment is configured to operate in the above-mentioned mode A and also in mode B. As a modified example, for example, the imaging device 1 may be configured to operate only in mode A (a mode in which machine learning is performed and an output image Yp is generated). In that case, the signal processing unit 104 performs learning processing of the model (neural network) held by the image generating unit 109 every time the image Yp for obtaining and outputting an image from the multiple image sensors 103 is generated, and updates the internal parameters stored in the storage unit. As another modified example, the imaging device 1 may be configured to operate only in mode B (a mode in which machine learning processing is not performed and an output image Yp is generated). In that case, sufficient learning processing of the model of the image generating unit 109 has already been completed. In other words, the storage unit stores the internal parameters of the learned model, and the internal parameters are only referred to and not updated. When the imaging device 1 operates only in mode B, the internal parameters of the model are already written in the storage unit, for example, at the time when the imaging device 1 is shipped from the factory where it was manufactured and initially set up.

[0041] That is, an imaging device that operates only in mode B includes a plurality of imaging elements 103 that capture images, and an image generation unit 109 that generates an output image based on the images obtained by the imaging elements. The image generation unit 109 has a model that inputs an image and outputs an output image. This model has internal parameters that are the result of learning in advance by the method already described.

[0042] In the above description, image Yf is taken as the learning image and image Yn is taken as the teacher image, for example. This may be reversed, with image Yf (image captured by an image sensor located at a shorter distance from the flange surface of the lens) as the teacher image and image Yn (image captured by an image sensor located at a longer distance from the flange surface of the lens) as the learning image.

[0043] The image output unit 111 outputs the image Yp generated by the image generating unit 109 to the outside.

[0044] That is, the imaging device 1 includes two imaging elements. These imaging elements capture an image Yf and an image Yn, respectively. The learning area priority determination unit 108 determines, for each area in the image, a priority P indicating which of the two images has a higher spatial frequency component, based on the spatial frequency component maps corresponding to the two imaging elements. This priority P is used for machine learning in the image generation unit 109.

[0045] As described above, in the imaging device 1 according to the present embodiment, the signal processing unit 104 extends the depth of field of the lens for imaging. That is, the signal processing unit 104 generates and outputs an image Yp (although calculated by a model) whose depth of field range is a range including both the depth of field range of the image Yf and the depth of field range of the image Yn. The depth of field range of the image Yf and the depth of field range of the image Yn may overlap partially, may be just adjacent, or may be separated from each other. When the depth of field range of the image Yf and the depth of field range of the image Yn are separated from each other, the depth of field range of the generated image Yp will have a plurality of ranges separated from each other. In this way, including the case where the depth of field range of the image Yf and the depth of field range of the image Yn are separated from each other, extending the depth of field of the imaging device 1 is useful because it results in generating a clear image at low cost.

[0046] Next, a more detailed configuration example of the optical system 102 and the image sensor 103 in this embodiment will be described with reference to FIGS. 2, 3, and 4. FIG.

[0047] 2 is a cross-sectional view showing a first configuration example of the optical system 102 and the image sensor 103. This example is a configuration example of a two-plate optical system. In the figure, 201 is a beam splitter. 203 and 204 are image sensors (corresponding to the image sensor 103 in FIG. 1). 202 indicates the path of a light beam that enters from the lens side, passes through the beam splitter 201, and reaches each of the image sensors 203 and 204.

[0048] The beam splitter 201 splits the incident light beam and directs the split light beams to the image pickup elements 203 and 204, respectively. The beam splitter 201 is realized using, for example, a half mirror, but the beam splitter may be realized by other means. The beam splitter 201 transmits approximately 50% of the incident light beam and directs it to the image pickup element 203. The beam splitter 201 also reflects the remaining 50% of the incident light beam and directs it to the image pickup element 204.

[0049] The imaging surface of the imaging element 203 and the imaging surface of the imaging element 204 are provided so that the flange back distances in the optical system 102 are different from each other. The difference in the flange back distances may be, for example, within a range of several micrometers to several tens of micrometers. A micrometer is 10 to the power of minus 6 meters. The difference in the distance from the flange surface does not have to be within the range exemplified above. In the illustrated example, the distance from the flange surface of the lens to the imaging surface of the imaging element 203 is shorter than the distance from the flange surface of the lens to the imaging surface of the imaging element 204. The image output by the imaging element 203 is called Yf. The image output by the imaging element 204 is called Yn. Here, the subscript "f" stands for far (the subject farther from the lens is in focus), and "n" stands for near (the subject closer to the lens is in focus). The distances from the flange surface of the lens may be reversed between the imaging elements 203 and 204.

[0050] 2, each of the multiple image pickup elements may capture a color image, or the image pickup elements may capture a monochrome image (including a black and white image (grayscale image)).

[0051] 3 is a cross-sectional view showing a second configuration example of the optical system 102 and the image sensor 103. This example is a configuration example of a three-plate optical system. In the figure, 301 is a beam splitter. 303, 304, and 305 are image sensors (corresponding to the image sensor 103 in FIG. 1), respectively. 302 indicates the path of a light beam that enters from the lens side, passes through the beam splitter 301, and reaches each of the image sensors 303, 304, and 305.

[0052] The beam splitter 301 splits the incident light beam and directs the split light beams to the image pickup elements 303, 304, and 305, respectively. The beam splitter 301 is realized, for example, by using a plurality of dichroic mirrors, but the beam splitter may be realized by other means. The light beam that passes through a first surface next to the light beam incident surface in the beam splitter 301 and is reflected by a second surface reaches the image pickup element 303. The light beam that passes through both the first surface and the second surface in the beam splitter 301 reaches the image pickup element 304. The light beam that is reflected by the first surface in the beam splitter 301 reaches the image pickup element 305. The beam splitter 301 splits the light beam so that the amount of light beam that reaches each image pickup element is approximately the same, for example.

[0053] Each of the image sensors 303, 304, and 305 outputs an image of a particular color component. In this example, the image sensor 303 outputs an image of a red (R) component. The image sensor 304 outputs an image of a green (G) component. The image sensor 305 outputs an image of a blue (B) component.

[0054] In this example, the image output by the imaging element 303 is called Yn'. The image output by the imaging element 304 is called Yf. The image output by the imaging element 305 is called Yn. The subscripts "f" and "n" are the same as those described in FIG. 2. That is, the imaging surface of the imaging element 303 and the imaging surface of the imaging element 305 are provided at positions where the distances (distances on the optical path) from the flange surface of the lens in the optical system 102 are the same. The imaging surface of the imaging element 304 is provided at a position where the distances (distances on the optical path) from the flange surface of the lens in the optical system 102 are different from those of the imaging elements 303 and 305. The difference in distance from the flange surface of the lens to the imaging surface of the imaging element may be, for example, within a range from several micrometers to several tens of micrometers. Alternatively, the difference in distance may not be within the range exemplified above. In the illustrated example, the distance from the lens flange surface to the imaging surface of the imaging element 304 is set to be shorter than the distance from the lens flange surface to the imaging surfaces of the imaging elements 303 and 305 .

[0055] That is, in this example, the image sensor 304 that outputs an image of a green component is provided at a position that is relatively closer to the flange surface of the lens, while the image sensor 303 that outputs an image of a red component and the image sensor 305 that outputs an image of a blue component are provided at positions that are relatively farther from the flange surface of the lens.

[0056] 3, the multiple imaging elements include a red component image imaging element for capturing a red component image and a blue component image imaging element for capturing a blue component image, which are provided so that the ranges of depth of field are the same as each other. The multiple imaging elements also include a green component image imaging element for capturing a green component image, which is provided so that the ranges of depth of field are different from those of the red component image imaging element and the blue component image imaging element.

[0057] 4 is a cross-sectional view showing a third example of the configuration of the optical system 102 and the image sensor 103. This example is a configuration example of a four-plate optical system. In the figure, 401 is a beam splitter. 403, 404, 405, and 406 are image sensors (corresponding to the image sensor 103 in FIG. 1), respectively. 402 indicates the path of a light beam that enters from the lens side, passes through the beam splitter 401, and reaches each of the image sensors 403, 404, 405, and 406.

[0058] The beam splitter 401 splits the incident light beam and directs the split light beams to the image pickup elements 403, 404, 405, and 406, respectively. The beam splitter 401 is realized, for example, by using a plurality of dichroic mirrors and a half mirror, but the beam splitter may be realized by other means. The light beam that passes through the first surface next to the light beam incident surface in the beam splitter 401 and is reflected by the second surface reaches the image pickup element 403. The light beam that passes through all of the first surface, the second surface, and the third surface in the beam splitter 401 reaches the image pickup element 404. The light beam that is reflected by the first surface in the beam splitter 401 reaches the image pickup element 406. The light beam that is reflected by the third surface after passing through both the first surface and the second surface in the beam splitter 401 reaches the image pickup element 405. The beam splitter 401 splits the light beam so that the amount of light beam reaching each image sensor is approximately the same. The first and second surfaces are dichroic mirrors with wavelength selection, and the third surface is a half mirror.

[0059] Each of the image sensors 403, 404, 405, and 406 outputs an image of a particular color component. In this example, the image sensor 403 outputs an image of a red (R) component. The image sensors 404 and 405 output an image of a green (G1 and G2, respectively) component. The image sensor 406 outputs an image of a blue (B) component.

[0060] In this example, the image output by the imaging element 404 is called Yf. Also, the image output by the imaging element 405 is called Yn. The subscripts "f" and "n" are the same as those explained in Fig. 2 and Fig. 3. In this configuration, the signal processing unit 104 extends the depth of field using the images Yf and Yn output from the two imaging elements that output images of the green component. The depth of field is not extended for images of other color components (red and blue).

[0061] That is, the imaging surface of the imaging element 404 and the imaging surface of the imaging element 405 are provided at positions such that the distances (distances on the optical path) from the flange surface of the lens in the optical system 102 are different from each other. The difference in distance from the flange surface of the lens to the imaging surface of the imaging element may be, for example, within a range of several micrometers to several tens of micrometers. Alternatively, the difference in distance does not have to be within the range exemplified above. In the illustrated example, the distance from the flange surface of the lens to the imaging surface of the imaging element 404 (the imaging element that outputs the image Yf) is set to be shorter than the distance from the flange surface of the lens to the imaging surface of the imaging element 405 (the imaging element that outputs the image Yn).

[0062] That is, in the configuration as shown in Fig. 4, the multiple imaging elements include a first green component image imaging element for capturing a first green component image and a second green component image imaging element for capturing a second green component image, which are provided so that the ranges of the depth of field are different from each other. For example, the first green component image is an image Yf, and the second green component image is an image Yn. However, Yf and Yn may be reversed.

[0063] As described above with reference to FIGS. 2, 3, and 4, in the imaging device 1, the imaging element that outputs the image Yf and the imaging element that outputs the image Yn (or Yn') are disposed at positions slightly offset from each other in the depth direction of the optical path (the distances from the flange surface of the lens are different). In the two-plate optical system shown in FIG. 2, the two color imaging elements (the imaging element that outputs the image Yf and the imaging element that outputs the image Yn) are disposed at positions slightly offset from each other in the depth direction of the optical path. In the three-plate optical system shown in FIG. 3, the imaging element for the green component (the imaging element that outputs the image Yf) and the imaging elements for the red and blue components (the images Yn' and Yn, respectively) are disposed at positions slightly offset from each other in the depth direction of the optical path. In the four-plate optical system shown in FIG. 4, the imaging elements for the two green components (the imaging element that outputs the image Yf and the imaging element that outputs the image Yn) are disposed at positions slightly offset from each other in the depth direction of the optical path.

[0064] In each of the configuration examples described above, the focal position (distance) in image Yf is different from the focal position (distance) in image Yn. Compared to image Yf, the focal position of image Yn is closer to the front when viewed from the lens side. In other words, compared to image Yf, the distance from the lens side to the focal position of image Yn is shorter. In other words, since the conditions such as the lens aperture are the same, the range of the depth of field of the captured image Yf is different from the range of the depth of field of image Yn.

[0065] 5 is a flowchart showing the procedure of processing by the signal processing unit 104 in the imaging device 1. The premise of this procedure is that whether the imaging device 1 operates in mode A or mode B has been determined at the latest by the time of mode determination in step S12. The operation procedure will be described below with reference to this flowchart.

[0066] First, in step S11, the image acquisition unit 105 acquires images from the multiple image pickup elements 103 provided at positions with different flange back distances. Here, the image on the side with the shorter flange back distance is designated Yf, and the image on the side with the longer flange back distance is designated Yn. In other words, the ranges of the depth of field of these multiple image pickup elements are different.

[0067] Next, in step S12, the signal processing unit 104 determines the operation mode. If the operation mode is mode A (learning and image generation mode), the process proceeds to the next step S13. If the operation mode is mode B (image generation mode), the process jumps to step S18.

[0068] Next, when the process proceeds to step S13, the spatial frequency component map generating unit 106 generates spatial frequency component maps of the images Yf and Yn. Here, the spatial frequency component map of the image Yf is designated as FH(Yf), and the spatial frequency component map of the image Yn is designated as FH(Yn).

[0069] Next, in step S14, the comparison unit 107 compares the spatial frequency component maps FH(Yf) and FH(Yn). Then, the learning region priority determination unit 108 determines the learning region priority P for each region in the image based on the comparison result of the spatial frequency component maps. The priority P is as described above.

[0070] Here, the region in the image may be, for example, a region that is fixedly determined in advance using the coordinate values ​​of pixels. Alternatively, the region may be determined individually for each image. For example, the region may be a region in which pixels having similar characteristics are consecutive. The characteristic for determining the region here is, for example, the distribution of spatial frequency for each pixel position. In addition, the distribution of color components for each pixel position may be added to the element of the characteristic.

[0071] Next, in step S15, the image generating unit 109 performs machine learning of a model that the image generating unit 109 has internally, with one of the images Yf and Yn as a learned image and the other as a teacher image based on the priority P determined in step S14. For example, the image generating unit 109 performs machine learning of a model with the image Yf as a learned image and the image Yn as a teacher image. Note that the learned image and teacher image may be reversed between the images Yf and Yn. In other words, the image Yf may be the teacher image and the image Yn may be the learned image. For example, when a neural network is used, the image generating unit 109 performs a learning process of the neural network.

[0072] Next, in step S16, the image generating unit 109 determines updated values ​​of the internal parameters (weight parameters of each node) of the neural network as a result of the learning process. As described above, for example, the backpropagation method can be used to update the parameters.

[0073] Next, in step S17, the image generating unit 109 updates the storage unit with the new internal parameter values ​​determined in step S16.

[0074] Depending on the result of the determination in step S12, the process proceeds to step S18 from step S12 (if YES), or from step S17 (if YES). In step S18, the image generating unit 109 selects weight parameters for the neural network. The image generating unit 109 usually reads out the latest weight parameters after learning from the storage unit.

[0075] Next, in step S19, the image generating unit 109 generates an image Yp using the weighting parameters of the neural network acquired in step S 18. Specifically, the image generating unit 109 inputs either the image Yf or the image Yn to the neural network, and calculates the image Yp using the weighting parameters.

[0076] Next, in step S20, the image output unit 111 outputs the output image Yp generated in step S19. For example, the image output unit 111 outputs data of the output image Yp as a file on a file system. Alternatively, the image output unit 111 performs output by a method of displaying the output image Yp on a display device (such as a liquid crystal display). The image output unit 111 may output the output image Yp by a method other than these.

[0077] Although the embodiment has been described above, the present invention can also be embodied in the following modified examples. Note that multiple modified examples may be combined to the extent that they are possible.

[0078] [First Modification] In the above embodiment, the image Yn captured by the imaging element placed at a position on the side closer to the flange surface of the lens is used as the teacher image, and the image Yf captured by the imaging element placed at a position on the side closer to the flange surface of the lens is used as the learned image. This may also be reversed. That is, the image Yf may be used as the teacher image, and the image Yn may be used as the learned image. On the other hand, in this modified example, an image Yfn is generated by combining the images Yf and Yn, which have different depth of field ranges, and this image Yfn is used as the teacher image. In this modified example, either the image Yf or the image Yn is used as the learned image, or both the images Yf and Yn are used as the learned images. In this manner, the signal processing unit 104 of this modified example learns a model (a neural network, for example) for image generation. In this modified example, a specific image is used as the teacher image when learning the model. The image Yfn used as the teacher image is generated, for example, as follows. That is, the spatial frequency component map generating unit 106 generates spatial frequency component maps FH(Yf) and FH(Yn) for the images Yf and Yn, respectively. Each of these spatial frequency component maps FH(Yf) and FH(Yn) represents, for example, the degree of spatial frequency for each pixel. As an example, each of the spatial frequency component maps FH(Yf) and FH(Yn) represents the degree of intensity of high-frequency components higher than a predetermined spatial frequency for each pixel. On the premise of such spatial frequency component maps, an image (either Yf or Yn) having a stronger high-frequency component is extracted for each pixel or for each region in the image, and the extracted images are arranged over all pixels or all regions to generate a teacher image Yfn. Except for the point that the teacher image Yfn is generated in this way, this modified example is similar to the embodiment already described.

[0079] As described above, in the first modified example, the image generation unit 109 synthesizes multiple images obtained from multiple imaging elements based on the priority P to obtain an image Yfn as a teacher image, and selects one of the multiple images obtained from the multiple imaging elements as an image to be learned.

[0080] [Second modified example] In this modification, the image generating unit 109 generates an output image Yp by inputting the image Yfn described in the first modification. Specifically, the image generating unit 109 generates the image Yfn. The image generating unit 109 inputs the image Yfn to an internal neural network. The neural network applies internal parameters based on the image Yfn to generate an image Yp. In this modification, the signal processing unit 104 of the imaging device 1 may operate in either mode A or mode B.

[0081] That is, in the second variant, the image generation unit 109 generates an output image Yp by inputting an image Yfn obtained by synthesizing multiple images obtained from multiple imaging elements based on the priority P into an internal model of the image generation unit 109.

[0082] [Third Modification] In the above embodiment, the model is learned based on two images, an image Yf captured by an image sensor closer to the flange surface of the lens, and an image Yn captured by an image sensor farther from the flange surface of the lens. In the third modified example, the number of images is expanded to three or more. Specifically, images are captured by N (N≧3) image sensors, all of which have different distances from the flange surface of the lens (distances on the optical path). The N images are Y(1), Y(2), . . . , Y(N). The image generating unit 109 selects one of the N images (e.g., image Y(1)) as the image to be learned, and the other (N-1) images (e.g., images Y(2) to Y(N)) as the teacher images. Which of the N images is to be the teacher image is the same as in the above embodiment. That is, for example, the spatial frequency for each region in each image is obtained, and the top (N-1) images having high frequency components are selected as the teacher images. The other points are as described in the above embodiment.

[0083] That is, in the third modified example, the plurality of imaging elements are three or more imaging elements, and the images obtained by each of the plurality of imaging elements are three or more images.

[0084] Here, details of the position of the image sensor and examples of its control in the above embodiment and modified examples will be described.

[0085] FIG. 6 shows an example of the arrangement of the optical system 102 and the image sensor 103 for capturing images Yf and Yn. In the figure, 501 is a beam splitter. 103f and 103n are image sensors. 502 shows a light beam that enters from the lens side, passes through the beam splitter 501, and reaches each of the image sensors 103f and 103n 406. Here, the image sensor 103f captures the image Yf. Also, the image sensor 103n captures the image Yn. Note that the optical path lengths of the light beams 502 that enter each of the image sensors 103f and 103n to the positions where they exit the beam splitter 501 are equal to each other. In the example shown in the figure, for example, the position of the image sensor 103f is fixed to the position of the flange back distance of the lens of the optical system 102, that is, the back focus position. The distance from the beam splitter 501 to the imaging surface of the image sensor 103f is d1. The position of the other image sensor, image sensor 103n, can be electrically controlled in the depth direction. The distance from the beam splitter 501 to the imaging surface of image sensor 103n is d2. In other words, the position of image sensor 103n is adjusted by electrical control so that d2>d1.

[0086] FIG. 7 is a schematic diagram showing the imaging elements 103f and 103n in FIG. 6 and the range of the depth of field of each imaging element. Note that FIG. 7 shows a case where the imaging elements 103f and 103n are virtually arranged on the same axis. As shown in the figure, the range of the depth of field corresponding to the imaging element 103f is DOF1. Also, the range of the depth of field corresponding to the imaging element 103n is DOF2. The imaging device 1 described above expands the depth of field. That is, by learning using the images Yf and Yn, the depth of field of the imaging device 1 is expanded to DOF1+2. Note that in the example shown in the figure, DOF1 and DOF2 have a common area, but the ranges of the depth of field of both do not necessarily have to overlap in this way. Note that the movable range of the imaging element 103n is determined based on the focal depth for the imaging element.

[0087] FIG. 8 is a schematic diagram for explaining the movable range of the image sensor 103n. In the figure, IS1 is the image sensor 103f. Also, IS2 is the image sensor 103n. That is, IS1 captures an image Yf. Also, IS2 captures an image Yn. ε1 shown in the figure is the length of the rear focal depth of IS1. Also, ε2 is the length of the front focal depth of IS2. The condition for the range of the depth of field of the image Yf and the range of the depth of field of the image Yn to overlap is as shown in the following formula (1).

[0088]

number

[0089] Here, Δd=d2-d1, δ is the diameter of the permissible circle of confusion, and F is the F-number of the lens. For example, when δ=2.5 [micrometers] and F=2.0, Δd≦10.0 [micrometers]. Note that the numerical values ​​of the sizes shown here are merely examples.

[0090] That is, of any two (at least any two sets) of the multiple imaging elements, the imaging element with the shorter flange back distance is the first imaging element (IS1 above) and the imaging element with the longer flange back distance is the second imaging element (IS2 above). In this case, the sum (ε1+ε2) of the length ε1 of the rear focal depth of the first imaging element (IS1) and the length ε2 of the front focal depth of the second imaging element is equal to or greater than the distance (Δd=d2-d1) obtained by subtracting the flange back distance of the first imaging element from the flange back distance of the second imaging element (IS2). When this condition is satisfied, the ranges of the depth of field of the images captured by these two imaging elements overlap. ε1 and ε2 depend on the F-number of the lens.

[0091] It should be noted that instead of making the imaging element 103f fixed and the imaging element 103n movable, the imaging element 103n may be fixed and the imaging element 103f may be movable.

[0092] As described above, according to this embodiment (including the modified example), the imaging device 1 performs machine learning of a model held by the image generating unit 109 based on a plurality of images having different ranges of depth of field. Furthermore, the image generating unit 109 generates an output image Yp using the trained model. This enables the imaging device 1 to extend the depth of field.

[0093] For example, by providing appropriate learned weighting data at the manufacturing stage of the imaging device 1, the user of the imaging device 1 can obtain an image with a deep depth of field immediately after obtaining the imaging device 1. Furthermore, it is possible for the user to add learning each time they use the imaging device 1. In other words, a higher performance depth of field improvement effect is achieved that matches the user's specific shooting conditions.

[0094] FIG. 9 is a block diagram showing an example of a partial internal configuration of the imaging device of the embodiment (including a modified example). The imaging device 1, particularly the signal processing unit 104, can be realized using a computer. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, and a bus 906. The computer itself can be realized using existing technology. The central processing unit 901 executes instructions included in a program read from the RAM 902 or the like. The central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations according to each instruction. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. Note that RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with an external input / output device or the like. The input / output devices 904 and 905 are input / output devices. Input / output devices 904 and 905 exchange data with the central processing unit 901 via an input / output port 903. A bus 906 is a common communication path used inside the computer. For example, the central processing unit 901 reads and writes data from and to a RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port via the bus 906.

[0095] At least some of the functions of the imaging device in the above-mentioned embodiment (particularly, the function of the signal processing unit 104) can be realized by a computer program. In that case, a program for realizing the functions may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the functions. Note that the term "computer system" here includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, CD-ROMs, DVD-ROMs, and USB memories, and storage devices such as hard disks built into a computer system. Furthermore, the term "computer-readable recording medium" may include those that temporarily and dynamically hold a program, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and those that hold a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in such a case. Furthermore, the above-mentioned program may be for realizing some of the above-mentioned functions, and may further be capable of realizing the above-mentioned functions in combination with a program already recorded in the computer system.

[0096] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of the present invention are also included. [Industrial Applicability]

[0097] The present invention can be used, for example, in businesses that produce visual content such as photographs and videos. It can also be used in businesses that perform surveillance and the like by taking images or videos. It can also be used in businesses that manufacture and sell imaging devices. However, the scope of use of the present invention is not limited to the examples given here. [Explanation of symbols]

[0098] 1. Imaging device 102 Optical system 103, 103f, 103n Image sensor 104 Signal Processing Section 105 Image acquisition unit 106 Spatial frequency component map generator 107 Comparison Section 108 Learning Area Priority Determination Unit 109 Image Generation Unit 111 Image output unit 203,204,303,304,305,403,404,405,406 Image sensor 201,301,401,501 Beam splitter 202,302,402,502 luminous flux 901 Central Processing Unit 902 RAM 903 Input / Output Ports 904,905 Input / Output Devices 906 Bus

Claims

1. A plurality of image pickup elements each having a different depth of field range for capturing an image; a spatial frequency component map generating unit that generates a spatial frequency component map representing spatial frequency components obtained by calculating, for each pixel in an image obtained by each of the plurality of image pickup elements, an intensity of spatial frequency components based on a pixel patch including the pixel and surrounding the pixel; a learning target area priority determination unit that determines a priority for each image area in an image based on the spatial frequency component map corresponding to each of the images obtained by the multiple image pickup devices, the learning target area priority determination unit determining, for each image area, the priority indicating which image area in the image obtained by which image pickup device has a high spatial frequency component among the image areas in each of the multiple images; an image generating unit that generates an output image based on the image obtained by the imaging element; Equipped with the image generation unit is configured to include a model capable of machine learning, and the model updates internal parameters of the model by performing machine learning processing using the image to be learned and the other images obtained by the plurality of image pickup elements as teacher images based on the priority for each image region within the image, and using the teacher image and the image to be learned; The priority is expressed by a real number within a predetermined range, and is a numerical value indicating how much importance should be attached to the teacher image, the image generation unit extracts an image having a higher priority for each pixel or each image region within the multiple images obtained from the multiple image pickup elements based on the priority, and generates the output image by inputting the image obtained by arranging the extracted image across all pixels or all image regions into an internal model. Imaging device.

2. The plurality of image pickup elements include a red component image pickup element for capturing a red component image and a blue component image pickup element for capturing a blue component image, the red component image pickup element and the blue component image pickup element being provided so that the ranges of the depth of field are the same; a green component image pickup element for capturing a green component image, the green component image pickup element being provided so as to have a depth of field range different from that of the red component image pickup element and the blue component image pickup element; Including, The imaging device according to claim 1 .

3. The plurality of image pickup elements include a first green component image capturing element for capturing a first green component image and a second green component image capturing element for capturing a second green component image, the first green component image capturing element and the second green component image capturing element being provided so as to have different ranges of depth of field from each other; Including, 3. The imaging device according to claim 1.

4. when an imaging element having a shorter flange back distance among any two of the plurality of imaging elements is designated as a first imaging element and an imaging element having a longer flange back distance is designated as a second imaging element, a sum of a rear focal depth of the first imaging element and a front focal depth of the second imaging element is equal to or greater than a distance obtained by subtracting the flange back distance of the first imaging element from the flange back distance of the second imaging element, The imaging device according to claim 1 .

5. A plurality of image pickup elements each having a different depth of field range for capturing an image; a spatial frequency component map generating unit that generates a spatial frequency component map representing spatial frequency components obtained by calculating, for each pixel in an image obtained by each of the plurality of image pickup elements, an intensity of spatial frequency components based on a pixel patch including the pixel and surrounding the pixel; a learning target area priority determination unit that determines a priority for each image area in an image based on the spatial frequency component map corresponding to each of the images obtained by the multiple image pickup devices, the learning target area priority determination unit determining, for each image area, the priority indicating which image area in the image obtained by which image pickup device has a high spatial frequency component among the image areas in each of the multiple images; an image generating unit that generates an output image based on the image obtained by the imaging element; Equipped with the image generation unit is configured to include a model capable of machine learning, and the model updates internal parameters of the model by performing machine learning processing using the image to be learned and the other images obtained by the plurality of image pickup elements as teacher images based on the priority for each image region within the image, and using the teacher image and the image to be learned; The priority is expressed by a real number within a predetermined range, and is a numerical value indicating how much importance should be attached to the teacher image, when an imaging element having a shorter flange back distance among any two of the plurality of imaging elements is designated as a first imaging element and an imaging element having a longer flange back distance is designated as a second imaging element, a sum of a rear focal depth of the first imaging element and a front focal depth of the second imaging element is equal to or greater than a distance obtained by subtracting the flange back distance of the first imaging element from the flange back distance of the second imaging element, Imaging device.

6. a spatial frequency component map generating unit that generates a spatial frequency component map representing spatial frequency components obtained by calculating the intensity of spatial frequency components for pixels in an image obtained by each of a plurality of image pickup devices that are provided so as to have different depth of field ranges, based on a pixel patch including the pixel and surrounding the pixel; a learning target area priority determination unit that determines a priority for each image area in an image based on the spatial frequency component map corresponding to each of the images obtained by the multiple image pickup devices, the learning target area priority determination unit determining, for each image area, the priority indicating which image area in the image obtained by which image pickup device has a high spatial frequency component among the image areas in each of the multiple images; an image generating unit that generates an output image based on the image obtained by the imaging element; Equipped with the image generation unit is configured to include a model capable of machine learning, and the model updates internal parameters of the model by performing machine learning processing using the image to be learned and the other images obtained by the plurality of image pickup elements as teacher images based on the priority for each image region within the image, and using the teacher image and the image to be learned; The priority is expressed by a real number within a predetermined range, and is a numerical value indicating how much importance should be attached to the teacher image, the image generation unit extracts an image having a higher priority for each pixel or each image region within the multiple images obtained from the multiple image pickup elements based on the priority, and generates the output image by inputting the image obtained by arranging the extracted image across all pixels or all image regions into an internal model. A program that causes a computer to function as a signal processing unit.

Images