Imaging device

Abstract

An image pickup device (200) comprising an image pickup optical system (100), a feature amount extraction unit (31), and an image correction unit (41). The image pickup optical system (100) comprises an image pickup element (1) having a light-receiving surface (10) and executes image pickup. The feature amount extraction unit (31) extracts a feature amount from a noise image (21) containing noise and an ideal image (11) from which the noise has been removed. The image correction unit (41) uses the feature amount to correct a pickup image picked up by the image pickup optical system (100). Imaging performance of the image pickup optical system (100) in a peripheral part (1B) of the light-receiving surface (10) located on the outer side with respect to a center part (1A) thereof is higher than the imaging performance of the image pickup optical system (100) in the center part (1A).

Description

[Technical Field]

[0001] The present disclosure relates to an imaging device. [Background technology]

[0002] Generally, as an image processing technique performed inside an imaging device, a technique has been proposed for effectively correcting an image that has been affected by optical aberrations generated by the optical system inside the imaging device (also referred to as an imaging optical system) (see, for example, Patent Document 1). The imaging device described in Patent Document 1 accurately corrects an image whose image quality has been degraded by the influence of the optical system, based on a first image that has been affected by the aberrations of the optical system, a second image from which the influence of the aberrations has been removed, position information of each pixel obtained from information on these images, and a distortion aberration correction table. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2017 / 158690 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional technologies, imaging performance is highest at the center of the light-receiving surface of an imaging device, and decreases with distance from the center. As a result, the peripheral area located outside the center of the light-receiving surface is significantly affected by optical system aberrations. Furthermore, the peripheral area receives information outside the image range as stray light. In other words, while such conventional technologies can effectively correct pixels near the center of the light-receiving surface, their correction performance is limited for pixels near the outer edge of the light-receiving surface. This issue is particularly pronounced when using an optical system with a simple configuration, such as a single lens or a spherical lens.

[0005] The object of the present disclosure has been made to solve the above-described conventional problems, and is to provide an imaging device that can remove noise from the entire captured image with high accuracy, even when using an optical system with a simple configuration. [Means for solving the problem]

[0006] of the present disclosure In one aspect The imaging device an imaging optical system that has an imaging element having a light receiving surface and performs imaging; a feature extraction unit that extracts feature amounts from a noise image including noise due to the imaging optical system and an ideal image obtained by removing the noise from the noise image; an image correction unit that corrects an image captured by the imaging optical system using the feature amount; Equipped with The imaging performance of the imaging optical system in the peripheral portion located outside the center of the light receiving surface is higher than the imaging performance of the imaging optical system in the center of the light receiving surface. Ku, The feature extraction unit a first image storage unit in which the ideal image is stored; a second image storage unit in which the noise image is stored; a machine learning processing unit that extracts the feature amounts by performing convolution processing on the ideal image and the noise image; and The first image storage unit stores an image having a wider range than the range captured by the imaging optical system. . An imaging device according to another aspect of the present disclosure includes: an imaging optical system that has an imaging element having a light receiving surface and performs imaging; a feature extraction unit that extracts feature amounts from a noise image including noise due to the imaging optical system and an ideal image obtained by removing the noise from the noise image; an image correction unit that corrects an image captured by the imaging optical system using the feature amount; Equipped with The imaging performance of the imaging optical system in the peripheral portion located outside the center of the light receiving surface is higher than the imaging performance of the imaging optical system in the center of the light receiving surface. , The feature extraction unit a first image storage unit in which the ideal image is stored; a second image storage unit in which the noise image is stored; a machine learning processing unit that extracts the feature amounts by performing convolution processing on the ideal image and the noise image; and The second image storage unit stores an image having a wider range than the range captured by the imaging optical system. Imaging device. An imaging device according to yet another aspect of the present disclosure includes: an imaging optical system that has an imaging element having a light receiving surface and performs imaging; a feature extraction unit that extracts feature amounts from a noise image including noise due to the imaging optical system and an ideal image obtained by removing the noise from the noise image; an image correction unit that corrects an image captured by the imaging optical system using the feature amount; Equipped with the imaging performance of the imaging optical system in a peripheral portion located outside the center portion of the light receiving surface is higher than the imaging performance of the imaging optical system in the center portion of the light receiving surface, The imaging performance of the imaging optical system is highest at the outermost edge of the light receiving surface. Imaging device. An imaging device according to yet another aspect of the present disclosure includes: an imaging optical system that has an imaging element having a light receiving surface and performs imaging; a feature extraction unit that extracts feature amounts from a noise image including noise due to the imaging optical system and an ideal image obtained by removing the noise from the noise image; an image correction unit that corrects an image captured by the imaging optical system using the feature amount; Equipped with the imaging performance of the imaging optical system in a peripheral portion located outside the center portion of the light receiving surface is higher than the imaging performance of the imaging optical system in the center portion of the light receiving surface, The imaging performance of the imaging optical system is highest in a portion of the light receiving surface other than the central portion or a portion of the light receiving surface other than the outermost edge portion. Imaging device. [Effects of the Invention]

[0007] According to the present disclosure, it is possible to provide an imaging device that can remove noise from the entire captured image with high accuracy even when an optical system with a simple configuration is used. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram illustrating a schematic configuration of an imaging device. [Figure 2] FIG. 2 is a block diagram illustrating an example of a feature extraction unit. [Figure 3] 10(A) and 10(B) are block diagrams showing the configuration of an image acquisition unit. [Figure 4] FIG. 2 illustrates an example of a hardware configuration of a main control unit. [Figure 5] FIG. 10 is a diagram illustrating another example of the hardware configuration of the main control unit. [Figure 6] FIG. 1 is a diagram schematically illustrating an example of an imaging optical system. DETAILED DESCRIPTION OF THE INVENTION

[0009] An imaging device 200 according to an embodiment of the present disclosure will be described below with reference to the drawings. The embodiment described below is merely an example, and various modifications are possible.

[0010] FIG. 1 is a block diagram showing a schematic configuration of an imaging device 200 according to this embodiment. The imaging device 200 includes an imaging optical system 100, a feature extraction unit 31, an image correction unit 41, and an image display unit 51.

[0011] The imaging optical system 100 captures an image. The imaging optical system 100 has an imaging element 1 having a light receiving surface 10. An image captured by the imaging optical system 100 is also referred to as a "captured image."

[0012] The feature extraction unit 31 extracts feature amounts from the noise image 21, which includes noise due to the imaging optical system 100, and the ideal image 11, in which the noise has been removed from the noise image 21. The feature amount is the difference between the ideal image 11 and the noise image 21. In other words, the feature extraction unit 31 extracts noise (also referred to as noise components) from the noise image 21.

[0013] FIG. 2 is a block diagram showing an example of the feature extraction unit 31. As shown in FIG. In the example shown in FIG. 2, the feature extraction unit 31 includes a first image storage unit 12, a second image storage unit 22, and a machine learning processing unit 32.

[0014] The machine learning processing unit 32 performs machine learning using a machine learning model such as a genetic algorithm or a neural network, etc. By using a machine learning model such as a neural network, it is possible to collectively correct various aberrations such as image distortion or blur correction.

[0015] The neural network extracts features by convolution processing (also called convolution operation) of two-dimensional information using, for example, a Convolutional Neural Network (CNN). The data input to these machine learning models is two-dimensional information of the ideal image 11 and the noise image 21, or multidimensional information including RGB information.

[0016] The first image storage unit 12 can store images of a wider range than the range captured by the imaging optical system 100. Similarly, the second image storage unit 22 can store images of a wider range than the range captured by the imaging optical system 100. For example, if the imaging device 100 can capture an image of a range of ±30 degrees, a pair of a wide-angle ideal image 11 and a wide-angle noise image 21 can be obtained by performing calculations using ray tracing of ±35 degrees. If pixel loss occurs due to noise or distortion caused by stray light of ±30 degrees or more, the noise or pixel loss is estimated and corrected using the pair of wide-angle images.

[0017] Instead of using only one image, multiple images may be used. When multiple images are used, the ideal image 11 is stored in the first image storage unit 12, and the noise image 21 is stored in the second image storage unit 22. In this case, each ideal image 11 is tagged so that it can be associated with the noise image 21.

[0018] When a neural network is used as the machine learning model, the machine learning processing unit 32 extracts features by performing a convolution process on the ideal image 11 and the noise image 21.

[0019] 2 , when the ideal image 11 and the noise image 21 are input to the machine learning processing unit 32, the machine learning processing unit 32 can estimate the noise components contained in the noise image 21. The machine learning processing unit 32 repeatedly adjusts the weights of the machine learning model so that the value of each pixel of the noise image 21 approaches that of the ideal image 11. In this way, the machine learning model learns to restore the noise image 21 to the ideal image 11.

[0020] The ideal image 11 is an image from which noise due to the imaging optical system 100 has been removed. The ideal image 11 is, for example, a real image captured by an imaging device having an imaging optical system with higher precision than the imaging optical system 100 of the imaging device 200. The ideal image 11 may also be an image generated by computer graphics (CG). The ideal image 11 may also be stored in advance in a memory (for example, the first image storage unit 12) within the imaging device 200.

[0021] The noise image 21 is an image that includes noise caused by the imaging optical system 100. That is, the noise image 21 is an image (i.e., a photographed image) output from the imaging optical system 100. Noise refers to, for example, image distortion or image blur caused by optical aberration of the imaging optical system 100, or stray light generated by optical members or a housing of the imaging optical system 100.

[0022] The noise image 21 may be an image generated by data processing such as ray tracing, where the ray tracing can be calculated using, for example, a point spread function (PSF) or an optical transfer function (OTF).

[0023] The image correction unit 41 acquires the feature amounts from the feature amount extraction unit 31. The image correction unit 41 corrects the captured image captured by the imaging optical system 100 using the feature amounts. The captured image captured by the imaging optical system 100 is an image containing noise equal to the noise in the noise image 21. For example, the image correction unit 41 removes the feature amounts from the captured image to output an image equal to the ideal image 11.

[0024] The image output from the image correction unit 41 is displayed on the image display unit 51. The image displayed on the image display unit 51 is a clear image from which noise has been removed by the image correction unit 41. The image display unit 51 is, for example, a device such as a liquid crystal display or a projector.

[0025] 3A and 3B are block diagrams showing the configurations of the image acquisition units 110 and 111. FIG. The imaging device 200 may include an image acquisition unit 110 or an image acquisition unit 111 .

[0026] The image acquisition unit 110 has an optical calculation unit 120. The ideal image 11a is an image that does not contain noise caused by the imaging optical system 100. The optical calculation unit 120 calculates noise and stray light that occurs when the ideal image 11a is input to the imaging optical system 100, and generates a noise image 21a.

[0027] The image acquisition unit 111 has an imaging optical system 100 and an imaging optical system 101. The imaging optical system 101 is, for example, an imaging optical system with higher accuracy than the imaging optical system 100 of the imaging device 200. In this case, the ideal image 11b is an image captured by the imaging optical system 101. The noise image 21b is an image captured by the imaging optical system 100. In other words, the ideal image 11b is an image captured with higher accuracy than the noise image 21b.

[0028] FIG. 4 is a diagram showing an example of the hardware configuration of the main control unit 42. FIG. 5 is a diagram showing another example of the hardware configuration of the main control unit 42. In FIG.

[0029] The feature extraction unit 31 and the image correction unit 41 are configured, for example, by a main control unit 42. In this case, the main control unit 42 is configured, for example, by at least one processor 42a and at least one memory 42b. In this case, the first image storage unit 12 and the second image storage unit 22 shown in FIG. 2 may be the memory 42b.

[0030] The processor 42a is, for example, a central processing unit (CPU) that executes a program stored in the memory 42b. In this case, the functions of the feature extraction unit 31 and the image correction unit 41 are realized by software, firmware, or a combination of software and firmware. The software and firmware can be stored in the memory 42b as a program. With this configuration, the program for realizing the functions of the main control unit 42 is executed by the computer.

[0031] The memory 42b is a computer-readable recording medium, and is, for example, a volatile memory, a non-volatile memory, or a combination of a volatile memory and a non-volatile memory, such as a random access memory (RAM) or a read only memory (ROM).

[0032] The main control unit 42 may have a plurality of processors 42a and a plurality of memories 42b. In this case, the functions of the feature extraction unit 31 and the image correction unit 41 are realized by the plurality of processors 42a and the plurality of memories 42b.

[0033] The main control unit 42 may be configured with a processing circuit 42c as dedicated hardware, such as a single circuit or a composite circuit. The processing circuit 42c may be, for example, a system LSI. In this case, the functions of the feature extraction unit 31 and the image correction unit 41 are realized by the processing circuit 42c.

[0034] FIG. 6 is a diagram schematically illustrating an example of the imaging optical system 100. As shown in FIG. In the example shown in FIG. 6, the imaging optical system 100 includes an imaging element 1 and an optical member 2.

[0035] 6, the X-axis direction is the horizontal direction of the imaging device 200, the Y-axis direction is the vertical direction of the imaging device 200, and the Z-axis direction is the optical axis direction of the optical member 2. In the example shown in FIG. 6, the subject is present on the −Z side of the optical member 2, and the imaging element 1 is present on the +Z side of the optical member 2.

[0036] 6, the direction along the Y-axis direction is, for example, the vertical direction, in which the +Y direction is the upward direction in the vertical direction, and the −Y direction is the downward direction in the vertical direction.

[0037] The image sensor 1 is an element capable of converting the intensity of light into an electrical signal. The image sensor 1 is composed of, for example, a plurality of light receiving elements arranged two-dimensionally. The image sensor 1 converts imaging information (for example, light) emitted from a subject area into an electrical signal, and can generate an image of the subject area based on the converted electrical signal.

[0038] The imaging device 1 may be any device capable of converting received light into an electrical signal, and may be, for example, an image sensor such as a Charge Coupled Device (CCD) or a Complementary Metal Oxide Semiconductor (CMOS).

[0039] The optical element 2 is, for example, a single spherical lens or multiple spherical lenses. In the example shown in Figure 6, the optical element 2 is a single spherical lens.

[0040] 6, the imaging optical system 100 has a range of angle of view that can be received by the imaging element 1 of ±30 degrees in the X-axis direction and ±30 degrees in the Y-axis direction. In Fig. 6, light ray r0, light ray r10, light ray r20, light ray r30, and light ray r31 indicate light rays with angle of view of 0 degrees, 10 degrees, 20 degrees, 30 degrees, and 31 degrees, respectively.

[0041] Typically, in an imaging optical system, the imaging performance is highest at a position on the light receiving surface of the imaging element where light rays with a field angle of 0 degrees reach, and the imaging performance decreases as the field angle increases. That is, the imaging performance of the imaging optical system decreases as the distance from the center to the outside of the light receiving surface of the imaging element increases. The imaging performance is expressed, for example, by an index such as the modulation transfer function (MTF), which indicates the degree of contrast reproduction as a spatial frequency characteristic. A high MTF value indicates high imaging performance. Furthermore, for example, a small amount of optical aberration or stray light generated in the imaging optical system indicates high imaging performance.

[0042] On the other hand, image processing techniques using an ideal image and a noise image generate an image within the range of the light receiving surface of the image sensor. For example, in an imaging optical system with a field of view of ±30 degrees, image processing can be performed only within the range of ±30 degrees to effectively correct the image.

[0043] However, if the imaging performance of the imaging optical system is poor in the peripheral area outside the center of the image sensor's light-receiving surface, light rays outside the ±30-degree range will be received by the image sensor as stray light. This stray light is information that differs from known noise components that can be predicted using an ideal image and a noise image. Therefore, in the range where this stray light is received by the image sensor, it is difficult to restore a clear image using image processing technology.

[0044] In contrast, in the imaging device 200 according to the present embodiment, the imaging performance of the imaging optical system 100 in the peripheral portion 1B located outside the central portion 1A of the light receiving surface 10 is higher than the imaging performance of the imaging optical system 100 in the central portion 1A of the light receiving surface 10. That is, in the example shown in Fig. 6, the imaging performance in the peripheral portion 1B where the light ray r10 or the light ray reaches is higher than the imaging performance in the central portion 1A where the light ray r0 reaches. The central portion 1A of the light receiving surface 10 is the position where the optical axis of the optical member 2 and the imaging element 1 intersect.

[0045] 6, the imaging performance of the imaging optical system 100 is highest at the outermost edge 1C of the light receiving surface 10. The outermost edge 1C is located on the outermost side of the light receiving surface 10. That is, in the example shown in FIG. 6, the imaging performance at the outermost edge 1C where the light ray r30 reaches is higher than the imaging performance at the center 1A where the light ray r0 reaches.

[0046] The imaging performance of the imaging optical system 100 may be highest in a portion of the light receiving surface 10 other than the central portion 1A or a portion of the light receiving surface 10 other than the outermost edge portion 1C. In this case, the imaging performance of the imaging optical system 100 may be highest in the peripheral portion 1B of the light receiving surface 10, for example.

[0047] This prevents light rays outside the angle of view of the imaging optical system 100, such as light ray r31, from being received as stray light by the image sensor 1. Furthermore, although the imaging performance in the central portion 1A is degraded compared to the imaging performance in the peripheral portion 1B, information about the light rays is retained within the image sensor 1. Therefore, noise such as the amount of blur caused by degradation of imaging performance can be removed as a known noise component.

[0048] The imaging performance of the imaging optical system 100 in the peripheral portion 1B is made higher than the imaging performance of the imaging optical system 100 in the central portion 1A by optical design of the single lens or spherical lens used as the optical member 2. For example, when there is an imaging optical system on the light receiving surface where the imaging performance is highest in the central portion and lowest in the peripheral portion, the imaging optical system 100 can be obtained by translating the optical lens in the optical axis direction so that the imaging performance in the peripheral portion is highest.

[0049] The noise component may be removed by using, for example, an image processing technique. In this case, for example, each aberration, such as image distortion or blur correction, may be corrected individually. The method of aberration correction using the image processing technique is not limited to a specific method.

[0050] As described above, according to this embodiment, it is possible to provide the imaging device 200 that can remove noise from the entire captured image with high accuracy even when an optical system with a simple configuration is used.

[0051] According to this embodiment, it is possible to provide an imaging device 200 that can correct the entire captured image acquired by the imaging optical system 100 with high precision.

[0052] The features of the above-described embodiment and modifications can be combined with each other. [Explanation of symbols]

[0053] 1 imaging element, 1A central portion, 1B peripheral portion, 1C outermost edge portion, 2 optical member, 10 light receiving surface, 11 ideal image, 12 first image memory unit, 21 noise image, 22 second image memory unit, 31 feature extraction unit, 32 machine learning processing unit, 41 image correction unit, 51 image display unit, 100 imaging optical system, 200 imaging device.

Claims

1. an imaging optical system that has an imaging element having a light receiving surface and performs imaging; a feature extraction unit that extracts feature amounts from a noise image including noise due to the imaging optical system and an ideal image obtained by removing the noise from the noise image; an image correction unit that corrects an image captured by the imaging optical system using the feature amount; Equipped with the imaging performance of the imaging optical system in a peripheral portion located outside the center portion of the light receiving surface is higher than the imaging performance of the imaging optical system in the center portion of the light receiving surface, The feature extraction unit a first image storage unit in which the ideal image is stored; a second image storage unit in which the noise image is stored; a machine learning processing unit that extracts the feature amounts by performing convolution processing on the ideal image and the noise image; and The first image storage unit stores an image having a wider range than the range captured by the imaging optical system. Imaging device.

2. The imaging device according to claim 1 , wherein the second image storage unit stores an image having a wider range than the range captured by the imaging optical system.

3. An imaging optical system having an imaging element with a light receiving surface for capturing images; a feature extraction unit that extracts feature amounts from a noise image including noise due to the imaging optical system and an ideal image obtained by removing the noise from the noise image; an image correction unit that corrects an image captured by the imaging optical system using the feature amount; Equipped with the imaging performance of the imaging optical system in a peripheral portion located outside the center portion of the light receiving surface is higher than the imaging performance of the imaging optical system in the center portion of the light receiving surface, The feature extraction unit a first image storage unit in which the ideal image is stored; a second image storage unit in which the noise image is stored; a machine learning processing unit that extracts the feature amounts by performing convolution processing on the ideal image and the noise image; and The second image storage unit stores an image having a wider range than the range captured by the imaging optical system. Imaging device.

4. The imaging device according to claim 1 , wherein the imaging optical system includes a single spherical lens or a plurality of spherical lenses.

5. The imaging device according to claim 1 , wherein the imaging performance of the imaging optical system is highest at the outermost edge of the light receiving surface.

6. An imaging optical system having an imaging element with a light receiving surface for capturing images; a feature extraction unit that extracts feature amounts from a noise image including noise due to the imaging optical system and an ideal image obtained by removing the noise from the noise image; an image correction unit that corrects an image captured by the imaging optical system using the feature amount; Equipped with the imaging performance of the imaging optical system in a peripheral portion located outside the center portion of the light receiving surface is higher than the imaging performance of the imaging optical system in the center portion of the light receiving surface, The imaging performance of the imaging optical system is highest at the outermost edge of the light receiving surface. Imaging device.

7. The imaging device according to claim 1 , wherein the imaging performance of the imaging optical system is highest in a portion of the light receiving surface other than the central portion or in a portion of the light receiving surface other than the outermost edge portion.

8. An imaging optical system having an imaging element with a light receiving surface for capturing images; a feature extraction unit that extracts feature amounts from a noise image including noise due to the imaging optical system and an ideal image obtained by removing the noise from the noise image; an image correction unit that corrects an image captured by the imaging optical system using the feature amount; Equipped with the imaging performance of the imaging optical system in a peripheral portion located outside the center portion of the light receiving surface is higher than the imaging performance of the imaging optical system in the center portion of the light receiving surface, The imaging performance of the imaging optical system is highest in a portion of the light receiving surface other than the central portion or a portion of the light receiving surface other than the outermost edge portion. Imaging device.

9. 9. The imaging device according to claim 1, further comprising an image display unit on which the image output from the image correction unit is displayed.

Images