Imaging device
By adjusting calculation ranges based on PSF for each pixel, the imaging device improves blur correction accuracy and reduces computational load, addressing the trade-off in existing imaging devices.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- DENSO CORP
- Filing Date
- 2023-03-06
- Publication Date
- 2026-07-07
Smart Images
Figure 0007886288000001 
Figure 0007886288000002 
Figure 0007886288000003
Abstract
Description
Technical Field
[0001] The technology disclosed in this specification relates to an imaging device.
[0002] Patent Document 1 discloses an imaging device. In this imaging device, an image is divided into uniform sizes, and a blur correction process using a filter is performed for each of the divided images.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] When an image is divided into uniform sizes and a blur correction process is performed for each of the divided images, if the image is divided into sizes according to the areas with a small degree of blur, the accuracy of the blur correction process will decrease for the areas with a large degree of blur. Conversely, if the image is divided into sizes according to the areas with a large degree of blur, the computational load will be excessive for the areas with a small degree of blur. This specification provides a technology capable of increasing the accuracy of the blur correction process while reducing the computational load.
Means for Solving the Problems
[0005] The imaging device disclosed herein includes an image sensor that outputs image data showing an image captured through an optical system including a lens, and a blur correction processing unit. The blur correction processing unit is configured to perform the following: (a) extracting pixel data included in the calculation range from the image data to obtain extracted image data; (b) performing blur correction processing using a filter on the extracted image data to calculate corrected extracted image data; (c) extracting pixel data included in the reconstruction range from the corrected extracted image data to obtain reconstructed image data; (d) repeatedly performing the processes (a)-(c) by changing the calculation range, the filter, and the reconstruction range; and (e) combining a plurality of the reconstructed image data obtained by the process of (c) multiple times to obtain corrected image data. The position and shape of the calculation range are set according to the position and shape of the corresponding reconstruction range and the size of the PSF (Point Spread Function) corresponding to the position of the pixels included in the corresponding reconstruction range.
[0006] The degree of blur for each pixel in the image data depends on the size of the Point Spread Function (PSF) corresponding to the position of each pixel. In this specification, the size of the PSF refers to an index indicating the degree of PSF spread. For example, when the PSF is shown as a luminance distribution, the size of the PSF means the width of the range where the luminance is above a predetermined threshold. With the above configuration, the position and shape of the calculation range are set according to the position and shape of the reconstruction range and the size of the PSF corresponding to the position of the pixels included in the reconstruction range, so that the size of each extracted image data can be adjusted to match the degree of blur at that location. By adopting such a configuration, the accuracy of the blur correction process can be improved while reducing the computational load. [Brief explanation of the drawing]
[0007] [Figure 1] This diagram schematically shows the overall configuration of the imaging device 10 of the embodiment. [Figure 2] This is a flowchart of the processing performed by the blur correction processing unit 26 in the imaging device 10 of the embodiment. [Figure 3] This diagram schematically shows how corrected image data is generated from uncorrected image data in the imaging device 10 of the embodiment. [Figure 4] This figure schematically shows the optical system 18 in the imaging device 10 of the embodiment. [Figure 5] This graph shows the relationship between the number of pixels from the center and the size of the PSF in the imaging device 10 of the embodiment. [Figure 6] This figure schematically shows the reconstruction range and calculation range in the imaging device 10 of the embodiment. [Modes for carrying out the invention]
[0008] In one example imaging apparatus disclosed herein, the center position of the calculation range may be offset from the center position of the corresponding reconstruction range.
[0009] The size of the PSF corresponding to the position of a pixel included in the reconstruction range is not constant, but varies to some extent. Therefore, if the calculation range is set larger near pixels with a large corresponding PSF among the pixels included in the reconstruction range, and smaller near pixels with a small corresponding PSF, the center position of the calculation range will be offset from the center position of the reconstruction range. With the above configuration, it is possible to set the calculation range according to the variation in the PSF corresponding to the position of a pixel included in the reconstruction range.
[0010] In one example imaging apparatus disclosed herein, the error between the PSFs corresponding to the positions of any two pixels included in the reconstruction range may be below a predetermined threshold. The error in the PSF may be, for example, the Mean Squared Error, the Mean Absolute Error, or the difference in the magnitude of the PSFs.
[0011] According to the above configuration, the sizes of the PSFs corresponding to the positions of the pixels included in the reconstruction range can be made approximately the same. By adopting such a configuration, it is possible to increase the accuracy of the blur correction processing while reducing the computational load.
[0012] In an imaging device according to an example disclosed in this specification, the optical system may have characteristics symmetric with respect to the optical axis.
[0013] According to the above configuration, a common filter can be used for the calculation range and the reconstruction range set symmetrically with respect to the optical axis.
[0014] In an imaging device according to an example disclosed in this specification, the light receiving surface of the imaging element may have a curved shape that approaches the lens side as it moves away from the optical axis.
[0015] If the light receiving surface of the imaging element has a planar shape perpendicular to the optical axis, the size of the PSF increases rapidly as the distance from the optical axis increases. According to the above configuration, it is possible to suppress the increase in the size of the PSF according to the distance from the optical axis.
Embodiment
[0016] (Overall Configuration) The imaging device 10 of the embodiment shown in FIG. 1 is mounted on a vehicle and used, for example. The imaging device 10 includes an imaging unit 12 and an image processing unit 14.
[0017] The imaging unit 12 images the surroundings of the vehicle and outputs image data indicating the captured image to the image processing unit 14. The imaging unit 12 includes an optical system 18 including a lens 16, an imaging element 20, and an imaging control unit 22.
[0018] The optical system 18 projects light incident from outside the imaging device 10 onto the light-receiving surface 20a of the image sensor 20 via the lens 16. The image sensor 20 is equipped with a plurality of imaging sensors 20b arranged vertically and horizontally on the light-receiving surface 20a. Each imaging sensor 20b may be, for example, a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor. Each imaging sensor 20b generates an electrical signal indicating the intensity (luminance) of the red, green, and blue components of the received light. The image sensor 20 outputs digital data indicating the intensity (luminance) of the red, green, and blue components of the received light for each imaging sensor 20b, i.e., for each pixel, as image data. The imaging control unit 22 is implemented by a microcontroller equipped with, for example, a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc., and operates by the CPU executing processing according to a program stored in the ROM. The imaging control unit 22 controls the imaging operation of the image sensor 20.
[0019] The image processing unit 14 includes a development processing unit 24, a blur correction processing unit 26, and a storage unit 28. The development processing unit 24 and the blur correction processing unit 26 are implemented, for example, by a microcontroller including a CPU, a ROM, a RAM, etc., and operate by the CPU executing processing according to a program stored in the ROM. The storage unit 28 is implemented, for example, by a memory device such as a flash memory, a SSD (Solid State Drive), or a HDD (Hard Disk Drive). The image data input from the imaging device 10 is stored in the storage unit 28. The development processing unit 24 executes various development processes such as demosaicing, color correction, white balance, and gamma correction on the image data stored in the storage unit 28. The blur correction processing unit 26 executes a blur correction process, which will be described later, on the image data stored in the storage unit 28. The storage unit 28 stores not only the image data but also various filters, parameters, etc. required for the development process in the development processing unit 24 and the blur correction process in the blur correction processing unit 26. In the image processing unit 14, first, the development processing unit 24 may perform a development process on the image data input from the imaging unit 12, and then the blur correction processing unit 26 may execute a blur correction process. Conversely, first, the blur correction processing unit 26 may perform a blur correction process on the image data input from the imaging unit 12, and then the development processing unit 24 may execute a development process. The image processing unit 14 can output the image data subjected to the development process by the development processing unit 24 and / or the blur correction process by the blur correction processing unit 26 to various external devices (not shown).
[0020] (Processing performed by the blur correction processing unit 26) Hereinafter, the processing performed by the blur correction processing unit 26 will be described while referring to the flowchart of FIG. 2.
[0021] In S2, the blur correction processing unit 26 sets the operation count i to 0.
[0022] In S4, the blur correction processing unit 26 increases the operation count i by 1.
[0023] In S6, the blur correction processing unit 26 identifies the position and shape of the calculation range and the position and shape of the reconstruction range to be used in the i-th calculation. In this embodiment, the position and shape of the calculation range at calculation number i (i=1,...,N) and the position and shape of the reconstruction range at calculation number i (i=1,...,N) are stored in advance in the storage unit 28.
[0024] In S8, the blur correction processing unit 26 identifies the filter to be used in the i-th calculation. In this embodiment, the filters to be used in the calculation count i (i=1,...,N) are pre-stored in the storage unit 28 in the form of a filter bank.
[0025] In S10, the blur correction processing unit 26 extracts the data of pixels included in the calculation range identified in S6 from the image data before correction, and obtains the extracted image data.
[0026] In S12, the blur correction processing unit 26 performs blur correction processing on the cropped image data acquired in S10 using the filter identified in S8, and calculates the corrected cropped image data. In this embodiment, the imaging device 10 can perform various blur correction processes in S12. For example, blur correction processing involving Fourier transforms using an inverse filter generated based on the OTF (Optical Transfer Function) or a Wiener filter may be performed. Alternatively, blur correction processing may be performed using a pre-trained machine learning model such as a neural network, with the uncorrected image data containing blur and the filter as inputs, and the corrected image data with the blur removed as output.
[0027] In S14, the blur correction processing unit 26 extracts the data of pixels included in the reconstruction range identified in S6 from the corrected cropped image data calculated in S12, and obtains reconstructed image data.
[0028] In S16, the blur correction processing unit 26 determines whether the number of calculations i has reached a preset threshold N. If the number of calculations i has not reached the threshold N (NO), the process returns to S4. If the number of calculations i has reached the threshold N (YES), the process proceeds to S18.
[0029] In S18, the blur correction processing unit 26 combines the reconstructed image data acquired in S14 for each calculation from the 1st to the Nth time to obtain the corrected image data. After S18, the processing shown in Figure 2 is completed.
[0030] Figure 3 schematically illustrates how the corrected image data is generated from the uncorrected image data through the process shown in Figure 2. As shown in the upper part of Figure 3, in the nth operation (n=1,...,N), the pixel data included in the operation range n is extracted from the uncorrected image data, and the extracted image data is obtained. Also, the filter n to be used in the nth operation is identified from the filter bank. Then, blur correction processing using filter n is performed on the extracted image data of the operation range n. After that, the pixel data included in the reconstruction range n is extracted from the corrected extracted image data, and the reconstructed image data is obtained. As shown in the middle and lower parts of Figure 3, the acquisition of such reconstructed image data is performed similarly for the (n+1)th operation, the (n+2)th operation, and so on. In this way, the corrected image data can be obtained by combining the reconstructed image data obtained from the 1st operation to the Nth operation.
[0031] (Pre-settings of the blur correction processing unit 26) As described above, the position and shape of the calculation range, the position and shape of the reconstruction range, and the filter used in the i-th calculation (i=1,...,N) performed by the blur correction processing unit 26 are pre-set based on the size of the PSF (Point Spread Function) corresponding to each pixel position in the image data output by the imaging unit 12. Below, we will explain the size of the PSF corresponding to each pixel position in the image data, and the settings of the reconstruction range, calculation range, and filter used in the i-th calculation based on the size of the PSF.
[0032] (PSF size) As shown in Figure 4, the light-receiving surface 20a of the image sensor 20 is positioned such that the optical axis OX is located at the center of the light-receiving surface 20a. Furthermore, the light-receiving surface 20a of the image sensor 20 is curved so that it moves closer to the lens 16 as it moves away from the optical axis OX. However, compared to the image plane IP, the light-receiving surface 20a of the image sensor 20 has a shape that moves further away from the lens 16 as it moves away from the optical axis OX. Therefore, as shown in Figure 5, in the image data output from the image sensor 20, the larger the angle of view from the optical axis OX, that is, the larger the number of pixels from the center of the light-receiving surface 20a, the larger the PSF. The size of the PSF corresponding to each position of such pixels can be calculated in advance based on the characteristics of the optical system 18.
[0033] (Setting the reconstruction range) In the imaging device 10 of this embodiment, the position and shape of the reconstruction range used for calculation count i(i=1,...,N) are set so that when all the reconstruction ranges used for calculation count i(i=1,...,N) are lined up, the entire image data is filled without any overlap or gaps (see the right side of Figure 3). Furthermore, in the imaging device 10 of this embodiment, the position and shape of the reconstruction range used for calculation count i(i=1,...,N) are set so that pixels with approximately the same PSF size corresponding to their respective positions are included in the same reconstruction range, in order to perform blur correction processing on as many pixels as possible at once. Specifically, for a reference pixel PR and a pixel PT to be judged, if the error between the PSFs corresponding to their respective positions is below a predetermined threshold, the pixel PT to be judged is included in the same reconstruction range as the reference pixel PR. As the error between PSFs, for example, the Mean Squared Error may be used, the Mean Absolute Error may be used, or the difference in PSF size may be used. The position and shape of the reconstruction range used for the number of operations i (i=1,...,N), which are set in this manner, are stored in the storage unit 28 beforehand.
[0034] (Setting the calculation range) In the imaging device 10 of this embodiment, the calculation range used for calculation count i (i=1,...,N) is set to have a position and shape that at least encompasses the reconstruction range used for calculation count i (i=1,...,N). In this case, even pixels included in the same reconstruction range do not have exactly the same PSF size corresponding to their position. Therefore, in the imaging device 10 of this embodiment, the position and shape of the calculation range are set so that the PSF corresponding to each position of pixels on the boundary of the reconstruction range is included in the calculation range. For example, as shown in Figure 6, if the reconstruction range has a rectangular shape, the smallest calculation range is set to have a position and shape that encompasses the PSF corresponding to each position of pixels at the four corners. When the calculation range is set in this way, if the PSF sizes corresponding to each position of pixels at the four corners of the reconstruction range are not the same, the center position of the calculation range will be offset from the center position of the reconstruction range.
[0035] Furthermore, when Fourier transform is used in blur correction processing, the size of the calculation range is 2 m ×2 n If (m and n are natural numbers), it becomes possible to use FFT (Fast Fourier Transform), which reduces the computational load. For this reason, in the imaging device 10 of this embodiment, after setting the minimum calculation range so that the PSF corresponding to the position of each of the four corner pixels of the reconstruction range is included as described above, the minimum calculation range is included, and the center position coincides with that minimum calculation range 2 m ×2 n The range of size (where m and n are natural numbers) is set again as the calculation range. The position and shape of the calculation range used for the number of calculations i (i=1,...,N), set in this way, are stored in the storage unit 28 beforehand.
[0036] (Filter settings) In the imaging device 10 of this embodiment, the filters used for calculation count i (i=1,...,N) are set according to the size of the PSF corresponding to each position of the pixels included in the calculation range used for calculation count i (i=1,...,N), and are filters suitable for blur correction processing. The filters used for calculation count i (i=1,...,N), set in this way, are stored in the storage unit 28 in advance in the form of a filter bank.
[0037] In the imaging device 10 of this embodiment, the optical system 18 of the imaging unit 12 has characteristics that are symmetric with respect to the optical axis OX. Therefore, if the reconstruction range and calculation range used for calculation count i (i=1,···,N) are arranged symmetrically with respect to the optical axis OX with respect to the reconstruction range and calculation range used for calculation count j (j=1,···,N, j≠i), a common filter can be used. This makes it possible to reduce the number of filters stored in the storage unit 28.
[0038] The configuration of the imaging apparatus disclosed herein is listed below. (Composition 1) An image sensor (20) that outputs image data showing an image captured through an optical system (18) including a lens (16), It is equipped with a bokeh correction processing unit (26), The aforementioned blur correction processing unit, (a) A process of extracting the data of pixels included in the calculation range from the image data and obtaining the extracted image data, (b) A process to perform blur correction processing using a filter on the extracted image data to calculate the corrected extracted image data, (c) A process to extract the data of pixels included in the reconstruction range from the corrected extracted image data and obtain reconstructed image data, (d) A process that repeatedly performs the processes described in (a)-(c) by changing the calculation range, the filter, and the reconstruction range, (e) A process of combining multiple reconstructed image data obtained by the process in (c) multiple times to obtain corrected image data, It is configured to perform the following actions: An imaging device (10) in which the position and shape of the calculation range are set according to the position and shape of the corresponding reconstruction range and the size of the PSF (Point Spread Function) corresponding to the position of the pixels included in the corresponding reconstruction range. (Configuration 2) The imaging apparatus according to configuration 1, wherein the center position of the calculation range is offset from the center position of the corresponding reconstruction range. (Composition 3) The imaging apparatus according to configuration 2, wherein the error between the PSFs corresponding to the respective positions of any two pixels included in the reconstruction range is below a predetermined threshold. (Composition 4) The imaging apparatus according to any one of configurations 1 to 3, wherein the optical system has characteristics symmetrical with respect to the optical axis. (Composition 5) The imaging device according to configuration 4, wherein the light-receiving surface (20a) of the image sensor has a curved shape that moves closer to the lens side as it moves away from the optical axis.
[0039] Although embodiments have been described in detail above, these are merely illustrative and do not limit the scope of the claims. The technologies described in the claims include various modifications and changes to the specific examples illustrated above. The technical elements described in this specification or drawings exhibit technical usefulness individually or in various combinations, and are not limited to the combinations described in the claims at the time of filing. Furthermore, the technologies illustrated in this specification or drawings achieve multiple objectives simultaneously, and achieving even one of these objectives constitutes technical usefulness. [Explanation of symbols]
[0040] 10: Imaging device, 12: Imaging unit, 14: Image processing unit, 16: Lens, 18: Optical system, 20: Image sensor, 20a: Light-receiving surface, 20b: Imaging sensor, 22: Imaging control unit, 24: Development processing unit, 26: Blur correction processing unit, 28: Memory unit
Claims
1. An image sensor (20) that outputs image data showing an image captured through an optical system (18) including a lens (16), It is equipped with a blur correction processing unit (26), The aforementioned blur correction processing unit, (a) A process to extract the data of pixels included in the calculation range from the image data and obtain the extracted image data, (b) A process to perform blur correction processing using a filter on the extracted image data to calculate the corrected extracted image data, (c) A process of extracting the data of pixels included in the reconstruction range from the corrected extracted image data to obtain reconstructed image data, (d) A process that repeatedly executes the processes (a)-(c) by changing the calculation range, the filter, and the reconstruction range, (e) A process of combining multiple reconstructed image data obtained by the process of (c) multiple times to obtain corrected image data, It is configured to perform the following actions: The position and shape of the calculation range are set according to the position and shape of the corresponding reconstruction range and the size of the PSF (Point Spread Function) corresponding to the position of the pixels included in the corresponding reconstruction range. The reconstruction range includes two or more pixels, and the center position of the calculation range is offset from the center position of the corresponding reconstruction range. The imaging device (10) is configured such that the position and shape of the reconstruction range are set such that the error between the PSFs corresponding to the positions of any two pixels included in the reconstruction range falls below a predetermined threshold.
2. The imaging apparatus according to claim 1, wherein the optical system has characteristics symmetrical with respect to the optical axis.
3. The imaging device according to claim 2, wherein the light-receiving surface (20a) of the image sensor has a curved shape that moves closer to the lens side as it moves away from the optical axis.