Information processing device, imaging system, and information processing method

Abstract

An information processing device according to the present disclosure comprises: a first correction unit that corrects an output from a plurality of pixels by using first correction data which is based on a characteristic variation of the plurality of pixels, at least one of the plurality of pixels having a different characteristic; and a second correction unit that uses second correction data, which is generated on the basis of the corrected output corresponding to the plurality of pixels, to further correct the corrected output.

Description

Information processing device, imaging system, and information processing method

[0001] The present disclosure relates to an information processing device, an imaging system, and an information processing method.

[0002] 2. Description of the Related Art There is known a technology for detecting an object such as a person by performing a recognition process based on an image captured by a camera.

[0003] Japanese Patent Application Laid-Open No. 2004-222154

[0004] There is a demand for technology that can obtain more accurate recognition results based on captured images.

[0005] An object of the present disclosure is to provide an information processing device, an imaging system, and an information processing method that are capable of outputting images for obtaining more accurate recognition results.

[0006] The imaging system according to the present disclosure includes a sensor in which at least one pixel among a plurality of pixels has different characteristics, a first correction unit that corrects outputs from the plurality of pixels using first correction data based on the characteristic variations of the plurality of pixels in the sensor, and a second correction unit that further corrects the corrected outputs using second correction data generated based on corrected corrected outputs corresponding to the plurality of pixels.

[0007] FIG. 1 is a schematic diagram for explaining an overview of the technology of the present disclosure. FIG. 1 is a block diagram of an example showing a first system configuration applicable to the present disclosure. FIG. 2 is a block diagram of an example configuration of an imaging device having the first system configuration applicable to the present disclosure. FIG. 3 is a schematic diagram showing an example configuration of an imaging unit applicable to the present disclosure. FIG. 4 is a circuit diagram showing an example circuit configuration of a pixel applicable to the present disclosure. FIG. 5 is a perspective view schematically showing the structure of an example imaging device having the first system configuration. FIG. 6 is a block diagram of an example block diagram showing a second system configuration applicable to the present disclosure. FIG. 7 is a block diagram of an example block diagram showing a third system configuration applicable to the present disclosure. FIG. 8 is a block diagram showing a hardware configuration of an example information processing device applicable to the third system configuration. FIG. 9 is a functional block diagram of an example for explaining functions of an imaging system according to a first embodiment. FIG. 10 is a schematic diagram for explaining an overview of sensitivity variation correction processing according to the first embodiment. FIG. 11 is a schematic diagram for explaining the sensitivity variation correction processing according to the first embodiment in more detail. FIG. 12 is a schematic diagram for explaining a first example of a method for acquiring sensitivity variation correction data according to the first embodiment. FIG. 10 is a schematic diagram for explaining a second example of a method for acquiring sensitivity variation correction data according to the first embodiment. FIG. 11 is a schematic diagram for explaining processing when a flat portion is moving in a plurality of sensitivity variation images acquired in time series according to the first embodiment. FIG. 12 is a schematic diagram for explaining a correction target of sensitivity variation correction according to the first embodiment. FIG. 13 is a schematic diagram for explaining a third example of a method for acquiring sensitivity variation correction data according to the first embodiment. FIG. 14 is a schematic diagram for explaining a first example of saturated region estimation processing according to the first embodiment. FIG. 15 is a schematic diagram for explaining a second example of saturated region estimation processing according to the first embodiment. FIG. 16 is a diagram showing an example of a histogram created based on an image captured by a sensitivity variation expansion sensor according to the first embodiment. FIG. 17 is a schematic diagram for explaining saturated region correction processing according to the first embodiment. FIG. 18 is a schematic diagram for explaining saturated region correction processing according to the first embodiment. FIG. 19 is a schematic diagram for explaining processing when saturated pixels are corrected using information correlated with noise according to the first embodiment. FIG. 19 is a schematic diagram showing an example of pixel response characteristics.FIG. 1 is a functional block diagram of an example of a configuration for performing exposure plane synthesis according to a first example of exposure plane synthesis processing in the first embodiment. FIG. 2 is a functional block diagram of an example of a configuration for performing exposure plane synthesis according to a second example of exposure plane synthesis processing in the first embodiment. FIG. 3 is a schematic diagram for explaining processing by an exposure plane synthesis unit according to the first and second examples of exposure plane synthesis processing in the first embodiment. FIG. 4 is a schematic diagram for explaining processing by an exposure plane synthesis unit according to the second example of exposure plane synthesis processing in the first embodiment. FIG. 5 is a functional block diagram of an example of a configuration for performing exposure plane synthesis according to a third example of exposure plane synthesis processing in the first embodiment. FIG. 6 is a block diagram of an example of a configuration of an exposure plane synthesis unit according to the third example of exposure plane synthesis processing in the first embodiment. FIG. 7 is a schematic diagram for explaining processing by an exposure plane synthesis unit according to the third example of exposure plane synthesis processing in the first embodiment. FIG. 8 is a block diagram of an example of a configuration of an imaging system according to the first embodiment. FIG. 9 is a block diagram of an example of a configuration of an imaging system according to the first embodiment. FIG. 10 is a block diagram of an example of a configuration of an imaging system according to the first embodiment. FIG. 11 is a block diagram of an example of a configuration of an imaging system according to the first embodiment. FIG. 12 is a schematic diagram for explaining a first example of a recognition unit when auxiliary information is not used according to the first embodiment. FIG. 10 is a schematic diagram for explaining a first example of a recognition unit according to the first embodiment when auxiliary information is not used. FIG. 11 is a schematic diagram for explaining a first example of a recognition unit according to the first embodiment when auxiliary information is not used. FIG. 12 is a schematic diagram for explaining processing by a second example of a recognition unit according to the first embodiment when auxiliary information is not used. FIG. 13 is a schematic diagram for explaining processing by a third example of a recognition unit according to the first embodiment when auxiliary information is not used. FIG. 14 is a schematic diagram for explaining processing by a fourth example of a recognition unit according to the first embodiment when auxiliary information is not used. FIG. 15 is a schematic diagram for explaining a fifth example of a recognition unit according to the first embodiment when auxiliary information is not used. FIG. 16 is a schematic diagram for explaining a sixth example of a recognition unit according to the first embodiment when auxiliary information is not used. FIG. 17 is a schematic diagram showing an example configuration of a recognition unit applicable to the first embodiment.FIG. 1 is a schematic diagram showing an example of the configuration of a recognition unit applicable to the first embodiment. FIG. 2 is a schematic diagram for explaining the data structure of auxiliary information according to the first embodiment. FIG. 3 is a schematic diagram showing an example of the configuration for separating auxiliary information from a sensitivity variation image according to the first embodiment. FIG. 4 is a schematic diagram for explaining a configuration example for separating auxiliary information from a sensitivity variation image according to the first embodiment. FIG. 5 is a schematic diagram for explaining feedback of a recognition result to a sensitivity variation expansion sensor according to the first embodiment. FIG. 6 is a schematic diagram for explaining the hardware configuration of a sensitivity variation expansion sensor according to the second embodiment. FIG. 7 is a schematic diagram for explaining an example of the structure of a pixel applicable to the second embodiment. FIG. 8 is a schematic diagram for explaining sensitivity variation control in an electronic domain according to the second embodiment. FIG. 9 is a schematic diagram showing an example of the configuration for discarding charges accumulated in a photoelectric conversion unit during an accumulation period, applicable to the second embodiment. FIG. 10 is a schematic diagram for explaining sensitivity variation control in a subsequent domain according to the second embodiment. FIG. 11 is a schematic diagram for explaining a first example of a sensitivity variation pattern according to the second embodiment. FIG. 10 is a schematic diagram for explaining a second example of a sensitivity variation pattern according to a second embodiment. FIG. 11 is a schematic diagram for explaining a second example of a sensitivity variation pattern according to a second embodiment. FIG. 12 is a schematic diagram for explaining a second example of a sensitivity variation pattern according to a second embodiment. FIG. 13 is a schematic diagram for explaining a third example of a sensitivity variation pattern according to a second embodiment. FIG. 14 is a schematic diagram for explaining a fourth example of a sensitivity variation pattern according to a second embodiment. FIG. 15 is a schematic diagram for explaining a fourth example of a sensitivity variation pattern according to a second embodiment. FIG. 16 is a schematic diagram for explaining a rotation operation on an index arrangement according to a second embodiment. FIG. 17 is a schematic diagram for explaining a method of holding and outputting auxiliary information according to a second embodiment. FIG. 18 is a schematic diagram for explaining a second example of a method of holding auxiliary information according to a second embodiment.10 is a schematic diagram for explaining a third example of a method for holding auxiliary information according to the second embodiment. FIG. 11 is a schematic diagram for explaining a fourth example of a method for holding auxiliary information according to the second embodiment. FIG. 12 is a schematic diagram showing an example of the configuration of frame image data output from a sensitivity variation expansion sensor, applicable to the second embodiment. FIG. 13 is a schematic diagram for explaining a third example of a method for outputting auxiliary information according to the second embodiment. FIG. 14 is a schematic diagram showing an example of the configuration of an imaging system according to the third embodiment. FIG. 15 is a schematic diagram for explaining a sensitivity variation image according to the third embodiment. FIG. 16 is a schematic diagram showing an example of the configuration of a learning system according to a first example of the third embodiment. FIG. 17 is a schematic diagram for explaining basic learning processing of a recognizer learning unit according to the third embodiment. FIG. 18 is a schematic diagram for explaining an example of learning processing of a recognizer learning unit according to the third embodiment. FIG. 19 is a schematic diagram for explaining fingerprinting according to a second example of the third embodiment. FIG. 19 is a schematic diagram for explaining how to deal with aging of a color filter, applicable to the third embodiment. FIG. 19 is a schematic diagram showing an example of a pixel sensitivity change model, applicable to the third embodiment. FIG. 19 is a schematic diagram for explaining a higher-level conceptualized process for obtaining a more accurate recognition result based on a captured image, according to a fourth embodiment. FIG. 10 is a functional block diagram for more specifically illustrating processing according to a fourth embodiment. FIG. 11 is a flowchart of an example showing a first correction processing according to the fourth embodiment. FIG. 12 is a flowchart of an example showing a second correction processing according to the fourth embodiment. FIG. 13 is a schematic diagram for explaining variations in the execution order of the first correction processing and the second correction processing according to the fourth embodiment. FIG. 14 is a schematic diagram for explaining variations in output due to the correction processing according to the fourth embodiment. FIG. 15 is a schematic diagram showing an example structure of a variation magnification sensor applicable to the fourth embodiment. FIG. 16 is a schematic diagram showing an example correspondence relationship between the configuration according to the first embodiment and a generic conceptual configuration according to the fourth embodiment. FIG. 17 is a schematic diagram showing an example structure in which one OCL is shared by a plurality of pixels, applicable to a second example of the fourth embodiment. FIG. 18 is a schematic diagram showing an example of light incident on a pixel group in which one OCL is shared by two pixels. FIG. 19 is a schematic diagram for explaining processing according to a second example of the fourth embodiment.10 is a schematic diagram showing an example of changing the number of OCLs shared by pixels according to the second example of the fourth embodiment. FIG. 11 is a schematic diagram showing an example of changing the light-collecting range in a pixel according to the second example of the fourth embodiment. FIG. 12 is a schematic diagram for explaining an example of implementing sensor characteristic variations in a variation spreading sensor according to the second example of the fourth embodiment. FIG. 13 is a schematic diagram for explaining an example of implementing sensor characteristic variations in a variation spreading sensor according to the second example of the fourth embodiment. FIG. 14 is a schematic diagram showing an enlarged view of a polarizing filter having a wire grid polarizer. FIG. 15 is a schematic diagram showing an example of the structure of a variation spreading sensor applicable to the fourth embodiment. FIG. 16 is a schematic diagram for explaining processing according to the third example of the fourth embodiment. FIG. 17 is a schematic diagram showing an example of a polarizing filter in which the extinction ratio is spatially modulated within a polarizer group, applicable to the third example of the fourth embodiment. FIG. 18 is a schematic diagram for explaining an example of super-resolution according to existing technology. FIG. 19 is a schematic diagram showing an example of a low-resolution imaging system including aliasing. 10 is a schematic diagram showing an implementation example of a sensor in a case where pixels that do not attenuate or cut high-frequency components and normal pixels are mixed according to a fourth example of the fourth embodiment. FIG. 11 is a schematic diagram for explaining processing according to the fourth example of the fourth embodiment. FIG. 12 is a schematic diagram for explaining processing according to the fourth example of the fourth embodiment. FIG. 13 is a schematic diagram showing an implementation example in which the size of an OCL is spatially modulated according to a second example of the fourth example of the fourth embodiment. FIG. 14 is a schematic diagram showing an implementation example in which the shape of the aperture of each pixel is spatially modulated according to a third example of the fourth example of the fourth embodiment. FIG. 15 is a schematic diagram showing an implementation example in which the phase of the aperture of each pixel is spatially modulated according to a fourth example of the fourth example of the fourth embodiment. FIG. 16 is a schematic diagram showing an implementation example in which the phase of an OCL of each pixel is spatially modulated according to a fifth example of the fourth example of the fourth embodiment. FIG. 17 is a schematic diagram showing an implementation example in which at least one of the pitch and size of each pixel is spatially modulated according to a sixth example of the fourth example of the fourth embodiment. FIG. 18 is a schematic diagram showing an example of an optical filter according to a first example in the fifth example of the fourth embodiment. FIG. 10 is a schematic diagram for explaining a process according to a fifth example of the fourth embodiment.FIG. 10 is a schematic diagram for explaining photon counting by SPAD. FIG. 11 is a diagram showing the configuration of an example of a pixel applicable to a sixth example of the fourth embodiment. FIG. 12 is a block diagram showing the configuration of an example of a variation magnification sensor according to the sixth example of the fourth embodiment. FIG. 13 is a schematic diagram for explaining processing according to the sixth example of the fourth embodiment. FIG. 14 is a schematic diagram for explaining an example of the characteristics of the variation magnification sensor according to the sixth example of the fourth embodiment.

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are denoted by the same reference numerals, and redundant description will be omitted.

[0009] Hereinafter, embodiments of the present disclosure will be described in the following order: 1. Overview of technology related to the present disclosure 2. Technology applicable to the present disclosure 2-1. First system configuration example 2-2. Second system configuration example 2-3. Third system configuration example 3. First embodiment 3-1. Sensitivity variation correction processing according to the first embodiment 3-2. Processing related to saturated regions according to the first embodiment 3-2-1. Saturated region estimation processing 3-2-2. Saturated region correction processing 3-3. Exposure plane combination processing according to the first embodiment 3-4. Configuration example of a recognizer according to the first embodiment 3-4-1. Configuration example of a recognizer when auxiliary information is not used 3-4-2. Configuration example of a recognizer when auxiliary information is used 4. Second embodiment 4-1. Photon domain 4-2. Electron domain 4-3. Later domain 4-4. Specific example of sensitivity variation generation 4-5. Method of holding and outputting sensitivity variation information 5. Third Embodiment 5-1. Regarding improvement of anonymity according to the third embodiment 5-2. Regarding fingerprinting according to the third embodiment 5-3. Handling of erroneous verification due to aging degradation 6. Fourth Embodiment 6-1. Regarding superordinate conception according to the fourth embodiment 6-1-1. Superordinate correction processing according to the fourth embodiment 6-1-2. Regarding other configurations related to the correction processing according to the fourth embodiment 6-2. Example of sensor structure applicable to the fourth embodiment 6-3. First example of the fourth embodiment 6-4. Second example of the fourth embodiment 6-4-1. Overview of the second example of the fourth embodiment 6-4-2. Processing in the second example of the fourth embodiment 6-4-3. Example of sensor implementation in the second example of the fourth embodiment 6-5. Third example of the fourth embodiment 6-5-1. Overview of the third example of the fourth embodiment 6-5-2. Example of sensor implementation in the third example of the fourth embodiment 6-5-3. Processing in the third example of the fourth embodiment 6-6. Fourth example of the fourth embodiment 6-6-1. Super-resolution and aliasing 6-6-2. Overview of the fourth example of the fourth embodiment 6-6-3. Example of sensor implementation in the fourth example of the fourth embodiment 6-6-4. Processing in the third example of the fourth embodiment6-6-5. Another example of sensor implementation in the fourth example of the fourth embodiment 6-7. Fifth example of the fourth embodiment 6-7-1. Regarding the first example of the fifth example of the fourth embodiment 6-7-1-1. Sensor implementation example in the first example of the fifth example of the fourth embodiment 6-7-1-2. Processing in the first example of the fifth example of the fourth embodiment 6-7-2. Regarding the second example of the fifth example of the fourth embodiment 6-8. Sixth example of the fourth embodiment 6-8-1. Regarding the variation magnification sensor using SPAD 6-8-2. Overview of the sixth example of the fourth embodiment 6-8-3. Processing in the sixth example of the fourth embodiment

[0010] (1. Overview of Technology Relating to the Present Disclosure) First, an overview of the technology relating to the present disclosure will be described. FIG. 1 is a schematic diagram for explaining the overview of the technology relating to the present disclosure. In the present disclosure, for example, a sensor in which the sensitivity of pixels as a pixel characteristic varies on a pixel-by-pixel basis is used as a sensor for acquiring an image for a recognizer to perform recognition processing. Hereinafter, a sensor in which the sensitivity of pixels varies on a pixel-by-pixel basis will be referred to as a sensitivity variation expansion sensor as appropriate.

[0011] In a sensor with increased sensitivity variation, for example, a PRNU (Photo Response Non-Uniformity), which indicates sensitivity variation per pixel, is made larger than that of a general sensor. For example, in a sensor with increased sensitivity variation, the sensitivity ratio between the minimum sensitivity pixel and the maximum sensitivity pixel is set to, for example, several times to several hundred times.

[0012] The imaging system 1 according to the present disclosure performs imaging using a sensitivity variation expanding sensor and acquires a captured image as a sensor output of the sensitivity variation expanding sensor (step S1). The captured image contains noise due to sensitivity variations on a pixel-by-pixel basis, as schematically shown in image 4a. Image 4a is, for example, an image that is extremely difficult to visually recognize.

[0013] The imaging system 1 corrects the sensitivity variation of the captured image (image 4a) acquired by the sensitivity variation expanding sensor using sensitivity variation correction data 2, which is correction data for correcting sensitivity variation on a pixel-by-pixel basis in the sensitivity variation expanding sensor (step S2). As schematically shown in image 4b, the image corrected using the sensitivity variation correction data 2 has reduced noise compared to the uncorrected image 4a.

[0014] The sensitivity variation correction data 2 may be obtained by, for example, calibrating the sensitivity variation expansion sensor. However, the sensitivity variation correction data 2 may also be obtained by estimation (online estimation) in accordance with the operation of the sensitivity variation expansion sensor.

[0015] Image 4b ​​corrected in step S2 may contain saturated pixels whose pixel values ​​are saturated. Therefore, imaging system 1 performs saturation correction on image 4b ​​corrected in step S2 using saturated pixel information 3, which indicates saturated pixel regions caused by pixels that become saturated when exposed to a predetermined amount of light (step S3). An image corrected using saturated pixel information 3 is an image in which noise is almost completely eliminated, as schematically shown in image 4c. Unlike images 4a and 4b described above, image 4c is an image that is easy to view.

[0016] The saturated pixel information 3 may be obtained by, for example, threshold determination of the sensor output when a certain amount of light or more is irradiated onto the sensitivity variation expansion sensor. Alternatively, the saturated pixel information 3 may be obtained by calibration related to saturation of the sensitivity variation expansion sensor.

[0017] The imaging system 1 may perform a recognition process on the image 4c corrected in step S3 (step S4). The recognition process may be performed using a neural network to which a pre-trained model is applied. Image 4d schematically illustrates a rectangular bounding box 5 set for an object detected by the recognition process.

[0018] Note that the recognition process according to the present disclosure is not limited to a process using a neural network, and for example, other means may be applied to the recognition process according to the present disclosure as long as the recognition process is capable of learning and is executed using a trained model.

[0019] In the following, unless otherwise specified, neural networks will be referred to as networks. Networks for communication, such as the Internet, will be referred to as communication networks to distinguish them from networks based on neural networks.

[0020] The image 4c obtained by performing the sensitivity variation correction process (step S2) and the saturation correction process (step S3) on the image captured by the sensitivity variation expansion sensor has an expanded dynamic range compared to an image captured by a general sensor. Therefore, by applying the imaging system 1, it is possible to obtain a more accurate recognition result.

[0021] (2. Techniques Applicable to the Present Disclosure) Next, techniques applicable to the present disclosure will be described.

[0022] (2-1. First System Configuration Example) Fig. 2 is a block diagram of an example showing a first system configuration applicable to the present disclosure. The imaging system 1a shown in Fig. 2 has an imaging device 10a including an imaging unit 100, a signal processing unit 110, and a recognition unit 120.

[0023] The image capturing unit 100 includes the above-described sensitivity variation expanding sensor, and outputs a captured image captured by the sensitivity variation expanding sensor as a sensor output. The captured image output from the image capturing unit 100 is supplied to the signal processing unit 110.

[0024] The signal processing unit 110 includes a sensitivity variation correction unit and a saturation correction unit. The sensitivity variation correction unit corrects sensitivity variations in the captured image based on sensitivity variation correction data 2. The saturation correction unit performs saturation correction on the captured image, for which sensitivity variations have been corrected by the sensitivity variation correction unit, based on saturated pixel information 3. The captured image, for which sensitivity variation correction and saturation correction have been performed by the signal processing unit 110, is supplied to the recognition unit 120.

[0025] The recognition unit 120 performs recognition processing on the image supplied from the signal processing unit 110 and outputs the recognition result obtained by the recognition processing.

[0026] 3 is a block diagram showing an example of the configuration of an image capture device 10a having a first system configuration applicable to the present disclosure. In FIG. 3, the image capture device 10a is configured as a CIS (CMOS (Complementary Metal Oxide Semiconductor) image sensor) and includes an image capture block 1030 and a signal processing block 1040. The image capture block 1030 and the signal processing block 1040 are electrically connected by connection lines CL1, CL2, and CL3, which are internal buses, respectively.

[0027] The imaging block 1030 has an imaging unit 1031 (referred to as "imaging section" in FIG. 3), an imaging processing section 1032, an output control section 1033, an output I / F 1034, and an imaging control section 1035, and captures an image of a subject to obtain a captured image. The imaging block 1030 corresponds to the imaging section 100 described with reference to FIG. 2.

[0028] The imaging unit 1031 includes a pixel array in which a plurality of pixels, which are light receiving elements that output signals according to received light through photoelectric conversion, are arranged in a matrix. The imaging unit 1031 is driven by an imaging processing unit 1032 to capture an image of a subject.

[0029] That is, light from an optical system (not shown) is incident on the imaging unit 1031. The imaging unit 1031 receives the incident light from the optical system at each pixel included in the pixel array, performs photoelectric conversion, and outputs an analog image signal corresponding to the incident light.

[0030] The size of the image generated by the image signal output by the imaging unit 1031 can be selected from a number of sizes, such as width x height, 3968 pixels x 2976 pixels, 1920 pixels x 1080 pixels, and 640 pixels x 480 pixels. The image sizes that the imaging unit 1031 can output are not limited to these examples. Furthermore, the image output by the imaging unit 1031 can be selected to be, for example, a color image of RGB (red, green, blue) or a black-and-white image of luminance only. These selections for the imaging unit 1031 can be made as a type of shooting mode setting.

[0031] Note that information based on the output of each pixel arranged in a matrix in the pixel array is called a frame. The imaging device 10a repeatedly acquires information from the matrix of pixels in the imaging unit 1031 in chronological order at a predetermined rate (frame rate). The imaging device 10a compiles and outputs each piece of acquired information for each frame.

[0032] The imaging processing unit 1032 performs imaging processing related to capturing an image in the imaging unit 1031, such as driving the imaging unit 1031, AD (Analog to Digital) conversion of the analog image signal output by the imaging unit 1031, and imaging signal processing, under the control of the imaging control unit 1035.

[0033] The imaging signal processing performed by the imaging processing unit 1032 includes, for example, a process for determining the brightness of each small region of the image output by the imaging unit 1031 by calculating the average pixel value for each predetermined small region, an HDR (High Dynamic Range) conversion process for converting the image output by the imaging unit 1031 into an HDR image, defect correction, development, etc.

[0034] The imaging processing unit 1032 outputs, as a captured image, a digital image signal obtained by AD conversion or the like of the analog image signal output by the imaging unit 1031. The imaging processing unit 1032 can also output, as a captured image, a RAW image that has not been subjected to processing such as development. Note that an image in which a RAW image has been subjected to processing such as development and in which each pixel has information on each of the RGB colors is called an RGB image.

[0035] The captured image output by the image capture processing unit 1032 is supplied to the output control unit 1033 and also to the image compression unit 1045 of the signal processing block 1040 via a connection line CL2.

[0036] The output control unit 1033 is supplied with the captured image from the imaging processing unit 1032, and also with the signal processing results of signal processing using the captured image and the like from the signal processing block 1040 via a connection line CL3.

[0037] The output control unit 1033 performs output control to selectively output the captured image from the imaging processing unit 1032 and the signal processing result from the signal processing block 1040 to the outside (for example, a memory connected to the outside of the imaging device 10a) from (one) output I / F 1034. That is, the output control unit 1033 selects the captured image from the imaging processing unit 1032 or the signal processing result from the signal processing block 1040 and supplies it to the output I / F 1034.

[0038] The output I / F 1034 is an I / F that outputs the captured image and the signal processing results supplied from the output control unit 1033 to the outside. As the output I / F 1034, for example, a relatively high-speed serial I / F such as a Mobile Industry Processor Interface (MIPI) can be adopted.

[0039] The output I / F 1034 outputs to the outside the captured image from the imaging processing unit 1032 or the signal processing result from the signal processing block 1040 in accordance with the output control of the output control unit 1033. Therefore, for example, when only the signal processing result from the signal processing block 1040 is needed outside and the captured image itself is not needed, it is possible to output only the signal processing result, and it is possible to reduce the amount of data output from the output I / F 1034 to the outside.

[0040] Furthermore, signal processing block 1040 performs signal processing that produces the signal processing results required externally, and outputs the signal processing results from output I / F 1034, thereby eliminating the need to perform signal processing externally and reducing the load on external blocks.

[0041] The imaging control unit 1035 includes a communication I / F 1036 and a register group 1037 .

[0042] The communication I / F 1036 is, for example, 2 The first communication I / F is a serial communication I / F such as an Inter-Integrated Circuit (C), and exchanges necessary information, such as information to be read and written to the register group 1037, with the outside (for example, a control unit that controls the device in which this imaging device 10a is installed).

[0043] The register group 1037 has a plurality of registers and stores various other information, such as imaging information related to the imaging of an image by the imaging unit 1031. For example, the register group 1037 stores imaging information received from the outside by the communication I / F 1036 and the results of imaging signal processing by the imaging processing unit 1032 (for example, brightness of each small region of the captured image, etc.).

[0044] The imaging information stored in the register group 1037 includes, for example, ISO sensitivity (analog gain during AD conversion in the imaging processing unit 1032), exposure time (shutter speed), frame rate, focus, shooting mode, cropping range, etc. (information representing the same).

[0045] The shooting modes include a manual mode in which the exposure time, frame rate, etc. are manually set, and an automatic mode in which the settings are automatically set according to the scene. The automatic mode includes modes according to various shooting scenes, such as night scenes and human faces.

[0046] The cut-out range refers to the range to be cut out from the image output by the imaging unit 1031 when the imaging processing unit 1032 cuts out a part of the image output by the imaging unit 1031 and outputs it as a captured image. By specifying the cut-out range, it becomes possible to cut out, for example, only the range in which a person appears from the image output by the imaging unit 1031. Note that, in addition to a method of cutting out an image from the image output by the imaging unit 1031, there is also a method of reading out only the image (signal) of the cut-out range from the imaging unit 1031.

[0047] The imaging control unit 1035 controls the imaging processing unit 1032 in accordance with the imaging information stored in the register group 1037 , thereby controlling the imaging of images by the imaging unit 1031 .

[0048] The register group 1037 can store imaging information, the results of imaging signal processing in the imaging processing unit 1032, and output control information related to output control in the output control unit 1033. The output control unit 1033 can perform output control to selectively output the captured image and the signal processing results in accordance with the output control information stored in the register group 1037.

[0049] Furthermore, in the imaging device 10a, the imaging control unit 1035 and the CPU 1041 of the signal processing block 1040 are connected via a connection line CL1, and the CPU 1041 can read and write information from and to the register group 1037 via the connection line CL1. That is, in the imaging device 10a, reading and writing of information from and to the register group 1037 can be performed not only from the communication I / F 1036 but also from the CPU 1041.

[0050] The signal processing block 1040 has a CPU 1041, a DSP (Digital Signal Processor) 1042, a memory 1043, a communication I / F 1044, an image compression unit 1045, and an input I / F 1046, and performs predetermined signal processing using the captured image obtained by the imaging block 1030, etc. Note that the CPU 1041 is not limited to this, and may also be an MPU (Micro Processor Unit) or an MCU (Micro Controller Unit). The signal processing block 1040 includes the functions of the signal processing unit 110 described using FIG. 2 and the functions of the recognition unit 120.

[0051] The CPU 1041, DSP 1042, memory 1043, communication I / F 1044, and input I / F 1046 that constitute the signal processing block 1040 are interconnected via a bus, and can exchange information as necessary.

[0052] The CPU 1041 executes a program stored in the memory 1043 to control the signal processing block 1040, read and write information to the register group 1037 of the imaging control unit 1035 via the connection line CL1, and perform various other processes.

[0053] For example, by executing a program, the CPU 1041 functions as an imaging information calculation unit that calculates imaging information using the signal processing results obtained by signal processing in the DSP 1042, and feeds back and stores the new imaging information calculated using the signal processing results to the register group 1037 of the imaging control unit 1035 via the connection line CL1.

[0054] Therefore, the CPU 1041 can control the image pickup by the image pickup unit 1031 and the image pickup signal processing by the image pickup processing unit 1032 according to the signal processing result of the picked-up image.

[0055] Furthermore, the imaging information stored in the register group 1037 by the CPU 1041 can be provided (output) to the outside from the communication I / F 1036. For example, focus information among the imaging information stored in the register group 1037 can be provided from the communication I / F 1036 to a focus driver (not shown) that controls focus.

[0056] By executing a program stored in memory 1043, DSP 1042 functions as a signal processing unit that performs signal processing using the captured image supplied from the imaging processing unit 1032 to the signal processing block 1040 via connection line CL2 and information received from the outside by input I / F 1046.

[0057] The memory 1043 is configured by a static random access memory (SRAM) or a dynamic RAM (DRAM), and stores data necessary for processing by the signal processing block 1040 .

[0058] For example, the memory 1043 stores programs received from outside via the communication I / F 1044, captured images compressed by the image compression unit 1045 and used for signal processing in the DSP 1042, the signal processing results of the signal processing performed by the DSP 1042, information received by the input I / F 1046, etc.

[0059] The communication I / F 1044 is a second communication I / F, such as a serial communication I / F such as an SPI (Serial Peripheral Interface), and exchanges necessary information such as programs executed by the CPU 1041 and DSP 1042 with the outside (for example, a memory or control unit not shown).

[0060] For example, the communication I / F 1044 downloads programs to be executed by the CPU 1041 and the DSP 1042 from an external device, and supplies and stores the programs in the memory 1043. Therefore, the programs downloaded by the communication I / F 1044 enable the CPU 1041 and the DSP 1042 to execute various processes.

[0061] The communication I / F 1044 can exchange any data, in addition to programs, with the outside. For example, the communication I / F 1044 can output to the outside the signal processing results obtained by the signal processing in the DSP 1042. The communication I / F 1044 can also output information according to instructions from the CPU 1041 to an external device, thereby controlling the external device according to instructions from the CPU 1041.

[0062] The signal processing results obtained by the signal processing in the DSP 1042 can be output to the outside from the communication I / F 1044, and can also be written to the register group 1037 of the imaging control unit 1035 by the CPU 1041. The signal processing results written to the register group 1037 can be output to the outside from the communication I / F 1036. The same applies to the processing results performed by the CPU 1041.

[0063] The captured image is supplied from the imaging processing unit 1032 to the image compression unit 1045 via a connection line CL2. The image compression unit 1045 performs a compression process to compress the captured image, and generates a compressed image with a smaller amount of data than the captured image. The compressed image generated by the image compression unit 1045 is supplied to the memory 1043 via a bus and stored therein.

[0064] Here, the signal processing in the DSP 1042 can be performed using the captured image itself, or can be performed using a compressed image generated from the captured image by the image compression unit 1045. Since the compressed image has a smaller amount of data than the captured image, it is possible to reduce the load of signal processing in the DSP 1042 and save the storage capacity of the memory 1043 that stores the compressed image.

[0065] The compression process in the image compression unit 1045 can be, for example, scaling down to convert a captured image of 3968 pixels x 2976 pixels into an image of 640 pixels x 480 pixels. Furthermore, if the signal processing in the DSP 1042 is performed on luminance and the captured image is an RGB image, the compression process can be YUV conversion to convert the RGB image into a YUV image, for example.

[0066] The image compression unit 1045 can be realized by software or by dedicated hardware.

[0067] The DSP 1042 may further execute network processing using a network stored in the memory 1043. Alternatively, the DSP 1042 may acquire a network stored in the RAM 2002 or the storage device 2004 via the communication I / F 1044 and execute network processing.

[0068] The input I / F 1046 is an I / F that receives information from the outside. For example, the input I / F 1046 receives the output of an external sensor (external sensor output) from the external sensor, and supplies the output to the memory 1043 via a bus for storage. As with the output I / F 1034, for example, a parallel I / F such as MIPI can be used as the input I / F 1046.

[0069] Furthermore, as the external sensor, for example, a distance sensor that senses information regarding distance can be used. Furthermore, as the external sensor, for example, an image sensor that senses light and outputs an image corresponding to that light, that is, an image sensor separate from the imaging device 10a, can be used.

[0070] In addition to using the captured image or the compressed image generated from the captured image, the DSP 1042 can perform signal processing using the external sensor output received by the input I / F 1046 from the external sensor described above and stored in the memory 1043.

[0071] In the imaging device 10a configured as described above, signal processing including network processing is performed by the DSP 1042 using a captured image obtained by imaging with the imaging unit 1031 or a compressed image generated from the captured image, and the signal processing result of the signal processing and the captured image are selectively output from the output I / F 1034. Therefore, it is possible to configure a compact imaging device that outputs information required by the user.

[0072] 4 is a schematic diagram showing an example configuration of an imaging unit 1031 applicable to the present disclosure. In FIG. 4, the imaging unit 1031 includes a pixel array section 1011, a vertical scanning section 1012, an AD (Analog to Digital) conversion section 1013, pixel signal lines 1016, vertical signal lines 1017, an imaging operation control section 1019, and an imaging processing section 112.

[0073] The pixel array unit 1011 includes a plurality of pixels Pix, each having a photoelectric conversion element that performs photoelectric conversion on received light. Photodiodes (PDs) can be used as the photoelectric conversion elements. In the pixel array unit 1011, the plurality of pixels Pix are arranged, for example, in a two-dimensional grid pattern in the horizontal direction (row direction) and the vertical direction (column direction). In the pixel array unit 1011, the row direction arrangement of the pixels Pix is ​​called a line. One frame of image (image data) is formed by pixel signals read from a predetermined number of lines in the pixel array unit 1011. For example, if one frame of image is formed with 3,000 pixels x 2,000 lines, the pixel array unit 1011 includes at least 2,000 lines, each including at least 3,000 pixels Pix.

[0074] In the pixel array unit 1011, a rectangular area formed by pixels Pix that output pixel signals that are valid for forming image data is called an effective pixel area. An image for one frame is formed based on pixel signals from the pixels Pix in the effective pixel area.

[0075] In addition, in the pixel array section 1011, pixel signal lines 1016 are connected to the rows and columns of the pixels Pix for each row, and vertical signal lines 1017 are connected to each column.

[0076] The ends of the pixel signal lines 1016 that are not connected to the pixel array unit 1011 are connected to a vertical scanning unit 1012. The vertical scanning unit 1012 transmits control signals such as drive pulses used to read pixel signals from the pixels Pix to the pixel array unit 1011 via the pixel signal lines 1016 under the control of an imaging operation control unit 1019, which will be described later. The ends of the vertical signal lines 1017 that are not connected to the pixel array unit 1011 are connected to an AD conversion unit 1013. The pixel signals read from the pixels are transmitted to the AD conversion unit 1013 via the vertical signal lines 1017.

[0077] 5 is a circuit diagram showing an example of a circuit configuration of a pixel Pix applicable to the present disclosure. The pixel Pix has, as a photoelectric conversion element, for example, a photodiode 1021. In addition to the photodiode 1021, the pixel Pix has a pixel configuration including a transfer transistor 1022, a reset transistor 1023, an amplification transistor 1024, and a selection transistor 1025.

[0078] In the example of FIG. 5 , the four transistors, i.e., the transfer transistor 1022, the reset transistor 1023, the amplification transistor 1024, and the selection transistor 1025, are, for example, N-channel MOS field effect transistors (FETs). Hereinafter, N-channel MOS field effect transistors will be referred to as NMOS transistors. By configuring the pixel Pix using only NMOS transistors, it is possible to achieve optimization in terms of area efficiency and process reduction. Note that the combinations of conductivity types of the transfer transistor 1022, the reset transistor 1023, the amplification transistor 1024, and the selection transistor 1025 shown in FIG. 5 are merely examples, and the combinations are not limited to these.

[0079] For this pixel Pix, a plurality of control lines are wired in common to each pixel Pix in the same pixel row as the above-mentioned pixel signal line 1016. These plurality of control lines are connected on a pixel row basis to output terminals of the vertical scanning unit 1012 corresponding to each pixel row. The vertical scanning unit 1012 outputs a transfer signal TRG, a reset signal RST, and a selection signal SEL to the plurality of control lines as appropriate.

[0080] The photodiode 1021 has an anode electrode connected to a low-potential power supply (e.g., ground potential), and photoelectrically converts received light into charges (here, photoelectrons) of an amount corresponding to the amount of light, and accumulates the charges. The cathode electrode of the photodiode 1021 is electrically connected to the gate electrode of the amplification transistor 1024 via the transfer transistor 1022. Here, the region electrically connected to the gate electrode of the amplification transistor 1024 is the floating diffusion region FD. The floating diffusion region FD is a charge-voltage converter that converts charges into voltage.

[0081] The gate electrode of the transfer transistor 1022 is connected to a high level (for example, V DD A transfer signal TRG, which makes the signal level (transfer transistor 1022) active, is supplied from the vertical scanning unit 1012. The transfer transistor 1022 becomes conductive in response to the transfer signal TRG, and thereby undergoes photoelectric conversion in the photodiode 1021, and transfers the charge accumulated in the photodiode 1021 to the floating diffusion region FD.

[0082] The reset transistor 1023 supplies a voltage V DD The reset transistor 1023 is connected between the node of the power supply of the transistor 1023 and the floating diffusion region FD. A reset signal RST, which becomes active at high level, is supplied to the gate electrode of the reset transistor 1023 from the vertical scanning unit 1012. The reset transistor 1023 becomes conductive in response to the reset signal RST, and transfers the charge in the floating diffusion region FD to a voltage V DD The floating diffusion region FD is reset by discharging the power supply voltage Vcc to the power supply node.

[0083] The amplifier transistor 1024 has a gate electrode connected to the floating diffusion region FD and a drain electrode connected to the voltage V DD The amplifier transistor 1024 serves as an input section of a source follower that reads out a signal obtained by photoelectric conversion in the photodiode 1021. That is, the source electrode of the amplifier transistor 1024 is connected to the vertical signal line 1017 via the selection transistor 1025. The amplifier transistor 1024 and a current source I connected to one end of the vertical signal line 1017 form a source follower that converts the voltage of the floating diffusion region FD into the voltage of the vertical signal line 1017.

[0084] The selection transistor 1025 has a drain electrode connected to the source electrode of the amplification transistor 1024 and a source electrode connected to the vertical signal line 1017. A selection signal SEL, which becomes active at high level, is supplied to the gate electrode of the selection transistor 1025 from the vertical scanning unit 1012. The selection transistor 1025 becomes conductive in response to the selection signal SEL, thereby selecting the pixel Pix and transmitting a signal output from the amplification transistor 1024 to the vertical signal line 1017.

[0085] The readout process for pixel Pix shown in Fig. 5 will be briefly described below. In the initial state, the selection signal SEL, reset signal RST, and transfer signal TRG are all low. Furthermore, the photodiode 1021 is exposed to light, and the transfer transistor 1022 is turned off by the low-state transfer signal TRG. Therefore, the charge generated by the exposure is accumulated in the photodiode 1021.

[0086] At a predetermined timing, the selection signal SEL is set to a high state, turning on the selection transistor 1025. Next, the reset signal RST is set to a high state, and the charge of FD is changed to a voltage V DD The potential of the FD is reset to a predetermined potential by discharging the voltage to the power supply line of the FD. A predetermined time after the reset signal RST is returned to a low state, the transfer signal TRG is set to a high state, and the charge accumulated in the photodiode 1021 by exposure is supplied to the FD and accumulated therein. A voltage corresponding to the charge accumulated in the FD is generated, and this voltage is amplified by the amplifier transistor 1024 and transmitted to the vertical signal line 1017 as a pixel signal via the selection transistor 1025.

[0087] Here, a predetermined time after the reset signal RST is set to a high state, for example, at the timing when the FD state stabilizes, a reset level (black level) voltage P output to the vertical signal line 1017 is converted into a digital value by the corresponding AD converter 1300 included in the AD conversion unit 1013 and temporarily stored in, for example, a register of the AD converter 1300. This voltage P is offset noise. Reading this voltage P is called P-phase (pre-charge) reading, and the period during which the P-phase reading is performed is called a P-phase period.

[0088] Furthermore, after a processing time has elapsed since the transfer signal TRG was set to a high state, for example, when the FD state stabilizes, voltage Q, the signal level output to the vertical signal line 1017, is converted into a digital value by the AD converter 1300 and temporarily stored in, for example, a register of the AD converter 1300. This voltage Q is a signal that includes offset noise and a pixel signal. Reading out this voltage Q is called D-phase (Data Phase) reading, and the period during which D-phase reading is performed is called a D-phase period.

[0089] The AD converter 1300 calculates the difference between the stored voltages P and Q. This makes it possible to obtain a pixel signal from which offset noise has been removed.

[0090] Next, a general description will be given of the sweep scanning by the sweep scanning system in the pixel Pix in Fig. 5. In the sweep scanning, the transfer signal TRG and the reset signal RST are set to a high level, the selection signal SEL is set to a low level, and the cathode electrode of the photodiode 1021 is set to a voltage V DD This connects the charge on the cathode electrode of the photodiode 1021 to a power supply of voltage V DD After this sweep scan, the transfer signal TRG is set to low level, and the photodiode 1021 is supplied with a voltage V DD By disconnecting the power supply, an electronic shutter operation is performed, and the accumulation of charges by photoelectric conversion in the photodiode 1021 can begin.

[0091] Regarding the selection transistor 1025, the voltage V DD5, the pixel circuit of pixel Pix is ​​shown to have a 4Tr configuration including four transistors (Tr), namely, a transfer transistor 1022, a reset transistor 1023, an amplification transistor 1024, and a selection transistor 1025. However, the pixel circuit is not limited to this. For example, the selection transistor 1025 can be omitted, and a 3Tr configuration can be adopted in which the amplification transistor 1024 has the function of the selection transistor 1025. Alternatively, the number of transistors can be increased to 5Tr or more, as needed.

[0092] The AD conversion unit 1013 includes an AD converter 1300 provided for each vertical signal line 1017, a reference signal generation unit 1014, and a horizontal scanning unit 1015. The AD converter 1300 is a column AD converter that performs AD conversion processing on each column of the pixel array unit 1011. The AD converter 1300 performs AD conversion processing on pixel signals supplied from the pixels Pix via the vertical signal lines 1017, and generates two digital values ​​(values ​​corresponding to voltages P and Q, respectively) for correlated double sampling (CDS) processing that reduces noise.

[0093] The AD converter 1300 supplies the two generated digital values ​​to the image capture processing unit 112. The image capture processing unit 112 performs CDS processing on the two digital values ​​supplied from the AD converter 1300, and generates a pixel signal (pixel data) as a digital signal. The pixel data generated by the image capture processing unit 112 is output to the outside of the image capture unit 1031. One frame of pixel data output from the image capture processing unit 112 is supplied as image data to, for example, the output control unit 113 and the image compression unit 125.

[0094] The reference signal generation unit 1014 generates a ramp signal RAMP indicating a reference voltage for each AD converter 1300 to convert a pixel signal into two digital values, based on an ADC control signal input from the imaging operation control unit 1019. The ramp signal RAMP is a signal whose level (voltage value) decreases at a constant slope over time, or a signal whose level decreases in a step-like manner. The AD converter 1300 compares the input pixel signal with the ramp signal RAMP to convert the pixel signal into a digital pixel signal (pixel data). The reference signal generation unit 1014 supplies the generated ramp signal RAMP to each AD converter 1300. The reference signal generation unit 1014 is configured using, for example, a DA (Digital to Analog) conversion circuit or the like.

[0095] Under the control of the imaging operation control unit 1019, the horizontal scanning unit 1015 performs selective scanning to select each AD converter 1300 in a predetermined order, thereby causing each digital value temporarily held by each AD converter 1300 to be sequentially output to the imaging processing unit 112. The horizontal scanning unit 1015 is configured using, for example, a shift register, an address decoder, etc.

[0096] The imaging operation control unit 1019 controls the driving of the vertical scanning unit 1012, the AD conversion unit 1013, the reference signal generation unit 1014, the horizontal scanning unit 1015, etc. The imaging operation control unit 1019 generates various driving signals that serve as references for the operations of the vertical scanning unit 1012, the AD conversion unit 1013, the reference signal generation unit 1014, and the horizontal scanning unit 1015. The imaging operation control unit 1019 generates control signals that the vertical scanning unit 1012 supplies to each pixel Pix via pixel signal lines 1016, based on a vertical synchronization signal or an external trigger signal supplied from the outside (for example, a sensor control unit) and a horizontal synchronization signal. The imaging operation control unit 1019 supplies the generated control signals to the vertical scanning unit 1012.

[0097] The vertical scanning unit 1012 supplies various signals including drive pulses to pixel signal lines 1016 of a selected pixel row of the pixel array unit 1011 for each pixel Pix on a line-by-line basis based on a control signal supplied from the imaging operation control unit 1019, and causes each pixel Pix to output a pixel signal to a vertical signal line 1017. The vertical scanning unit 1012 is configured using, for example, a shift register, an address decoder, etc.

[0098] The imaging unit 1031 configured in this manner is a column AD type CMOS (Complementary Metal Oxide Semiconductor) image sensor in which AD converters 1300 are arranged for each column.

[0099] FIG. 6 is a perspective view schematically illustrating the structure of an example of the imaging device 10a according to the first system configuration described with reference to FIG.

[0100] The imaging device 10a can be configured as a one-chip semiconductor device having a stacked structure in which multiple dies are stacked, for example, as shown in Fig. 6. In the example of Fig. 6, the imaging device 10a is configured as a one-chip semiconductor device in which two dies, dies 1050 and 1051, are stacked.

[0101] A die refers to a small thin piece of silicon in which an electronic circuit is fabricated, and an individual in which one or more dies are sealed is called a chip.

[0102] 6 , an imaging unit 1031 is mounted on an upper die 1050. An imaging processing unit 1032, an output control unit 1033, an output I / F 1034, and an imaging control unit 1035 are mounted on a lower die 1051. As such, in the example of FIG. 6 , the imaging unit 1031 of the imaging block 1030 is mounted on the die 1050, and the parts other than the imaging unit 1031 are mounted on the die 1051. A signal processing block 1040 including a CPU 1041, a DSP 1042, a memory 1043, a communication I / F 1044, an image compression unit 1045, and an input I / F 1046 is further mounted on the die 1051.

[0103] The upper die 1050 and the lower die 1051 are electrically connected, for example, by forming a through-hole that penetrates the die 1050 and reaches the die 1051. Without being limited to this, the dies 1050 and 1051 may be electrically connected by metal-metal wiring such as Cu-Cu bonding that directly connects a metal wiring such as Cu exposed on the lower surface side of the die 1050 with a metal wiring such as Cu exposed on the upper surface side of the die 1051.

[0104] Here, the imaging processing unit 1032 may use, for example, a column-parallel AD method or an area AD method to perform AD conversion on the image signal output from the imaging unit 1031 .

[0105] In the column-parallel AD method, for example, an ADC (AD Converter) is provided for each column of pixels that make up the imaging unit 1031, and the ADC for each column is responsible for AD conversion of the pixel signals of the pixels in that column, thereby performing AD conversion of the image signals of the pixels in each column of one row in parallel. When the column-parallel AD method is adopted, part of the imaging processing unit 1032 that performs AD conversion in the column-parallel AD method may be mounted on the upper die 1050.

[0106] In the area AD system, the pixels that make up the imaging unit 1031 are divided into multiple blocks, and an ADC is provided for each block. The ADC in each block is responsible for AD conversion of the pixel signals of the pixels in that block, thereby performing AD conversion of the image signals of the pixels in multiple blocks in parallel. In the area AD system, the block is the smallest unit, and AD conversion (reading and AD conversion) of image signals can be performed only on necessary pixels among the pixels that make up the imaging unit 1031.

[0107] If the area of ​​the imaging device 10a is allowed to be large, the imaging device 10a can be configured with a single die.

[0108] 6, two dies 1050 and 1051 are stacked to form the one-chip imaging device 10a, but the one-chip imaging device 10a can be formed by stacking three or more dies. For example, when three dies are stacked to form the one-chip imaging device 10a, the memory 1043 mounted on the die 1051 in FIG. 6 can be mounted on a die different from the dies 1050 and 1051.

[0109] Here, in an imaging device in which the sensor chip, memory chip, and DSP chip are connected in parallel with multiple bumps (hereinafter also referred to as a bump-connected imaging device), the thickness increases significantly and the device becomes larger compared to the single-chip imaging device 10a configured in a stacked structure.

[0110] Furthermore, in a bump-connected imaging device, due to signal degradation at the bump connection points, it may be difficult to ensure a sufficient rate for outputting captured images from the imaging processing unit 1032 to the output control unit 1033.

[0111] The imaging device 10a with a stacked structure can prevent the device from becoming larger as described above and from being unable to ensure a sufficient rate between the imaging processing unit 1032 and the output control unit 1033. Therefore, the imaging device 10a with a stacked structure can make the imaging device that outputs information required for processing in a later stage of the imaging device 10a compact.

[0112] If the information required at a later stage is a captured image, the imaging device 10a can output the captured image (a RAW image, an RGB image, etc.). Also, if the information required at a later stage is obtained by signal processing using the captured image, the imaging device 10a can obtain and output the signal processing result as the information required by the user by performing the signal processing in the DSP 1042.

[0113] The signal processing performed by the image capturing device 10a, that is, the signal processing by the DSP 1042, may be, for example, a recognition process for recognizing a predetermined recognition target from a captured image.

[0114] (2-2. Second System Configuration Example) Fig. 7 is a block diagram of an example showing a second system configuration applicable to the present disclosure. The imaging system 1b shown in Fig. 7 includes an imaging device 10b including an imaging unit 100, a signal processing device 111, and a recognition processing device 121. The signal processing device 111 has functions equivalent to the signal processing unit 110 in the imaging system 1a described using Fig. 2. Similarly, the recognition processing device 121 has functions equivalent to the recognition unit 120 in the imaging system 1a.

[0115] That is, in the first system configuration described above, the imaging unit 100, the signal processing unit 110, and the recognition unit 120 are configured as included in a single imaging device 10a, whereas in the second system configuration, the imaging device 10b, the signal processing device 111, and the recognition processing device 121 are configured as independent devices.

[0116] In the second system configuration, either wired communication or wireless communication may be used as the communication method between the imaging device 10b and the signal processing device 111, and between the signal processing device 111 and the recognition processing device 121. Furthermore, one or both of the signal processing device 111 and the recognition processing device 121 may be connected by communication via a communication network, or may be configured on a cloud network.

[0117] 8 is a block diagram of an example of a third system configuration applicable to the present disclosure. The imaging system 1c shown in FIG. 8 includes an imaging device 10b including an imaging unit 100, and an information processing device 20 including a signal processing unit 110 and a recognition unit 120.

[0118] The information processing device 20 is, for example, a computer, and the signal processing unit 110 and the recognition unit 120 are configured, for example, by a predetermined program running on the computer. However, without being limited to this, one or both of the signal processing unit 110 and the recognition unit 120 may be configured by hardware circuits that operate in cooperation with each other.

[0119] In the third system configuration, either wired communication or wireless communication may be used as the communication method between the imaging device 10b and the information processing device 20. The information processing device 20 may be a server device connected to the imaging device 10b via a communication network, or may be configured on a cloud network.

[0120] 9 is a block diagram showing an example of a hardware configuration of an information processing device 20 applicable to the third system configuration. In Fig. 9, the information processing device 20 includes a CPU 2000, a ROM (Read Only Memory) 2001, a RAM (Random Access Memory) 2002, a display control unit 2003, a storage device 2004, an input device 2005, a data I / F 2006, a communication I / F 2007, and a camera I / F 2008, which are communicably connected to each other via a bus 2020.

[0121] The storage device 2004 is a non-volatile storage medium such as a flash memory or a hard disk drive. The CPU 2000 controls the overall operation of the information processing device 20 in accordance with programs stored in the storage device 2004 or the ROM 2001, using the RAM 2002 as a work memory.

[0122] The display control unit 2003 generates a display signal that can be displayed by the display device 2010, based on display control information generated by the CPU 2000 in accordance with a program. The display control unit 2003 supplies the generated display signal to the display device 2010. The display device 2010 displays a screen corresponding to the display signal supplied from the display control unit 2003.

[0123] The input device 2005 is a device that accepts input operations by the user, and may be, for example, a pointing device such as a mouse, a keyboard, or a touchpad. The input device 2005 outputs a control signal in response to the input operation by the user.

[0124] The data I / F 2006 inputs and outputs data to and from external devices via wired or wireless communication. The communication I / F 2007 performs communication via a communication network. The camera I / F 2008 is an interface for transmitting and receiving control commands and captured images to and from the imaging device 10b. The data I / F 2006 can also be used as the camera I / F 2008.

[0125] In the information processing device 20, the CPU 2000 executes the information processing program according to the present disclosure, thereby configuring the above-mentioned signal processing unit 110 and recognition unit 120, for example, as modules on the main storage area of ​​the RAM 2002.

[0126] The information processing program can be acquired from an external source via a communication network such as the Internet through communication via the communication I / F 2007, and can be installed on the information processing device 20. Alternatively, the information processing program may be provided from an external source via the data I / F 2006. Furthermore, the information processing program may be provided by being stored in a removable storage medium such as a CD (Compact Disk), a DVD (Digital Versatile Disk), or a USB (Universal Serial Bus) memory.

[0127] The configuration shown in FIG. 9 can also be applied to one or both of the signal processing device 111 and the recognition processing device 121 in the second system configuration described above.

[0128] 3. First Embodiment Next, a first embodiment of the present disclosure will be described. The first embodiment relates to signal processing in an imaging system 1 according to the present disclosure.

[0129] 10 is a functional block diagram illustrating an example of functions of the imaging system 1 according to the first embodiment. In FIG. 10, the imaging system 1 includes a sensitivity variation expansion sensor 300, a sensitivity variation correction unit 301, a saturated region correction unit 302, and a saturated region estimation unit 303. The imaging system 1 may further include a recognition unit 330.

[0130] Of these, the sensitivity variation correction unit 301, the saturated region correction unit 302, and the saturated region estimation unit 303 are functions included in the above-mentioned signal processing unit 110 or the signal processing device 111. The recognition unit 330 corresponds to the above-mentioned recognition unit 120 or the recognition processing device 121.

[0131] In the following description, the imaging system 1 is assumed to be the imaging system 1a including the imaging device 10a including the imaging unit 100, the signal processing unit 110, and the recognition unit 120 described with reference to FIG.

[0132] The sensitivity variation expanding sensor 300 is a sensor in which the sensitivity of the pixels Pix included in the pixel array unit 1011 is made different on a pixel-by-pixel basis. In the sensitivity variation expanding sensor 300, the pixels with different sensitivities may be arranged in an irregular pattern throughout the pixel array unit 1011, or pixel blocks including a predetermined number of pixels with different sensitivities may be arranged in a repeated pattern in the pixel array unit 1011.

[0133] As described above, the sensitivity variation expansion sensor 300 has a PRNU, which indicates the sensitivity variation per pixel, that is larger than that of a general sensor. As an example, the sensitivity variation expansion sensor 300 has a sensitivity ratio between the minimum sensitivity pixel and the maximum sensitivity pixel that is several times to several hundred times.

[0134] The sensitivity variation expansion sensor 300 supplies a sensor output (captured image) obtained by capturing an image to a sensitivity variation correction unit 301 and a saturated region estimation unit 303 as a sensitivity variation image.

[0135] The sensitivity variation correction unit 301 performs sensitivity variation correction processing on the sensitivity variation image supplied from the sensitivity variation expansion sensor 300 using a variation correction map 310. The variation correction map 310 is, for example, created in advance based on the sensitivity of each pixel Pix in the entire area of ​​the pixel array unit 1011 or at least each pixel Pix in the effective pixel area, and is stored in a storage medium such as a memory included in the imaging device 10a. The sensitivity variation correction unit 301 supplies the sensitivity variation corrected image, in which the sensitivity variation of the sensitivity variation image has been corrected, to the saturated area correction unit 302.

[0136] The saturated region estimation unit 303 estimates saturated regions in the pixel array unit 1011 based on the sensitivity variation image supplied from the sensitivity variation spread sensor 300. For example, the imaging system 1 creates in advance a saturation threshold map 320 that indicates regions in the sensitivity variation spread sensor 300 that become saturated due to a predetermined amount of light or more. The saturated region estimation unit 303 references the saturation threshold map 320 to estimate saturated pixels in the sensitivity variation image supplied from the sensitivity variation spread sensor 300, and generates a saturated region map based on the estimation result. The saturated region estimation unit 303 passes the generated saturated region map to the saturated region correction unit 302.

[0137] The saturated region correction unit 302 performs saturation correction on the sensitivity variation corrected image supplied from the sensitivity variation correction unit 301, using the saturated region map passed from the saturated region estimation unit 303. The saturated region correction unit 302 passes the saturated region corrected image, which is the sensitivity variation corrected image, to the recognition unit 330.

[0138] The recognition unit 330 performs recognition processing using a network to which a pre-trained model is applied on the saturated region corrected image supplied from the saturated region correction unit 302. The recognition unit 330 outputs the recognition result obtained by the recognition processing on the saturated region corrected image.

[0139] The learning data for training the recognition unit 330 may be acquired, for example, by constructing a database based on sensitivity variation images captured by the sensitivity variation expansion sensor 300 included in the imaging system 1 including the recognition unit 330. Without being limited to this, the learning data may be acquired using sensitivity variation images generated by simulating sensitivity variation based on images captured by a non-sensitivity variation sensor in which sensitivity variation is not expected.

[0140] (3-1. Sensitivity Variation Correction Processing According to First Embodiment) The above-described sensitivity variation correction processing by the sensitivity variation correction unit 301 will be described in more detail.

[0141] (Sensitivity Variation Correction Process) FIGS. 11 and 12 are schematic diagrams for explaining an overview of sensitivity variation correction process according to the first embodiment.

[0142] 11, the horizontal axis represents pixel position, and the vertical axis represents the level of a signal output by the pixel. Pixel output level 50 represents an example of the output level when the same amount of light is incident on each pixel of the sensitivity variation expansion sensor 300. Because the sensitivity of the sensitivity variation expansion sensor 300 varies from pixel to pixel, the output level 50 will be a different value depending on the sensitivity of the pixel, even when the same amount of light is incident. In other words, the output level 50 is higher for pixels with higher sensitivity.

[0143] The sensitivity variation correction unit 301 performs a predetermined calculation on the output level 50 of each pixel and corrects the output level 50 of each pixel to a fixed level 51 after correction. That is, as schematically shown in FIG. 12 , the sensitivity variation correction unit 301 performs a calculation to divide the output level of each pixel of the sensitivity variation image (image 4 a) supplied as a sensor output from the sensitivity variation expansion sensor 300 by the sensitivity variation ratio of each pixel. Through this calculation, the sensitivity variation correction unit 301 can obtain a sensitivity variation corrected image (image 4 b) in which the sensitivity variation has been corrected. If the sensitivity variation corrected image includes saturated pixels, the saturated pixels appear as noise when viewed.

[0144] The sensitivity variation ratio may be calculated as the ratio of the output level of each pixel Pix to the output level of the pixel with the highest sensitivity when uniform light is irradiated onto the pixel array unit 1011. The sensitivity variation ratio can be calculated based on the sensitivity variation correction data 2, for example.

[0145] In the following description, dividing the output level of a pixel by the sensitivity variation ratio of that pixel may be referred to as "reducing."

[0146] FIG. 13 is a schematic diagram for more specifically explaining the sensitivity variation correction process according to the first embodiment. In FIG. 13, each of charts 60a to 60d and 61a to 61d shows an example of the output level of each pixel when uniform light is irradiated onto the sensitivity variation expansion sensor 300, with the horizontal axis representing the pixel position and the vertical axis representing the output level. The output levels of each pixel are assumed to be 1 / 2 and 1 / 4, respectively, assuming that the output level of a pixel with maximum sensitivity in a non-saturated state is 1. Hereinafter, a pixel with an output level of 1 will be referred to as a pixel with a sensitivity ratio of "1," and pixels with output levels of 1 / 2 and 1 / 4 will be referred to as a pixel with a sensitivity ratio of "1 / 2" and a pixel with a sensitivity ratio of "1 / 4," respectively.

[0147] 13 show examples of a state in which sensitivity variation correction processing is not performed by the sensitivity variation correction unit 301, and the charts 61a to 61d in the lower row show examples of a state in which sensitivity variation correction processing is performed by the sensitivity variation correction unit 301. Furthermore, the amount of irradiated light increases toward the left in Fig. 13, with the leftmost charts 60a and 61a showing a state in which light is irradiated with an amount of light that does not saturate all pixels, and the rightmost charts 60d and 61d showing a state in which light is irradiated with an amount of light that saturates all pixels.

[0148] In chart 60a of Fig. 13, the output level of each pixel is below saturation level 52, indicating that none of the pixels are saturated. In chart 60b of Fig. 13, only the pixel with a sensitivity ratio of "1" is saturated, while the pixels with sensitivity ratios of "1 / 2" and "1 / 4" are not saturated. On the other hand, pixel 53 is a saturated pixel, and the output level of the saturated portion, i.e., the output level that could not be output due to saturation, is indicated by a diagonal line. Note that pixel 53 will hereinafter be referred to as saturated pixel 53.

[0149] Similarly, in the chart 60c of FIG. 13, pixels with a sensitivity ratio of "1" and a ratio of "1 / 2" are set as saturated pixels 53, and in the chart 60d, all pixels are set as saturated pixels 53.

[0150] A chart 61a shows an example of a corrected level 51a obtained by performing sensitivity variation correction processing on the state of the chart 60a. The output level of each pixel is divided by the sensitivity ratio to obtain a uniform output level.

[0151] Chart 61b shows an example of corrected levels 51b obtained by performing sensitivity variation correction processing on the state of chart 60b. The pixels with a sensitivity ratio of "1 / 2" and the pixels with a sensitivity ratio of "1 / 4" are not saturated, so by dividing the output level by the sensitivity ratio, the output level (true value level 55) when not saturated can be obtained as corrected level 51b. On the other hand, the pixel with a sensitivity ratio of "1" is saturated, so the output level is saturation level 52, and even if the output level is divided by the sensitivity ratio of "1", a true value cannot be obtained.

[0152] Chart 61c shows an example of corrected level 51c obtained by performing sensitivity variation correction processing on the state of chart 60c. In this case, pixels with a sensitivity ratio of "1 / 4" are not saturated, so true level 55 can be obtained by dividing the output level by the sensitivity ratio. On the other hand, pixels with a sensitivity ratio of "1 / 2" are saturated, so their output level is set to saturation level 52. When the output level is divided by the sensitivity ratio of "1 / 2", an output level twice the saturation level 52 can be obtained as corrected level 51c. However, because the calculation of corrected level 51c does not include the saturation level, true level 55 cannot be obtained.

[0153] Chart 61d shows an example of a corrected level 51d obtained by performing sensitivity variation correction processing on the state of chart 60d. In this case, because all pixels are saturated, the corrected level 51c divided by the sensitivity ratio is only the reciprocal multiple of the sensitivity ratio of the saturation level 52, and the true value level 55 is indefinite. For this reason, the true value level 55 may be set to, for example, the maximum value in the bit depth of the sensor output of the sensitivity variation expansion sensor 300.

[0154] (Method of Obtaining Sensitivity Variation Correction Data) Next, a method of obtaining sensitivity variation correction data 2 according to the first embodiment will be described.

[0155] In the first embodiment, proposed methods for acquiring the sensitivity variation correction data 2 include a method of acquiring the sensitivity variation correction data 2 by prior calibration and a method of acquiring the sensitivity variation correction data 2 online, i.e., in parallel with the imaging operation of the sensitivity variation expansion sensor 300. In the method of acquiring the sensitivity variation correction data 2 by prior calibration, the acquired sensitivity variation correction data 2 is held in the sensitivity variation expansion sensor 300 or in a host device (such as the signal processing unit 110 in the example of FIG. 2 ) connected to the sensitivity variation expansion sensor 300 as a subsequent stage.

[0156] 14 is a schematic diagram for explaining a first example of a method for acquiring sensitivity variation correction data 2 according to the first embodiment. This first example corresponds to the above-described method of acquiring sensitivity variation correction data 2 by advance calibration.

[0157] In the first example, the sensitivity variation correction data 2 (variation correction map 310) is generated using a test chart 70. It is preferable to use a chart with uniform brightness at least in the imaging range imaged by the sensitivity variation expansion sensor 300 as the test chart 70. For example, a white piece of paper or a white board with a non-glossy surface can be used as the test chart 70.

[0158] The sensitivity variation expanding sensor 300 captures an image of the test chart 70 and outputs, as a captured image, a sensitivity variation image 71. The sensitivity variation image 71 is an image in which the shading varies on a pixel-by-pixel basis in accordance with the sensitivity variation in the sensitivity variation expanding sensor 300.

[0159] The sensitivity variation correction unit 301 generates a variation correction map 310 based on the sensitivity variation image 71 and a maximum value image 73 in which the pixel value of each pixel is maximized, through a sensitivity variation estimation process 72. More specifically, the sensitivity variation correction unit 301 obtains the variation relative to the maximum sensitivity by dividing the pixel value of each pixel in the sensitivity variation image 71 by the pixel value of each pixel in the maximum value image 73 through the sensitivity variation estimation process 72. Based on this variation relative to the maximum sensitivity, for example, a sensitivity ratio of each pixel is obtained, and the variation correction map 310 is generated.

[0160] In the above description, the sensitivity variation estimation process 72 is described as being executed in the sensitivity variation correction unit 301, but this is not limited to this example, and the sensitivity variation estimation process 72 may be executed outside the imaging device 10a.

[0161] 15 is a schematic diagram for explaining a second example of a method for acquiring sensitivity variation correction data 2 according to the first embodiment. This second example corresponds to the above-described method for acquiring sensitivity variation correction data 2 online.

[0162] In the second example, the sensitivity variation correction unit 301 generates the variation correction map 310 by performing sensitivity variation estimation processing 72 ′ to perform online estimation in the time series direction while performing imaging using the sensitivity variation expansion sensor 300 .

[0163] The sensitivity variation expanding sensor 300 captures an image of a normal subject 75 and outputs a sensitivity variation image 76 as the captured image. The sensitivity variation image 76 is an image in which the shading of the image of the subject 75 varies on a pixel-by-pixel basis in accordance with the sensitivity variation in the sensitivity variation expanding sensor 300. When viewed, the sensitivity variation image 76 appears to be an image in which noise has been added to the original image of the subject 75.

[0164] The sensitivity variation correction unit 301 estimates sensitivity variations in the sensitivity variation image 76 by performing a sensitivity variation estimation process 72', assuming that brightness is locally uniform. Specifically, the sensitivity variation correction unit 301 generates a variation correction map 310 based on an image 77 of a local region in the sensitivity variation image 76 and the maximum value image 73' by performing the sensitivity variation estimation process 72'. More specifically, the sensitivity variation correction unit 301 obtains variations relative to the maximum sensitivity by performing the sensitivity variation estimation process 72' to divide pixel values ​​of the image 77 of the local region in the sensitivity variation image 76 by the pixel values ​​of each pixel in the maximum value image 73'. Based on this variation relative to the maximum sensitivity, the sensitivity variation correction unit 301 obtains, for example, a sensitivity ratio of each pixel to generate the variation correction map 310.

[0165] The sensitivity variation correction unit 301 may perform flatness detection of pixel values ​​on a local region of the sensitivity variation image 76 through the sensitivity variation estimation process 72', and acquire the detected flat portion as the local region. Alternatively, the sensitivity variation correction unit 301 may perform the sensitivity variation estimation process 72' to acquire multiple sensitivity variation images 76 in time series from the sensitivity variation expansion sensor 300, average and stabilize them, and perform flatness detection based on the stabilized image information. In this case, a flatness detection score may be calculated for each of the multiple acquired images, and the calculated score may be used as a weight for averaging.

[0166] When a plurality of sensitivity variation images 76 are acquired in time series, a situation may occur in which flat portions move in the acquired plurality of sensitivity variation images 76. Fig. 16 is a schematic diagram for explaining processing according to the first embodiment when flat portions move in the plurality of sensitivity variation images 76 acquired in time series. Note that Fig. 16 is an example that is applied to a case in which the sensitivity variation in the sensitivity variation expansion sensor 300 is not a variation having a specific period.

[0167] 16 , the sensitivity variation expansion sensor 300 outputs a sensitivity variation image 76 at time t. In a sensitivity variation estimation process 72′, the sensitivity variation correction unit 301 estimates a flat portion by, for example, flatness detection, and acquires an image 77 of a local region based on the estimation result. The sensitivity variation correction unit 301 generates a variation correction map 310 based on this image 77.

[0168] The sensitivity variation expansion sensor 300 outputs a sensitivity variation image 762 at time t2, which is after time t1. In a sensitivity variation estimation process 72', the sensitivity variation correction unit 301 estimates a flat portion by, for example, flatness detection, and acquires an image 772 of a local region based on the estimation result. Here, it is assumed that the image 772 has moved from the position of the image 771 at time t1.

[0169] Similarly, the sensitivity variation expansion sensor 300 detects the sensitivity variation expansion at time t ３ The sensitivity variation correction unit 301 outputs a sensitivity variation image 763. In a sensitivity variation estimation process 72', the sensitivity variation correction unit 301 estimates flat portions by, for example, flatness detection, and acquires an image 773 of a local region based on the estimation result. Here, it is assumed that the image 773 has moved further from the position of the image 772 at time t2.

[0170] 16 , at time t2, the sensitivity variation correction unit 301 combines an image 77 acquired at time t1 with an image 772 acquired at time t2. The sensitivity variation correction unit 301 can generate a variation correction map 310 based on the image obtained by combining the images 771 and 772. In this case, the sensitivity variation correction unit 301 may update the variation correction map 310 generated based on the image 771 at time t1 with the variation correction map 310 generated based on the image obtained by combining the images 771 and 772.

[0171] Similarly, at time t3, the sensitivity variation correction unit 301 further combines image 773 with the image obtained by combining images 771 and 772 at time t2. The sensitivity variation correction unit 301 can generate the variation correction map 310 based on the image obtained by combining images 771 and 772 with image 773. In this case, the sensitivity variation correction unit 301 may update the variation correction map 310 generated at time t2 based on the image obtained by combining images 771 and 772 with the variation correction map 310 generated based on the image obtained by combining images 771 to 773.

[0172] (Regarding Correction Targets of Sensitivity Variation Correction) Here, a description will be given of correction targets in the sensitivity variation correction process by the sensitivity variation correction unit 301. Fig. 17 is a schematic diagram for explaining correction targets of sensitivity variation correction according to the first embodiment.

[0173] As illustrated in FIG. 17 , in the first embodiment, (A) sensitivity variation due to fabrication and (B) sensitivity variation determined by accidental factors such as variation occurring during manufacturing are targets of sensitivity variation correction.

[0174] The "sensitivity variation determined by accidental factors" in (B) may be appropriately described as "accidental sensitivity variation" or the like.

[0175] Of these, (A) sensitivity variation due to fabrication causes sensitivity variation on a pixel-by-pixel basis in the sensitivity variation expanded sensor 300, for example, at the time of designing the sensitivity variation expanded sensor 300. As an example, it is conceivable to change the size of an opening in a photomask used in the manufacturing process of the sensitivity variation expanded sensor 300, thereby making the sizes of the photodiode, floating diffusion layer FD, each transistor, etc. in the sensitivity variation expanded sensor 300 different on a pixel-by-pixel basis.

[0176] The built-in sensitivity variation correction data 2 may be information obtained at the time of designing the sensitivity variation expanded sensor 300. The built-in sensitivity variation correction data 2 may be obtained by actually measuring the sensitivity variation expanded sensor 300, or may be an online estimated value. The built-in sensitivity variation correction data 2 may be provided such that sensitivity variations are irregularly distributed over the entire pixel array unit 1011, or may be provided periodically so that variations are repeated for multiple pixels.

[0177] Furthermore, (B) sensitivity variation determined by accidental factors utilizes, for example, variations during manufacturing of the sensitivity variation expanding sensor 300. Examples of variations during manufacturing include variations occurring in the sensitivity variation expanding sensor 300 due to differences in lot size or wafer position.

[0178] The sensitivity variation correction data 2 due to accidental variation may be obtained by actually measuring the sensitivity variation expansion sensor 300, or may use an online estimated value. When the sensitivity variation correction data 2 due to accidental variation is obtained using design values, it is possible to obtain the sensitivity variation correction data 2 using the variance σ of the design values. The sensitivity variation correction data 2 due to accidental variation is often given irregularly throughout the pixel array unit 1011.

[0179] 18 is a schematic diagram for explaining a third example of a method for acquiring sensitivity variation correction data 2 according to the first embodiment. This third example is an example in which sensitivity variation in the sensitivity variation expansion sensor 300 is corrected using information correlated with noise.

[0180] 18, pixels with a sensitivity ratio of "1 / 2" or "1 / 4", which have lower sensitivities than pixels with a sensitivity ratio of "1", are less likely to saturate and can capture images without saturation even in bright scenes. However, as described above, the sensitivity variation correction unit 301 corrects sensitivity variations by dividing the output level 50 by the sensitivity ratio, so pixels with low sensitivity have a poor S / N ratio and may contain a lot of noise when divided down. In contrast, pixels with a sensitivity ratio of "1" in FIG. 18 have high sensitivity and are more likely to saturate, but have a better S / N ratio than pixels with a sensitivity ratio of "1 / 2" or "1 / 4".

[0181] Therefore, for example, when performing noise reduction on the output from each pixel, the sensitivity variation correction unit 301 uses a sensitivity ratio map (variation correction map 310) to weight each pixel according to its sensitivity. For example, the sensitivity variation correction unit 301 performs noise reduction by assigning a larger weight to pixels with low sensitivity and a smaller weight to pixels with high sensitivity. The sensitivity variation correction unit 301 may perform noise reduction processing immediately after the sensitivity variation correction processing.

[0182] (3-2. Processing Related to Saturated Regions According to the First Embodiment) Next, processing related to saturated regions according to the first embodiment will be described.

[0183] (3-2-1. Saturated Region Estimation Processing) First, the saturated region estimation processing according to the first embodiment will be described. In the first embodiment, the saturated region estimation unit 303 estimates a saturated region in the sensitivity variation image output from the sensitivity variation expansion sensor 300. More specifically, the saturated region estimation unit 303 applies a threshold value to the sensitivity variation image to estimate a saturated region.

[0184] 19 corresponds to the diagram in the upper part of Fig. 13 described above, and is a schematic diagram for explaining a first example of the saturated region estimation process according to the first embodiment. For the sake of explanation, the example of Fig. 19 shows an example of the output level of each pixel when the sensitivity variation expansion sensor 300 is irradiated with uniform light, similar to Fig. 13 described above.

[0185] In this first example, the saturated region estimation unit 303 estimates a saturated region using a saturation threshold 56a that is common to each pixel of the pixel array unit 1011. The saturated region estimation unit 303 estimates that, among the pixels of the sensitivity variation image, pixels whose output level 50 is higher than the saturation threshold 56a are pixels included in the saturated region. For example, the saturation level 52 may be used as the saturation threshold 56a.

[0186] Alternatively, the saturation threshold 56a may be set to a level that is offset toward the lower level side with respect to the saturation level 52. In the example of Fig. 19, the saturation threshold 56a is set to a level that is lower than the saturation level 52 and higher than the maximum output level of the pixel with the highest sensitivity in the pixel array unit 1011 when the pixel is not saturated. Setting the saturation threshold 56a in this manner makes it possible to reduce the influence of saturation variations, which will be described later.

[0187] In FIG. 19 , in the example of chart 60a, which is illuminated with a light intensity that does not saturate all pixels, none of the pixels are saturated, and therefore no saturated region exists. In the example of chart 60b, which is illuminated with light brighter than chart 60a, only pixels with a sensitivity ratio of "1" may be saturated. Therefore, in the example of chart 60b, pixels with a sensitivity ratio of "1" may be included in the saturated region. In the example of chart 60c, which is illuminated with light brighter than chart 60b, pixels with a sensitivity ratio of "1" and a sensitivity ratio of "½" may be included in the saturated region. Therefore, in the example of chart 60c, pixels with a sensitivity ratio of "1" and a sensitivity ratio of "½" may be included in the saturated region. Furthermore, in the example of chart 60d, which is illuminated with a light intensity that saturates all pixels, all pixels may be saturated, and therefore all pixels may be included in the saturated region.

[0188] Second Example In a second example, the saturated region estimation unit 303 sets a different saturation threshold 56 b for each pixel in the pixel array unit 1011 .

[0189] 20 is a schematic diagram illustrating a second example of the saturated region estimation process according to the first embodiment. In sections (a) and (b) of FIG. 20, each pixel (three pixels in each example) has a different saturation level 52'. For the sake of explanation, the example of FIG. 20 illustrates an example of the output level of each pixel when the sensitivity variation expansion sensor 300 is irradiated with uniform light, similar to the example of FIG. 13 described above.

[0190] In Figure 20, section (a) shows an example in which a common saturation threshold 56a is used for all pixels. Because each pixel has a different saturation level 52', a common saturation threshold 56a for all pixels may not correctly detect saturated regions. In the example shown, the center pixel 54a is not detected as a saturated pixel even though it is saturated because its saturation level 52' is lower than the saturation threshold 56a. On the other hand, the second pixel from the right, 54b, is detected as a saturated pixel even though it is not saturated because its saturation level 52' is higher than the saturation threshold 56a.

[0191] 20, section (b) shows an example in which different saturation thresholds 56b are used for each pixel. By using different saturation thresholds 56b for each pixel in this way, it is possible to properly detect as saturated pixels even pixels 54a whose saturation level 52' shown in section (a) is lower than the saturation threshold 56a, and pixels 54b whose saturation level 52' is higher than the saturation threshold 56a.

[0192] (Method of Obtaining Saturation Threshold) The saturation threshold can be obtained using a test chart. For example, the test chart is irradiated with a quantity of light that saturates all pixels. The test chart is then imaged by the sensitivity variation expansion sensor 300. It is preferable that the light irradiated onto the test chart or the light reflected from the test chart has uniform brightness at least within the imaging range imaged by the sensitivity variation expansion sensor 300. For example, a piece of white paper or a white board with a matte surface can be used as the test chart.

[0193] The sensitivity variation expansion sensor 300 captures an image of this test chart and outputs, as the captured image, a sensitivity variation image in which the shade varies pixel by pixel according to the sensitivity variation in the sensitivity variation expansion sensor 300. For example, the saturated region estimation unit 303 may detect, as saturated pixels, pixels in the sensitivity variation image whose pixel value is, for example, the maximum value in the bit depth of the sensor output of the sensitivity variation expansion sensor 300. The saturated region estimation unit 303 estimates saturated regions for all pixels in the pixel array unit 1011 based on information about the pixels detected as saturated pixels, and generates a saturation threshold map 320.

[0194] As an example, in a sensitivity variation expansion sensor 300 in which the bit depth of the sensor output is 8 bits and the maximum pixel value is "255," pixel A can output a pixel value of up to "255" (high sensitivity), while another pixel B can only output a pixel value of up to "200," and yet another pixel C can only output a pixel value of up to "127" (low sensitivity).

[0195] Consider a case where sensitivity variation expanding sensor 300 having pixels A, B, and C is irradiated with an amount of light such that the pixel value exceeds the maximum value of "255," for example, to "1000." In this case, by reading the pixel values ​​of pixels A, B, and C in the sensor output of sensitivity variation expanding sensor 300, it is possible to obtain saturation levels 52' of these pixels A, B, and C. Based on the saturation levels 52' of each of pixels A, B, and C, it is possible to obtain a saturation threshold 56b for each of pixels A, B, and C. It is also possible to obtain a saturation threshold 56a common to all pixels based on the saturation levels 52' of each of pixels A, B, and C.

[0196] The saturation threshold value is not limited to being obtained using a test chart, but can also be obtained online, that is, by a method of obtaining the value in parallel with the imaging operation of the sensitivity variation expansion sensor 300. As an example, the saturated region estimation unit 303 classifies and accumulates the pixel values ​​of each pixel in each captured image captured by the sensitivity variation expansion sensor 300, for example, at a frame cycle, to create a histogram.

[0197] 21 is a diagram showing an example of a histogram created based on an image captured by the sensitivity variation expanding sensor 300 according to the first embodiment. In FIG. 21, the vertical axis represents frequency, and the horizontal axis represents pixel value classes. The representative pixel value of each class increases toward the right on the horizontal axis. The saturated region estimation unit 303 determines the representative pixel value of the class with the rightmost peak in this histogram as the saturation level 52 or saturation threshold 56.

[0198] (3-2-2. Saturated Region Correction Processing) Next, a description will be given of the saturated region correction processing according to the first embodiment. Figures 22A and 22B are schematic diagrams for explaining the saturated region correction processing according to the first embodiment.

[0199] In Figure 22A, the saturated area correction unit 302 corrects saturated pixels in the saturated area based on the sensitivity variation corrected image obtained by correcting the captured image by the sensitivity variation correction unit 301 and the saturated area map created by the saturated area estimation unit 303, and outputs the saturated area corrected image.

[0200] The saturated region correction process according to the first embodiment will be described in more detail with reference to Fig. 22B. The sensitivity variation correction unit 301 supplies a sensitivity variation corrected image 90 obtained by performing sensitivity variation correction on a captured image. The sensitivity variation corrected image 90 includes saturated pixels 53, as shown in an enlarged view as a small region 91. If the sensitivity variation corrected image 90 includes saturated pixels 53, the saturated pixels 53 will appear as noise when viewed, for example.

[0201] Based on the sensitivity variation corrected image 90 and the saturated region map, the saturated region correction unit 302 estimates the position of a saturated pixel 53 included in the sensitivity variation corrected image 90. For the sake of explanation, the drawing shows only a small region 91. The saturated region correction unit 302 corrects the pixel value of the saturated pixel 53 by interpolation or the like using pixel values ​​of non-saturated pixels surrounding the saturated pixel 53, which is shown as a region 92 in the drawing.

[0202] The saturated region correction unit 302 raster scans the sensitivity variation corrected image 90 pixel by pixel, and sequentially corrects the saturated pixels 53. The right side of Fig. 22B shows an example of a saturated region corrected image 93 in which the saturated pixels 53 of the sensitivity variation corrected image 90 have been corrected. For example, in a small region 91, the pixel values ​​of the saturated pixels 53 in the sensitivity variation corrected image 90 are corrected to become saturated corrected pixels 94, thereby obtaining the saturated region corrected image 93. Since the saturated pixels 53 included in the sensitivity variation corrected image 90 have been corrected, the saturated region corrected image 93 is an image in which noise has been suppressed, as illustrated in the figure.

[0203] (Example of Correction Using Information Correlated with Noise) The saturated region correction unit 302 may correct saturated pixels using information correlated with noise.

[0204] 23 is a schematic diagram illustrating processing for correcting saturated pixels using information correlated with noise according to the first embodiment. In Fig. 23, the saturated region correction unit 302 corrects saturated pixels in the saturated region using a sensitivity ratio map (variation correction map 310) as information correlated with noise in addition to the sensitivity variation corrected image and saturated region map, and outputs the saturated region corrected image.

[0205] As described with reference to FIG. 18 , pixels with low sensitivity have a poor S / N ratio, while pixels with high sensitivity have a good S / N ratio. Therefore, the saturated region correction unit 302 may use a sensitivity ratio map to weight pixels according to their sensitivity and perform saturated pixel correction processing. For example, the saturated region correction unit 302 may weight pixels with good S / N ratios based on the sensitivity map, perform predetermined signal processing, and then perform the saturated pixel correction processing by interpolation described above. In this case, noise reduction processing may be applied as the predetermined signal processing. However, other high-image-quality signal processing may also be applied as the predetermined signal processing.

[0206] Without being limited to this, in the saturated pixel correction process by interpolation described above, the saturated region correction unit 302 may weight the pixels used to interpolate the saturated pixels 53 according to their sensitivity ratios (for example, weighting pixels with higher sensitivity more heavily) based on the sensitivity ratio map.

[0207] (Example of Correction Using Nonlinear Characteristics in Pixel Output Level) The saturated region correction unit 302 may perform correction processing on the saturated pixels 53 by further using nonlinear characteristics in pixel output levels.

[0208] Fig. 24 is a schematic diagram showing an example of pixel response characteristics. In sections (a) and (b) of Fig. 24, the horizontal axis indicates the brightness (amount of light) of incident light, and the vertical axis indicates the output level (quantized output level). As shown in section (a) of Fig. 24, the response characteristics of the output level to the brightness of a pixel are linear from brightness "0" to a predetermined brightness Lux0, and become nonlinear when the brightness exceeds Lux0. The output level corresponding to brightness Lux0 at which the response characteristics change from linear to nonlinear is called the output level Lv sat Let's say.

[0209] Generally, the output of a pixel has a linear characteristic output level Lv sat An output level Lv below which an output is used and which has nonlinear characteristics sat Any output beyond this will be ignored.

[0210] In the first embodiment, the saturated region correction unit 302 corrects the output level Lv sat 24, the saturated region correction unit 302 may also use the output exceeding the linear region output of the pixel as well as the output level Lv sat to a predetermined output level Lv max The saturated pixel 53 is corrected using the output of the

[0211] More specifically, the saturated region correction unit 302 converts the quantization step of the output level in the nonlinear region into the quantization step of the output level with linear characteristics using a table, a function, or the like. The saturated region correction unit 302 corrects the saturated pixel 53 using this converted output level. For example, in the correction using the outputs of pixels around the saturated pixel 53 described with reference to FIG. 22B , the saturated region correction unit 302 may use the outputs of pixels whose output levels are in the nonlinear region in combination.

[0212] In this case, the output of the nonlinear region of the pixel may contain a large amount of quantization error due to the conversion process of the quantization step, so the saturated region correction unit 302 may, for example, assign a smaller weight to the output of the nonlinear region and use it to correct the saturated pixel 53.

[0213] (3-3. Exposure Plane Combining Process According to First Embodiment) Next, exposure plane combining process according to the first embodiment will be described.

[0214] The exposure plane refers to an image output under certain exposure conditions, for example, an image output by a single exposure of a specific pixel group. For example, the sensitivity variation expansion sensor 300 may perform multiple exposures in a single imaging operation. In this case, the exposure plane is composed of multiple sensitivity variation images obtained by multiple exposures with different exposure times.

[0215] For example, when considering a plurality of sensitivity variation expansion sensors 300 each having a different sensitivity variation pattern, a pixel group is formed by each pixel included in the pixel array unit 1011 of each of the plurality of sensitivity variation expansion sensors 300. In this case, an exposure surface is formed by each of a plurality of sensitivity variation images output by each sensitivity variation expansion sensor 300 in a single exposure.

[0216] Furthermore, for example, the sensitivity variation expansion sensor 300 may include two types of pixels (large pixels and small pixels) with different pixel areas in the pixel array unit 1011. In this case, an image based on a pixel group of large pixels and an image based on a pixel group of small pixels output in a single exposure each constitute an exposure surface.

[0217] The imaging system 1 according to the first embodiment can combine multiple exposure planes and output them as a single image. That is, the imaging system 1 according to the first embodiment combines exposure planes corresponding to the sensitivity variation images output from the multiple sensitivity variation expansion sensors 300 and outputs them as a single image. For the sake of explanation, it is assumed below that two sensitivity variation expansion sensors 300 are used.

[0218] 25 is a functional block diagram showing an example of a configuration for performing exposure plane synthesis according to a first example of exposure plane synthesis processing in the first embodiment. The first example is an example in which gain processing by sensitivity variation correction is performed independently for each exposure plane.

[0219] 25, correction processing units 350a1 and 350a2 each include a sensitivity variation correction unit 301, a saturated region correction unit 302, and a saturated region estimation unit 303, similar to the configuration described using Fig. 10. Correction processing units 350a1 and 350a2 each receive a sensitivity variation image output from each of a plurality of (two in this example) sensitivity variation expansion sensors 300. The functions of the respective units in these correction processing units 350a1 and 350a2 are similar to the functions described using Fig. 10 etc., and therefore will not be described here.

[0220] Correction processing units 350a1 and 350a2 each supply the saturated region corrected images as exposure planes output from saturated region correction unit 302 and the saturated region map created by saturated region estimation unit 303 to exposure plane synthesis unit 340a. Exposure plane synthesis unit 340a synthesizes the exposure planes based on the saturated region corrected images supplied from correction processing units 350a1 and 350a2 using the saturated region maps, and outputs the result as a single image.

[0221] The exposure plane synthesis unit 340a may synthesize pixels that correspond in position in each saturated region corrected image by alpha blending, for example.

[0222] Alternatively, the exposure plane composition unit 340a may combine the saturated region corrected images by switching the pixels to be combined in each saturated region corrected image on a pixel-by-pixel basis based on each saturated region map. For example, if one of the pixels at corresponding positions in each saturated region corrected image is a saturated pixel 53 and the other pixel is not a saturated pixel 53, the exposure plane composition unit 340a uses only the other pixel at that pixel position. In this case, simply switching the pixel to be used may result in a brightness step in the image, so alpha blending may be used in combination.

[0223] 26 is a functional block diagram showing an example of a configuration for performing exposure plane synthesis according to a second example of the exposure plane synthesis process in the first embodiment. Similar to the first example described above, the second example is an example in which gain processing by sensitivity variation correction is performed independently for each exposure plane.

[0224] 26, correction processing units 350b1 and 350b2 have a configuration in which the saturated region correction unit 302 is omitted from the correction processing units 350a1 and 350a2 described using Fig. 25. That is, in each of the correction processing units 350b1 and 350b2, the sensitivity variation corrected images output from the sensitivity variation correction unit 301 are supplied to the exposure plane composition unit 340b.

[0225] The exposure plane synthesis unit 340b synthesizes each exposure plane based on each sensitivity variation corrected image output from each sensitivity variation correction unit 301 while correcting the saturated regions using each saturated region map supplied from the correction processing units 350b1 and 350b2.

[0226] 27 is a schematic diagram for explaining processing by the sensitivity variation correction unit 301 according to first and second examples of exposure plane combination processing in the first embodiment. In Fig. 27, the upper part shows processing for the sensitivity variation image 95 input to the sensitivity variation correction unit 301 as exposure plane #1, and the lower part shows processing for the sensitivity variation image 95 input to the sensitivity variation correction unit 301 as exposure plane #2.

[0227] In addition, in FIG. 27 and FIG. 30 described later, the pixel bit depth is assumed to be 12 bits.

[0228] 27 shows an example in which the exposure plane composition unit 340a or 340b performs sensitivity variation correction and saturation correction independently for exposure plane #1 and exposure plane #2. As shown in the upper part of FIG. 27, in the sensitivity variation image 95, the pixel value of the pixel 57 to be corrected is set to "710." Furthermore, the sensitivity ratio at position 58 corresponding to pixel 57 in the sensitivity ratio map 311 based on the variation correction map 310 is set to "1 / 100." The sensitivity variation correction unit 301 performs sensitivity variation correction processing on pixel 57 based on the sensitivity ratio map 311, thereby obtaining a corrected pixel 57' in which the sensitivity of pixel 57 has been corrected. For corrected pixel 57', the pixel value of pixel 57 is divided by the sensitivity ratio, resulting in a pixel value of "71,000." The sensitivity variation correction unit 301 performs common gain processing on all pixels of the sensitivity variation image 95 1 to obtain a sensitivity variation corrected image 90 1 .

[0229] In the example in the lower part of FIG. 27 , in the sensitivity variation image 95 , the pixel 57 to be corrected has a pixel value of “34,561.” Furthermore, the sensitivity ratio map 311 based on the variation correction map 310 has a sensitivity ratio of “½” at a position 58 corresponding to the pixel 57 . The sensitivity variation correction unit 301 performs sensitivity variation correction processing on the pixel 57 based on the sensitivity ratio map 311 , and obtains a corrected pixel 57 ′ in which the sensitivity of the pixel 57 ′ has been corrected. The pixel value of the pixel 57 ′ is divided by the sensitivity ratio, and the pixel value of the corrected pixel 57 ′ is set to “70,922.” The sensitivity variation correction unit 301 performs a common gain processing on all pixels in the sensitivity variation image 95 , and obtains a sensitivity variation corrected image 90 .

[0230] 25 relating to the first example of exposure plane composition processing, for example, in the correction processing unit 350a, the saturated region correction unit 302 performs the saturated region correction processing as described above on the sensitivity variation corrected image 90 thus acquired to generate a saturated region corrected image. The saturated region correction unit 302 outputs the generated saturated region corrected image to the exposure plane composition unit 340a.

[0231] 26 relating to the first example of exposure plane combination processing, for example, in correction processing unit 350b1, the sensitivity variation correction unit 301 outputs a sensitivity variation corrected image to exposure plane combination unit 340b. The exposure plane combination unit 340b performs saturated region correction processing on each of the sensitivity variation corrected images output from correction processing units 350b1 and 350b2.

[0232] 28 is a schematic diagram for explaining the processing in the exposure plane synthesis unit 340b according to a second example of the exposure plane synthesis processing in the first embodiment. In sections (a) and (b) of Fig. 28, it is assumed that the sensitivity variation corrected images 901 and 902 each have a size of 10 pixels horizontally by 8 pixels vertically. As shown in section (a) of Fig. 28, the exposure plane synthesis unit 340b is supplied with the sensitivity variation corrected image 901 from the correction processing unit 350b1 and with the sensitivity variation corrected image 902 from the correction processing unit 350b2.

[0233] The exposure plane synthesis unit 340b synthesizes the saturated pixel 53 in the sensitivity variation corrected image 901. tgt In this case, the exposure plane synthesis unit 340b performs saturation correction processing on the saturated pixel 53 shown as the region 92 in the sensitivity variation correction image 90 as shown in section (b) of FIG. tgt The pixel values ​​of the surrounding non-saturated pixels and the saturated pixel 53 in the sensitivity variation corrected image 902 are tgt and the pixel position corresponds to the saturated pixel 53 tgt ') and the saturated pixel 53 tgt The exposure plane synthesis unit 340b performs saturation correction on the saturated pixel 53 tgt The pixel values ​​of the pixels surrounding the saturated pixel 53′ (shown as region 92 2 ) are further used to calculate the saturated pixel 53′. tgt A saturation correction may be performed on

[0234] The exposure plane synthesis unit 340b raster scans the sensitivity variation correction image 901 pixel by pixel, and sequentially extracts the saturated pixels 53 tgtThe exposure plane synthesis unit 340b may perform exposure plane synthesis using alpha blending or the like for pixels that correspond to each other in the sensitivity variation corrected images 901 and 902 and are not saturated.

[0235] The exposure plane synthesis unit 340b synthesizes the saturated pixel 53 of the sensitivity variation corrected image 901. tgt When performing the saturation correction of the sensitivity variation correction image 901, the sensitivity ratio map is used to compare the pixels in the region 92 in the sensitivity variation correction image 901 with the saturated pixels 53 in the sensitivity variation correction image 902. tgt ' and its surrounding pixels, and pixels with good S / N ratios in ' may be weighted and subjected to saturation correction.

[0236] In addition, the exposure plane synthesis unit 340b uses other high-quality image signal processing such as a median filter, a bilateral filter, or noise reduction processing to synthesize the saturated pixels 53. tgt A saturation correction for .times. ...

[0237] In this way, in the second example of the exposure plane synthesis process in the first embodiment, one saturated pixel 53 tgt On the other hand, the saturated pixel 53 tgt 25 , for example, compared to the case where saturation correction processing is performed based on pixel information of a single exposure surface using the configuration described with reference to FIG. 25 , it is possible to appropriately use pixel information with a better S / N ratio, thereby obtaining an image with higher image quality.

[0238] 29A is a functional block diagram showing an example of a configuration for performing exposure plane synthesis according to a third example of the exposure plane synthesis process in the first embodiment. The third example is an example in which, for example, sensitivity variation correction processing for exposure plane #1 and sensitivity variation correction processing for exposure plane #2 are performed using information on each other.

[0239] In FIG. 29A, correction processing units 350c1 and 350c2 have a configuration in which the sensitivity variation correction unit 301 is further omitted from the correction processing units 350b1 and 350b2 described with reference to FIG. 26, respectively.

[0240] 29A , in correction processing unit 350b1, a sensitivity variation image output from a sensitivity variation expansion sensor 300 (not shown) is supplied to exposure plane synthesis unit 340c. Also, a sensitivity variation image output from a sensitivity variation expansion sensor 300 different from the sensitivity variation expansion sensor 300 is supplied to exposure plane synthesis unit 340c. Also, in correction processing units 350c1 and 350c2, each variation correction map 310 is supplied to exposure plane synthesis unit 340c.

[0241] The exposure surface synthesis unit 340c synthesizes each exposure surface based on each sensitivity variation image output from each sensitivity variation expansion sensor 300, while performing sensitivity variation correction and saturated area correction using each saturated area map and each variation correction map 310 supplied from the correction processing units 350c1 and 350c2.

[0242] 29B is a block diagram showing the configuration of an example of an exposure plane combination unit according to a third example of exposure plane combination processing in Embodiment 1. As shown in Fig. 29B, the exposure plane combination unit 340c includes sensitivity variation correction units 341 and 341 corresponding to exposure planes #1 and #2, respectively, and an exposure plane combination unit 342.

[0243] The sensitivity variation correction unit 341 receives the sensitivity variation image of exposure surface #1, the variation correction map 310, and the saturated region map output from the correction processing unit 350c. Similarly, the sensitivity variation correction unit 341 receives the sensitivity variation image of exposure surface #2, the variation correction map 310, and the saturated region map output from the correction processing unit 350c.

[0244] The sensitivity variation correction unit 341 further receives the variation correction map 310 and saturation region map for exposure plane #2 as input. For example, the sensitivity variation correction unit 341 performs sensitivity variation correction processing for exposure plane #1 on the sensitivity variation image for exposure plane #1 using the variation correction map 310 and saturation region map for exposure plane #1 as well as the variation correction map 310 and saturation region map for exposure plane #2. The sensitivity variation correction unit 341 outputs the sensitivity variation corrected image for exposure plane #1, obtained by correcting the sensitivity variation image, to the exposure plane synthesis unit 342.

[0245] The sensitivity variation correction unit 341 also generates a map of pixels for which gain processing has not been performed, i.e., pixels for which sensitivity variation correction processing has not been performed in the sensitivity variation image, for exposure plane #1. The sensitivity variation correction unit 341 outputs the generated invalid area map to the exposure plane synthesis unit 342.

[0246] The processing in the sensitivity variation correction unit 341 is the same as the processing in the sensitivity variation correction unit 341, and therefore a description thereof will be omitted here. The sensitivity variation correction unit 341 outputs the sensitivity variation corrected image and invalid area map for exposure plane #2 to the exposure plane synthesis unit 342.

[0247] 30 is a schematic diagram for explaining the sensitivity variation correction process in the exposure plane combination unit 340c according to the third example of the exposure plane combination process in the first embodiment. In Fig. 30, the upper part shows the process for the sensitivity variation image 951 input to the correction processing unit 350c1 as exposure plane #1, and the lower part shows the process for the sensitivity variation image 952 input to the correction processing unit 350c2 as exposure plane #2.

[0248] Figure 30 shows an example in which the exposure plane synthesis unit 340c performs sensitivity variation correction and saturation correction for exposure planes #1 and #2 using information on both exposure planes #1 and #2. The pixel values ​​of pixels 57 and 57 to be corrected and the sensitivity ratios at positions 58 and 58 corresponding to these pixels 57 and 57 in sensitivity variation images 95 and 95 are the same as those in the example of Figure 27. In Figure 30, the process up to the sensitivity variation correction process for exposure planes #1 and #2 is the same as the process shown in Figure 27 above, so a description thereof will be omitted here.

[0249] The exposure plane synthesis unit 340c selects pixels with a sensitivity equal to or higher than a predetermined value (pixels with a good S / N ratio) from the sensitivity variation image 95, and performs gain processing based on the sensitivity ratio map 311. This gain processing results in a sensitivity variation corrected image 90'. Similarly, the exposure plane synthesis unit 340c selects pixels with a predetermined sensitivity from the sensitivity variation image 95, and performs gain processing based on the sensitivity ratio map 311. This gain processing results in a sensitivity variation corrected image 90'.

[0250] In Figure 30, the black pixels 591 and 592 indicate pixels in the invalid area that have not been subjected to gain processing. As described above, information on the pixels in the invalid area is generated as invalid area maps in the sensitivity variation correction units 3411 and 3412, respectively, and supplied to the exposure plane synthesis unit 342. In this example, there is a clear difference in sensitivity between exposure plane #1 and exposure plane #2 (1 / 100 and 1 / 2). Therefore, gain processing with a sensitivity ratio of "1 / 100" is not performed, and only gain processing with a sensitivity ratio of "1 / 2" is executed. In this way, when there is a clear difference in sensitivity between exposure plane #1 and exposure plane #2, selectively performing gain processing on pixels with a sensitivity above a predetermined level can reduce unnecessary gain processing.

[0251] Furthermore, the exposure plane synthesis unit 340c may omit saturated region processing for invalid pixels based on the invalid region map in the exposure plane synthesis unit 342. Furthermore, the exposure plane synthesis unit 340c may set the reference weight of invalid pixels to 0 based on the invalid region map when performing synthesis processing of exposure plane #1 and exposure plane #2 in the exposure plane synthesis unit 342. In this way, the exposure plane synthesis unit 340c can reduce the load caused by the exposure plane synthesis processing.

[0252] (3-4. Example of the Configuration of the Recognizer According to the First Embodiment) Next, an example of the configuration of the recognizer according to the first embodiment will be described.

[0253] 31A to 31D are block diagrams showing examples of the configuration of an imaging system according to the first embodiment.

[0254] The imaging system 1d shown in Fig. 31A includes, for each exposure plane, a sensitivity variation expansion sensor 300, a sensitivity variation correction unit 301, a saturated region correction unit 302, an exposure plane synthesis unit 340, and a recognition unit 330a. In the example of Fig. 31A, the imaging system 1d includes a configuration of n exposure planes (n is an integer of 2 or more). Each saturated region corrected image output from the multiple saturated region correction units 302 is supplied to the exposure plane synthesis unit 340. The exposure plane synthesis unit 340 synthesizes the saturated region corrected images of each exposure plane and supplies the synthesized image to the recognition unit 330a.

[0255] The recognition unit 330a executes a recognition process on the supplied composite image.

[0256] The imaging system 1e shown in Fig. 31B has a configuration in which the exposure plane composition unit 340 is omitted from the configuration of the imaging system 1d in Fig. 31A. In the imaging system 1e, the saturated region corrected images output from the saturated region corrector 302 of each exposure plane are supplied to the recognition unit 330b.

[0257] The recognition unit 330b has inputs of multiple channels for each exposure surface, and receives saturated region-corrected images of the exposure surface from each of the multiple channels. The recognition unit 330b performs recognition processing based on the images of each exposure surface input to each channel and outputs the recognition results.

[0258] The imaging system 1f shown in Fig. 31C has a configuration in which the saturated region correction unit 302 for each exposure surface is further omitted from the configuration of the imaging system 1e in Fig. 31B. In the imaging system 1f, the sensitivity variation corrected images for each exposure surface output from the multiple sensitivity variation correction units 301 are supplied to a recognition unit 330c.

[0259] The recognition unit 330c has inputs of multiple channels, for example, for each exposure surface, and sensitivity variation-corrected images of the exposure surface are input from each of the multiple channels. The recognition unit 330c performs recognition processing based on the images of each exposure surface input to each channel and outputs the recognition results. In this example, the recognition unit 330c performs recognition processing for each exposure surface, including saturated regions.

[0260] An imaging system 1g shown in Fig. 31D has a configuration in which the sensitivity variation correction unit 301 for each exposure surface is further omitted from the configuration of the imaging system 1f in Fig. 31C. In other words, the imaging system 1g does not include the signal processing unit 110 described using Fig. 2, for example. In the imaging system 1g, each sensitivity variation image output from a plurality of sensitivity variation magnification sensors 300 is supplied to a recognition unit 330c.

[0261] The recognition unit 330d has, for example, inputs from multiple channels, and sensitivity variation images output from each sensitivity variation expansion sensor 300 are input from each of the multiple channels. The recognition unit 330d performs recognition processing based on the images input to each channel and outputs the recognition results. In this example, the recognition unit 330d performs recognition processing on each exposure surface in a state including sensitivity variation pixels. Furthermore, depending on the model used, the recognition unit 330d can be configured to output information regarding the sensitivity variation of the images input to each channel.

[0262] Of the image capture systems 1d to 1g, the image capture system 1d has a recognition unit 330a that can be configured to have a single input channel, and therefore detailed description thereof will be omitted here. That is, the recognition unit 330a receives a composite image in which each correction process has been completed and the exposure surface has been combined.

[0263] On the other hand, in the imaging systems 1e and 1f, the recognition units 330b to 330d are required to have a plurality of channels since images are input for each exposure surface.

[0264] The imaging systems 1e to 1g may perform recognition processing independently for each exposure surface, or may have multiple channels corresponding to multiple exposure surfaces and perform recognition processing by integrating each exposure surface.

[0265] Also, in the examples of Figures 31A to 31D, the number of exposure surfaces n is shown as being 2 or more, but this is not limited to this example, and each example is also applicable when the number of exposure surfaces n is 1.

[0266] In the above description, the imaging system 1d has been described as inputting a composite image, a saturated region corrected image, a sensitivity variation corrected image, and a sensitivity variation image to the recognition units 330a to 330d, respectively. However, this is not limited to this example. That is, auxiliary information used to assist in the recognition process may also be input to each of the recognition units 330a to 330d. For example, auxiliary information including the variation correction map 310 and a saturation threshold map corresponding to each exposure surface may be input to the recognition unit 330b. Auxiliary information including the variation correction map 310 corresponding to each exposure surface may also be input to the recognition unit 330c. The auxiliary information that can be input to each of the recognition units 330a to 330d is not limited to the variation correction map 310 and the saturation threshold map; for example, a sensitivity ratio map or a saturation region map may also be applied.

[0267] (3-4-1. Example of the configuration of the recognition unit when auxiliary information is not used) Here, an example of the configuration of the recognition units 330a to 330d according to the first embodiment when auxiliary information is not used will be described. Note that, unless otherwise specified, the following description will be given assuming that the imaging system 1 is the imaging system 1g shown in FIG. 31D. Furthermore, in the following, it is assumed that the sensitivity of each pixel of the sensitivity variation expansion sensor 300 varies among four types of sensitivity: 1x, 1 / 2x, 1 / 4x, and 1 / 8x.

[0268] (First Example of Recognition Unit When Auxiliary Information Is Not Used) A first example of processing by the recognition unit 330d when auxiliary information is not used according to the first embodiment will be described. In the first example, the sensitivity variation image output from the sensitivity variation expansion sensor 300 is decomposed into planes for each pixel's sensitivity, and the planes are input to a network included in the recognition unit 330d. Figures 32A to 32C are schematic diagrams for describing a first example of the recognition unit 330d when auxiliary information is not used according to the first embodiment.

[0269] 32A to 32C, the sensitivity of a pixel or plane is represented by the darkness of the fill, with darker fill indicating a pixel or plane with lower sensitivity.

[0270] 32A is a schematic diagram showing an example in which image 30, which is a sensitivity variation image, is input to recognition unit 330d without being decomposed into sensitivities. In recognition unit 330d-10 shown in FIG. 32A, the output of sensitivity variation expansion sensor 300 is input directly to network (NW) 3300. That is, in the example of FIG. 32A, a plane including all of the pixels 500 included in image 30 is input to network 3300.

[0271] 32B and 32C show examples in which the sensitivity variation image is decomposed into planes 510a, 510b, 510c, and 510d for each sensitivity. In the examples of Fig. 32B and 32C, the sensitivity variation image is decomposed into a plane 510a with pixels having 1x sensitivity, a plane 510b with pixels having 1 / 2x sensitivity, a plane 510c with pixels having 1 / 4x sensitivity, and a plane 510d with pixels having 1 / 8x sensitivity.

[0272] 32B includes a network 3301 capable of receiving four-channel input, and each of the planes 510a, 510b, 510c, and 510d is input to the network 3301. The network 3301 combines the input planes 510a to 510d, for example, at the first stage, and performs recognition processing, etc. The recognition unit 330d-10 may train the network 3300 in advance, for example, based on information on the predetermined combination of each of the planes 510a to 510d.

[0273] The recognition unit 330d-12 shown in FIG. 32C includes networks 3302a, 3302b, 3302c, and 3302d, each of which accepts one channel of input, and a network 3303 capable of accepting four channels of input. Each of the planes 510a, 510b, 510c, and 510d is input to the networks 3302a, 3302b, 3302c, and 3302d, respectively, and undergoes, for example, the first-stage network processing in the recognition process. The recognition unit 330d-11 may train each of the networks 3302a to 3302d in advance, for example, based on the planes of their corresponding sensitivities. The outputs of the networks 3302a to 3302d are input to the network 3303, integrated, and then undergo the second-stage network processing in the recognition process.

[0274] 32B and 32C , image 30 as a sensitivity variation image is decomposed into planes 510a to 510d using sensitivity as a condition, but the condition for decomposing image 30 is not limited to sensitivity. That is, recognition unit 330d may achieve processing equivalent to decomposition according to sensitivity by adjusting the number of convolution steps (stride number) in the network included in recognition unit 330d, dilation which is a parameter for expanding the cell width of data to be convolved, the number of convolution taps, and the like.

[0275] According to this first example, a network trained according to the sensitivity of pixels can be used, making it possible to perform recognition processing with higher accuracy.

[0276] (Second Example of Recognition Unit When Auxiliary Information Is Not Used) A second example of processing by the recognition unit 330d when auxiliary information is not used according to the first embodiment will be described. The second example is an example in which sample points are dynamically changed when network processing is performed on the sensitivity variation image output from the sensitivity variation expansion sensor 300. Fig. 33 is a schematic diagram for describing processing by the recognition unit 330d according to the second example when auxiliary information is not used according to the first embodiment.

[0277] Section (a) of Figure 33 shows a schematic diagram of sampling using existing technology. In a network, a convolution operation for a pixel of interest 501 is generally performed in units of a block 520 consisting of nine pixels 500, for example, 3 pixels x 3 pixels, with each pixel 500 included in the block 520 being used as a sampling point. In existing technology, the convolution operation is performed while shifting the position of the block 520 within the image 30. In other words, in existing technology, the relative positional relationship of each pixel used in the convolution operation is fixed.

[0278] Section (b) of Figure 33 schematically illustrates a second example of sampling by the recognition unit 330d according to the first embodiment when auxiliary information is not used. In this second example, the recognition unit 330d performs a convolution operation on the image 30, using as sampling points each pixel included in a pixel group 521 formed by a predetermined number of pixels selected according to pixel characteristics. The recognition unit 330d may, for example, select a predetermined number of pixels 500' having similar sensitivities to form the pixel group 521. The recognition unit 330d also selects the pixel 500' included in the pixel group 521 from pixels near the pixel of interest 501, for example. The pixels near the pixel of interest 501 refer to pixels within a predetermined distance from the pixel of interest 501.

[0279] The recognition unit 330d may select the pixels 500' included in the pixel group 521 based on a condition other than sensitivity. For example, the recognition unit 330d may select pixels 500' having an S / N ratio equal to or greater than a predetermined value from among the pixels included in the image 30, and set them as the pixel group 521, or may acquire the pixels 500' included in the pixel group 521 through learning.

[0280] The recognition unit 330d sequentially sets all pixels included in the image 30 as a pixel of interest 501, and performs convolution using a network based on a pixel group 521' that dynamically selects pixels 500' with sensitivity corresponding to the pixel of interest 501 from pixels in the vicinity of the pixel of interest 501.

[0281] The pixel 500' included in the pixel group 521 may be dynamically selected, for example, according to an object included in a range corresponding to the pixel group 521 in the image 30. Furthermore, as described above, the pixel 500' included in the pixel group 521 may be dynamically selected according to the sensitivity variation for each pixel of the sensitivity variation expansion sensor 300. For example, when the position of the target pixel 501 changes, the sensitivity of the target pixel 501 changes according to the sensitivity variation, and the sensitivity of pixels in the vicinity of the target pixel 501 also changes. Therefore, the pixel 500' included in the pixel group 521 is dynamically selected according to the position of the target pixel 501.

[0282] The recognition unit 330d may also weight each pixel 500 included in the block 520 based on the existing technology in section (a) of Fig. 33, or each pixel 500' included in the pixel group 521 according to the first embodiment in section (b) of Fig. 33. In this case, the recognition unit 330d may dynamically weight each pixel 500 or each pixel 500' according to the characteristics of the pixel.

[0283] According to this second example, the recognition unit 330d can perform convolution using a pixel group 521 that selectively extracts pixels 500' that are appropriate for the recognition process. For example, as described above, the recognition unit 330d can selectively extract pixels 500' that have an S / N ratio equal to or higher than a predetermined value or pixels 500' that have sensitivity close to that of the target pixel 501 to form the pixel group 521, thereby enabling the recognition process to be performed with higher accuracy.

[0284] Furthermore, according to this second example, for example, when the target pixel 501 is included in a relatively small object, the range of the pixel group 521 can be narrowed, and when the target pixel 501 is included in a large object, the range of the pixel group 521 can be widened, thereby enabling the recognition unit 330d to perform recognition processing with higher accuracy.

[0285] (Third Example of Recognition Unit When Auxiliary Information Is Not Used) A third example of processing by the recognition unit 330d according to the first embodiment when auxiliary information is not used will be described. The third example is an example in which pixels with little information in the input image are masked and calculations using those pixels are skipped, thereby reducing the amount of calculation. Saturated pixels, for example, may be used as pixels with little information.

[0286] 34 is a schematic diagram illustrating the processing of the recognition unit 330d according to a third example in which auxiliary information is not used, according to the first embodiment. The recognition unit 330d performs convolution processing on a pixel of interest 501 in units of 3-pixel by 3-pixel blocks 522, each block consisting of the pixel of interest 501 and each pixel adjacent to the pixel of interest 501, with each pixel included in the block 522 being used as a sampling point. To perform convolution on 3-pixel by 3-pixel blocks, nine product-sum operations are required.

[0287] 34 , three of the nine pixels included in block 522 are shown as filled-in pixels 503 with little information. The recognition unit 330d may, for example, estimate the pixels 503 with little information based on the pixel values ​​of the pixels included in block 522. The recognition unit 330d masks the estimated pixels 503 with little information, and performs a convolution operation using the unmasked pixels 504.

[0288] In this example, three of the nine pixels included in block 522 are set as pixels 503 with little information, and the recognition unit 330d performs convolution calculations using the other six pixels 504 included in block 522. The number of calculations for the convolution calculations in 3 pixel x 3 pixel block 522 is reduced from nine when no mask is used to six, thereby realizing faster convolution calculations and reducing the load on the convolution calculations.

[0289] When saturated pixels are used as the pixels 503 with little information, the recognition unit 330d is not limited to estimating saturated pixels based on pixel values, and may estimate saturated pixels using the saturation threshold map 320, for example.

[0290] (Fourth Example of Recognition Unit When Auxiliary Information Is Not Used) A fourth example of processing by the recognition unit 330d according to the first embodiment when auxiliary information is not used will be described. The fourth example is an example in which network processing is performed using global illumination information related to the input image. Global illumination refers to illumination that combines direct illumination and indirect illumination related to the subject.

[0291] FIG. 35 is a schematic diagram for explaining the processing of the recognition unit 330d according to the fourth example of the recognition unit when auxiliary information is not used, according to the first embodiment.

[0292] 35, section (a) schematically illustrates an example of a process for acquiring global illumination information 523. In section (a), network 3320 is pre-trained to acquire global illumination information from a reduced image obtained by reducing the size of an input image. Network 3320 acquires global illumination information 523 from a reduced image 32 obtained by reducing the size of the input image.

[0293] For example, the network 3320 may calculate the average value of the pixel values ​​of each pixel in the reduced image 32 and use the average value as the global illumination information. Alternatively, the network 3320 may use other statistical values, such as the distribution of pixel values ​​of each pixel in the reduced image 32, as the global illumination information. Alternatively, the input image may be divided into multiple blocks, and global illumination information may be obtained for each block. Furthermore, the means for obtaining global illumination information from the reduced image 32 is not limited to the network 3320. For example, the global illumination information may be obtained from the reduced image 32 by another image processing means.

[0294] In Figure 35, section (b) schematically illustrates the processing of the recognition unit 330d-20 according to the fourth example of the recognition unit when auxiliary information is not used. An input image, an image 30 corresponding to a reduced image 32, is input to the network 3304. Note that the reduced image 32 may be an image obtained by reducing the image 30 by thinning or the like. However, the reduced image 32 may be an image similar to the image 30, for example, an image captured at a different time of the same subject as the image 30 in an environment with the same global illumination as the image 30.

[0295] The network 3304 performs a recognition process or the like on the image 30 using the global illumination information 523 acquired in the process of section (a) of Fig. 35 as a priori. For example, the network 3304 may perform the recognition process by selectively using pixels having an appropriate sensitivity for the brightness of the captured scene from among the pixels included in the image 30 based on the global illumination information 523. Also, for example, the network 3304 may weight the output of the recognition process that uses pixels having an appropriate sensitivity for the brightness of the captured scene from among the pixels included in the image 30 based on the global illumination information 523.

[0296] According to this fourth example, the recognition unit 330d-20 performs recognition processing on the image 30 using global illumination information 523 acquired based on the image 30 or a reduced image 32 obtained by reducing an image similar to the image 30. Therefore, the recognition unit 330d-20 can perform recognition processing according to the photographed scene, and can perform recognition processing with higher accuracy.

[0297] (Fifth Example of Recognition Unit When Auxiliary Information Is Not Used) A fifth example of processing by the recognition unit 330d when auxiliary information is not used according to the first embodiment will be described. The fifth example is an example in which, when performing time-series processing on an input image, the reference length of the time-series information or the weighting of past information is changed according to scene information and sensitivity information related to the input image.

[0298] 36A and 36B are schematic diagrams illustrating a fifth example of a recognition unit 330d according to the first embodiment when auxiliary information is not used. In FIGS. 36A and 36B, the sensitivity variation image is decomposed into planes 510a, 510b, 510c, and 510d for each sensitivity. In the example of FIGS. 36A and 36B, the sensitivity variation image is decomposed into a plane 510a with pixels having 1x sensitivity, a plane 510b with pixels having 1 / 2x sensitivity, a plane 510c with pixels having 1 / 4x sensitivity, and a plane 510d with pixels having 1 / 8x sensitivity.

[0299] 36A and 36B, the sensitivity of a plane is represented by the darkness of the fill, with darker fill indicating a plane with lower sensitivity.

[0300] Using Figure 36A, we will explain the processing in the case where the reference length of the time series information is changed for each sensitivity in a fifth example of processing by the recognition unit 330d in the first embodiment when auxiliary information is not used.

[0301] 36A, recognition unit 330d-30 includes networks 3305a, 3305b, 3305c, and 3305d, each of which accepts input from multiple channels, and network 3306, which can accept input from four channels. Each of planes 510a, 510b, 510c, and 510d receives input frame by frame in chronological order to multiple channels of networks 3302a, 3302b, 3302c, and 3302d, respectively, and undergoes, for example, network processing at the early stage of the recognition process. For example, if network 3305a can accept input from three channels, each of the past three frames of plane 510a is input to each of the three channels.

[0302] In this case, the recognition unit 330d-30 uses more past frames for planes with pixels having improper exposure. In the example of FIG. 36A , planes 510b and 510d are considered to be planes with improper exposure, and three past frames for each are input to multiple channels of networks 3305b and 3305d, respectively. Furthermore, plane 510a is considered to be a plane with an even higher degree of improper exposure than planes 510b and 510d, and four past frames for each are input to multiple channels of network 3305a, respectively. For images with pixels having a high degree of improper exposure, a noise reduction effect can be expected by inputting multiple frames, enabling highly accurate recognition processing.

[0303] On the other hand, plane 510c is a plane made up of pixels with proper exposure (low degree of inadequacy), and one frame is input to network 3305c. Using an image made up of pixels with proper exposure alone reduces motion blur and enables highly accurate recognition processing.

[0304] The outputs of the networks 3305a to 3305d are input to a network 3306, where they are integrated and subjected to subsequent network processing in the recognition process.

[0305] Using Figure 36B, we will explain the processing in the fifth example of processing by the recognition unit 330d in the first embodiment when auxiliary information is not used, in which the weighting for past information is changed depending on the scene information and sensitivity information related to the input image.

[0306] 36B, recognition unit 330d-31 includes networks 3307a, 3307b, 3307c, and 3307d, each of which accepts one channel of input, and network 3308, which can accept four channels of input. Planes 510a, 510b, 510c, and 510d are input to networks 3302a, 3302b, 3302c, and 3302d, respectively, and are subjected to, for example, network processing at the front end of the recognition process.

[0307] Here, the recognition unit 330d-31 uses, for example, a recurrent neural network (RNN) network as each of the networks 3307a, 3307b, 3307c, and 3307d. The RNN is a network that performs recursive processing, and for example, the recognition process is performed using a network whose internal information is updated using the immediately preceding recognition result. The RNN can improve recognition accuracy by sequentially updating the internal state of the RNN using, for example, frame images that are updated in chronological order.

[0308] In this example, the recognition unit 330d-31 increases the weighting of past information (past feature amounts) for planes with pixels of improper exposure. In the example of Fig. 36B, in each of the networks 3307a to 3307d, the size of the circle of the arrow that returns to itself indicates the magnitude of the weighting of the past information.

[0309] Planes 510a, 510b, and 510d are planes of pixels with improper exposure, with plane 510d having the greatest degree of improper exposure and plane 510b having the least. On the other hand, plane 510c is a plane with pixels with proper exposure (the least degree of improper exposure).

[0310] Network 3307d assigns the largest weighting among networks 3307a to 3307d to feature quantity 3330d based on input plane 510d in accordance with the degree of exposure inadequacy, and updates the internal information of network 3307d based on weighted feature quantity 3330d. Similarly, networks 3307a and 3307b assign weighting amounts according to the respective degrees of exposure inadequacy, and update the internal information of networks 3307a and 3307b based on weighted feature quantities 3330a and 3330b, respectively. Meanwhile, network 3307c updates its internal information by using feature quantity 3330c based on input plane 510c as is, for example, or by assigning the smallest weighting among networks 3307a to 3307d.

[0311] 36A and 36B, the degree of inadequacy in the exposure of each of the planes 510a to 510d may be found based on scene information of the input image and sensitivity information of each pixel of the sensitivity variation expansion sensor 300. For example, in a scene with higher brightness, the degree of inadequacy of the plane due to high-sensitivity pixels may be high, and the degree of inadequacy of the plane due to low-sensitivity pixels may be low.

[0312] The outputs of the networks 3307a to 3307d are input to a network 3308, where they are integrated and subjected to subsequent network processing in the recognition process.

[0313] According to this fifth example, the weights of the time-series information are changed depending on the degree of inadequacy in the exposure of the pixels included in each of the planes 510a to 510d. Therefore, by applying the recognition units 330d-30 and 330d-31 according to this fifth example, it is possible to perform the recognition process with higher accuracy.

[0314] (Sixth Example of Recognition Unit When Auxiliary Information Is Not Used) A sixth example of processing by the recognition unit 330d when auxiliary information is not used according to the first embodiment will be described. The sixth example is an example in which pixel information is divided in the luminance direction and calculation is performed.

[0315] 37 is a schematic diagram illustrating a sixth example of the recognition unit 330d according to the first embodiment when auxiliary information is not used. In FIG. 37, the recognition unit 330d-40 according to the sixth example includes a network 3309 and a luminance direction attention unit 3340. The network 3309 performs arithmetic processing such as a convolution operation on the image 30 and outputs the arithmetic result.

[0316] The luminance direction attention unit 3340 specifies a value range of interest 3341 indicating a value range to be focused on in the luminance direction for each pixel 500 included in the image 30 based on the output of the network 3309. The luminance direction attention unit 3340 may specify the value range of interest 3341 for the image 30 by learning, for example. Without being limited to this, the luminance direction attention unit 3340 may specify the value range of interest 3341 based on sensitivity information for each pixel of the sensitivity variation expansion sensor 300 that has been acquired in advance. The luminance direction attention unit 3340 feeds back information indicating the specified value range of interest 3341 to the network 3309.

[0317] The network 3309 performs calculations on the image 30, focusing on the luminance range indicated in the range of interest 3341. Alternatively, the network 3309 may perform calculations on the image 30 by masking all luminance ranges other than the reliable range based on the range of interest 3341. The calculation output of the network 3309 is output from the recognition unit 330d-40 as the recognition result by the recognizer 33d-40. The calculation output of the network 3309 is also supplied to the luminance direction attention unit 3340.

[0318] In the sixth example, the network 3309 limits the range of brightness values ​​in the pixel information used for calculation, so that the load of calculation processing on the network 3309 can be reduced.

[0319] In the above description, the network 3309 performs calculations on the image 30 in the attention value range 3341 specified by the luminance direction attention unit 3340. In this case, for example, the luminance direction attention unit 3340 may delete pixels whose luminance values ​​exceed the attention value range 3341. Furthermore, the luminance direction attention unit 3340 may convert the luminance values ​​of each pixel in the image 30 whose luminance values ​​are within the attention value range 3341 so that the variance of the luminance values ​​is 1 and the average is 0, and specify the attention value range 3341. Without being limited to this, the luminance direction attention unit 3340 may perform a conversion using a predetermined tone curve or an arbitrary linear conversion on the image 30, for example.

[0320] In the above description, the luminance direction attention unit 3340 specifies the region of interest 3341 based on the output of the network 3309, but this is not limited to this example. For example, the luminance direction attention unit 3340 may specify the region of interest 3341 based on information extracted from an intermediate layer of the network 3309. In the above description, the network 3309 applies the region of interest 3341 to the image 30 input to the input layer, but this is not limited to this example. For example, the network 3309 may apply the region of interest 3341 to an intermediate layer. Furthermore, in the above description, the network 3309 applies the region of interest 3341 to the entire image 30, but the region of interest 3341 may be specified and applied for each divided region obtained by dividing the image 30. Furthermore, multiple regions of interest 3341 may be specified for one piece of pixel information.

[0321] The first to sixth examples of the recognition unit when auxiliary information is not used can be used in any combination as long as they do not contain contradictory elements.

[0322] (3-4-2. Example of the Configuration of the Recognition Unit When Auxiliary Information is Used) FIGS. 38A and 38B are schematic diagrams showing an example of the configuration of the recognition unit 330 applicable to the first embodiment.

[0323] The recognition unit 330e shown in Fig. 38A is an example having a structure in which input channels are multi-channelized for a general network (shown as "NW" in the figure) 332. An image 30 and auxiliary information 31, 31, ... are input to the recognition unit 330e. Note that in the figure, the auxiliary information 31, 31, ... are also shown as auxiliary information #1, auxiliary information #2, ....

[0324] Here, the image 30 is a sensitivity variation image (in the case of the imaging system 1g), a sensitivity variation correction image (in the case of the imaging system 1f), or a saturated region correction image (in the case of the imaging system 1e). The auxiliary information 31, 31, ... is a variation correction map 310, a saturation threshold map, etc. The saturation threshold map may be a scalar value when a saturation threshold common to all pixels is used.

[0325] The images 30 and auxiliary information 31, 31, ... input to the recognition unit 330e are input to the multi-channel processing unit 331. The multi-channel processing unit 331 combines the input images 30 and auxiliary information 31, 31, .... The method by which the multi-channel processing unit 331 combines the images 30 and auxiliary information 31, 31, ... is not particularly limited, and may be a combination that simply increases the dimension, or it is also possible to apply an element sum or element product.

[0326] The multi-channel processing unit 331 inputs combined data 35, which is obtained by combining the image 30 and auxiliary information 31, 31, ..., to the network 332. In other words, the multi-channel processing unit 331 has a function of multi-channelizing the input of the network 332. The network 332 performs recognition processing on the input combined data 35 and outputs the recognition result.

[0327] A recognition unit 330f shown in FIG. 38B is an example in which an image 30 and auxiliary information 311, 312, . . . are input to general networks 333a, 333b1, 333b2, . . . , respectively, and network processing is performed, and the results are combined.

[0328] 38B , an image 30 and auxiliary information 31, 31, ... are input to a recognition unit 330f. In a recognition unit 33b′, the image 30 and auxiliary information 31, 31, ... are input to general networks 333a, 333b, 333b, ..., respectively. Each network 333a, 333b, 333b, ... performs a predetermined network process on the input image 30 and auxiliary information 31, 31, ..., and outputs intermediate features 33a, 33b, 33b, ..., each having a predetermined dimension, for example. The intermediate features 33a, 33b, 33b, ... are each input to a combination unit 334.

[0329] The combining unit 334 combines the input intermediate features 33 a, 33 b, 33 b, ..., and outputs the combined intermediate feature 34. The method by which the combining unit 334 combines the image 30 and the auxiliary information 31, 31, ... is not particularly limited, and may be, for example, a combination that simply increases the dimension, or it may also be possible to apply an element sum or element product.

[0330] The combined intermediate features 34 output from the combining unit 334 are input to a general network 335. The network 335 performs recognition processing on the input combined features 32 and outputs the recognition results.

[0331] The configuration of these recognition units 330e and 330f may be applied to, for example, the above-mentioned recognition units 330b to 330d.

[0332] Fig. 39 is a schematic diagram for explaining the data structure of auxiliary information 31, 31, ... according to the first embodiment. In sections (a) and (b) of Fig. 39, networks 3308 and 3308' schematically show the configurations included in recognition unit 330e shown in Fig. 38A or recognition unit 330f shown in Fig. 38B, respectively.

[0333] Section (a) of FIG. 39 shows auxiliary information 31 in a two-dimensional array. 2D 3 shows an example in which auxiliary information 31 is used. 2D is input to the network 3308 as information of a two-dimensional array corresponding to the array of pixels Pix included in the pixel array unit 1011 of the sensitivity variation expansion sensor 300. 2D For example, a sensitivity variation map or a saturation region map may be applied.

[0334] Section (b) of FIG. 39 shows auxiliary information 31 in the form of an n-dimensional array (n>2). nD For example, the auxiliary information 31 shown in section (a) is transmitted using a network. 2D is projected onto a high-dimensional vector, and auxiliary information 31 is obtained from the high-dimensional vector. nD In this example, the auxiliary information 31 nD is set to n=256, and is 256-dimensional information.

[0335] In other words, in a network, it may be more efficient to convert data into a high-dimensional vector rather than treating it as a scalar value. For example, networks such as CNNs (Convolutional Neural Networks) often have difficulty directly handling scalar values. In such cases, converting information based on scalar values ​​in a two-dimensional array into information based on a high-dimensional vector can be expected to speed up network processing.

[0336] As an example, as shown on the right side of section (b) in Fig. 39, when there is information of a scalar value of 3 / 4, a method is known in which this value of 3 / 4 is projected and converted into an n-dimensional (n>2) vector, for example, a 256-dimensional vector where n = 256. When processing is performed using this method in the previous stage, when combined with processing in the subsequent stage, there are cases in which the redundant representation of a high-dimensional vector is more efficient than a scalar value.

[0337] (Auxiliary Information Separation Processing) FIGS. 40A and 40B are schematic diagrams showing configuration examples for separating auxiliary information from a sensitivity variation image according to the first embodiment.

[0338] Fig. 40A shows an example configuration using the above-described RNN-based network 336. In section (a) of Fig. 40A , the network 336 receives the sensitivity variation image I(t) output from the sensitivity variation expansion sensor 300. The sensitivity variation image I(t) input to the network 336 includes an image for each frame that includes time-series fluctuations, and sensitivity variation information that is static information related to sensitivity variations in pixel units.

[0339] The network 336 is trained to separate and retain static information without time-series fluctuations from input information including time-series fluctuations, and to output the difference between the input information and the retained static information. The network 336 trained in this manner performs network processing on the sensitivity variation image I(t) input as information including time-series fluctuations, and separates sensitivity variation information S(t) as static information from the sensitivity variation image I(t). The separated sensitivity variation information S(t) is auxiliary information for the sensitivity variation image I(t).

[0340] The network 336 stores the separated sensitivity variation information S(t) as a variation correction map 310. The variation correction map 310 is updated by the sensitivity variation information S(t) separated from the sensitivity variation image I(t) that is sequentially input. The network 336 corrects the sensitivity variation image I(t) based on the stored variation correction map 310 and outputs a sensitivity variation corrected image I(t)' that includes time-series fluctuations. The network 336 also updates its internal state using the stored variation correction map 310.

[0341] In the above description, the network 336 stores the sensitivity variation information S(t) separated from the sensitivity variation image I(t) as the variation correction map 310, but this is not limited to this example. For example, as shown in section (b) of FIG. 40A , the network 336′ may store intermediate feature values ​​F(t) generated by performing network processing on the sensitivity variation image I(t). The network 336′ updates its internal state using the stored intermediate feature values ​​F(t). Furthermore, it is possible to generate the variation correction map 310 by performing predetermined network processing on the stored intermediate feature values ​​F(t).

[0342] Fig. 40B shows an example configuration using a network 337 that is not an RNN. In section (a) of Fig. 40B, sensitivity variation images I(t), I(t-1), and I(t-2) acquired at multiple times (t), (t-1), and (t-2), respectively, are input to the network 337. That is, the sensitivity variation images including time-series fluctuations and sensitivity variation information, which is static information, are input to the network 337.

[0343] Network 337 is trained to separate and retain static information that does not fluctuate over time from a plurality of pieces of input information acquired at different times, and to output the difference between the input information and the retained static information. Network 337 trained in this manner performs network processing on input sensitivity variation images I(t), I(t-1), and I(t-2) acquired at different times (for example, for each frame), and separates sensitivity variation information S(t) as static information from the sensitivity variation images I(t), I(t-1), and I(t-2). The separated sensitivity variation information S(t) is auxiliary information for the sensitivity variation image I(t).

[0344] The network 337 stores the separated sensitivity variation information S(t) as the variation correction map 310. The network 337 updates the variation correction map 310 with the sensitivity variation information S(t+1) separated from the sensitivity variation images I(t+1), I(t), and I(t−1) acquired at the next timing, for example.

[0345] In the above description, the network 337 stores the sensitivity variation information S(t) separated from the sensitivity variation images I(t), I(t-1), and I(t-2) as the variation correction map 310, but this is not limited to this example. For example, as shown in section (b) of FIG. 40B , the network 337′ may store intermediate feature values ​​F(t) generated by performing network processing on the sensitivity variation images I(t), I(t-1), and I(t-2). The variation correction map 310 can be generated by performing predetermined network processing on the stored intermediate feature values ​​F(t).

[0346] (Regarding Individual Differences in Sensitivity Variation) Strictly speaking, the sensitivity variation in the sensitivity variation expanded sensor 300 differs for each individual sensitivity variation expanded sensor 300. It is preferable that the individual difference in sensitivity variation among the sensitivity variation expanded sensors 300 be as small as possible. In order to reduce this individual difference, it is possible to consider learning the difference for each individual sensitivity variation expanded sensor 300. However, in practical applications, this method of learning the difference for each individual sensor is difficult, considering the cost of creating learning data and a recognizer for each individual sensor. Therefore, several methods for absorbing the individual difference among the sensitivity variation expanded sensors 300 are proposed below.

[0347] The first method is to create a recognizer that absorbs the expected individual variations. In this case, training data that covers the expected distribution of sensitivity variations in the sensitivity variation expanded sensor 300 is prepared, and a recognizer trained using this training data is used as, for example, the recognition unit 330.

[0348] The second method is to provide the recognizer (recognition unit 330) with individual variation information indicating the individual variation of the sensitivity variation expanded sensor 300. Based on this individual variation information, the recognizer outputs a recognition result that absorbs the individual variation of the sensitivity variation expanded sensor 300. As the individual variation information, a σ value indicating the variance of deviation from the true value may be used.

[0349] The third method is a method of absorbing the individual variation of the sensitivity variation expansion sensor 300 online. For example, as described with reference to FIG. 15 , the individual variation is estimated by online calibration and reflected in the variation correction map 310. As a modification of the third method, the individual variation may be estimated within the recognizer. For example, the recognizer (recognition unit 330) may automatically estimate and learn the individual variation over time.

[0350] (Regarding Feedback to Sensor Based on Recognition Result) The imaging system 1 according to the first embodiment can feed back the recognition result by the recognition unit 330 to the sensitivity variation expansion sensor 300. Fig. 41 is a schematic diagram for explaining feedback of the recognition result to the sensitivity variation expansion sensor 300 according to the first embodiment.

[0351] 41 generates parameters for controlling sensitivity variation in the sensitivity variation expanding sensor 300 based on the recognition result by the recognition unit 330. The parameters may be generated in the recognition unit 330 or in the sensitivity variation expanding sensor 300.

[0352] As parameters for controlling sensitivity variations, for example, it is conceivable to apply parameters for controlling the operation of the AD conversion unit 1013 in the imaging unit 1031, as described with reference to Fig. 4. However, it is also conceivable to apply parameters for controlling the operation of the pixels Pix, as described with reference to Fig. 5, as parameters for controlling sensitivity variations. Parameters for controlling sensitivity variations are not limited to these examples.

[0353] 4. Second Embodiment Next, a second embodiment of the present disclosure will be described. The second embodiment relates to a hardware configuration of a sensitivity variation expanding sensor 300 according to the present disclosure, particularly a configuration in units of pixels.

[0354] When sensing and recognizing a subject that exists in both bright and dark areas, such as in an imaging system for automotive applications, the dynamic range of the sensor becomes an issue. Existing technologies have addressed the issue of the sensor's dynamic range by improving the device to increase the number of saturation electrons in each pixel of the sensor, or by performing multiple exposures through signal processing. However, these existing methods have the problem of increasing the power consumption of the sensor.

[0355] In contrast, in the present disclosure, the dynamic range is expanded by varying the sensitivity of the sensor on a pixel-by-pixel basis, that is, by varying the sensitivity of each pixel of the sensor on a pixel-by-pixel basis. The technique according to the present disclosure makes it possible to expand the dynamic range while suppressing increases in power consumption using an easily realizable device.

[0356] 42 is a schematic diagram for explaining the hardware configuration of the sensitivity variation expansion sensor 300 according to the second embodiment. In FIG. 42 , the sensitivity variation expansion sensor 300 according to the second embodiment includes pixels Pix, an auxiliary information holding unit 410, and an auxiliary information output unit 412.

[0357] In the sensitivity variation expanding sensor 300, photons are incident on a pixel Pix from a scene 400, i.e., a subject. The pixel Pix generates a pixel signal in response to the incident photons. The pixel signal generated by each pixel Pix included in a pixel array unit 1011 (not shown) is output from the sensitivity variation expanding sensor 300 as a sensitivity variation sensor output 401 (sensitivity variation image).

[0358] Furthermore, the pixel Pix has auxiliary information 411 according to its structure. The auxiliary information 411 held by the pixel Pix is ​​held in an auxiliary information holding unit 410. The auxiliary information holding unit 410 may be, for example, a flash ROM provided in the sensitivity variation expansion sensor 300 for storing parameters and the like.

[0359] The auxiliary information 411 stored in the auxiliary information storage unit 410 is output from the sensitivity variation expansion sensor 300 by an auxiliary information output unit 412 as auxiliary information output 413. The auxiliary information output unit 412 is an interface for communicating with, for example, the signal processing unit 110 serving as a host device downstream of the sensitivity variation expansion sensor 300.

[0360] 42 , the pixel Pix includes a photon domain 420, an electron domain 440, and a subsequent domain 460. The photon domain 420 is a configuration related to the incidence of photons on the sensitivity variation expansion sensor 300. The electron domain 440 includes a configuration that performs photoelectric conversion to convert incident photons into electric charges in the photon domain 420. The subsequent domain 460 includes a configuration that converts the electric charges resulting from the conversion of photons in the electron domain 440 into voltages to generate pixel signals. The subsequent domain 460 is not limited to the configuration shown in FIG. 42 , and some of its functions may be provided outside the pixel Pix.

[0361] In the second embodiment, the domains for varying the sensitivity of the sensitivity variation expanding sensor 300 on a pixel-by-pixel basis are classified into the three domains described above: the photon domain 420, the electron domain 440, and the subsequent domain 460. The photon domain 420 achieves sensitivity variation on a pixel-by-pixel basis by using a structure that controls incident photons. The electron domain 440 achieves sensitivity variation on a pixel-by-pixel basis by controlling the efficiency of photoelectric conversion and the transfer of electrons that have undergone photoelectric conversion. The subsequent domain 460 achieves sensitivity variation on a pixel-by-pixel basis by controlling the conversion efficiency during electron-to-voltage conversion and the amplifier gain. The subsequent domain 460 may also include an A / D conversion circuit that converts analog pixel signals into digital pixel signals, and sensitivity variation on a pixel-by-pixel basis can also be achieved by controlling the A / D conversion.

[0362] In addition, the second embodiment also proposes a method for holding auxiliary information including sensitivity variation and outputting it to the host side (the subsequent stage of the sensitivity variation expansion sensor 300).

[0363] (4-1. Photon Domain) First, the control of sensitivity variations in the photon domain 420 according to the second embodiment will be described.

[0364] 43 is a schematic diagram illustrating an example of the structure of a pixel Pix applicable to Embodiment 2. In FIG. 43, the pixel Pix includes a light collection layer 421, a semiconductor layer 422, and a wiring layer 423.

[0365] The light-collecting layer 421 includes an on-chip lens 430, an optical filter 431, a light-shielding film 432, and an insulating film 433. The on-chip lens 430 collects light incident on the pixel Pix and guides it to the semiconductor layer 422. The light-shielding film 432 prevents the incident light from leaking into adjacent pixels Pix. The optical filter 431 is, for example, a color filter with spectral characteristics, and selectively transmits light with wavelengths in a specific wavelength range. The light-collecting layer 421 is connected to the semiconductor layer 422 via the insulating film 433.

[0366] The semiconductor layer 422 is made of, for example, a silicon (Si) substrate or a compound semiconductor substrate such as indium gallium arsenide (InGaAs). The semiconductor layer 422 includes, for each pixel Pix, a photoelectric conversion unit 440 (photodiode 1021) and a plurality of pixel transistors (not shown) (transfer transistor 1022, reset transistor 1023, amplification transistor 1024, and selection transistor 1025). The semiconductor layer 422 also includes a diffractive / scattering structure 434 with a fine concave-convex structure on the incident surface side. The diffractive / scattering structure 434 traps photons from incident light and improves incidence efficiency.

[0367] The wiring layer 423 includes a wiring section 450 and an insulating film 451. The wiring section 450 forms signal lines and power supply lines that transmit pixel signals generated by the pixels Pix and signals applied to pixel circuits in the pixels Pix.

[0368] (Control of Light-Collecting Layer) The sensitivity of the pixel Pix may be changed by changing the shape of the on-chip lens 430, for example, the height, curvature, or dimensions of the on-chip lens 430. This increases or decreases the number of photons incident on the photoelectric conversion unit, making it possible to increase or decrease the sensitivity of the pixel Pix. The sensitivity of the pixel Pix may also be changed by changing the reflectance of an anti-reflection film provided on the incident surface side of the on-chip lens 430. More specifically, the reflectance of the anti-reflection film may be changed by changing the refractive index or thickness of the anti-reflection film.

[0369] The sensitivity of the pixel Pix may be changed by changing the shape of the light-shielding film 432 or by thinning the film thickness to allow light to pass through. The light-shielding film 432 is formed using a metal mask in the manufacturing process, so the shape and film thickness can be controlled with high precision, and sensitivity variations can also be easily controlled. Specifically, the light-shielding film 432 can be changed by changing the metal mask or the film formation conditions. Changing the light-shielding film 432 makes it easy to reduce the number of photons incident on the photoelectric conversion unit.

[0370] The sensitivity of the pixel Pix may be changed by changing the shape, transmittance, color combination, and material of the optical filter 431. For example, by changing the optical filter 431 as a color filter, the sensitivity related to the spectral characteristics can be changed. The shape of the optical filter 431 includes changing the thickness of the optical filter 431 and the shape of the incident surface. The color combination is achieved, for example, by overlapping color filters of different colors (wavelength ranges to be transmitted) for one pixel Pix. As an example, an R (red) color filter and a B (blue) color filter are overlapped. Regarding the material of the optical filter 431, the composition, concentration, and film thickness of the material of the optical filter 431 may be changed.

[0371] The sensitivity of the pixel Pix may be changed by changing the structure of the diffraction / scattering structure 434. For example, the sensitivity of the pixel Pix can be changed by changing the shape (such as a square pyramid or a cone), number, pitch, or height (depth) of the fine concave-convex structure. By changing the diffraction / scattering structure 434, it is possible to increase or decrease the number of photons incident on the photoelectric conversion unit, and the sensitivity of the pixel Pix can be changed to either higher or lower sensitivity.

[0372] (Control of Wiring Layer) The sensitivity of the pixel Pix may be changed by changing the wiring section 450 in the wiring layer 423. That is, photons incident on the pixel Pix are converted into electric charges by the photoelectric conversion section 440, but some are dropped by the photoelectric conversion section 440 and reach the wiring layer 423. The photons that reach the wiring layer 423 are reflected by the wiring section 450 and are again incident on the photoelectric conversion section 440. Therefore, by utilizing the reflection by the wiring section 450, the number of photons photoelectrically converted by the photoelectric conversion section 440 can be increased, thereby enabling higher sensitivity. For example, the sensitivity of the pixel Pix may be changed by changing the area, wiring pattern, etc. of the wiring section 450 through a design change of the wiring layer 423.

[0373] (4-2. Electronic Domain) Next, sensitivity variation control in the electronic domain 440 according to the second embodiment will be described.

[0374] 44 is a schematic diagram for explaining sensitivity variation control in an electron domain 440 according to the second embodiment. Fig. 44 shows an excerpt of the semiconductor layer 422 and the wiring layer 423 in Fig. 43 described above.

[0375] 44, in the semiconductor layer 422, the photoelectric conversion unit 440 converts incident photons into charges (electrons) through photoelectric conversion. The photoelectric conversion unit 440 accumulates the charges converted from the photons. The accumulated charges are transferred to the FD via the transfer transistor 1022 (see FIG. 5) formed in the semiconductor layer 422 by drive control.

[0376] In the electronic domain 440, the sensitivity of the pixel Pix is ​​changed by changing the characteristics of the photoelectric conversion unit 440.

[0377] The photoelectric conversion unit 440 is formed across the entire thickness of the semiconductor layer 422, which is made of, for example, a silicon (Si) substrate or a compound semiconductor substrate such as indium gallium arsenide (InGaAs), and is configured as a pn junction photodiode consisting of a first conductivity type, for example, an n-type semiconductor region, and a second conductivity type, for example, a p-type semiconductor region, facing both the front and back surfaces of the substrate. The p-type semiconductor regions facing both the front and back surfaces of the substrate also serve as hole charge accumulation regions for suppressing dark current. Each pixel Pix is ​​isolated by an element isolation unit (not shown) provided in the semiconductor layer 422.

[0378] In the semiconductor layer 422, a pattern is formed using resist on the surface side of a semiconductor substrate, which is, for example, a silicon substrate, and a p-type well region and an n-type semiconductor region are formed in the semiconductor substrate, and a photoelectric conversion section 440 and the like are formed by ion implantation.

[0379] The sensitivity of the pixel Pix may be changed by changing the shape of the photoelectric conversion unit 440, such as the area of ​​the incident surface or the depth (thickness). The area of ​​the incident surface of the photoelectric conversion unit 440 can be changed by changing the resist mask that determines the implantation regions into which the p-type and n-type implants that form the photoelectric conversion unit 440 are implanted. At this time, the height of the resist mask may be further changed, or unevenness may be tolerated. The depth of the photoelectric conversion unit 440 may be changed by changing the mask pattern for the n-type implant that is implanted deep into each pixel. This change changes the photoelectric conversion efficiency in the photoelectric conversion unit 440, making it possible to achieve both high and low sensitivity.

[0380] The sensitivity of the pixel Pix may be changed by providing an absorption region that absorbs photons inside the photoelectric conversion unit 440. Furthermore, the sensitivity of the pixel Pix may be changed by changing the width of the element isolation unit to change the area of ​​the light-receiving region on the incident surface. Furthermore, the sensitivity of the pixel Pix may be changed by providing a potential barrier in the photoelectric conversion unit 440, collapsing the potential barrier, drilling a hole, or the like. The potential barrier is determined by n-type implantation. The sensitivity of the pixel Pix may be changed by changing the depletion region of the photoelectric conversion unit 440. This change also changes the conversion efficiency of photoelectric conversion in the photoelectric conversion unit 440, making it possible to achieve both high and low sensitivity.

[0381] In the electronic domain 440, the sensitivity of the pixel Pix may be changed by controlling the driving of the photoelectric conversion unit 440. For example, it is possible to control the photoelectric conversion unit 440 so that the charge accumulated by the photoelectric conversion unit 440 during the exposure period is discarded during the charge accumulation period.

[0382] 45 is a schematic diagram showing an example of a configuration applicable to the second embodiment for discarding the charge accumulated in the photoelectric conversion unit 440 during an accumulation period. As shown in Fig. 45, an OFG (overflow gate) transistor 1026 is connected to a photodiode 1021 serving as the photoelectric conversion unit 440. The ON / OFF of the OFG transistor 1026 is controlled by a signal OFG supplied to the gate.

[0383] When the transfer transistor 1022 is turned on at a predetermined timing in response to a transfer signal TRG supplied from the vertical scanning unit 1012, the charge accumulated in the photodiode 1021 is transferred to the DF. During a predetermined period from when charge accumulation in the photoelectric conversion unit 440 begins due to exposure to when the transfer transistor 1022 is turned on, the OFG transistor 1026 is turned on by a signal OFG. As a result, the charge accumulated in the photodiode 1021 until the OFG transistor 1026 is turned on is released to the OGD signal line and discarded.

[0384] This change reduces the amount of charge transferred from the photodiode 1021 to the FD, making it possible to lower the sensitivity.

[0385] (4-3. Backward Domain) Next, sensitivity variation control in the backward domain 460 according to the second embodiment will be described.

[0386] In the subsequent domain 460, sensitivity variation control may be achieved by control related to the floating diffusion region FD. Fig. 46 is a schematic diagram for explaining sensitivity variation control related to the floating diffusion region FD in the subsequent domain 460 according to the second embodiment. Fig. 46 schematically shows the floating diffusion region FD in Fig. 5 described above.

[0387] 46, the charge accumulated by the photoelectric conversion unit 440 is transferred to and stored in the floating diffusion region FD. The charge is read out from the floating diffusion region FD and converted into a voltage. The voltage into which the charge is converted in the FD is supplied to an amplifier (Amp) 4400, amplified to a required voltage, and output. The amplifier 4400 corresponds to, for example, the amplifying transistor 1024 in FIG. 5.

[0388] The sensitivity of the pixel Pix may be changed by changing the capacitance of the floating diffusion region FD. The capacitance of the FD can be changed by changing the area of ​​the wiring that constitutes the FD, the pn junction capacitance, the wiring capacitance, etc. For example, the capacitance of the floating diffusion region FD can be changed by changing the implant that becomes the pn junction capacitance.

[0389] In the subsequent domain 460, the sensitivity of the pixel Pix may be changed by changing the source follower gain of the transistor (amplifying transistor 1024) of the amplifier 4400, the thickness and dielectric constant of the oxide film forming the transistor, and the gate capacitance and size of the transistor. The characteristics of the transistor can be changed by changing the source well concentration of the transistor and the material used to form the oxide film.

[0390] By changing these elements, it is possible to change the charge-voltage conversion efficiency in the floating diffusion region FD. Note that there is a trade-off between the saturation capacitance in the floating diffusion region FD and the quantization error in the subsequent circuit.

[0391] Furthermore, in the subsequent domain 460, sensitivity variation control may be realized by controlling A / D conversion processing that converts analog pixel signals output from the amplifier 4400 into digital pixel signals, for example.

[0392] As an example, consider an A / D conversion circuit that converts an analog pixel signal into a digital pixel signal with a bit depth of 10 bits. Generally, an A / D conversion circuit converts an input analog signal into a digital signal by comparing it with a reference voltage. For example, the A / D conversion circuit converts the input analog signal into a digital signal by comparing it with each voltage obtained by dividing the reference voltage according to the grayscale.

[0393] In an A / D conversion circuit configured as described above, the apparent gain of the output signal can be controlled by changing the reference voltage. More specifically, when an A / D conversion circuit converts an input analog signal into a digital signal with a bit depth of 10 bits, the input analog signal is converted into a digital signal with 1024 gradations having gradation values ​​of 0 to 1023. In this case, the output range when converting the analog signal into a digital signal can be set by the set value of the full-range output voltage, thereby determining the sensitivity of the A / D conversion. For example, when the reference voltage corresponding to the full range is set to 1 V and 5 V, the 5 V setting appears to have five times the gain of the 1 V setting, and therefore can be said to have five times the sensitivity.

[0394] A specific control example will be described using the imaging unit 1031 shown in Fig. 4 as an example. For example, the reference signal generation unit 1014 controls the ramp signal RAMP supplied to each AD converter 1300 for each scan by each pixel signal line 1016 and each vertical signal line 1017, based on the ADC control signal supplied from the imaging operation control unit 1019. By controlling the ramp signal RAMP in this way, it is possible to control sensitivity variations for each pixel Pix.

[0395] As described above, the subsequent domain 460 may include an A / D conversion circuit that performs the A / D conversion process, either as a component included in the pixel Pix or as a component external to the pixel Pix. For example, in the example of FIG. 4 , each AD converter 1300 is a column AD converter connected to each vertical signal line 1017, and the A / D conversion circuit is shown as a component external to the pixel Pix. However, the present invention is not limited to this, and a pixel ADC (Analog to Digital Converter) configuration in which each pixel Pix has its own AD converter may also be used.

[0396] (4-4. Specific Example of Sensitivity Variation Generation) Next, sensitivity variation in pixel units in the sensitivity variation expanding sensor 300 according to the second embodiment will be described in more detail.

[0397] In the second embodiment, the sensitivity variation in pixel units is set by, for example, assigning an index to each pixel (pixel position) and associating a sensitivity ratio with each index. The association between pixels and sensitivity ratios is not limited to the method using an index, and sensitivity ratios may be directly assigned to pixels (pixel positions), for example.

[0398] (First Example of Sensitivity Variation Pattern) A first example of a sensitivity variation pattern according to the second embodiment will be described. Figures 47A and 47B are schematic diagrams for explaining the first example of the sensitivity variation pattern according to the second embodiment.

[0399] In the example of Figure 47A, 16 indices are assigned to a block array consisting of 16 pixels (4 pixels horizontally x 4 pixels vertically) (called a 4x4 block) using an arrangement method called the Bayer type.

[0400] Bayer type 2 n ×2 n When n=1, the matrix is ​​defined by the following equation (1).

[0401]

[0402] 2 where all values ​​are 1 n ×2 n If the square matrix is ​​expressed as the following equation (2), then n ×2n Matrix D2 n is recursively defined by equation (3).

[0403]

[0404]

[0405] When n=4, the matrix D4 is expressed as in the following equation (4): The matrix D4 corresponds to the array shown in FIG.

[0406]

[0407] The association between pixel positions and indices is not limited to this example.

[0408] Fig. 47B shows a schematic example of the association of sensitivity ratios with the indices assigned in Fig. 47A. In Fig. 47B, in section (a) and section (b), the horizontal axis indicates the index and the vertical axis indicates the sensitivity ratio.

[0409] In the example of section (a), a sensitivity ratio expressed linearly is associated with each index. For example, a sensitivity ratio of "1.0" is assigned to index "0", and a sensitivity ratio of "0.5" is assigned to intermediate index "8". In the example of section (b), a sensitivity ratio expressed logarithmically is associated with each index. For example, a sensitivity ratio of "1.0" is assigned to index "0", and a sensitivity ratio of "0.1" is assigned to intermediate index "8".

[0410] In this way, the sensitivity variation pattern constitutes a variation correction map 310 .

[0411] Note that the examples in sections (a) and (b) of FIG. 47B are merely examples and are not limited to these. For example, the vertical axis may be a scale other than linear or logarithmic. Also, in the example of section (a), the interval between the sensitivity ratios is "0.1," but the interval between the sensitivity ratios may be a value other than this.

[0412] (Second Example of Sensitivity Variation Pattern) Next, a second example of the sensitivity variation pattern according to the second embodiment will be described. Figures 48A to 48C are schematic diagrams for explaining the second example of the sensitivity variation pattern according to the second embodiment.

[0413] In a second example of the sensitivity variation pattern, the sensitivity variation is set based on a Bayer array, which is an example of a color filter array. The Bayer array is generally an array in which R (red) pixels, G (green) pixels, and B (blue) pixels are arranged in a 2x2 block in a ratio of 1:2:1 such that pixels of the same color are not adjacent to each other. As an example, in a Bayer array of 2x2 blocks, the upper left and lower right of the 2x2 block are G pixels, the upper right is an R pixel, and the lower left is a B pixel.

[0414] In a second example of a sensitivity variation pattern, an index is set for each 2 × 2 block color filter array. Here, the 2 × 2 block color filter array uses the Bayer array described above, in which the upper left and lower right are G pixels, the upper right is R pixels, and the lower left is B pixels.

[0415] 48A to 48C, and later-described FIGS. 50A, 50C, and 50D, as well as FIG. 51, G pixels are indicated as Pix(G), R pixels as Pix(R), and B pixels as Pix(B).

[0416] 48A, an index is assigned to each 2×2 block in which color filters are arranged in a Bayer array. In the example shown, of the 2×2 blocks obtained by dividing a 4×4 block into four, the index “0” is assigned to the four pixels in the upper left 2×2 block, the index “1” is assigned to the four pixels in the lower right 2×2 block, the index “2” is assigned to the four pixels in the upper right 2×2 block, and the index “3” is assigned to the four pixels in the lower left 2×2 block.

[0417] In the example of Figure 48B, an array of 8 pixels horizontally by 8 pixels vertically is divided into four horizontally and vertically, resulting in 16 2x2 blocks, each of which is assigned indexes "0" to "15", in the same way as in the case of the pixel-by-pixel array described using Figure 47A.

[0418] Fig. 48C shows an example in which indexes are not assigned in units of a 2 × 2 block color filter array, in which a Bayer array of 2 × 2 blocks is applied to each of the 4 × 4 blocks to which indexes are assigned as shown in Fig. 47A, divided into four 2 × 2 blocks.

[0419] (Third Example of Sensitivity Variation Pattern) Next, a third example of the sensitivity variation pattern according to the second embodiment will be described. Figures 49A and 49B are schematic diagrams for explaining the third example of the sensitivity variation pattern according to the second embodiment. The third example of the sensitivity variation pattern is an example in which the scale of sensitivity variation is made different for each exposure surface.

[0420] Fig. 49A shows an example in which the sensitivity range is different for each exposure plane. In Fig. 49A, characteristic line 800 shows the correspondence between index and sensitivity ratio for the first exposure plane, and characteristic line 801 shows the correspondence between index and sensitivity ratio for the second exposure plane. In the example of Fig. 49A, the slope of characteristic line 801 is smaller than the slope of characteristic line 800, and it can be seen that the interval between the index and the sensitivity ratio for the second exposure plane is narrower than that for the first exposure plane, and that in the range of indexes "0" to "15," the sensitivity ratio for the second exposure plane is set in a narrower range than that for the first exposure plane.

[0421] Figure 49B shows an example in which the scale is different for each exposure plane. In Figure 49B, section (a) shows the correspondence between the index and the sensitivity ratio for the first exposure plane. In the example of section (a), the sensitivity ratio expressed linearly is associated with the index. In Figure 49B, section (b) shows the correspondence between the index and the sensitivity ratio for the second exposure plane. In the example of section (b), the sensitivity ratio expressed logarithmically is associated with the index.

[0422] (Fourth Example of Sensitivity Variation Pattern) Next, a fourth example of the sensitivity variation pattern according to the second embodiment will be described. Figures 50A to 50D are schematic diagrams for explaining the fourth example of the sensitivity variation pattern according to the second embodiment. The fourth example of the sensitivity variation pattern is an example in which the sensitivity variation pattern is different for each exposure surface.

[0423] In Fig. 50A, section (a) shows an example of a sensitivity variation pattern of the first exposure surface. As shown in section (a), on the first exposure surface, as in Fig. 48A described above, an index is assigned to each 2x2 block in which color filters are arranged in a Bayer array. Specifically, in the example of the sensitivity variation pattern of the first exposure surface shown in section (a), of the 2x2 blocks obtained by dividing a 4x4 block into four, an index "0" is assigned to four pixels in the upper left 2x2 block, an index "1" is assigned to four pixels in the lower right 2x2 block, an index "2" is assigned to four pixels in the upper right 2x2 block, and an index "3" is assigned to four pixels in the lower left 2x2 block.

[0424] In Fig. 50A, section (b) shows an example of a sensitivity variation pattern of the second exposure plane. In this example, the rows and columns of the 2x2 blocks of the first exposure plane shown in section (a) are swapped on the second exposure plane. That is, in the example of the sensitivity variation pattern of the second exposure plane shown in section (b), index "3" is assigned to four pixels in the upper left 2x2 block, index "2" is assigned to four pixels in the lower right 2x2 block, index "1" is assigned to four pixels in the upper right 2x2 block, and index "0" is assigned to four pixels in the lower left 2x2 block.

[0425] In this fourth example, as shown in Fig. 50B, the association between the index and the sensitivity ratio is further changed for each exposure plane. That is, for the first exposure plane, as shown by characteristic line 804, on the linear scale, the index "0" is associated with a sensitivity ratio of "1" and the index "3" is associated with a sensitivity ratio of "0.5." On the other hand, for the second exposure plane, as shown by characteristic line 805, on the linear scale, the index "0" is associated with a sensitivity ratio of "2.0" and the index "3" is associated with a sensitivity ratio of "1.0."

[0426] Fig. 50C shows the ratio of the sensitivity ratio of a pixel at the same position on the second exposure plane shown in section (b) to the sensitivity ratio on the first exposure plane shown in section (a) of Fig. 50A. Also, Fig. 50D shows the ratio of the sensitivity ratio of a pixel at the same position on the second exposure plane to the sensitivity ratio on the first exposure plane when the sensitivity variation patterns are the same on the first and second exposure planes.

[0427] In the example of Figure 50D, the ratio is the same for all pixels in the 4x4 block, making the sensitivity ratio spatially uniform.In contrast, in the example of Figure 50C, the sensitivity ratio is 1.0 times in the 2x2 block in the upper left of the 4x4 block, 2.5 times in the 2x2 block in the upper right, 4.0 times in the 2x2 block in the lower left, and 1.6 times in the 2x2 block in the lower right, making the sensitivity ratio spatially non-uniform.Therefore, in the example of Figure 50C, it is possible to create areas where the sensitivity variation correction unit 301 is good at correction processing and areas where it is not good at it.

[0428] (Fifth Example of Sensitivity Variation Pattern) Next, a fifth example of the sensitivity variation pattern according to the second embodiment will be described. The fifth example of the sensitivity variation pattern is an example in which the sensitivity variation pattern is a pattern different from the Bayer array.

[0429] For example, indexes sequentially associated with sensitivity ratios may be consecutively arranged for each pixel. Specifically, it is conceivable to consecutively arrange the indexes for the pixels in the entire pixel array unit 1011 or for each block by raster scanning from the top left. Also, for example, the indexes sequentially associated with sensitivity ratios may be irregularly arranged. In this case, the indexes may be irregularly arranged within a block, such as a 4x4 block, and the block may be repeated, or the indexes may be irregularly arranged for all pixels in the pixel array unit 1011.

[0430] Furthermore, for example, the index arrangement in the Bayer array may be modified. For example, as described with reference to FIG. 50A , the index arrangement of rows and columns of blocks of a predetermined size on the first exposure surface may be interchanged to form the index arrangement on the second exposure surface.

[0431] The present invention is not limited to this, and a rotation operation may be performed on an index arrangement in a block of a predetermined size, for example. Fig. 51 is a schematic diagram for explaining a rotation operation on an index arrangement according to the second embodiment.

[0432] First, consider the index arrangement in a 4×4 block shown on the left side of Fig. 51. This index arrangement is the same as the arrangement shown in Fig. 48C above, so a detailed description will be omitted.

[0433] In this index arrangement, a 4x4 block is divided into four to generate four 2x2 blocks, and the index arrangement is rotated in each of these four 2x2 blocks.As an example, as shown in the center of Fig. 51, in the 2x2 block at the top left of the 4x4 block, the index arrangement is left as is, and in the 2x2 block at the top right, the index arrangement is rotated 90° clockwise.Similarly, in the 2x2 block at the bottom right, the index arrangement is rotated 180° clockwise, and in the 2x2 block at the bottom left, the index arrangement is rotated 270° clockwise.

[0434] Such block-by-block matrix permutation and block-by-block rotation of the index arrangement can be realized using functions, etc. Furthermore, the index arrangement itself can also be realized using functions, etc., if it is a regular arrangement.

[0435] (4-5. Method of Retaining and Outputting Sensitivity Variation Information) Next, a method of retaining and outputting the auxiliary information 411 according to the second embodiment will be described.

[0436] Fig. 52 is a schematic diagram for explaining a method for holding and outputting auxiliary information 411 according to the second embodiment. Fig. 52 shows an excerpt of a portion closely related to a method for holding and outputting sensitivity variation information from the hardware configuration of the sensitivity variation expansion sensor according to the second embodiment shown in Fig. 42.

[0437] 52, the auxiliary information 411 includes the variation correction map 310, the sensitivity ratio map, the saturation region map, and the saturation threshold map 320. Of these, the sensitivity ratio map may be information included in the variation correction map 310. Each of these maps is related to the above-mentioned sensitivity variation pattern, and the auxiliary information 411 is information generated for each sensitivity variation pattern.

[0438] The auxiliary information 411 is written to a memory or the like when the sensitivity variation expanding sensor 300 is manufactured, for example.

[0439] (First Example of Method for Storing Auxiliary Information) A first example of a method for storing auxiliary information 411 according to the second embodiment will be described. In the first example of the method for storing auxiliary information 411, all of the map information included in the auxiliary information 411 is stored by the auxiliary information storage unit 410. The auxiliary information storage unit 410 may store the auxiliary information 411 in the memory 1043 in the example of Fig. 3, or in the storage device 2004 in the example of Fig. 7 or 8.

[0440] The auxiliary information storage unit 410 stores the auxiliary information 411 by the method of this first example when the sensitivity variation pattern is set irregularly for all pixels of the pixel array unit 1011. In this case, according to this first example, it is necessary to store information for all pixels, which requires a large-capacity memory for storage.

[0441] On the other hand, if the sensitivity variation pattern is composed of a repeated pattern, the auxiliary information storage unit 410 may store only one cycle of the repeated sensitivity variation pattern as auxiliary information 411. Furthermore, if the sensitivity variation pattern can be generated by a function or the like, the auxiliary information storage unit 410 may store only parameters of the function as auxiliary information 411.

[0442] (Second Example of Method for Holding Auxiliary Information) A second example of a method for holding the auxiliary information 411 according to the second embodiment will be described.

[0443] There may be a plurality of sensitivity variation patterns applicable to the sensitivity variation expansion sensor 300. For example, there may be cases where several types of sensitivity variation patterns are prepared in advance and, during manufacturing, the sensitivity variation patterns are assigned to each sensitivity variation expansion sensor 300. In the second example of holding the auxiliary information 411 according to the second embodiment, when the variation in sensitivity variation patterns is limited as described above, identification information (ID) that identifies each of the auxiliary information 411 corresponding to each sensitivity variation pattern is assigned, and only the ID is stored on the sensitivity variation expansion sensor 300 side.

[0444] 53 is a schematic diagram for explaining a second example of a method for storing auxiliary information 411 according to the second embodiment. As shown in FIG. 53, auxiliary information 411, 411, ..., 411 assigned to the sensitivity variation expanding sensor 300 is stored in the memory 360 of the sensitivity variation expanding sensor 300. N 3, a register included in the register group 1037 may be used as the memory 360. The memory 1043 may also be used as the memory 360.

[0445] The host-side storage unit 240 stores auxiliary information 4111, 4112, ..., 4113, which is actual data of each sensitivity variation pattern #1, #2, ..., #N. N are stored in association with the IDs. In the configurations of Fig. 7 and Fig. 8, the storage device 2004 or the like may be applied as the host-side storage unit 240. In the example of Fig. 3, the memory 1043 may be applied as the host-side storage unit 240.

[0446] For example, when the sensitivity variation expanding sensor 300 is started up, the signal processing unit 110 (or the signal processing device 111) reads auxiliary information 411 1 , 411 2 , . . . , 411 N The signal processing unit 110 reads out auxiliary information 411 associated with the ID read out from the memory 360 from the host-side storage unit 240. The signal processing unit 110 uses the auxiliary information 411 read out from the host-side storage unit 240 to execute sensitivity variation correction processing and saturated region correction processing.

[0447] (Third Example of Method for Holding Auxiliary Information) Next, a third example of a method for holding the auxiliary information 411 according to the second embodiment will be described. The third example of a method for holding the auxiliary information 411 according to the second embodiment is an example in which the sensitivity variation expansion sensor 300 physically holds the auxiliary information 411.

[0448] One example of physically storing the auxiliary information 411 is to use dummy pixels in the pixel array unit 1011. The dummy pixels may be, for example, pixels in an invalid pixel area in the pixel array unit 1011 (for example, optical black pixels).

[0449] More specifically, in the third example of the method for storing the auxiliary information 411, the sensitivity variation pattern is acquired by actually measuring the pixel values ​​of the dummy pixels. In this case, it can be said that the dummy pixels themselves store the auxiliary information 411.

[0450] 54 is a schematic diagram illustrating a third example of a method for retaining auxiliary information 411 according to the second embodiment. In FIG. 54 , among the pixels Pix included in the pixel array unit 1011, for example, pixels Pix included in a predetermined region on the periphery of the pixel array unit 1011 may be used as dummy pixels. The dummy pixels are, for example, light-shielded and used to extract black-level pixel signals. When sensitivity variations that cause black level variations are set for the pixels Pix, the pixel signals output from these dummy pixels can be used as sensitivity variation information.

[0451] In this case, for example, if the sensitivity variation pattern is a repeat of the sensitivity variation pattern within a 4 × 4 block, and if, for example, four rows at the top or bottom end or four columns at the left or right end of the pixel array unit 1011 are set as dummy pixels, the sensitivity variation pattern can be obtained within the dummy pixels.

[0452] As an example, the signal processing unit 110 may read the pixel values ​​of the dummy pixels when the sensitivity variation expansion sensor 300 is started, obtain a sensitivity variation pattern based on the read pixel values, and generate the auxiliary information 411 .

[0453] (Fourth Example of Method for Holding Auxiliary Information) Next, a fourth example of a method for holding the auxiliary information 411 according to the second embodiment will be described. Similar to the third example described above, the fourth example of the method for holding the auxiliary information 411 according to the second embodiment is an example in which the sensitivity variation expansion sensor 300 physically holds the auxiliary information 411.

[0454] In this fourth example, dummy information is given to dummy pixels, such as optical black pixels, in the pixel array section 1011 to obtain a sensitivity variation pattern.

[0455] FIG. 55 is a schematic diagram for explaining a fourth example of a method for holding auxiliary information 411 according to the second embodiment.

[0456] As shown in section (a) of FIG. 55, the pixel region 250 in the pixel array section 1011 has the sensitivity variation patterns of patterns #1 and #2. 10 and 250 11 In each of the above, a predetermined region at the top end is set as a dummy region 251 in which dummy pixels are arranged.

[0457] During normal operation of the pixel Pix in the dummy region 251, the photodiode 1021 accumulates charge corresponding to incident photons. The charge accumulated in the photodiode 1021 is transferred to the floating diffusion region FD at a predetermined timing. The charge transferred to and accumulated in the floating diffusion region FD is converted into a voltage when read out from the floating diffusion region FD and supplied to the AD converter 1300. The AD converter 1300 converts the supplied voltage into a digital value and outputs it.

[0458] In a fourth example of a method for retaining the auxiliary information 411, the auxiliary information 411 is acquired by shielding the photodiode (PD) 1021 from light and then injecting a predetermined amount of charge into the photodiode (PD) 1021 and the floating diffusion region FD. A predetermined voltage is also applied to the AD converter 1300. In this state, the digital value output from the AD converter 1300 is measured.

[0459] By performing the above-described operation of acquiring the auxiliary information 411 for each pixel in the dummy area 251, it is possible to acquire a sensitivity variation pattern for each pixel in the dummy area 251. When patterns #1 and #2 are each configured by a repeated pattern of pixels in the dummy area 251, the operation of acquiring the auxiliary information 411 can acquire the sensitivity variation pattern for each pixel in the pixel area 250. 10 and 250 11 Each auxiliary information 411 can be obtained.

[0460] (First Example of Output Method of Auxiliary Information) Next, a first example of an output method of the auxiliary information 411 according to the second embodiment will be described. In the first example of the output method of the auxiliary information 411, the auxiliary information 411 is output when the first frame is output, for example, when the sensitivity variation expansion sensor 300 is started up.

[0461] Fig. 56 is a schematic diagram showing an example of the configuration of frame image data applicable to the second embodiment and output from the sensitivity variation expansion sensor 300. In the example of Fig. 56, the frame image data 252 is transferred, for example, by serial communication, with line data transferred in the horizontal direction from left to right in the figure, and then repeatedly transferred in the vertical direction from top to bottom.

[0462] 56, a header section in which header information is stored is arranged at the beginning of one frame of frame image data 252, followed by a data section in which image data, etc. is stored. In the data section, predetermined areas at both ends of each line and areas formed by predetermined lines at the beginning and end are invalid areas in which no image data is stored, and the area excluding the invalid areas is a valid area in which image data is stored.

[0463] The auxiliary information output unit 412 may write the auxiliary information 411 read by the auxiliary information holding unit 410 into the header section or invalid area and output it as auxiliary information output 413. Because the auxiliary information 411 is expected not to change in the short term, the auxiliary information output 413 may be stored and output only in the frame image data 252 of the initial frame output when the sensitivity variation expansion sensor 300 is started up.

[0464] (Second Example of Auxiliary Information Output Method) Next, a second example of the auxiliary information 411 output method according to the second embodiment will be described. In the second example of the auxiliary information 411 output method, the auxiliary information output unit 412 stores the auxiliary information 411 in the frame image data 252 for each frame output from the sensitivity variation expansion sensor 300, for example, and outputs the stored auxiliary information as auxiliary information output 413. However, the auxiliary information output unit 412 may output the auxiliary information output 413 every few frames.

[0465] (Third Example of Output Method of Auxiliary Information) Next, a third example of the output method of the auxiliary information 411 according to the second embodiment will be described. The third example of the output method of the auxiliary information 411 is an example in which the auxiliary information 411 is output in response to an external request.

[0466] Fig. 57 is a schematic diagram for explaining a third example of the output method of auxiliary information 411 according to the second embodiment. In Fig. 57, the host device 200 is a device that performs predetermined processing on image data output from the sensitivity variation expansion sensor 300, and may include a signal processing unit 110 and a signal processing device 111.

[0467] 57, the sensitivity variation expansion sensor 300 transmits an image output based on a captured image to the host device 200 via a transmission path 253. The interface of the transmission path 253 is not particularly limited, but a Mobile Industry Processor Interface (MIPI) can be applied.

[0468] Furthermore, control signals are input and output between the sensitivity variation expansion sensor 300 and the host device 200 via a transmission path 254 that is a predetermined communication interface such as a serial bus.

[0469] For example, in the sensitivity variation expansion sensor 300, the auxiliary information output unit 412 transmits auxiliary information output 413 to the host device 200 in response to a request transmitted from the host device 200 via the transmission path 254. The auxiliary information output unit 412 transmits the auxiliary information output 413 to the host device 200 via the transmission path 254. The auxiliary information output unit 412 may store the auxiliary information output 413 in frame image data 252 and transmit it to the host device 200 via the transmission path 253, for example, as described using FIG.

[0470] The timing at which the auxiliary information output unit 412 outputs the auxiliary information 413 is not limited to the timing in response to a request from the host device 200. For example, the auxiliary information output unit 412 may transmit the auxiliary information output 413 to the host device 200 at a timing at which the sensitivity variation in the sensitivity variation expansion sensor 300 is controlled.

[0471] In the first to third examples of the auxiliary information output method described above, it is assumed that the auxiliary information 411 is stored uniformly, but this is not limited to this example. For example, when storing the auxiliary information 411, the auxiliary information storage unit 410 may weight the auxiliary information 411 according to its importance in the recognition process. The importance in the recognition process may be a value according to, for example, the position of the pixel corresponding to the auxiliary information 411 in the image, the sensitivity of the pixel, whether the pixel is a saturated pixel, or the like. Furthermore, the weighting may be, for example, by allocating more bits to information with higher importance, deleting redundant information, or treating information with the same importance if the difference in importance is equal to or less than a predetermined value.

[0472] 5. Third Embodiment Next, a third embodiment of the present disclosure will be described. The third embodiment is an example in which security-related processing is performed using the output of the sensitivity variation expansion sensor 300.

[0473] In recent years, the performance of camera modules used in surveillance cameras, in-vehicle cameras, and the like has increased, increasing the possibility that information that can identify individuals may be captured even in applications that do not require personal information. For example, an application that applies emergency braking based on images captured by an in-vehicle camera does not require the identification of personal information. However, the captured image may include the license plates of preceding or surrounding vehicles, or the faces of occupants. Similarly, when recording captured images, even if personal information is not required, if the captured image contains information that can identify individuals, the information will be recorded.

[0474] Furthermore, images captured by in-vehicle cameras may be used for critical applications that may affect human lives, such as emergency braking control. In in-vehicle cameras used for such purposes, vehicle control may be attacked by maliciously replacing the camera module. For example, if a camera module is maliciously replaced, emergency braking information may not be output even when necessary, causing the vehicle to run out of control.

[0475] Therefore, there is a demand for anonymity that conceals information that identifies individuals from images captured by camera modules used in surveillance cameras, in-vehicle cameras, etc. There is also a demand for a security mechanism that can easily detect the replacement of camera modules.

[0476] In the third embodiment of the present disclosure, a method for realizing anonymity and security using an image captured by the sensitivity variation expanding sensor 300 is proposed.

[0477] Fig. 58 is a schematic diagram showing an example configuration of an imaging system according to the third embodiment. In Fig. 58, the imaging system 1i includes a camera module 390, a recognition processing unit 371, an auxiliary information holding unit 372, and a result verification unit 373.

[0478] The camera module 390 includes a sensitivity variation expansion sensor 300. Note that, hereinafter, the sensitivity variation expansion sensor 300 may be referred to as a sensitivity variation expansion sensor #A in order to identify the individual sensitivity variation expansion sensor 300. The sensitivity variation expansion sensor #A has a sensitivity variation #A as a sensitivity variation pattern. The sensitivity variation expansion sensor #A supplies a sensitivity variation image 600 obtained by capturing an image of a subject 620 to the recognition processing unit 371.

[0479] Fig. 59 is a schematic diagram illustrating a sensitivity variation image 600 according to the third embodiment. In Fig. 59 , section (a) schematically illustrates a captured image 611 of a subject 620 captured by a general camera without sensitivity variation magnification. In the captured image 611, the subject 620 is clearly recognizable. Section (b) schematically illustrates an example of a sensitivity variation image 600, which is an image of the same subject 620 captured by the sensitivity variation magnification sensor 300 according to the present disclosure. As described above, in the sensitivity variation image 600, the sensitivity variation in pixel units due to the sensitivity variation magnification sensor 300 appears as noise, making it difficult to visually recognize the subject 620.

[0480] The recognition processing unit 371 includes the functions of the signal processing unit 110 described above, has a pre-trained recognizer, and includes the functions of the recognition unit 120. The recognition processing unit 371 performs sensitivity variation correction processing and saturated region correction processing on the sensitivity variation image 600 supplied from the camera module 390 using auxiliary information #A corresponding to the sensitivity variation expanded sensor #A stored in the auxiliary information storage unit 372, thereby generating a saturated region corrected image. Note that the auxiliary information storage unit 372 corresponds to the auxiliary information storage unit 410 described above.

[0481] The recognition processing unit 371 further performs recognition processing on the saturated region corrected image using a recognizer. The recognition processing unit 371 outputs a recognition result 610 by the recognizer. In the example shown in the figure, the recognition processing unit 371 outputs the recognition result 610 as an image in which the subject 620 is visible. The output of the recognition result 610 by the recognition processing unit 371 is not limited to an image, and may be, for example, meta information related to the subject 620.

[0482] The result verification unit 373 verifies the recognition result 610 based on the recognition result 610 output from the recognition processing unit 371 and the auxiliary information #A stored in the auxiliary information storage unit 372 .

[0483] (5-1. Improvement of Anonymity According to the Third Embodiment) As a first example of the third embodiment, improvement of anonymity will be described.

[0484] In a first example of the third embodiment, a recognizer included in the recognition processing unit 371 is trained to improve the anonymity of a captured image. Briefly, in the first example, the recognizer is trained by a recognizer training unit using a sensitivity variation image 600, which includes sensitivity variations on a pixel-by-pixel basis due to the sensitivity variation expansion sensor 300, as training data. Furthermore, another recognizer is trained using anonymized information, which is information to be anonymized, and the training result is inverted and provided to a recognition unit trained using the sensitivity variation image 600 as training data. Using such a mechanism makes it possible to recognize a subject from the sensitivity variation image 600 while suppressing recognition of anonymized information contained in the subject, etc.

[0485] Fig. 60 is a schematic diagram showing the configuration of an example of a learning system according to a first example of the third embodiment. In Fig. 60, the learning system includes a variance optimization unit 700, a recognizer training unit 710, an anonymization training unit 720, and a dataset 730. The recognizer training unit 710 also includes two networks 711 and 712. After being trained, the networks 711 and 712 are applied to a recognizer included in the recognition processing unit 371.

[0486] The anonymization learning unit 720 includes a GRL (Gradient Reversal Layer) 721 and a network 722 .

[0487] The variation optimization unit 700 generates parameters for optimizing the sensitivity variation in pixel units in the sensitivity variation expansion sensor 300 based on the recognition results of the recognizer trained by the recognizer training unit 710 .

[0488] The dataset 730 for training the recognizer training unit 710 and the anonymization training unit 720 includes an image 731, a recognition result 732, and anonymized information 733. Of these, the image 731 is a sensitivity variation image captured by the target sensitivity variation expanding sensor 300 (e.g., sensitivity variation expanding sensor #A).

[0489] The anonymized information is information that should not be recognized when a recognition process is performed on the sensitivity variation image output from the sensitivity variation expansion sensor 300 by the recognizer trained by the recognizer training unit 710. The anonymized information may be the license plate of a vehicle or information that can identify an individual (face, gender, etc.).

[0490] 61 is a schematic diagram for explaining the basic learning process of the recognizer learning unit 710 according to the third embodiment. The recognizer learning unit 710 basically follows the direction of the arrow in the figure, uses an image 731 as learning data, and a recognition result 732 as evaluation data to learn networks 711 and 712, and uses the recognition result 732 as feedback to the recognizer learning unit 710 in the direction opposite to the arrow in the figure by the error backpropagation method, thereby updating the networks 711 and 712.

[0491] FIG. 62 is a schematic diagram for explaining an example of the learning process of the recognizer learning unit 710 according to the third embodiment.

[0492] 62, networks 711 and 712 of the recognizer training unit 710 may be fixed in a trained state. The anonymization training unit 720 acquires the output of the network 711 and performs recognition processing of the anonymized information 733 using the network 722 based on the acquired output. At this time, if the processing is proceeding in the direction of the arrow in the figure, the GRL 721 does nothing.

[0493] The anonymization learning unit 720 uses backpropagation to feed back anonymized information 733 to the network 722 in the direction opposite to the arrow in the figure. The anonymization learning unit 720 trains the network 722 to recognize the anonymized information 733 based on the fed-back anonymized information 733, and inputs the learning result to the GRL 721. In the processing opposite to the arrow in the figure, the GRL 721 inverts the input learning result and outputs it. By inverting the learning result in the GRL 721, an output is obtained that suppresses recognition of the anonymized information 733.

[0494] The output of the GRL 721 is traced in the opposite direction to the arrow in the figure and input to the optimization unit 700' via the network 711. The optimization unit 700' executes an optimization process in accordance with the output of the network 711 to output a recognition result 732 while preventing the output of anonymized information. Based on the results of the optimization process, the optimization unit 700' generates parameters for controlling the sensitivity variation per pixel of the sensitivity variation expansion sensor 300. The parameters generated by the optimization unit 700' are, for example, a sensitivity ratio map. The parameters generated by the optimization unit 700' may be reflected in, for example, the design of the sensitivity variation expansion sensor 300.

[0495] As described above, in the first example of the third embodiment, the learning result of learning the anonymized information 733 is inverted by the GRL 721 and fed back to the network 711 that performs recognition processing on the image 731 output from the sensitivity variation expanding sensor 300. Then, based on the output of the network 711 in response to this feedback, parameters for optimizing the sensitivity variation in pixel units of the sensitivity variation expanding sensor 300 are generated.

[0496] Therefore, the sensitivity variation expansion sensor 300 to which this parameter is applied can output a sensitivity variation image that can suppress the recognition of anonymized information while still recognizing the target.

[0497] (5-2. Fingerprinting According to the Third Embodiment) As a second example of the third embodiment, fingerprinting using a sensitivity variation image will be described.

[0498] Fingerprinting (device fingerprinting) refers to a technology for identifying a device using device-specific attributes. In a second example of the third embodiment, fingerprinting is performed using sensitivity variation information of the sensitivity variation expanded sensor 300.

[0499] Fig. 63 is a schematic diagram for explaining fingerprinting according to a second example of the third embodiment. In Fig. 63, section (a) shows an example (normal state) in which a subject 620 is imaged by a valid camera module 390 (referred to as camera module #A), and section (b) shows an example (abnormal state) in which a subject 620 is imaged by an unauthorized camera module 390 (referred to as camera module #B).

[0500] 63, the camera module #A includes a sensitivity variation expansion sensor 300 (referred to as sensitivity variation expansion sensor #A) having a sensitivity variation #A as a sensitivity variation pattern. The auxiliary information storage unit 372 stores auxiliary information #A corresponding to the sensitivity variation #A.

[0501] Camera module #A captures an image of subject 620 using sensitivity variation expansion sensor #A and outputs, as the captured image, a sensitivity variation image 600 reflecting sensitivity variation #A. Recognition processing unit 371 performs variation correction processing and saturated region correction processing on sensitivity variation image 600 output from sensitivity variation expansion sensor #A using auxiliary information #A stored in auxiliary information storage unit 372, executes recognition processing on the saturated region corrected image, and outputs recognition result 610.

[0502] In the case of section (a), the recognition processing unit 371 performs each complementation using auxiliary information #A corresponding to sensitivity variation #A on the sensitivity variation image 600 reflecting sensitivity variation #A, and then performs recognition processing. Therefore, the recognition processing unit 371 can output the recognition result 610 as an image that is easily visible.

[0503] The result verification unit 373 finds the correlation between the recognition result 610 output from the recognition processing unit 371 and the auxiliary information #A stored in the auxiliary information storage unit 372. For example, if the value of the correlation coefficient indicating the found correlation is within a predetermined range, the result verification unit 373 determines that the camera module #A is authentic, that is, that the camera module #A has sensitivity variation #A corresponding to the auxiliary information #A.

[0504] On the other hand, in the configuration of section (b) in Fig. 63 , an unauthorized camera module #B different from camera module #A is used as camera module 390. In camera module #B, sensitivity variation expanding sensor 300 (referred to as sensitivity variation expanding sensor #B) has sensitivity variation #B as the sensitivity variation pattern.

[0505] Camera module #B captures an image of subject 620 using sensitivity variation expansion sensor #B and outputs, as the captured image, a sensitivity variation image 600′ reflecting sensitivity variation #B. Recognition processing unit 371 performs variation correction processing and saturated region correction processing on sensitivity variation image 600′ output from sensitivity variation expansion sensor #B using auxiliary information #A stored in auxiliary information storage unit 372, executes recognition processing on the saturated region corrected image, and outputs recognition result 612.

[0506] In the case of section (b), the recognition processing unit 371 performs each complementation using auxiliary information #A corresponding to sensitivity variation #A on the sensitivity variation image 600' reflecting sensitivity variation #B, and then performs recognition processing. Therefore, the recognition processing unit 371 outputs the recognition result 612 as an image that is difficult to see.

[0507] The result verification unit 373 determines the correlation between the recognition result 612 output from the recognition processing unit 371 and the auxiliary information #A stored in the auxiliary information storage unit 372. In the example of section (b), the sensitivity variation expansion sensor #B in the camera module #B has, as a sensitivity variation pattern, a sensitivity variation #B that does not correspond to the auxiliary information #A stored in the auxiliary information storage unit 372. Therefore, the value of the correlation coefficient indicating the correlation between the recognition result 612 and the auxiliary information #A is outside the predetermined range. Therefore, the result verification unit 373 determines that the camera module #B is fraudulent.

[0508] As described above, according to the second example of the third embodiment, a correlation is found between the sensitivity variation image 600, which is an image captured by the sensitivity variation expanding sensor 300 having a specific sensitivity variation, and the auxiliary information corresponding to the specific sensitivity variation, thereby determining whether the sensitivity variation expanding sensor 300 is genuine. Therefore, by using the imaging system 1i according to the second example of the third embodiment, it is possible to determine whether the camera module 390 included in the imaging system 1i is genuine.

[0509] The method for determining whether the camera module 390 is authentic is not limited to the method using the correlation described above.

[0510] For example, in a saturated region corrected image obtained by performing variation correction processing and saturated region correction processing on a sensitivity variation image output from camera module 390, the imaging setup, such as the angle of view of camera module 390, is generally fixed. Therefore, for example, by learning a recognizer included in recognition processing unit 371, it may be possible to distinguish between a state in which the angle of view has been restored in the saturated region corrected image (normal) and a state in which the saturated region corrected image outputs noise due to sensitivity variation, saturated regions, or the like (abnormal).

[0511] Furthermore, for example, if camera module 390 is replaced by a malicious third party, the positional relationship of an object whose position is fixed relative to camera module 390 and is included in an image captured by camera module 390 may be different from that before the replacement. Therefore, object detection may be performed in advance on the image captured by camera module 390, and the position of the detected object may be stored. Then, object detection may be performed on the image captured by camera module 390 by recognition processing unit 371, and whether camera module 390 is genuine may be determined based on the positional relationship between the position of the detected object and the position of the stored object.

[0512] Furthermore, the fingerprinting according to the second example of the third embodiment can also be used for general purposes.

[0513] For example, the result verification unit 373 may verify the recognition result 610 using as a key an object that is always included within the imaging range of the sensitivity variation expansion sensor 300. If the sensitivity variation expansion sensor 300 is mounted on a vehicle and its imaging range is in the front direction, an example of an object that is always included within the imaging range of the sensitivity variation expansion sensor 300 may be an emblem attached to the tip of the vehicle hood.

[0514] Furthermore, for example, if the sensitivity variation per pixel of the sensitivity variation expansion sensor 300 is caused not by the pixel structure but by electronic or downstream circuits, the pixel variation pattern may be verified from the output pattern of dummy pixels such as optical black pixels.

[0515] Furthermore, in the above description, verification is performed based on the correlation between the recognition result and the auxiliary information, but this is not limited to this example. For example, there is a certain correlation between the sensitivity variation image 600 input to the recognition processing unit 371 and the auxiliary information stored in the auxiliary information storage unit 372. Therefore, the sensitivity variation image 600 input to the recognition processing unit 371 may be verified using this correlation.

[0516] It should be noted that there are two possible methods for providing a sensitivity variation pattern: one is to provide a sensitivity variation pattern for each model of the sensitivity variation expanding sensor 300, and the other is to provide a sensitivity variation pattern for each individual sensor. Of these, in the method for providing a sensitivity variation pattern for each model of the sensitivity variation expanding sensor 300, the sensitivity variations #A and #B are treated like secret keys in a public key cryptosystem. On the other hand, in the method for providing a sensitivity variation pattern for each individual sensor 300, the sensitivity variations #A and #B are treated like public keys in a public key cryptosystem.

[0517] (5-3. Measures for False Verification Due to Aging Deterioration) Next, measures for false verification due to aging deterioration in the sensitivity variation expansion sensor 300 according to the third embodiment will be described.

[0518] The characteristics of some materials used in the sensitivity variation expanding sensor 300 change over time. The sensitivity variation expanding sensor 300, whose materials have changed over time, may have a different degree of sensitivity variation from that at the time of initial manufacture, and if the verification described with reference to Fig. 63 is performed based on the sensitivity variation image output from such a sensitivity variation expanding sensor 300, there is a risk of erroneous verification.

[0519] In the third embodiment, the following two types of measures are proposed to deal with such erroneous verification: (1) Ensuring by using other elements that have not changed over time, and (2) Estimating and correcting for changes over time.

[0520] First, the method (1) of ensuring the quality by using other elements that do not change over time will be described. Here, the explanation will be given using a color filter as an example of a component that changes over time. Figure 64 is a schematic diagram for explaining how to deal with the change over time of a color filter, which can be applied to the third embodiment.

[0521] In Figure 64, section (a) shows the change in pixel sensitivity due to aging of the color filter. Depending on the color and material, color filters may become discolored over time. Pixels equipped with a discolored color filter have a higher sensitivity than their initial sensitivity. In this case, the difference between the sensitivity after aging and the initial sensitivity is the degree of degradation, but the specific degree of degradation is unknown.

[0522] The change in sensitivity due to aging can be known by referring to the sensitivity of non-deteriorated pixels that do not experience sensitivity degradation due to aging. Section (b) of Figure 64 shows an example of predicting the change in sensitivity due to aging of a bleached color filter using non-deteriorated pixels that do not experience sensitivity degradation due to aging. As for non-deteriorated pixels, in the case of optical filters, pixels equipped with a colorless clear filter (called clear pixels) can be considered.

[0523] As shown in section (b) of Figure 64, the difference between the sensitivity of the clear pixel and the sensitivity of the pixel that has changed due to aging is calculated. This difference can be estimated as the amount of deterioration in the sensitivity of the pixel that has changed due to aging, relative to the initial sensitivity. For example, the recognition processing unit 371 may correct the auxiliary information (sensitivity variation map, saturation region map, etc.) stored in the auxiliary information storage unit 372 based on this estimated amount of deterioration.

[0524] Alternatively, the sensitivity of pixels whose sensitivity has changed due to aging may be corrected by performing calibration using a test chart, etc. Also, dummy pixels may be used instead of the above-described clear pixels.

[0525] Furthermore, the sensitivity change of the pixel due to aging may be estimated using a model to complement the auxiliary information. Fig. 65 is a schematic diagram showing an example of a pixel sensitivity change model applicable to the third embodiment.

[0526] 65 , the horizontal axis represents time and the vertical axis represents sensitivity. A characteristic line 810 shows an example of a change in sensitivity over time based on a sensitivity change model. For example, the imaging system 1i creates and stores in advance a sensitivity change model shown by the characteristic line 810. For example, the recognition processing unit 371 may refer to the stored sensitivity change model in accordance with the cumulative operation time of the sensitivity variation expansion sensor 300, and correct the auxiliary information stored in the auxiliary information storage unit 372 when there has been a change in sensitivity of a predetermined amount or more since the previous correction timing.

[0527] In the imaging system 1i according to the third embodiment, the sensitivity change due to aging of pixels is corrected in this manner, and therefore it is possible to suppress false detections due to aging of the materials used in the sensitivity variation expansion sensor 300.

[0528] (6. Fourth Embodiment) Next, a fourth embodiment of the present disclosure will be described. In the above description, attention is focused on sensitivity as a characteristic of each pixel of a sensor, and by varying the sensitivity for each pixel, the dynamic range of a captured image is expanded, making it possible to obtain a more accurate recognition result based on the captured image. The method for obtaining a more accurate recognition result based on a captured image is not limited to the method of varying the sensitivity for each pixel. In the fourth embodiment, the above-described method for obtaining a more accurate recognition result based on a captured image is given a general concept, and an example applicable to the present disclosure is proposed based on this general concept.

[0529] (6-1. Regarding Superordinate Conception According to the Fourth Embodiment) The superordinate conception of the method for obtaining a more accurate recognition result based on a captured image according to the fourth embodiment will be described.

[0530] (6-1-1. Generic Conceptual Correction Processing According to Fourth Embodiment) Fig. 66 is a schematic diagram illustrating a generic conceptual processing for obtaining a more accurate recognition result based on a captured image according to the fourth embodiment. The variation expansion sensor 3000 is a sensor in which pixel characteristics are made different on a pixel-by-pixel basis. In the variation expansion sensor 3000, the characteristics of at least one pixel among the multiple pixels included in the variation expansion sensor 3000 may be made different. There are several possible targets for characteristic variation in the variation expansion sensor 3000, but these will be described in detail below.

[0531] The first correction process 3100 is a process for correcting the characteristics of each pixel in the captured image output from the variation expansion sensor 3000, based on the variation characteristics of each pixel of the variation expansion sensor 3000. The second correction process 3200 corrects the characteristics of a target pixel for the output by the first correction process 3100, using information on surrounding pixels around the target pixel.

[0532] In this way, in the fourth embodiment, as a higher-level concept, correction is performed for each pixel by the first correction process 3100, and correction is performed on the result of the first correction process 3100 by the second correction process 3200 using information on surrounding pixels.

[0533] Fig. 67 is a functional block diagram for more specifically explaining the processing according to the fourth embodiment. In Fig. 67, a first correction processing 3100 includes processing by a characteristic correction unit 3101 and processing by a characteristic conversion unit 3102. Furthermore, a second correction processing 3200 includes processing by a correction area calculation unit 3201 and processing by a wide-area correction unit 3202.

[0534] The captured image output from the variation expansion sensor 3000 is input to the characteristic correction unit 3101 as an input image 3300. The characteristic conversion unit 3102 converts characteristic information 3103 for each pixel of the variation expansion sensor 3000, which has been acquired in advance, in accordance with the correction processing executed by the characteristic correction unit 3101, and passes the converted information to the characteristic correction unit 3101. The characteristic correction unit 3101 performs correction processing for each pixel of the input image 3300 based on the converted characteristic information 3103 passed from the characteristic conversion unit 3102. However, the characteristic correction unit 3101 may also perform correction processing for each pixel group of several to several tens of pixels on the input image 3300.

[0535] The characteristic correction unit 3101 outputs the image obtained by correcting the input image 3300 as an intermediate image 3301 to the second correction processing 3200. In the second correction processing 3200, the intermediate image 3301 is passed to a correction area calculation unit 3201 and a wide area correction unit 3202.

[0536] The correction area calculation unit 3201 calculates a correction area for performing correction on the intermediate image 3301 based on the intermediate image 3301. For example, if the variation expansion sensor 3000 has sensitivity variation, a saturated area in the intermediate image 3301 may be calculated as the correction area. The wide-area correction unit 3202 performs wide-area correction (called wide-area correction) on the correction area calculated by the correction area calculation unit 3201. Here, a wide area refers to an area that exceeds a single pixel or a pixel group. For example, the wide-area correction unit 3202 performs wide-area correction on the correction area using information on pixels or areas surrounding the correction area.

[0537] The wide-area correction unit 3202 performs wide-area correction on the intermediate image 3301 and outputs the result as an output image 3302 .

[0538] FIG. 68 is a flowchart illustrating an example of the first correction process 3100 according to the fourth embodiment.

[0539] In step S10, the characteristic correction unit 3101 acquires an input image 3300. In the next step S11, the characteristic conversion unit 3102 acquires characteristic information 3103 indicating the characteristics of each pixel of the variation expansion sensor 3000. The characteristic information 3103 may be acquired in advance and stored in a storage unit (not shown). In the next step S12, the characteristic conversion unit 3102 converts the characteristic information 3103 acquired in step S11 in accordance with the correction process executed by the characteristic correction unit 3101. In the next step S13, the characteristic correction unit 3101 corrects the input image 3300 acquired in step S10 for each pixel or for each pixel group based on the characteristic information 3103 converted in step S12. In the next step S14, the characteristic correction unit 3101 outputs an intermediate image 3301 obtained by correcting the input image 3300.

[0540] FIG. 69 is a flowchart illustrating an example of second correction processing 3200 according to the fourth embodiment.

[0541] In step S20, the correction area calculation unit 3201 and the wide area correction unit 3202 acquire an intermediate image 3301 obtained by correcting the input image 3300 through the first correction process 3100. In the next step S21, the correction area calculation unit 3201 calculates a correction area based on the intermediate image 3301 acquired in step S20. In the next step S22, the wide area correction unit 3202 performs wide area correction on the intermediate image 3301 acquired in step S20 based on the correction area calculated by the correction area calculation unit 3201 in step S21. In the next step S23, the wide area correction unit 3202 performs output processing to output the image obtained by performing wide area correction on the intermediate image 3301 as an output image 3302.

[0542] (6-1-2. Other Configurations for Correction Processing According to the Fourth Embodiment) Here, variations in the configuration for correction processing according to the fourth embodiment will be described.

[0543] FIG. 70 is a schematic diagram for explaining variations in the execution order of the first correction process 3100 and the second correction process 3200 according to the fourth embodiment.

[0544] 70 , section (a) is an example similar to that of FIG. 66 described above, in which a first correction process 3100 is performed on the output of the variation expansion sensor 3000, and a second correction process 3200 is performed on the correction result of the first correction process 3100. Section (b) of FIG. 70 is an example in which the first correction process 3100 and the second correction process 3200 are performed in parallel. For example, when the first correction process 3100 and the second correction process 3200 are realized using a DNN that has been machine-trained to perform both the first correction process 3100 and the second correction process 3200 on the correction result of the first correction process 3100, the configuration of section (b) can be adopted.

[0545] Section (c) of Fig. 70 is an example in which the second correction process 3200 is first performed on the output of the variation expansion sensor 3000, and then the first correction process 3100 is performed on the correction result of the second correction process 3200. Section (d) of Fig. 70 is an example in which a correction process 3300, which is different from the first correction process 3100 and the second correction process 3200, is performed on the output of the variation expansion sensor 3000. In this case, the first correction process 3100 is performed on the correction result of the correction process 3300, and then the second correction process 3200 is performed on the correction result of the first correction process 3100. The correction process 3300 may be, for example, a preparatory process for the first correction process 3100.

[0546] Fig. 71 is a schematic diagram illustrating variations in output due to correction processing according to the fourth embodiment. Note that in sections (a) to (c) of Fig. 71, a second correction processing 3200 is performed on the correction results of a first correction processing 3100, as in the example of section (a) of Fig. 70.

[0547] 71 , section (a) shows an example of a configuration in which a recognition result 3401 is output based on a captured image output from the variation magnification sensor 3000. In this example, a recognition process is performed on the correction result of the second correction process 3200 using a machine learning recognizer 3400 that has been trained in advance. The machine learning recognizer 3400 performs a recognition process on the correction result of the second correction process 3200, and outputs the recognition result 3401.

[0548] 71 , section (b) shows an example of a configuration for outputting an image 3411 based on a captured image output from the variation magnification sensor 3000. In this example, a predetermined signal processing (image processing) is performed on the correction result of the second correction process 3200 using a machine learning signal processor 3410 that has been trained in advance, and an image 3411 is output. The signal processing (image processing) performed by the machine learning signal processor 3410 is not particularly limited, and may be high image quality processing, high resolution processing, etc.

[0549] 71 , section (c) shows another example of a configuration for outputting an image 3411 based on a captured image output from the variation magnification sensor 3000. In this example, the correction result of the second correction process 3200 is directly output as an image 3421.

[0550] The configurations in sections (a) to (c) of FIG. 71 may be combined with the configurations shown in sections (a) to (d) of FIG. 70 described above.

[0551] (6-2. Structural example of sensor applicable to fourth embodiment) Next, a structural example of the variation magnification sensor 3000 applicable to the fourth embodiment will be described. Fig. 72 is a schematic diagram showing a structural example of the variation magnification sensor 3000 applicable to the fourth embodiment. In the example of Fig. 72, the structure of the variation magnification sensor 3000 is shown as a cross section of a plane parallel to the incident light.

[0552] 72, a color filter 3005 is provided on the light receiving surface side of a pixel array section 3006 (corresponding to the pixel array section 1011 in FIG. 4), and a condenser lens unit 3004 using an OCL (On Chip Lens) is provided on the incident side of the color filter 3005. The pixel array section 3006, color filter 3005, and condenser lens unit 3004 are generally formed integrally.

[0553] An optical low-pass filter 3003 and a UV (Ultraviolet) / IR (Infrared) cut filter 3002 are provided on the incident side of the condenser lens unit 3004. An imaging lens 3001 is provided on the incident side of the UV / IR cut filter 3002. The imaging lens 3001 is composed of, for example, a single lens or a combination of multiple lenses.

[0554] Light (incident light) incident on the variation magnification sensor 3000 passes through an imaging lens 3001, a UV / IR cut filter 3002, and an optical low-pass filter 3003 and enters a condenser lens unit 3004. The condenser lens unit 3004 has, for example, microlenses corresponding to each pixel Pix included in the pixel array section 3006, and guides the incident light to each pixel Pix via a color filter 3005. The color filter 3005 may have, for example, R, G, and B colors arranged in a Bayer array.

[0555] 43 , the pixel Pix includes a light-collecting layer 421, a semiconductor layer 422, and a wiring layer 423, and the light-collecting layer 421 includes an on-chip lens 430, an optical filter 431, a light-shielding film 432, and an insulating film 433. Hereinafter, the on-chip lens 430 will be described as an OCL 430 where appropriate.

[0556] The OCL 430 corresponds to the OCL for each pixel in the condenser lens unit 3004 in Fig. 72. The optical filter 431 corresponds to, for example, the color filter 3005. The light-shielding film 432 prevents incident light from leaking into an adjacent pixel Pix. The light incident on the pixel Pix is ​​guided to the semiconductor layer 422 from the opening of the light-shielding film 432.

[0557] (6-3. First Example of Fourth Embodiment) Next, a first example of the fourth embodiment will be described. The first example of the fourth embodiment is an example in which the characteristics of each pixel of the variation expansion sensor 3000 are varied by varying the sensitivity of the pixels. That is, the variation expansion sensor 3000 according to the first example of the fourth embodiment corresponds to the sensitivity variation expansion sensor 300 according to the first embodiment described above.

[0558] Fig. 73 is a schematic diagram showing an example of the correspondence between the configuration according to the first embodiment shown in Fig. 10 and the configuration of a higher-level concept according to the fourth embodiment. In Fig. 73, the variation magnification sensor 3000a has characteristic variations for each pixel by varying the sensitivity of each pixel.

[0559] The first correction process 3100a includes a process in which the sensitivity variation correction unit 301 uses the variation correction map 310 to even out the variations for each pixel in the sensitivity variation image output from the variation expansion sensor 3000a. More specifically, the first correction process 3100a evens out the variations for each pixel in the sensitivity variation image by the sensitivity variation correction unit 301 dividing the output level of each pixel in the sensitivity variation image by the sensitivity variation ratio of that pixel in the variation correction map 310.

[0560] As described with reference to FIG. 14 , the variation correction map 310 may be acquired in advance by calibration using a test chart 70 or the like. The acquired variation correction map 310 may be stored in the variation expansion sensor 3000a using, for example, dummy pixels (see FIG. 54 ), or may be stored in the host-side storage unit 240 (see FIG. 53 ). An example in which the variation correction map 310 is acquired in advance and stored in the host-side storage unit 240 has the advantage of facilitating processing by the sensitivity variation correction unit 301. Furthermore, an example in which the variation correction map 310 is acquired in advance and stored in the variation expansion sensor 3000a has the advantage of facilitating processing by the sensitivity variation correction unit 301 to some extent and eliminating the need for support for host firmware.

[0561] 15 , in the first correction process 3100a, the sensitivity variation correction unit 301 performs sensitivity variation estimation process 72′ to perform online estimation in the time series direction while capturing images using the variation magnification sensor 3000a, thereby generating the variation correction map 310. The example in which the variation correction map 310 is obtained online has the advantages of eliminating the need for prior calibration and reducing the data size of the variation correction map 310.

[0562] The sensitivity variation corrected image output as the correction result from the sensitivity variation correction unit 301 by the first correction process 3100a may contain saturated pixels. Therefore, there is a risk that the sensitivity variation image cannot be restored simply by dividing it by the sensitivity variation ratio of the variation correction map 310.

[0563] Therefore, in the second correction process 3200a, the saturated region estimation unit 303 uses the saturation threshold map 320 to estimate saturated pixels in the sensitivity variation image output from the variation spread sensor 3000a, and generates a saturated region map based on the estimation result. The saturated region correction unit 302 uses the saturated region map to perform saturation correction on the sensitivity variation corrected image, which is the correction result of the first correction process 3100a.

[0564] Here, in the second correction process 3200a, as explained using Figures 22A and 22B, wide-area correction is performed on saturated pixels 53 in the sensitivity variation correction image using non-saturated pixels surrounding the saturated pixels 53, and the pixel value of the saturated pixels 53 is corrected.

[0565] Furthermore, in the second correction process 3200a, correction may be performed using information correlated with noise, as described with reference to Fig. 23. That is, as shown in Fig. 23, in the second correction process 3200a, the saturated region correction unit 302 may correct saturated pixels in the saturated region by further using a sensitivity ratio map (variation correction map 310) as information correlated with noise, in addition to the sensitivity variation corrected image and the saturated region map.

[0566] However, the second correction process 3200a may correct saturated pixels in the saturated region using other information correlated with noise. That is, as described with reference to FIG. 18 , pixels with low sensitivity have a poor S / N ratio, while pixels with high sensitivity have a good S / N ratio. Therefore, in the second correction process 3200a, the saturated region correction unit 302 may weight pixels with a good S / N ratio based on the sensitivity map, perform predetermined signal processing, and then perform saturated pixel correction processing. In this case, the predetermined signal processing may be noise reduction processing or other high-image-quality signal processing.

[0567] (6-4. Second Example of Fourth Embodiment) Next, a second example of the fourth embodiment will be described. The second example of the fourth embodiment is an example in which the inter-pixel distance is varied in the variation magnification sensor 3000.

[0568] (6-4-1. Overview of a Second Example of the Fourth Embodiment) For example, for a pixel group consisting of multiple pixels Pix that share an optical structure, the amount of light incident on pixels Pix at different positions within the pixel group varies depending on the angle of incidence on the pixel group. Parallax can be calculated based on this change in light amount, and information on the depth direction can be obtained using this parallax. By varying the distance between pixels, a wide parallax range can be accommodated, making it possible to obtain depth information with higher accuracy.

[0569] For example, the plurality of pixels Pix may share an optical structure by sharing the OCL 430. Fig. 74 is a schematic diagram showing an example of a structure in which one OCL 430 is shared by a plurality of pixels Pix, which is applicable to the second example of the fourth embodiment.

[0570] In FIG. 74 and subsequent similar figures, the pixel Pix is ​​shown as a cross-sectional view taken along a plane parallel to the incident light.

[0571] In Figure 74, section (a) shows an example in which one OCL 430a is provided for each pixel Pix. In each pixel Pix, incident light is incident on the semiconductor layer 422 through the OCL 430a and the opening Apt in the light-shielding film 432. On the other hand, section (b) of Figure 74 shows an example in which two adjacent pixels Pix form a pixel group PixGr, and one OCL 430b is provided for each pixel group PixGr. That is, in section (b), two adjacent pixels Pix share one OCL 430b.

[0572] FIG. 75 is a schematic diagram showing an example of light incident on a pixel group PixGr in which two pixels Pix share one optical control lens (OCL) 430b. Light from a light source (not shown) enters the photographing lens 3001 and exits, for example, from point 4000 on the photographing lens 3001. The light exiting the photographing lens 3001 follows different optical paths 4001a and 4001b depending on the distance from the light source to the photographing lens 3001, and is incident on the left and right pixels Pix of the pixel group PixGr via the optical control lens 430b. In this example, the inclination of optical path 4000b is smaller than the inclination of optical path 4000a, so the amount of light incident on the right pixel Pix is ​​greater than the amount of light incident on the left pixel Pix. This difference in light intensity at each pixel Pix in the pixel group PixGr corresponds to the parallax of each pixel Pix relative to the light source, and depth direction information can be obtained based on this parallax.

[0573] In such a configuration, the distance between pixels determines the parallax range. That is, a longer distance between pixels allows for a larger parallax range, while a shorter distance between pixels improves accuracy for smaller parallax ranges. Therefore, varying the distance between pixels allows for a wider parallax range.

[0574] (6-4-2. Processing in Second Example of Fourth Embodiment) Fig. 76 is a schematic diagram for explaining processing according to the second example of the fourth embodiment. In Fig. 76, the variation magnification sensor 3000b has characteristics in which the inter-pixel distances are varied, as described above. The first correction processing 3100b calculates the parallax for each pixel group PixGr using the characteristics of the variation magnification sensor 3000b based on the captured image output from the variation magnification sensor 3000b by a parallax calculation processing 3110, and outputs a parallax image based on the calculated parallax.

[0575] In the second correction process 3200b, a parallax interpolation process 3210 interpolates the parallax image output from the first correction process 3100b using the parallax of each pixel group PixGr, thereby calculating a more accurate parallax image. For example, in the second correction process 3200b, the parallax interpolation process 3210 weights each calculated parallax based on the parallax image in accordance with the inter-pixel distance. The parallax interpolation process 3210 applies a weight in accordance with the inter-pixel distance to the parallax around the pixel of interest, correcting the parallax of the pixel of interest, thereby performing wide-area correction on the pixel of interest and calculating a more accurate parallax image.

[0576] The second correction process 3200b outputs a high-precision disparity image calculated based on the disparity information of each pixel Pix corrected by the wide-area correction. By using this high-precision disparity image, it is possible to obtain more accurate depth information.

[0577] (6-4-3. Example of Sensor Implementation in Second Example of Fourth Embodiment) An example of implementation of sensor characteristic variation in the variation magnifying sensor 3000b according to the second example of the fourth embodiment will be described with reference to FIGS.

[0578] 77 to 80, the upper diagram of each section shows a cross-sectional view of each pixel Pix in a plane parallel to the incident light, and the lower diagram shows a view of each pixel Pix as seen from the incident surface side. The arrangement of pixels Pix or the size of pixels Pix is ​​shown as "number of pixels in the horizontal direction x number of pixels in the vertical direction," with the size of one pixel Pix as a unit.

[0579] FIG. 77 is a schematic diagram showing an example of changing the number of pixels Pix that share the OCL 430 according to a second example of the fourth embodiment.

[0580] 77, section (a) shows an example in which a pixel group PixGr(1) consisting of three pixels Pix arranged in a 3×1 matrix shares one OCL 430c, and section (b) shows an example in which a pixel group PixGr(2) consisting of four pixels Pix arranged in a 2×2 matrix shares one OCL 430d.

[0581] Note that examples of pixel groups PixGr that share one OCL 430 are not limited to these. For example, a pixel group PixGr consisting of two pixels Pix arranged in a 2×1 matrix may share one OCL 430. Also, for example, a pixel group PixGr consisting of nine pixels Pix arranged in a 3×3 matrix may share one OCL, or a pixel group PixGr consisting of a larger number of pixels Pix or a pixel group PixGr arranged in a different matrix may share one OCL.

[0582] 78 is a schematic diagram showing an example of changing the light collection range of a pixel Pix according to a second example of the fourth embodiment. In Fig. 78, section (a) shows an example in which a pixel group PixGr(3) consisting of two pixels Pix arranged in a 2x1 matrix shares one OCL 430b. In this case, each pixel Pix in the pixel group PixGr(3) has a light collection range equivalent to two pixels.

[0583] 78, in section (b), a pixel group PixGr(4) consisting of two pixels Pix arranged in a 2 × 1 matrix and two pixels Pix, each 0.5 pixels in size, on either side of the two pixels Pix shares one OCL 430e whose size corresponds to an array of pixels Pix of 3 × 1. Furthermore, pixel groups PixGr(5) on both sides of pixel group PixGr(4) each include one pixel Pix and a pixel Pix of 0.5 pixels in size, and share an OCL 430f whose size is 1.5 pixels.

[0584] In the example of section (b), each of the two central pixels Pix in the pixel group PixGr(4) has a light-collecting range of three pixels. In contrast, the pixel Pix at the right end of the pixel group PixGr(4), for example, shares the OCL 430f with the pixel group PixGr(5). By providing a light-shielding film 432 on the shared portion of the OCL 430f in the pixel Pix at the right end of the pixel group PixGr(4), for example, leakage of light incident on the OCL 430f into the pixel group PixGr(4) and leakage of light incident on the OCL 430e into the pixel group PixGr(5) are prevented. This limits the angle of incidence of light incident on the pixel group PixGr(4) and the pixel group PixGr(5). Therefore, it is possible to make the amount of light incident on pixel Pix of pixel group PixGr(4) via OCL 430e from point 4000 (see FIG. 74 ) of the photographing lens 3001 different from the amount of light incident on pixel Pix of pixel group PixGr(5) via OCL 430f from point 4000. The same is true for pixel Pix at the left end of pixel group PixGr(4).

[0585] Fig. 79 is a schematic diagram showing an example of changing the size of the OCL 430 according to a second example of the fourth embodiment. Section (a) of Fig. 79 shows an example in which one OCL 430b is shared by two pixels Pix arranged in a 2x1 matrix and included in a pixel group PixGr(6). In this case, the OCL 430b has a size corresponding to the two pixels Pix.

[0586] Section (b) of Figure 79 shows an example in which a single pixel Pix and a pixel group PixGr(7) including two pixels Pix are mixed. In this case, an OCL 430a having a size corresponding to one pixel Pix is ​​provided for the single pixel Pix, and an OCL 430b having a size corresponding to two pixels Pix is ​​provided for the pixel group PixGr(7). Furthermore, in the example of section (b), an OCL 430a having a circular shape corresponding to a 1x1 size, for example, and an OCL 430b having a different shape from the OCL 430a, i.e., an elliptical shape corresponding to a 2x1 size, are mixed.

[0587] Section (c) of Figure 79 shows an example in which two individual pixels Pix and a pixel Pix-L having a size of 2x2 are mixed together. In this case, it corresponds to the individual pixel Pix. An OCL 430a having a size of 1x1 and an OCL 430g having a size corresponding to the pixel Pix-L having a size of 2x2 are mixed together.

[0588] FIG. 80 is a schematic diagram showing an example of changing the light-shielding film 432 according to a second example of the fourth embodiment. In section (a) of FIG. 80 , a 1×1 OCL 430a is provided for each 1×1 pixel Pix, and the positions of the openings formed by the light-shielding film 432 are different. In this example, the two central pixels Pix out of the four pixels Pix have openings formed by the light-shielding film 432 in the center of the pixel Pix, and the right half of the pixel Pix on the left edge is an opening. The left half of the pixel Pix on the right edge is an opening. By varying the positions of the openings between adjacent pixels Pix, it is possible to change the amount of incident light for each pixel Pix.

[0589] Section (b) of Figure 80 shows an example in which pixels Pix-L of 2x2 size and pixels Pix-S of 1 / 2x1 / 2 size are mixed. Note that in section (b), the light-shielding film 432 in pixels Pix-S is omitted. Furthermore, four pixels Pix-S in a 2x2 array constitute a pixel group PixGr(8).

[0590] In section (b), two pixels Pix-L are arranged adjacent to each other, with the left half of the left pixel Pix-L being an opening and the right half of the right pixel Pix-L being an opening. In this way, by mixing pixels Pix of different sizes and further biasing the light incident on each pixel Pix using the opening, it is possible to change the amount of light incident on each pixel Pix.

[0591] (Application of Image Plane Phase Difference Method to AF) Image plane phase difference method AF (autofocus) control is known, which measures distance based on the phase difference of light received by a plurality of pixels and executes focus control in accordance with the distance measurement result. In image plane phase difference method AF control, two pixels are paired, one pixel having a light-shielding portion that shields, for example, the left half of the light-receiving surface of the pixel, and the other pixel having a light-shielding portion that shields the right half of the light-receiving surface of the pixel, and distance measurement is performed based on the phase difference of light received by these two pixels. This image plane phase difference method AF control can be realized by using a modification of the light-shielding film 432 according to the second example of the fourth embodiment.

[0592] (6-5. Third Example of Fourth Embodiment) Next, a third example of the fourth embodiment will be described. The third example of the fourth embodiment is an example in which the variation expansion sensor 3000 is configured to extract a portion of the characteristics of light, and the degree of extraction is spatially changed. More specifically, the third example of the fourth embodiment is an example in which a polarizer that polarizes incident light is provided to the variation expansion sensor 3000, and polarization characteristics are varied for each pixel Pix.

[0593] (6-5-1. Overview of the Third Example of the Fourth Embodiment) A wire-grid polarizer is known as an example of a polarizer that can be used in the third example of the fourth embodiment. Fig. 81 is an enlarged schematic diagram of a polarizing filter 3070 having a wire-grid polarizer.

[0594] The polarizing filter 3070 shown in Fig. 81 includes four polarizers 3007a, 3007b, 3007c, and 3007d, each with a different polarization direction. Polarizer 3007a has a grid arranged vertically in the figure, while polarizer 3007d has a grid arranged horizontally in the figure. Polarizer 3007b has a grid arranged from the upper left to the lower right (referred to as right diagonal), while polarizer 3007c has a grid arranged from the upper right to the lower left (referred to as left diagonal).

[0595] In each of the polarizers 3007a to 3007d, the direction along the grid is the transmission axis, and the direction perpendicular to the grid is the absorption axis. When circularly polarized light is incident on the polarizing filter 3070, the incident light is extracted as polarized light polarized in the direction of the transmission axis, i.e., the grid direction, in each of the polarizers 3007a to 3007d.

[0596] Fig. 82 is a schematic diagram showing a structural example of a variation magnification sensor 3000c applicable to the fourth embodiment. In the example of Fig. 82, similar to Fig. 72 described above, the structure of the variation magnification sensor 3000 is shown as a cross section of a plane parallel to the incident light.

[0597] 82, the variation spread sensor 3000c is provided with a polarizing filter 3070 instead of the color filter 3005 of the variation spread sensor 3000 described with reference to FIG. 72. In other words, the variation spread sensor 3000c is configured as a polarization sensor that polarizes incident light in a specific polarization direction and receives the light. Also, as with the example of FIG. 72, the pixel array section 3006, polarizing filter 3070, and condenser lens unit 3004 are generally formed integrally. Furthermore, although not shown, each of the polarizers 3007a to 3007d of the polarizing filter 3070 is provided for one pixel Pix.

[0598] In the variation expansion sensor 3000c applicable to the fourth embodiment, the polarization filter 3070 may be provided, and a color filter 3005 (not shown) may also be provided.

[0599] The extinction ratio is an index that indicates the polarization characteristics of the polarizers 3007a to 3007d. The extinction ratio is expressed as the ratio between the transmission axis sensitivity and the absorption axis sensitivity, as shown in the following equation (5). The larger the extinction ratio, the better the characteristics of the polarizer. Extinction ratio = transmission axis sensitivity / absorption axis sensitivity (5)

[0600] Generally, a polarizer has a structure that selectively transmits light of a specific polarization direction, so a pixel provided with a polarizer receives less light than a normal pixel without a polarizer. In other words, in principle, there is a trade-off between the extinction ratio and the amount of received light.

[0601] (6-5-2. Example of Sensor Implementation in the Third Example of the Fourth Embodiment) FIG. 83 is a schematic diagram showing the configuration of an example of a polarizing filter 3070 applicable to the third example of the fourth embodiment.

[0602] 83, as shown on the left side, polarizer group 3007H includes polarizers 3007Ha, 3007Hb, 3007Hc, and 3007Hd, each having an extinction ratio higher than the first threshold and with a grid oriented vertically, diagonally right, diagonally left, and horizontally in the figure. A pixel group PixGrH (not shown) is configured by four pixels Pix, each of which is provided with a polarizer 3007Ha to Hd of polarizer group 3007H.

[0603] The polarizer group 3007L includes polarizers 3007La, 3007Lb, 3007Lc, and 3007Ld, each having an extinction ratio lower than a second threshold that is lower than the first threshold, and whose grids are oriented vertically, diagonally right, diagonally left, and horizontally in the drawing. A pixel group PixGrL (not shown) is configured by four pixels Pix, each of which is provided with one of the polarizers 3007La to 3007Ld of the polarizer group 3007L.

[0604] 83, the polarizing filter 3070 has a checkerboard arrangement of polarizer groups 3007H and 3007L, resulting in a mixture of pixels with high extinction ratios and pixels with low extinction ratios. The first correction process 3100 corrects the polarization information for each pixel Pix, and the wide-area correction of the second correction process 3200 corrects the polarization information using information on pixels surrounding the pixel of interest, making it possible to obtain highly accurate modification information.

[0605] (6-5-3. Processing in the third example of the fourth embodiment) Fig. 84 is a schematic diagram for explaining processing according to the third example of the fourth embodiment. In Fig. 84, the variation expansion sensor 3000c has characteristics in which the polarization characteristics vary for each pixel Pix, as described above.

[0606] In the first correction process 3100c, a polarization information calculation process 3120 calculates polarization information for each pixel group PixGrH and PixGrL using the characteristics of each polarizer based on the captured image output from the variation expansion sensor 3000c, and outputs a polarization information image based on the calculated polarization information. The polarization information may be the degree of polarization or the polarization direction.

[0607] In the second correction process 3200c, a polarization information correction process 3220 performs wide-area correction using the polarization information image output by the first correction process 3100c to calculate a polarization information image with a high S / N ratio. More specifically, in the polarization information correction process 3220, the polarization information of the pixel of interest is corrected based on the polarization information and extinction ratio of the surrounding area of ​​the pixel of interest.

[0608] At this time, if the pixel of interest receives a sufficient amount of light, the correction is performed by weighting the polarization information of pixels Pix with high extinction ratios and low light amounts around the pixel of interest, whereas if the pixel of interest does not receive a sufficient amount of light, the correction is performed by weighting the polarization information of pixels Pix with low extinction ratios and high light amounts around the pixel of interest.

[0609] Whether the variation spreading sensor 3000c is receiving a sufficient amount of light may be determined, for example, by whether the average value of the amount of light incident on the light receiving surface of the variation spreading sensor 3000c exceeds a threshold value. Also, whether the extinction ratio is high or low may be determined by whether the extinction ratio value exceeds a threshold value.

[0610] In this way, by selecting a pixel Pix to be weighted according to the light intensity of the pixel of interest, a high S / N polarization information image with a higher S / N ratio can be calculated. The calculated high S / N polarization information image is then passed to, for example, subsequent signal processing.

[0611] (6-5-4. Another example of sensor implementation in the third example of the fourth embodiment) In the above-mentioned Figure 83, the extinction ratio is spatially modulated in units of, for example, a 2 pixel x 2 pixel polarizer group, but the implementation example of the sensor characteristic variation in the variation magnification sensor 3000b according to the third example of the fourth embodiment is not limited to this example.

[0612] For example, the extinction ratio may be spatially modulated within a polarizer group. Fig. 85 is a schematic diagram showing an example of a polarizing filter 3070' in which the extinction ratio is spatially modulated within the polarizer group, which is applicable to the third example of the fourth embodiment. In the example of Fig. 85, the polarizer group 3007HLa in the upper left corner contains a mixture of polarizers 3007Lc and 3007Lb, in which grids with low extinction ratios are oriented diagonally right and left in the figure, and polarizers 3007Hd and 3007Ha, in which grids with high extinction ratios are oriented vertically and horizontally in the figure. Furthermore, polarizer group 3007HLb adjacent to polarizer group 3007HLa on the right side contains a mixture of polarizers 3007La and 3007Ld, each having a low extinction ratio grid oriented horizontally and vertically in the figure, and polarizers 3007Hc and 3007Hb, each having a high extinction ratio grid oriented diagonally to the right and left in the figure. When the extinction ratio is spatially modulated within a polarizer group, the bias in the extinction ratio and the grid direction (polarization direction) may be suppressed across the entire polarizing filter 3070'.

[0613] Furthermore, non-polarized pixels may be mixed within a polarizer group. As an example, in polarizer group 3007H shown on the left side of Fig. 83, one to three of polarizers 3007Ha, 3007Hb, 3007Hc, and 3007Hd are non-polarized elements. Pixels without a grid or pixels with a colorless optical filter instead of a grid may also be used as non-polarized elements.

[0614] Additionally, specific wavelengths of incident light may be targeted for polarization, for example by using color filters.

[0615] (6-6. Fourth Example of Fourth Embodiment) Next, a fourth example of the fourth embodiment will be described. The fourth example of the fourth embodiment is an example in which the resolution or spatial frequency is varied in the variation expansion sensor 3000.

[0616] (6-6-1. Super-resolution and Aliasing) To facilitate understanding, a brief explanation of super-resolution will be provided. A technique called super-resolution is known that obtains an image with a higher resolution than the captured image acquired from a sensor. One method proposed for achieving super-resolution is to physically shift the pixels of the sensor for each frame when capturing images.

[0617] (Regarding super-resolution) Fig. 86 is a schematic diagram for explaining an example of super-resolution using existing technology. In Fig. 86, as shown in section (a), an image Img at position Posi#1, which is generated by each pixel PixD corresponding to each pixel Pix of the pixel array unit 3006 (pixel array unit 1011), is shifted by 1 / 2 pixel in both the horizontal and vertical directions to generate each pixel PixD-sft corresponding to each pixel PixD. That is, an image Img-sft using each pixel PixD-sft is generated at position Posi#2, which is shifted by 1 / 2 pixel in both the horizontal and vertical directions to the image Im...

Claims

1. An information processing device comprising: a first correction unit that corrects an output from a plurality of pixels using first correction data based on characteristic variations of the plurality of pixels where the characteristics of at least one pixel among the plurality of pixels are different; and a second correction unit that further corrects the corrected output using second correction data generated based on the corrected corrected output corresponding to the plurality of pixels.

2. The information processing device of claim 1, wherein the characteristic is the sensitivity of each of the multiple pixels, the first correction unit corrects sensitivity variations of the outputs from the multiple pixels based on the first correction data, and the second correction unit performs saturation correction on the outputs from the multiple pixels whose sensitivity variations have been corrected by the first correction unit, based on the second correction data including saturation information of each of the multiple pixels.

3. The information processing device according to claim 2, wherein the first correction data includes a correction value for each pixel included in at least an effective pixel area among the plurality of pixels, or includes a correction value for each pixel included in a pattern formed by a plurality of pixels among the plurality of pixels.

4. The information processing device according to claim 2, wherein the second correction unit estimates a saturated region including saturated pixels that become saturated when a predetermined amount of light is irradiated onto the multiple pixels from the region including the multiple pixels based on the second correction data.

5. The information processing device according to claim 2, wherein the first correction unit corrects the sensitivity variation by multiplying pixel values ​​in the output from the plurality of pixels by a gain based on a pixel-by-pixel correction value included in the first correction data.

6. The information processing device according to claim 2, further comprising a recognition unit that performs recognition processing based on an input image, wherein the recognition unit receives one or more outputs of the multiple pixels, the first correction unit, and the second correction unit as the input image.

7. The information processing device according to claim 1, wherein the characteristic is a pixel-to-pixel distance among the plurality of pixels, the first correction unit calculates a disparity for each pixel group composed of adjacent pixels among the plurality of pixels based on the first correction data to obtain a disparity image, and the second correction unit performs an interpolation process on the disparity image using the disparity calculated for each pixel group by the first correction unit.

8. The information processing device of claim 1, wherein the characteristics are polarization characteristics of each of the plurality of pixels, the first correction unit calculates polarization information based on the first correction data for each pixel group consisting of adjacent pixels among the plurality of pixels and each pixel group having a polarizer with a different polarization direction, and the second correction unit corrects the polarization information using the polarization information calculated for each pixel group by the first correction unit as the second correction data.

9. The information processing device of claim 8, wherein the second correction unit corrects the polarization information by weighting pixels among the pixels having an extinction ratio higher than a second threshold and an amount of incident light equal to or less than a third threshold when the amount of incident light to the plurality of pixels is higher than a first threshold, and corrects the polarization information by weighting pixels among the pixels having an extinction ratio equal to or less than the second threshold and an amount of incident light exceeding the third threshold when the amount of incident light to the plurality of pixels is equal to or less than the first threshold.

10. The information processing device of claim 1, wherein the characteristic is a spatial frequency of each of the multiple pixels, the first correction unit corrects sensitivity variations of the outputs from the multiple pixels based on the first correction data, and the second correction unit performs super-resolution processing using the outputs from the multiple pixels in which the sensitivity variations have been corrected by the first correction unit as the second correction data.

11. The information processing device of claim 1, wherein the characteristics are characteristics relating to the wavelength of light incident on each of the multiple pixels, the first correction unit corrects sensitivity variations of the multiple pixels based on the first correction data for outputs from the multiple pixels, and the second correction unit estimates an image based on the characteristics of pixels surrounding a target pixel among the multiple pixels for outputs from the multiple pixels in which the sensitivity variations have been corrected by the first correction unit.

12. The information processing device according to claim 11, wherein the characteristic is a spectral characteristic of each of the plurality of pixels, and the width of a transmission band of at least one of the plurality of pixels is made different.

13. The information processing device of claim 12, wherein the second correction unit estimates the image by weighting pixels among the pixels whose width of the transmission band is narrow and whose amount of incident light is equal to or less than a fifth threshold when the amount of incident light to the multiple pixels is higher than a fourth threshold, and estimates the image by weighting pixels among the pixels whose width of the transmission band is broader than the narrow band and whose amount of incident light exceeds a fifth threshold when the amount of incident light to the multiple pixels is equal to or less than the fourth threshold.

14. The information processing device according to claim 11, wherein the characteristic is a characteristic of attenuating or cutting light in the invisible light wavelength region of each of the plurality of pixels, and the characteristic of at least one pixel among the plurality of pixels is made different.

15. The information processing device of claim 14, wherein the second correction unit estimates the image by weighting pixels among the pixels that attenuate or cut light in the invisible light wavelength region when the amount of incident light to the multiple pixels is higher than a sixth threshold, and estimates the image by weighting pixels among the pixels that do not attenuate or cut light in the invisible light wavelength region when the amount of incident light to the multiple pixels is equal to or less than the sixth threshold.

16. An imaging system comprising: a sensor in which at least one pixel among a plurality of pixels has different characteristics; a first correction unit that corrects outputs from the plurality of pixels using first correction data based on characteristic variations of the plurality of pixels in the sensor; and a second correction unit that further corrects the corrected output using second correction data generated based on corrected corrected outputs corresponding to the plurality of pixels.

17. The imaging system described in claim 16, wherein the characteristic is the sensitivity of each of the multiple pixels, the sensor makes the sensitivity of at least one of the multiple pixels different so that PRNU (Photo Response Non-Uniformity) is a predetermined value or greater, the first correction unit corrects sensitivity variations of the multiple pixels based on the first correction data for outputs from the multiple pixels, and the second correction unit performs saturation correction on the outputs from the multiple pixels whose sensitivity variations have been corrected by the first correction unit, based on the second correction data including saturation information of each of the multiple pixels.

18. The imaging system described in claim 16, wherein each of the multiple pixels includes an element capable of detecting photons and a counter whose count value increases each time a photon is detected by the element, the characteristic is the sensitivity of each of the multiple pixels, the first correction unit corrects sensitivity variations and linearity of the outputs from the multiple pixels based on the first correction data, and the second correction unit performs saturation correction on the outputs from the multiple pixels in which the sensitivity variations have been corrected by the first correction unit, based on the second correction data including saturation information of each of the multiple pixels.

19. The imaging system of claim 18, wherein the sensitivity of the characteristic is a rate of increase in the count value in response to detection of the photon by the element.

20. The imaging system according to claim 18, wherein an upper limit of the count value is set for the counter.

21. An information processing method including: a first correction step executed by a processor, correcting outputs from the multiple pixels using first correction data based on characteristic variations of the multiple pixels in a sensor in which at least one of the multiple pixels has different characteristics; and a second correction step, further correcting the corrected output using second correction data generated based on the corrected corrected output corresponding to the multiple pixels.

Images