Imaging device and electronic device

Abstract

[Solution] The present invention provides an imaging element configured as a semiconductor chip, wherein an optical member is provided outside the semiconductor chip, and said imaging element comprises: an acquisition unit for acquiring information regarding the optical member; a pixel unit including N (N is an integer) types of pixels each having different spectral characteristics with respect to the wavelength of input light input via the optical member; a conversion unit that converts an output signal from the pixel unit into a digital output signal; a processing unit that uses the output signal output by the conversion unit as a basis to perform conversion processing of at least N + 1 processing signals each having different spectral characteristics using the aforementioned information; and an output unit that outputs a signal based on the processing signals to the exterior of the semiconductor chip.

Description

Technical Field

[0001] Embodiments of the present invention relate to an image sensor and an electronic device.

Background Art

[0002] A multi-spectrum image is an image that records electromagnetic waves in a plurality of wavelength bands. In addition to the information of electromagnetic waves in the wavelength band of visible light, a multi-spectrum image also has information of electromagnetic waves in the wavelength bands of invisible light such as ultraviolet rays, infrared rays, and far-infrared rays as image signals. Therefore, a multi-spectrum image is used to visualize things that cannot be identified by the naked eye, or to identify objects and determine states.

[0003] That is, unlike general image sensors for image capture, it is necessary to have more spectral characteristics than the primary colors red (R) / green (G) / blue (B) (hereinafter may be referred to as R, G, B), or cyan (C), magenta (M), and yellow (Y) (hereinafter may be referred to as C, M, Y). Therefore, an image sensor for a multi-spectrum image generally has five or more wavelength characteristics depending on the application.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] On the one hand, in order to have fine spectral characteristics (see, for example, FIG. 20), it is necessary to provide that number of filters. For this reason, assuming mounting within an image sensor, the number of pixels per wavelength band with respect to the total number of pixels decreases, and there is a risk that the resolution for one wavelength band will decrease.

[0006] Also, if an attempt is made to convert to different spectra using calculations after outputting signals corresponding to each wavelength band from the image sensor, the processing time will increase due to an increase in the amount of calculation and the like. For this reason, in software processing in electronic devices such as smartphones, real-time processing may be inhibited. In particular, it is necessary to redundantly output an image signal with respect to the finally generated image signal, which also leads to an increase in the data volume.

[0007] Also, the pixel array of an image sensor for multi-spectral images (the image sensor may be described as an Image Sensor) is different from the 2x2 unit pixel array (for example, Bayer) of a normal camera imaging image sensor. For example, the pixel array of an image sensor for multi-spectral images has a pixel output of 2x4 units or 3x3 units or more. For this reason, in a subsequent application processor or the like that processes the image signal output by the image sensor for multi-spectral images, it is necessary to correspond to such an array period.

Means for Solving the Problems

[0008] In order to solve the above problems, according to the present disclosure, there is provided an image sensor configured as a semiconductor chip, an acquisition unit that is provided with an optical member outside the semiconductor chip and acquires information regarding the optical member; a pixel unit having N (N is an integer) types of pixels with different spectral characteristics for the wavelength of input light input through the optical member; a conversion unit that converts the output signal of the pixel unit into a digital output signal; a processing unit that performs conversion processing on the output signal output by the conversion unit into N + 1 or more processing signals with different spectral characteristics using the information; An output unit that outputs a signal based on the processing signal outside the semiconductor chip, A imaging device is provided that includes this.

[0009] The pixel unit may include a plurality of photoelectric conversion elements that convert incident light into the output signal through N types of filters.

[0010] N may be 5 or more.

[0011] The imaging device may be configured within one semiconductor chip portion or a plurality of adjacent semiconductor chip portions.

[0012] The information relates to the optical characteristics of the optical member between the imaging device and the subject, and may relate to at least one of transmittance, reflectance, refractive index, emission wavelength, and wavelength dependence.

[0013] The optical member may be at least any one of a color filter, a plasmon, and an organic photoelectric conversion film.

[0014] The N types of filters may include four or more types of filters that transmit any one of red light, green light, blue light, cyan light, magenta light, and yellow light.

[0015] The acquisition unit is a memory (EEPROM) capable of storing the information from outside the semiconductor chip, and the information from outside the semiconductor chip stored in the memory may be supplied to the processing unit.

[0016] The optical member may be a band-pass filter.

[0017] The band-pass filter may transmit light in a predetermined visible light region and a predetermined infrared (IR) region.

[0018] The processing unit is capable of performing processing using parameters related to the shooting environment. The acquisition unit may be capable of acquiring the parameters including at least information related to the light source estimation result.

[0019] The processing signal output by the processing unit is image data according to predetermined array information. The acquisition unit may be capable of acquiring at least one of the information related to the N + 1 or more spectral characteristics and the information related to the array information.

[0020] Each of the processing signals has a peak of light sensitivity in each of the N + 1 or more wavelength bands in the input light within a predetermined wavelength range. The processing unit may be capable of changing at least one of the ranges of the N + 1 or more wavelength bands by parameter setting from outside the semiconductor chip.

[0021] The processing signal output by the processing unit is image data according to predetermined array information. The processing unit can change the pixel array of the image data by parameter setting from outside the semiconductor chip.

[0022] The optical member is at least a display panel for display. The processing unit may generate the processing signal using at least information related to the optical characteristics of the display panel.

[0023] The processing unit may also generate the processing signal based on output signals generated by different image sensors.

[0024] The pixel unit may have either an organic optoelectronic conversion film or a divided photodiode divided in the cross-sectional direction.

[0025] The combination in the processing signal generated for the first frame and the combination in the processing signal generated for the second frame generated after the first frame may be different.

[0026] The processing unit may generate M (M is an integer and M < N + 1) processing signals among the N + 1 processing signals as the first frame, and generate the remaining processing signals among the N + 1 processing signals as the second frame.

[0027] The pixel unit may perform different exposure controls between frames or between pixels.

[0028] The pixel unit may have at least one of a white pixel and a gray pixel that have a sensitivity in a wide wavelength band overlapping with a wavelength band having the sensitivity of other pixels with respect to input light in the predetermined wavelength range.

[0029] In the spectral characteristics of the N types of filters, there may be an overlap at one or more places in the wavelength band to be transmitted.

[0030] The spectral characteristic indicates a variation in the magnitude of the processing signal with respect to input light in a predetermined wavelength range. When the optical member is a band pass filter, the processing unit may perform a process of making the half-value width of the variation value of the processing signal with respect to the wavelength in at least one of the input lights of the N + 1 or more processing signals narrower.

[0031] The processing unit may each include at least one processing signal having a common spectral characteristic in the processing signal in the first frame and the processing signal in the second frame.

[0032] The processing unit may be capable of performing dynamic correction of a subject using the processing signal having the common spectral characteristic.

[0033] In order to solve the above problems, according to the present disclosure, an electronic device having an image sensor may be used.

Brief Description of the Drawings

[0034]

Figure 1

Figure 2

Figure 3A

Figure 3B

Figure 4

Figure 5

Figure 6A

Figure 6B

Figure 6C

Figure 7

Figure 8A

Figure 8B

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Embodiments for Carrying Out the Invention

[0035] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings attached to this specification, for the convenience of illustration and easy understanding, the scale, the aspect ratio of the vertical and horizontal dimensions, etc. are appropriately changed and exaggerated from those of the actual object.

[0036] (First Embodiment) FIG. 1 is a diagram showing an example of a basic unit of a pixel portion 120 according to the present embodiment. In the present embodiment, when the light receiving surface is divided into rectangular pixel blocks composed of a plurality of pixels, the pixel blocks are referred to as basic units. In FIG. 1, 4×4 pixels are shown as a pixel group constituting the basic unit. The pixel portion 120 according to the present embodiment is configured such that, for example, such basic units are repeatedly arranged horizontally and vertically. Note that the pixel portion 120 according to the present embodiment may be configured to have such basic units on the order of several million.

[0037] Pixels R, B, G, Y, C, IR, M, and W are classified into types according to their sensitivity characteristics with respect to the received light wavelength. For each pixel, for example, on-chip color filters of red (R: Red), blue (B: Blue), green (G: Green), yellow (Y: Yello), cyan (C: Cyan), infrared (IR), magenta (M: Magenta), and white (W: White) are formed. That is, the type of each pixel is made to correspond to a color filter, and the symbols R, B, G, Y, C, IR, M, and W are assigned. Each of the red (R: Red), blue (B: Blue), green (G: Green), yellow (Y: Yello), cyan (C: Cyan), infrared (IR), magenta (M: Magenta), and white (W: White) filters has the characteristic of transmitting light in the red band, blue band, green band, yellow band, cyan band, infrared band, magenta band, and white band, respectively.

[0038] As shown in FIG. 1, in the top row, pixels corresponding to green (G: Green), blue (B: Blue), yellow (Y: Yello), and cyan (C: Cyan) are arranged in order from the left. In the row below that, pixels corresponding to red (R: Red), infrared (IR, IR may be expressed as Black), magenta (M: Magenta), and white (W: White) are arranged in order from the left. In the row further below that, pixels corresponding to yellow (Y: Yello), cyan (C: Cyan), green (G: Green), and blue (B: Blue) are arranged in order from the left. In the bottom row, pixels corresponding to magenta (M: Magenta), white (W: White), red (R: Red), and infrared (IR) are arranged in order from the left.

[0039] FIG. 2 is a diagram showing the spectral characteristics of pixels R, B, G, Y, C, IR, M, and W. The vertical axis represents the quantum effect (QE), and the horizontal axis represents the wavelength. The quantum effect QE is a value obtained by dividing the light reception sensitivity of pixels R, B, G, Y, C, IR, M, and W with respect to the wavelength by the wavelength. In FIG. 2, spectral curves of eight types of quantum effects QE corresponding to the outputs of pixels R, B, G, Y, C, IR, M, and W are shown. As shown in FIG. 2, the input light input to pixels R, B, G, Y, C, IR, M, and W is measured (spectrally) for each wavelength, and what represents the ratio of the signal value to the light is called the spectral characteristic (spectral distribution). Thereby, information such as what color wavelength range the pixels R, B, G, Y, C, IR, M, and W have and what the shape of the peak is like is shown by the spectral curve showing the spectral characteristics of the pixels R, B, G, Y, C, IR, M, and W. In FIG. 2, eight types of spectral curves corresponding to the outputs of pixels R, B, G, Y, C, IR, M, and W are shown.

[0040] Each of the spectral curves of pixels R, B, G, Y, C, IR, M, and W has a broad spectral width (half-value width). In the present embodiment, the number of signal values for a predetermined wavelength range, for example, 300 to 900 nanometers, is called the spectral number. For example, in FIG. 2, since there are eight spectral curves, the spectral number is eight. Also, in the present embodiment, the signal value may be referred to as a pixel value. Information indicating colors such as R, B, G, Y, C, IR, M, and W is associated with the signal value. Alternatively, information indicating colors such as R, B, G, Y, C, IR, M, and W is associated with the information of the array of image data composed of signal values.

[0041] FIG. 3A is a diagram showing an example of output values O22 and O24 with respect to the wavelength of two certain pixels. The vertical axis represents the output value, and the horizontal axis represents the wavelength. In the example of FIG. 3A, the output values O22 and O24 with respect to the wavelength of the two pixels are an example without overlap. On the other hand, FIG. 3B is a diagram showing an example of output values O26 and O28 with respect to the wavelength of another two pixels. The vertical axis represents the output value, and the horizontal axis represents the wavelength. In the present embodiment, when there is an overlap like the output values O26 and O28, a spectral generation process for reducing the overlap is performed.

[0042] For example, coefficients are multiplied by combinations of output signal values of pixels R, B, G, Y, C, IR, M, and W to generate new output signals according to spectral characteristics. More specifically, as shown in Equation (1), the signal value of the Y pixel is multiplied by the coefficient a, the signal value of the G pixel is multiplied by the coefficient -b, the signal value of the B pixel is multiplied by the coefficient -c, and they are added together. Thereby, an output signal value α having new spectral characteristics can be generated. [Equation 1] α = aY - bG - cB (1)

[0043] In this way, for example, by performing operations on the output signal values of pixels R, B, G, Y, C, IR, M, and W using an N-row 8-column color matrix, it is possible to obtain N signal values having new spectral characteristics. For example, the color matrix can be set in advance through initial experiments during manufacturing, computational simulations, or the like.

[0044] FIG. 4 is a diagram showing an example of a processing signal obtained by performing a 12-row 8-column color matrix operation on the output signal values of the pixels R, B, G, Y, C, IR, M, and W shown in FIG. 2. α indicates the position of the peak of the spectral characteristics shown in Equation (1). That is, in the 12-row 8-column color matrix, the coefficient, which is the matrix parameter of the row corresponding to the output α, is, for example, (0, -c, -b, a, 0, 0, 0, 0). More specifically, when the output values of the pixels R, B, G, Y, C, IR, M, and W are taken as the signal value sequence (R, B, G, Y, C, IR, M, W), the row coefficient (0, -c, -b, a, 0, 0, 0, 0) × the signal value sequence (R, B, G, Y, C, IR, M, W) becomes Equation (1). Further, in the present embodiment, for example, since the W pixel has a wider half-value width in the quantum effect (QE), it is possible to cover a wide spectral region. Therefore, it is possible to obtain a processing signal having new spectral characteristics that cover a wide spectral region by performing an operation with the output signals of the other pixels R, B, G, Y, C, IR, and M. Note that the same applies to Gray with the transmittance of the W pixel reduced, and a Gray pixel may be used. Thus, the pixel according to the present embodiment has at least one of a White pixel and a Gray pixel having sensitivity in a wide wavelength band overlapping with the wavelength bands having the sensitivity of the other pixels R, B, G, Y, C, IR, and M with respect to input light in a predetermined wavelength range, for example, 300 to 1000 nanometers.

[0045] Further, by performing arithmetic processing on the output signals of the pixels R, B, G, Y, C, IR, M, and W, the spectral characteristics of the imaging device can be changed according to the purpose. For example, in an N-row 8-column color matrix, it is possible to make N larger than 8. Thus, when the number of color filters is N, it is possible to generate an output having N + 1 or more spectral numbers. In other words, the spectral characteristics (see, for example, FIG. 20) can be made finer while suppressing the number of color filters. Thereby, it is possible to make the spectral characteristics finer while suppressing a decrease in resolution. Note that, in the present embodiment, making the spectral characteristics finer means making the half-value width of the spectral curve (see FIGS. 2 and 4) narrower than before and increasing the number of spectral curves.

[0046] Here, with reference to FIG. 5, the basic schematic configuration of the image sensor according to the present embodiment will be described. FIG. 5 is a diagram showing a configuration example of the image sensor according to the present embodiment. As shown in FIG. 5, in the present embodiment, for example, a processing unit that performs signal processing as described above is configured in the logic circuits 334, 345, 355 in the image sensors (image sensors) 330, 340, 350.

[0047] As a first example, the image sensor 330 shown in the upper part of FIG. 5 is configured by mounting a pixel region 332, a control circuit 333, and a logic circuit 334 including the above-described signal processing circuit in one semiconductor chip 331.

[0048] As a second example, the image sensor 340 shown in the middle part of FIG. 5 is composed of a first semiconductor chip part 341 and a second semiconductor chip part 342. The first semiconductor chip part 341 is mounted with a pixel region 343 and a control circuit 344, and the second semiconductor chip part 342 is mounted with a logic circuit 345 including the above-described signal processing circuit. Then, the first semiconductor chip part 341 and the second semiconductor chip part 342 are electrically connected to each other, thereby constituting the image sensor 340 as one semiconductor chip.

[0049] As a third example, the image sensor 350 shown in the lower part of FIG. 5 is composed of a first semiconductor chip part 351 and a second semiconductor chip part 352. The first semiconductor chip part 351 is mounted with a pixel region 353, and the second semiconductor chip part 352 is mounted with a control circuit 354 and a logic circuit 355 including the above-described signal processing circuit. Then, the first semiconductor chip part 351 and the second semiconductor chip part 352 are electrically connected to each other, thereby constituting the image sensor 350 as one semiconductor chip.

[0050] FIG. 6A is a diagram showing a configuration example of the imaging module 110 according to the present embodiment. The imaging module 110 includes a lens system 112, an optical filter 114, an image sensor 116, and a storage unit (memory: EEPROM) 118.

[0051] The lens system 112 forms an image of the light from the subject on the pixel section 120 via the optical filter 114. The optical filter 114 is, for example, an infrared cut filter (IR-Cut Filter). Note that the optical filter 114 may be omitted. Also, generally, the optical filter 114 is changed according to the processing purpose for the imaging element 116. For example, as will be described later, the optical filter 114 may use a band pass filter, a plasmon, an organic photoelectric conversion film, or the like. That is, in the present embodiment, an object that affects the spectral shape (see FIG. 2) is collectively referred to as a filter. In this way, since the lens system 112 and the optical filter 114 are attached after the imaging element 116 is manufactured, the signal processing section of the imaging element 116 cannot recognize the characteristics of the filter including these lens system 112 and optical filter 114 at the time of manufacture.

[0052] In the present embodiment, the entire acquisition of two-dimensional image data is referred to as imaging. That is, imaging also includes outputting an output signal from the imaging element 11 as sensing data such as object identification and state recognition.

[0053] The imaging element 116 corresponds to, for example, the imaging elements (image sensors) 330, 340, and 350 shown in FIG. 5, and includes a pixel section 120, an AD converter 122, a light source estimation section 124, a spectral generation processing section 126, and an output interface 128. Note that the pixel section 120 is configured, for example, within the pixel regions 332, 343, and 353 shown in FIG. 5. Also, the AD converter 122, the light source estimation section 124, and the spectral generation processing section 126 are configured, for example, within the logic circuits 334, 345, and 355 shown in FIG. 5. Note that the AD converter 122 according to the present embodiment corresponds to the conversion section. Also, the light source estimation section 124 and the spectral generation processing section 126 according to the present embodiment correspond to the processing section. Also, the output interface 128 according to the present embodiment corresponds to the output section.

[0054] The pixel unit 120 is composed of, for example, the basic units described in FIG. 1. The reflected light from the subject passing through the lens system 112 and the optical filter 114 is photoelectrically converted by each pixel of the pixel unit 120. That is, the pixel unit 120 here may have the pixel layout shown in FIG. 1, or may have other layouts.

[0055] The AD converter 122 converts the output signal of each pixel of the pixel unit 120 into a digital signal value. In this embodiment, data having a signal value and arrangement information of the signal value is referred to as image data or an image. That is, the AD converter 122 converts the output signal of each pixel of the pixel unit 120 into a digital signal value and generates a multi-spectrum image.

[0056] The light source estimation unit 124 performs light source estimation processing. The light source estimation unit 124 performs background detection, for example, in the multi-spectrum image generated by the AD converter 122, and sets a light source estimation region based on the result of the background detection. Then, the light source estimation unit 124 performs estimation processing of the type of the light source when the multi-spectrum image is captured based on the light source estimation region.

[0057] The input light input to the imaging device 116 is represented by, for example, subject reflectance × light source spectrum × lens transmittance × optical filter transmittance × spectral characteristics of the pixel (see FIG. 2 for example). Therefore, when the characteristic to be obtained from the multi-spectrum image is the subject reflectance, in order to accurately recognize it, the light source spectrum, that is, the processing result of the light source estimation unit 124 is used. At this time, since the lens transmittance and the optical filter transmittance are unknown characteristics for the imaging device at the time of manufacture, by inputting filter characteristics including information such as the lens transmittance and the optical filter transmittance from the outside to the imaging device 116, it is possible to further improve the accuracy of the light source estimation processing.

[0058] The spectral generation processing unit 126 generates a processing signal corresponding to the spectrum by performing a color matrix operation including, for example, equation (1). At this time, arithmetic processing is performed based on the sensor spectrum. That is, when the spectral characteristics change due to the lens system 112 and the optical filter 114, the calculation is performed using this as the original spectrum. For example, although the color matrix operation including equation (1) is used as the basis of the arithmetic processing, when the spectral characteristics change due to the lens system 112 and the optical filter 114, the coefficients of the color matrix operation including equation (1) are changed using the optical characteristics.

[0059] Note that the spectral generation processing unit 126 according to the present embodiment generates a processing signal (which may be referred to as a pixel signal or a pixel value) corresponding to the spectrum by performing a color matrix operation including, for example, equation (1), but is not limited thereto. For example, the neural network (NN) may be trained using the output signals of pixels R, B, G, Y, C, IR, M, W (see FIG. 2) as input signals and the processing signal corresponding to the target spectrum as teacher data. That is, the spectral generation processing unit 126 may be configured using this trained neural network (NN). Also in this case, it is desirable to input the characteristics of the lens system 112 and the optical filter 114. Further, from the outside, the spectral characteristics themselves may be input as optical characteristics, or parameters after some calculation may be input. For example, matrix parameters used for matrix operations such as linear matrices and inverse matrix operations may be input. Also, the spectral generation processing unit 126 generates array information of the generated signal values based on the information supplied from the storage unit 118. In this way, the spectral generation processing unit 126 converts image data having N colors into image data having N + 1 or more colors.

[0060] The processing signal having a predetermined spectral characteristic generated by the spectral generation processing unit 126 is output via the output IF 128. Note that, as will be described later, the processing signal after performing processing for changing the specifications of the output image such as pixel interpolation and pixel rearrangement may be output via the output IF.

[0061] In the conventional process, the processing of the spectral generation processing unit 126 is executed by software processing or the like after output from the imaging element 116. Therefore, in the case of processing in the subsequent stage as in the conventional process, it is necessary to output an output signal (image data) in which all pixels are output from the imaging element 116. For example, in the above-described example, in order to obtain the spectrum α, at least three output signals of pixels Y, G, and B are required. Thus, when the data size of the output signal increases, there are drawbacks such as an increase in the data amount, which affects the frame rate of the imaging element, or an increase in power consumption. On the other hand, the spectral generation processing unit 126 according to the present embodiment can perform arithmetic processing within the imaging element 116. Therefore, the spectral generation processing unit 126 according to the present embodiment can suppress an increase in the data amount and suppress the influence on the frame rate of the imaging element 116 and the increase in power consumption.

[0062] On the other hand, the processing of the spectral generation processing unit 126 is implemented within the imaging element 116. If a calculation circuit such as the spectral generation processing unit 126 is mounted within the imaging element 116, many spectral variation factors such as variations in the manufacturing process and the characteristics of the optical filter 114 mounted in the subsequent process of the sensor manufacturing process cannot be taken into account. Therefore, as shown in FIG. 6, the storage unit 118 enables information such as filter characteristics to be input from the outside of the imaging element 116. That is, the storage unit 118 supplies information including parameters required by the light source estimation unit 124 and the spectral generation processing unit 126. As described above, the parameters include matrix parameters used for matrix operations such as linear matrices and inverse matrix operations. Note that the method of inputting information such as filter characteristics may be arbitrary, and it may be directly input from the application processor via an arbitrary IF (I2C, I3C, SPI, MIPI, etc.).

[0063] FIG. 6B is a diagram showing another configuration example of the imaging module 110 according to the present embodiment. The imaging module 110 includes a lens system 112, an optical filter 114, an imaging device 116, and an information input unit 1180. In the information input unit 1180, adjustment values such as filter characteristics and necessary wavelengths, and optical characteristic information are stored in association with parameters such as "1", "2". Thereby, when parameters such as "1", "2" are transmitted from the application processor 1120, for example, the adjustment values and optical characteristics associated with the parameters such as "1", "2" are set in the light source estimation unit 124, the spectral generation processing unit 126, and the like.

[0064] FIG. 6C is a diagram showing yet another configuration example of the imaging module 110 according to the present embodiment. The imaging module 110 includes a lens system 112, an optical filter 114, an imaging device 116, and an internal storage unit (OTP) 1122. The internal storage unit 1122 has the same configuration as the storage unit 118. That is, the internal storage unit 1122 supplies information including parameters required by the light source estimation unit 124 and the spectral generation processing unit 126. Note that the storage unit 118, the information input unit 1180, the internal storage unit 1122, and an arbitrary IF (I2C, I3C, SPI, MIPI, etc.) according to the present embodiment correspond to the acquisition unit.

[0065] As described above, according to this embodiment, the spectroscopic generation processing unit 126 uses the signal values of the pixels R, B, G, Y, C, IR, M, and W corresponding to the N color filters as input signals to generate N+1 or more processing signals having predetermined spectroscopic characteristics. As a result, it becomes possible to generate signal values having N+1 or more spectral numbers. Further, since the spectroscopic generation processing unit 126 is configured inside the image sensor 116, it becomes possible to generate N+1 or more signal values having predetermined spectroscopic characteristics without outputting the image data outside the image sensor 116, and the load of data transfer can be suppressed. At this time, since information required for arithmetic processing such as filter characteristics can be input from the storage unit 118 and an arbitrary IF (I2C, I3C, SPI, MIPI, etc.), even if the filter characteristics are changed or determined after the manufacture of the image sensor 116, it is possible to perform more accurate arithmetic processing of the light source estimation unit 124 and the spectroscopic generation processing unit 126.

[0066] (Second Embodiment) The imaging module 110 according to the first embodiment uses an infrared cut filter (IR-Cut Filter) for the optical filter 114, but the imaging module 110 according to the first embodiment is different in that a band pass filter (Band Pass Filter (BPF)) is used for the optical filter 114. Hereinafter, the differences from the imaging module 110 according to the first embodiment will be described.

[0067] FIG. 7 is a diagram showing an example of an output signal of the image sensor 116 when a band pass filter is used. The left diagram A is an output example showing the spectroscopic characteristics of the pixel unit 120 when there is no optical filter 114. The vertical axis represents the output value (Outpou), for example, showing the quantum effect, and the horizontal axis represents the wavelength (Wavelength). The middle diagram B is a diagram showing the wavelength transmission characteristics of the band pass filter. The vertical axis represents the transmittance (Transpareency), and the horizontal axis represents the wavelength (Wavelength). The right diagram C is an output example showing the spectroscopic characteristics of the spectroscopic generation processing unit 126 when there is an optical filter 114.

[0068] As shown in the left figure A, when there is no band-pass filter, for example, in the infrared (IR) region, the spectral peaks are reduced. Therefore, even if the spectral processing operation of the spectral generation processing unit 126 is performed, it becomes difficult to generate a spectrum with a narrow half-value width.

[0069] As shown in the middle figure B, the band-pass filter can narrow the transmitted light in the infrared (IR) region to, for example, only 800 - 900 nanometers. Similarly, this band-pass filter can narrow the transmitted light in the visible light region to, for example, 400 - 700 nanometers. As a result, when the spectral processing operation of the spectral generation processing unit 126 is performed, as shown in the right figure C, in the infrared (IR) region, a spectral curve with a narrow half-value width can be obtained. Similarly, in the visible light region, a spectral curve with a narrow half-value width can be obtained.

[0070] In such a case, it is possible to input a signal including the characteristic information of the band-pass filter from the storage unit 118 or the input system (see FIG. 6). In this case, other parameters calculated based on the characteristics of the band-pass filter may be used. Thereby, for example, the coefficient, which is a parameter of the linear matrix used in the operation of the spectral generation processing unit 126, can be changed to a coefficient more suitable for the characteristics of the band-pass filter. That is, the parameters of the linear matrix corresponding to the band-pass filter may be stored in the storage unit 118 in advance, or may be input from the input system (see FIG. 6). Note that the band-pass filter according to the present embodiment is arranged in the optical filter 114, but is not limited thereto. For example, a band-pass filter may be configured within the imaging element 116.

[0071] As described above, according to the present embodiment, a band-pass filter having transmission characteristics in specific wavelength regions (400 - 700 nm, 800 - 900 nm) is used for the optical filter 114. Thereby, the spectral generation processing unit 126 can obtain a spectral curve with a narrow half-value width in the wavelength region corresponding to the specific wavelength regions (400 - 700 nm, 800 - 900 nm).

[0072] (Third Embodiment) In the imaging module 110 according to the first embodiment, spectral processing is performed to make the number of spectral components of each pixel in a unit unit in the pixel unit 120 larger than the number N of types of color filters arranged in the unit unit. However, the imaging module 110 according to the third embodiment is different in that spectral processing can also be performed to make the number of spectral components of the signal output of each pixel in the unit unit in the pixel unit 120 smaller than the number N of types of color filters arranged in the unit unit. Hereinafter, the differences from the imaging module 110 according to the first embodiment will be described.

[0073] FIG. 8A is a diagram showing a configuration example of the imaging module 110 according to the third embodiment. The imaging module 110 further includes a second input unit 132 that inputs wavelength information (AP). The second input unit 132 is configured by, for example, a memory. Alternatively, the second input unit 132 may be configured as an input processing unit that directly acquires information from an application processor via an arbitrary IF (I2C, I3C, SPI, MIPI, etc.). For this reason, even if the wavelength information (AP) is changed after the manufacturing of the image sensor 116, it is possible to cope with it. The image sensor 116 further includes a pixel interpolation processing unit 130.

[0074] The pixel unit 120 has 4×4 pixels as a pixel group constituting a basic unit. In addition, for example, eight types of color filters are arranged on-chip in the pixel group constituting the basic unit. Therefore, the number of spectral components of the pixel group constituting the basic unit of the pixel unit 120 is eight.

[0075] Based on the wavelength information (AP) input from the second input unit 132, the spectral generation processing unit 126 acquires, for example, parameters used for matrix calculation from the storage unit 118. For example, when the number of spectral components of the input wavelength information (AP) is four, the spectral generation processing unit 126 performs arithmetic processing on a determinant of four rows and eight columns with respect to the output signal values of the 4×4 pixels. Note that the output signals of the 4×4 pixels are converted into digital signal values by the AD conversion unit 122.

[0076] The pixel interpolation processing unit 130 uses the processing result of the spectral generation processing unit 126 to convert an array of pixel values of 4×4 pixels into an array of only four colors. The pixel interpolation processing unit 130 can also perform pixel interpolation processing when converting an array of pixel values of 4×4 pixels into an array of only four colors. For this pixel interpolation processing, interpolation processing (Demosic) from the information of surrounding pixels as used in conventional camera signal processing may be performed, or interpolation processing using a neural network may be used. Regarding the array order at this time, the pixel interpolation processing unit 130 can perform the array based on the wavelength information (AP) input from the second input unit 132 and based on the information input from the outside. Note that the spectral generation processing unit 126 and the pixel interpolation processing unit 130 according to the present embodiment correspond to the processing units. Also, the second input unit 132 according to the present embodiment corresponds to the acquisition unit.

[0077] In this way, through the processing of the spectral generation processing unit 126 and the pixel interpolation processing unit 130, it is possible to convert from an 8-color coding to a 4-color coding. Usually, the camera signal processing system is often constructed to use 3 - 4 colors, and by outputting 4 colors or less, there is also the merit that much of the conventional signal processing can be reused. Note that in this embodiment, it is converted to 4 colors, but this number is arbitrary and not limited to this. Similar to the first embodiment, the spectral generation processing unit 126 can also be changed to output signals with a larger number of spectral components than the type N of the color filter by changing the parameters used for the matrix operation. In this case, the pixel interpolation processing unit 130 can be changed to convert from an N-color coding to a coding having more colors than N colors.

[0078] FIG. 8B is a diagram showing a configuration example of the imaging module 110 having no lens. As shown in FIG. 8B, the imaging module 110 may be a lens-less imaging system, for example, a pinhole camera or a structure of pinhole pixels.

[0079] As described above, according to the present embodiment, the imaging module 110 performs a spectral processing to make the number of spectral components of the signal output of each pixel of the unit unit in the pixel unit 120 less than the number N of types of color filters arranged in the unit unit. As a result, it becomes possible to change the image data format of the subsequent processing unit that processes using the image data of the imaging module 110.

[0080] (Fourth Embodiment) The imaging module 110 according to the fourth embodiment is different from the imaging module 110 according to the third embodiment in that an output signal having a new spectral characteristic is generated using the output signals of the stereo cameras. Hereinafter, the differences from the imaging module 110 according to the third embodiment will be described.

[0081] FIG. 9 is a diagram showing a configuration example of the imaging module 110 according to the fourth embodiment. The imaging module 110 according to the fourth embodiment includes a first lens system 112a, a first optical filter 114a, a first image sensor (Image Sensor1) 116a, a second lens system 112b, and a second image sensor (Image Sensor2) 116b. FIG. 9 further shows an application processing circuit (Application Processor) 200. In FIG. 9, the configurations related to the first image sensor 116a are denoted by a, and the configurations related to the second image sensor 116b are denoted by b. Also, the same configurations as those described above are denoted by the same numbers, and the description thereof is omitted. That is, the first image sensor 116a includes a pixel unit 120a, an AD converter 122a, a clamp unit 132a, and an output interface 128a. On the other hand, the first image sensor 116b includes a pixel unit 120b, an AD converter 122b, a clamp unit 132b, an input interface 134b, a memory unit 136b, a light source estimation unit 124, a spectral generation processing unit 126b, a pixel interpolation processing unit 130b, an output interface 128b, and an information input unit 138b. Note that the information input unit 138b according to the present embodiment corresponds to the acquisition unit.

[0082] FIG. 10 is a diagram showing an example of a pixel array of a basic unit of the pixel portion 120a and the pixel portion 120b. As shown in FIG. 10, the pixel portion 120a has 4×4 pixels as a basic unit and is composed of pixels of cyan, magenta, and yellow.

[0083] On the other hand, the pixel portion 120b has 4×4 pixels as a basic unit and is composed of pixels of red, green, blue, and infrared (IR). That is, the cyan pixel and the red pixel have a complementary color relationship, the magenta pixel and the green pixel have a complementary color relationship, and the yellow pixel and the blue pixel have a complementary color relationship.

[0084] Referring to FIG. 9 again, the clamp portion 132a executes, for example, processing related to the ground level in the image. The clamp portion 132a, for example, defines a black level, subtracts the defined black level from the image data output from the AD converter 122a, and outputs the result.

[0085] The input interface 134b inputs the first image data output by the output interface 128a. The memory portion 136b stores the first image data and the second image data output by the clamp portion 132b in association with each other. The information input portion 138b acquires a signal including information on filter characteristics and necessary wavelengths from the application processing circuit 200 and supplies the signal to the light source estimation portion 124 and the spectral generation processing portion 126b.

[0086] The spectral generation processing unit 126b generates an output signal having a new spectral characteristic for each basic unit by using the first signal included in each basic unit of the first image data and the second signal included in the corresponding basic unit of the second image data. In the present embodiment, the first signal has output signals of pixels of cyan, magenta, and yellow colors. On the other hand, the second signal has output signals of pixels of red, green, blue, and infrared (IR) colors. Thereby, the spectral generation processing unit 126b can process signals corresponding to seven colors for each basic unit. Then, the spectral generation processing unit 126b generates signals having M new spectral characteristics, for example, by matrix operations of M rows and 7 columns. In this case, it is possible to make M larger than 7. Thus, by using the output signals of the plurality of imaging modules 110, it is possible to make M larger than 7. In this case, compared to using only one of the first imaging element 116a and the second imaging element 116b, signals in more wavelength bands can be used, and the accuracy of the spectral processing can be further improved. As can be seen from these, it becomes possible to use different optical systems 112a and 112b and different first filters 114a for the first imaging element 116a and the second imaging element, and the spectral generation processing unit 126b can use signals in more wavelength bands.

[0087] Note that when having pixels of infrared (IR) color, since an infrared cut filter (IR-Cut Filter) cannot be used, the second imaging element may be configured not to have an infrared cut filter (IR-Cut Filter). Alternatively, a band pass filter may be used. Thus, for each of the first imaging element 116a and the second imaging element, it becomes possible to use an optical filter and a lens suitable for each.

[0088] FIG. 11 is a diagram showing an example of a pixel arrangement of basic units of a pixel portion 120a and a pixel portion 120b different from that in FIG. 10. As shown in FIG. 11, the pixel portion 120a has 4×4 pixels as a basic unit and is composed of pixels of cyan, magenta, and yellow colors.

[0089] On the other hand, the pixel portion 120b is composed of pixels of red, green, and blue with 4×4 pixels as the basic unit. That is, the cyan pixel and the red pixel have a complementary color relationship, the magenta pixel and the green pixel have a complementary color relationship, and the yellow pixel and the blue pixel have a complementary color relationship.

[0090] FIG. 12 is a diagram showing an example of a pixel arrangement of basic units of different pixel portions 120a and 120b. As shown in FIG. 12, the pixel portion 120a is composed of pixels of cyan, magenta, yellow, and green with 4×4 pixels as the basic unit.

[0091] On the other hand, the pixel portion 120b is composed of pixels of red, green, and blue with 4×4 pixels as the basic unit. That is, the cyan pixel and the red pixel have a complementary color relationship, the magenta pixel and the green pixel have a complementary color relationship, and the yellow pixel and the blue pixel have a complementary color relationship.

[0092] FIG. 13 is a diagram showing an example of a pixel arrangement of basic units of pixel portions 120a and 120b different from FIGS. 10 to 12. As shown in FIG. 13, the pixel portion 120a is composed of pixels of cyan, magenta, and yellow with 4×4 pixels as the basic unit.

[0093] On the other hand, the pixel section 120b is composed of pixels of red, green, blue, and white colors with 4×4 pixels as a basic unit. That is, a cyan pixel and a red pixel have a complementary color relationship, a magenta pixel and a green pixel have a complementary color relationship, and a yellow pixel and a blue pixel have a complementary color relationship.

[0094] As described above, according to the present embodiment, the spectroscopic generation processing section 126b generates an output signal having a new spectroscopic characteristic for each basic unit by using the first signal included in each basic unit of the first image data generated by the first image sensor 116a and the second signal included in the corresponding each basic unit of the second image data generated by the second image sensor 116b. Thereby, more signals in a wider wavelength band can be used than when only one of the first image sensor 116a and the second image sensor 116b is used, and the accuracy of the spectroscopic processing can be further improved.

[0095] (Fifth Embodiment) FIG. 14 is a diagram showing an example of the basic configuration of the pixel section 120 according to the fifth embodiment. As shown in FIG. 14, the pixels of the basic configuration are configured as 8×8. Also, adjacent pixels on the left and right have one of the same color filters M, R, Y, G, and C. That is, the rectangles divided left and right correspond to the photodiodes. Thus, any applicable pixel arrangement may be used. Also, one on-chip lens is provided for each color. The shape of the on-chip lens may be a square on-chip lens or other shapes such as a rectangle. The pixel section 120 according to the fifth embodiment can be used for the pixel sections 120 of the first to fourth embodiments.

[0096] FIG. 15 is a diagram showing an example of another basic configuration of the pixel portion 120 according to the fifth embodiment. As shown in FIG. 15, the pixels of the basic configuration are arranged in a 4×4 pattern. Also, 2×2 pixels can be read out analogously by addition, and different spectra can be obtained by the addition readout (see Patent Document 2). That is, the upper left 2×2 pixels are composed of pixels M and R, the upper right 2×2 pixels are composed of pixels Y and G, the lower left 2×2 pixels are composed of pixels Y and G, and the lower right 2×2 pixels are composed of pixels C and B. Then, the additive colors Cmr of pixels M and R, the additive colors Cyg of pixels Y and G, the additive colors Cyg of pixels Y and G, and the additive colors Ccb of pixels C and B are output as shown in the right figure.

[0097] FIG. 16 is a diagram showing an example of pixels that can be read out analogously by addition as in FIG. 15. As shown in FIG. 16, the pixels of the basic configuration are arranged in a 6×6 pattern. Also, 3×3 pixels can be read out analogously by addition, and different spectra can be obtained by the addition readout. That is, the upper left 3×3 pixels are composed of pixels M and R, the upper right 3×3 pixels are composed of pixels Y and G, the lower left 3×3 pixels are composed of pixels Y and G, and the lower right 43×3 pixels are composed of pixels C and B. Then, the additive colors Cmr of pixels M and R, the additive colors Cyg of pixels Y and G, the additive colors Cyg of pixels Y and G, and the additive colors Ccb of pixels C and B are output in the same manner as the right figure in FIG. 15. The pixel portion 120 shown in FIGS. 15 and 16 can be used for the pixel portion 120 of the first to fourth embodiments.

[0098] (Sixth Embodiment) FIG. 17 is a diagram showing a part of a cross-sectional view of the pixel portion 120 according to the sixth embodiment. As shown in FIG. 17, the pixel portion 120 includes, for example, an on-chip lens 400, organic photoelectric conversion films 402, 404, 406, and photoelectric conversion elements (PhotoDide) 404, 406, 408. The organic photoelectric conversion film 402 has a function equivalent to that of a color filter that does not transmit green light. Therefore, from the light transmitted through the organic photoelectric conversion film 402, magenta light, which is the remaining light, is photoelectrically converted by the photoelectric conversion element 408. Similarly, the organic photoelectric conversion film 404 has a function equivalent to that of a color filter that does not transmit red light. Therefore, from the light transmitted through the organic photoelectric conversion film 404, cyan light, which is the remaining light, is photoelectrically converted by the photoelectric conversion element 410. Similarly, the organic photoelectric conversion film 406 has a function equivalent to that of a color filter that does not transmit blue light. Therefore, from the light transmitted through the organic photoelectric conversion film 404, cyan light, which is the remaining light, is photoelectrically converted by the photoelectric conversion element 410. In the case of the case as shown in FIG. 17, it is considered that six colors are included as filter characteristics. That is, by implementing the present invention, an output of seven or more colors of 6 + 1 is possible. The pixel portion 120 shown in FIG. 17 can be used for the pixel portion 120 of the first to fifth embodiments.

[0099] Thus, in the pixel according to the present embodiment, the filter refers to all those that affect spectroscopy. For example, the organic photoelectric conversion films 402, 404, 406 and the spectroscopy characteristics of the photoelectric conversion elements (PhotoDide) 404, 406, 408 themselves are referred to as filters. For example, when a plasmon filter using plasmon resonance or a Fabry-Perot using a refractive index difference is used for spectroscopy, the filter includes the plasmon filter and the Fabry-Perot.

[0100] (Seventh Embodiment) FIG. 18 is a diagram showing an example in which the imaging modules (stereo camera modules) 110a and 110b according to the present embodiment are applied to a smartphone 1000b as an electronic device. The smartphone 1000a arranges the camera 1002a on the display 1006a side. For this reason, it is necessary to arrange the camera 1002a on the bezel, and there are design restrictions on the arrangement of the display 1006a.

[0101] The smartphone 1000b according to the present embodiment shown in FIG. 18 is a smartphone incorporating the imaging module (image generation device) 110 described in the first to sixth embodiments of the present disclosure. It is an example of a stereo type having two imaging modules 110. In the smartphone 1000b according to the present embodiment, the imaging modules 110a and 110b are arranged on the back side of the display 1006b together with the component layer 1004b, and imaging is performed through the display 1006b. For this reason, there are no restrictions on the design of the display 1006b, and it becomes possible to arrange the display 1006b over the entire surface of the smartphone 1000b. The display 1006a is, for example, an OLED panel.

[0102] As described above, when the imaging modules 110a and 110b according to the present embodiment are arranged in the smartphone 1000a, it becomes possible to generate an image signal having spectral characteristics according to the purpose such as object identification. For this reason, it is possible to improve the accuracy of object identification during camera shooting. As a result, the accuracy of scene determination and the like during camera shooting is improved by object identification, and appropriate shooting can be performed. In addition, when the imaging modules 110a and 110b according to the present embodiment are applied to the smartphone 1000a, the data efficiency is further improved. That is, since the output wavelength can be limited according to the purpose, the effect of improving the identification accuracy and the wavelength generation process can be performed within the imaging modules 110a and 110b, so that the power consumption can be suppressed.

[0103] In addition, as shown in Fig. 18, by disposing the imaging modules 110a and 110b under the display (under the touch panel), it becomes possible to sense the user's finger and detect blood circulation and oxygen saturation. In this case, the light source of the OLED can also be used to irradiate the finger. In the case of such an example, when inputting the filter characteristics, it is also possible to input information such as the transmittance of the OLED panel. Further, since the light source is known, wavelength information of the OLED light source may also be included. As a result, the imaging modules 110a and 110b can generate an output signal having spectral characteristics suitable for sensing. In addition, the input of the filter characteristics may be general information related to the wavelength information received by the imaging modules 110a and 110b in addition to the information regarding the optical filter described above, and may further include information such as light source spectroscopy and manufacturing variations.

[0104] FIG. 19 is a diagram showing an example in which the imaging modules (image generation devices) 110a and 110b according to the present embodiment are applied to a VR / AR / MR HMD (head-mounted display) as an electronic device. The head-mounted display shown in FIG. 19 is a head-mounted display (head-mounted display) incorporating the imaging module (image generation device) 110 described in the first to sixth embodiments of the present disclosure. The head-mounted display shown in FIG. 18 includes a glasses-type frame 10 attached to the head of the observer 40 and two imaging modules (image generation devices) 110a and 110b. The coupling member 20 is attached to the side facing the observer of the central portion 10C of the frame 10 located between the two pupils 41 of the observer 40. The frame 10 includes a front portion 10B disposed in front of the observer 40, two temple portions 12 rotatably attached to both ends of the front portion 10B via hinges 11, and modern portions 13 attached to the tip ends of the respective temple portions 12. The coupling member 20 is attached to the central portion 10C of the front portion 10B located between the two pupils 41 of the observer 40. The wiring 17 for the headphone unit extends from the tip end of the modern portion 13 to the headphone unit 16 through the inside of the temple portion 12 and the modern portion 13. More specifically, the wiring 17 for the headphone unit extends from the tip end of the modern portion 13 to the headphone unit 16 so as to go around the back side of the auricle (pinna).

[0105] Thus, the electronic device incorporating the imaging module 110 described in the first to sixth embodiments of the present disclosure is not limited to a smartphone, and may be a VR / AR / MR HMD (head-mounted display) as shown in FIG. 19, or a single-lens reflex camera or a capsule endoscope. Further, the imaging module 110 is not necessarily limited to an object for photographing a picture, and may include sensing for purposes such as an authentication sensor, skin analysis, and healthcare. Alternatively, the imaging module 110 may have other sensing functions.

[0106] (Eighth Embodiment) The imaging module 110 according to this embodiment is different from the imaging modules 110 in the first to seventh embodiments in that the spectroscopic generation processing unit 126 may further have a function of changing spectroscopic processing for each imaging frame. Hereinafter, differences from the imaging modules 110 according to the first to seventh embodiments will be described.

[0107] The spectroscopic generation processing unit 126 changes, for example, the parameters of a linear matrix for each imaging frame. For example, in the first frame, signals corresponding to three wavelengths (three colors) of 400, 450, and 500 nm are output, and in the next frame, signals corresponding to wavelengths of 550, 600, and 650 nm are output. Since the arithmetic processing of the spectroscopic generation processing unit 126 is performed within the imaging module 110, the control in the time axis direction can be arbitrarily changed. As a result, it is possible to output output signals having different spectroscopic characteristics for each frame in real time. The application processor that receives the output signal of the imaging module 110 is generally constructed on the premise of receiving an arrangement of generally three to four colors, as described above.

[0108] For example, the imaging module 110 according to this embodiment outputs signals corresponding to three wavelengths of 400, 450, and 500 nm in the first frame, and outputs signals corresponding to three wavelengths of 550, 600, and 650 nm in the second frame. Therefore, it is possible to transmit image data having an arrangement of six colors even to an application processor constructed on the premise of receiving, for example, an arrangement of three colors. Thus, when there are restrictions on the number of color combinations, by outputting signals corresponding to different wavelength bands for each frame as in this embodiment, it is possible to remove the restrictions on the number of color combinations. Note that the number of time divisions and the wavelength bands corresponding to the output signals can be arbitrarily set.

[0109] In addition, the spectral generation processing unit 126 may output signals corresponding to the same wavelength band between frames. Also, the spectral generation processing unit 126 can perform dynamic correction of the subject using processing signals having common spectral characteristics. For example, in frame 1, signals corresponding to three wavelengths of 400 nm, 500 nm, and 600 nm may be output, and in frame 2, signals corresponding to three wavelengths of 300 nm, 500 nm, and 700 nm may be output. That is, signals corresponding to the same wavelength may be output between frames. For example, when the subject is a moving object (a moving subject), by outputting signals corresponding to the same wavelength between frames, it becomes possible to more accurately correct the movement of the same subject.

[0110] Also, as shown in FIG. 2, the sensitivity of each pixel to the input light is different. In particular, the photosensitivity of the infrared (IR, Black) pixels becomes low. In order to correct such a sensitivity difference by color, in the imaging module 110, the exposure control may be changed for each pixel, for each frame, or between imaging modules 110. Also, the color filter may create colors by stacking a plurality of color filters. For example, when a Blue and a Red color filter are physically laminated, it becomes a Black (transmitting only IR) color filter.

[0111] As described above, according to the present embodiment, the spectral generation processing unit 126 is further configured to have a function of changing the spectral processing for each imaging frame. Thereby, even for an application processor having a restriction on the number of color combinations when handling a plurality of colors simultaneously, it becomes possible to output signals.

[0112] FIG. 20 is a diagram showing an example of spectral characteristics. The horizontal axis represents wavelength, and the vertical axis represents an example of signal values. Signal values O1 to O12 indicate variations in the magnitude of the signal with respect to the input light in a predetermined wavelength range, for example, 300 to 1000 nanometers. Each of the signal strength values O1 to O12 divides, for example, the range from the ultraviolet region to the infrared region into 12 wavelength bands and has the maximum value of the signal value in each wavelength band. In FIG. 20, the wavelength bands corresponding to the maximum values of the signal values (which may also be referred to as output) are sequentially assigned the symbols O1 to O12 from the low wavelength side to the high wavelength side.

[0113] Note that the present technology can adopt the following configuration.

[0114] (1) An image sensor configured as a semiconductor chip, an acquisition unit that is provided with an optical member outside the semiconductor chip and acquires information regarding the optical member; a pixel unit having N (N is an integer) types of pixels each having different spectral characteristics with respect to the wavelength of the input light input through the optical member; a conversion unit that converts the output signal of the pixel unit into a digital output signal; a processing unit that performs conversion processing into N + 1 or more processing signals each having different spectral characteristics based on the output signal output by the conversion unit using the information; an output unit that outputs a signal based on the processing signal outside the semiconductor chip; An image sensor comprising the above.

[0115] (2) The pixel unit has a plurality of photoelectric conversion elements that convert the input light into the output signal through N types of filters. The image sensor according to (1).

[0116] (3) N is 5 or more. The image sensor according to (2).

[0117] (4) The image sensor is configured within one semiconductor chip portion or a plurality of adjacent semiconductor chip portions. The image sensor according to (1).

[0118] (5) The information relates to the optical characteristics of the optical member between the imaging device and the subject, and relates to at least one of transmittance, reflectance, refractive index, emission wavelength, and wavelength dependence, the imaging device according to (1).

[0119] (6) The optical member is at least any one of a color filter, a plasmon, and an organic optoelectronic conversion film, the imaging device according to (1).

[0120] (7) The N types of filters include four or more types of filters among the filters that transmit any one of red light, green light, blue light, cyan light, magenta light, and yellow light, the imaging device according to (2).

[0121] (8) The acquisition unit is a memory (EEPROM) capable of storing the information from outside the semiconductor chip, and the information from outside the semiconductor chip stored in the memory is supplied to the processing unit, the imaging device according to (1).

[0122] (9) The optical member is a band-pass filter, the imaging device according to (1).

[0123] (10) The band-pass filter transmits light in a predetermined visible light region and a predetermined infrared (IR) region, the imaging device according to (9).

[0124] (11) The processing unit is capable of performing processing using parameters related to the shooting environment, The acquisition unit is capable of acquiring the parameters including at least information related to the light source estimation result, the imaging device according to (1).

[0125] (12) The processing signal output by the processing unit is image data according to predetermined array information, The acquisition unit is capable of acquiring at least any one of the information regarding the N+1 or more spectral characteristics and the information regarding the array information, the image sensor according to (1).

[0126] (13) Each of the processing signals has a peak in photosensitivity in each of the N+1 or more wavelength bands in the input light in a predetermined wavelength range, The processing unit is capable of changing at least any one of the ranges of the N+1 or more wavelength bands by parameter setting from outside the semiconductor chip, the image sensor according to (1).

[0127] (14) The processing signal output by the processing unit is image data according to predetermined array information, The processing unit is capable of changing the pixel array of the image data by parameter setting from outside the semiconductor chip, the image sensor according to (1).

[0128] (15) The optical member is at least a display panel for display, The processing unit generates the processing signal using at least information regarding the optical characteristics of the display panel, the image sensor according to (1).

[0129] (16) The processing unit generates the processing signal based also on output signals generated by different image sensors, the image sensor according to (1).

[0130] (17) The pixel unit has either an organic optoelectronic conversion film or a divided photodiode divided in a cross-sectional direction, the image sensor according to (1).

[0131] (18) The processing unit is the image sensor according to (1), wherein a combination in the processing signal generated for the first frame is different from a combination in the processing signal generated for a second frame generated after the first frame.

[0132] (19) The processing unit is the image sensor according to (18), wherein M (M is an integer and M < N + 1) processing signals among the N + 1 processing signals are generated as the first frame, and the remaining processing signals among the N + 1 processing signals are generated as the second frame.

[0133] (20) The pixel unit is the image sensor according to (1), wherein different exposure controls are performed between frames or between pixels.

[0134] (21) The pixel unit is the image sensor according to (13), wherein the pixel unit has at least one of a white pixel and a gray pixel having a sensitivity in a wide wavelength band overlapping a wavelength band having a sensitivity of other pixels with respect to input light in the predetermined wavelength range.

[0135] (22) The image sensor according to (2), wherein there is an overlap in at least one place of the wavelength bands transmitted in the spectral characteristics of the N types of filters.

[0136] (23) The spectral characteristics indicate a variation in the magnitude of the processing signal with respect to input light in a predetermined wavelength range. The processing unit is the image sensor according to (1), wherein when the optical member is a band pass filter, a process is performed to make a half-value width of a variation value of the processing signal with respect to the wavelength in at least any one of the input lights of the N + 1 or more processing signals narrower.

[0137] (24) The processing unit is the image sensor according to (18), which includes at least one processing signal having a common spectral characteristic in the processing signal in the first frame and the processing signal in the second frame.

[0138] (25) The processing unit is the image sensor according to (24), which can perform dynamic correction of a subject using the processing signal having the common spectral characteristic.

[0139] (26) An electronic device having the image sensor according to (1).

[0140] Aspects of the present disclosure are not limited to the individual embodiments described above, but also include various modifications that can be conceived by those skilled in the art, and the effects of the present disclosure are not limited to the above-described content. That is, various additions, changes, and partial deletions are possible without departing from the conceptual ideas and spirit of the present disclosure derived from the content defined in the claims and their equivalents.

Claims

1. An image sensor configured as a semiconductor chip, comprising: an acquisition unit that is provided outside the semiconductor chip with an optical member and acquires information regarding the optical member; a pixel unit having N (N is an integer) types of pixels each having spectral characteristics different from each other with respect to the wavelength of input light input through the optical member; a conversion unit that converts an output signal of the pixel unit into a digital output signal; a processing unit that performs conversion processing to convert the output signal output by the conversion unit into N + 1 or more processing signals each having different spectral characteristics, using the information; an output unit that outputs a signal based on the processing signal outside the semiconductor chip; wherein the information relates to the optical characteristics of the optical member between the pixel unit and the subject, and the image sensor relates to at least one of transmittance, reflectance, refractive index, emission wavelength, and wavelength dependence.

2. The image sensor according to claim 1, wherein the pixel unit has a plurality of photoelectric conversion elements that convert input light into the output signal through N types of filters.

3. The image sensor according to claim 2, wherein N is 5 or more.

4. The image sensor according to claim 1, which is configured within one semiconductor chip unit or within each of a plurality of semiconductor chip units that are electrically connected.

5. The image sensor according to claim 1, wherein the optical member is at least one of a color filter, a plasmon, and an organic photoelectric conversion film.

6. The image sensor according to claim 3, wherein the N types of filters include 4 or more types of filters that transmit any one of red light, green light, blue light, cyan light, magenta light, and yellow light.

7. The image sensor according to claim 1, wherein the acquisition unit is a memory (EEPROM) capable of storing the information from outside the semiconductor chip, and the information from outside the semiconductor chip stored in the memory is supplied to the processing unit.

8. The image sensor according to claim 1, wherein the optical member is a band-pass filter.

9. The image sensor according to claim 8, wherein the band-pass filter transmits light in a predetermined visible light region and a predetermined infrared (IR) region.

10. Each of the processing signals has a peak of light sensitivity in each of the N + 1 or more wavelength bands in input light in a predetermined wavelength range. The imaging device according to claim 1, wherein the processing unit can change at least one of the ranges of the N+1 or more wavelength bands by parameter setting from outside the semiconductor chip.

11. The processing signal output by the processing unit is image data according to predetermined array information, The imaging device according to claim 1, wherein the processing unit can change the pixel array of the image data by parameter setting from outside the semiconductor chip.

12. The optical member is at least a display panel for display, The imaging device according to claim 1, wherein the processing unit generates the processing signal using at least information regarding the optical characteristics of the display panel.

13. The imaging device according to claim 1, wherein the processing unit generates the processing signal based also on output signals generated by different imaging devices.

14. The imaging device according to claim 1, wherein a combination in the processing signal generated for a first frame is different from a combination in the processing signal generated for a second frame generated after the first frame.

15. The imaging device according to claim 14, wherein the processing unit generates M (M is an integer and M < N+1) processing signals in the N+1 processing signals as the first frame, and generates the remaining processing signals in the N+1 processing signals as the second frame.

16. The imaging device according to claim 1, wherein different exposure controls are performed between frames or between pixels in the pixel unit.

17. The pixel unit has at least one of a white pixel having sensitivity in a wide wavelength band overlapping with a wavelength band having the sensitivity of other pixels with respect to input light in the predetermined wavelength range, and a gray pixel, according to claim 10. The imaging device described.

18. In the spectral characteristics of the N types of filters, there is an overlap at one or more places in the wavelength band to be transmitted, according to the imaging device of claim 2.

19. The imaging device according to claim 14, wherein each of the processing signals in the first frame and the processing signals in the second frame includes at least one processing signal having a common spectral characteristic.

20. The imaging device according to claim 19, wherein the processing unit can perform dynamic correction of a subject using the processing signal having the common spectral characteristic.

21. An electronic device having the imaging device according to claim 1.

Images