Image sensor and its signal processing circuit

Abstract

To provide an image pick-up device and a signal processing circuit thereof which can perform processing of a plurality of images inside and reduce the output data volume to the outside and reduce the data readout time.SOLUTION: In a signal processing circuit having a pulse generation circuit which generates a pulse correspondingly to a charge amount of a detection object and a counter circuit which counts the pulses, the counter circuit up-counts the pulse number corresponding to the first charge amount generated in a first detection period, and down-counts the pulse number corresponding to the second charge amount generated in a second detection period to output the pulse number corresponding to a difference between the first charge amount and the second charge amount. An image pick-up device includes the signal processing circuit in each pixel.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an imaging element and a signal processing circuit therefor, and more particularly to an imaging element and a signal processing circuit therefor that can be used for digital holography or the like. [Background technology]

[0002] In recent years, research has been conducted into technologies that generate highly functional images and videos by applying various processes to image data acquired by image sensors. Digital holography is one such technology.

[0003] Digital holography is an imaging method capable of acquiring the complex amplitude (amplitude-phase distribution) of a subject. It has many advantages, including excellent spatial and depth resolution and the ability to adjust the focus position through calculations. In digital holography, to acquire a hologram at high resolution using an image sensor, it is desirable to use an in-line optical configuration in which the two interfering light waves that form the hologram are both incident on the image sensor nearly perpendicularly. In this in-line holography, phase-shifting digital holography is known, in which the phase of one of the two interfering light waves is shifted multiple times to acquire subject light while avoiding the superposition of direct light and conjugate light. The four-step phase-shifting method is particularly popular due to its high robustness against noise (Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Ito and Shimobaba, "Introduction to Holography: 3D Images and Measurements Using Computers," Kodansha, 2017, pp. 119-122 Summary of the Invention [Problem to be solved by the invention]

[0005] In the four-step phase-shifting method, to obtain one reconstructed image of an object, the phase is shifted in four steps, and a hologram is acquired by the image sensor each time, which results in a large amount of data.In addition, it takes four times as long to output the hologram signal acquired by the image sensor for one object image, which makes it difficult to increase the frame rate of the digital holography system.

[0006] Another method for acquiring multi-phase holograms is to use multiple image sensors to simultaneously capture holograms with different optical path length phases, but this method results in large optical systems and camera systems, and the amount of data required for the entire system remains large.Another method involves spatially multiplexing hologram information consisting of multiple phases into a single hologram and acquiring it with a single image sensor, but this method suffers from the problem of degraded resolution of the hologram information corresponding to each phase.

[0007] In this way, in a video system that processes a plurality of images, an increase in the amount of output data from the image sensor and an increase in readout time as the number of images increases have become an issue.

[0008] Therefore, in consideration of the above-mentioned problems, an object of the present invention is to provide an imaging element and its signal processing circuit that are capable of processing multiple images internally and that can reduce the amount of data output to the outside and the data readout time. [Means for solving the problem]

[0009] In order to solve the above problems, a signal processing circuit according to the present invention includes a pulse generating circuit that generates pulses corresponding to the amount of charge to be detected, and a counter circuit that counts the pulses. The counter circuit counts up the number of pulses corresponding to a first amount of charge generated in a first detection period, and counts down the number of pulses corresponding to a second amount of charge generated in a second detection period, thereby outputting the number of pulses corresponding to the difference between the first amount of charge and the second amount of charge. The counter circuit further includes a first memory corresponding to the number of bits of the counter circuit, a counter output switch connecting the counter circuit and the first memory, a second memory corresponding to the number of bits of the first memory, and a memory output switch connecting the first memory and the second memory.It is characterized by:

[0010] It is also desirable that the signal processing circuit use the charge generated by the photoelectric conversion element as the charge amount to be detected, and output the result of subtraction processing of the light intensity in the first detection period and the light intensity in the second detection period.

[0011] Furthermore, it is desirable that the signal processing circuit include a pulse generation circuit that includes a voltage detection node whose voltage changes in accordance with the amount of charge of the detection target, reset means that sets the voltage detection node to a reset voltage, and an inverter circuit to which the voltage of the voltage detection node is input.

[0014] In order to solve the above problems, a signal processing circuit according to the present invention includes a pulse generation circuit that generates pulses corresponding to an amount of charge to be detected, and a counter circuit that counts the pulses, wherein the counter circuit counts up the number of pulses corresponding to a first amount of charge generated in a first detection period, and counts down the number of pulses corresponding to a second amount of charge generated in a second detection period, thereby outputting a number of pulses corresponding to a difference between the first amount of charge and the second amount of charge; The number of bits of the number of output pulses is made smaller than the number of bits of the counter circuit. Characterized by .

[0015] In order to solve the above problem, an image pickup device according to the present invention is characterized in that each pixel is provided with the above signal processing circuit.

[0016] It is also desirable that the imaging element detect a hologram obtained by a phase shift method. [Effects of the Invention]

[0017] According to the image sensor and its signal processing circuit of the present invention, it is possible to process a plurality of images internally, thereby reducing the amount of data output to the outside and the time required to read the data. [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 1 is a diagram of a holographic imaging device using an imaging element of the present invention. [Figure 2] FIG. 2 is a diagram illustrating an example of the configuration of a signal processing circuit according to the first embodiment. [Figure 3] 10 is a timing chart showing the relationship between a photodiode voltage and a pulse output. [Figure 4]4 is a timing chart illustrating the operation of the signal processing circuit according to the embodiment. [Figure 5] FIG. 10 is a diagram illustrating an example of the configuration of a signal processing circuit according to a second embodiment. [Figure 6] FIG. 10 is a diagram illustrating an example of the configuration of a signal processing circuit according to a third embodiment. [Figure 7] FIG. 1 is a diagram illustrating a flow of a simulation of hologram formation and object image reconstruction. [Figure 8] FIG. 1 is a diagram showing an example in which the imaging element of the present invention is configured with a three-dimensional structure. DETAILED DESCRIPTION OF THE INVENTION

[0019] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0020] A holographic imaging device using the imaging element of the present invention is shown in Figure 1. This holographic imaging device constitutes an incoherent digital holographic imaging device using the optical system of a Michelson interferometer. The imaging device includes a lens 2, a beam splitter 3, a concave mirror 4, a plane mirror 5, a piezoelectric element 6, an imaging element 7, a phase synchronization control device 8, and a processing unit 9.

[0021] Object light from object (subject) 1 is converted by lens 2 into parallel light and enters beam splitter 3, where it is split into two split beams. One of the split beams heads toward concave mirror 4, is reflected by concave mirror 4, passes through beam splitter 3, and enters image sensor 7. The other split beam heads toward plane mirror 5, is reflected by plane mirror 5, has its direction changed by beam splitter 3, and then enters image sensor 7. The two split beams are given different curvatures by concave mirror 4 and plane mirror 5, and the two light waves interfere to form a hologram on the imaging surface of image sensor 7.

[0022] This holographic imaging device uses a four-step phase shift method to acquire object light while avoiding the superposition of direct and conjugate light in in-line holography. The plane mirror 5 is moved in the optical axis direction by a phase shift means (piezo element) 6 by distances of 1 / 8, 2 / 8, and 3 / 8 times the wavelength (the optical path length on the plane mirror side changes by 1 / 4, 2 / 4, and 3 / 4 times the wavelength), thereby changing the phase in four steps: 0, π / 2, π, and 3π / 2.

[0023] The image sensor 7 acquires the necessary information from the four holograms formed on its imaging surface using a signal processing circuit (described later) and outputs the information to the arithmetic processing unit 9. The arithmetic processing unit 9 calculates the amplitude and phase distribution on the image sensor surface based on the output signal from the image sensor 7. Furthermore, by performing a backpropagation calculation from the image sensor surface to the object surface of the subject 1, the amplitude and phase distribution at the object position can be reconstructed. A phase synchronization control unit 8 is connected to the image sensor 7 and the phase shift means (piezo element) 6, so that the signal processing and phase shift operation of the image sensor 7 (described later) can be synchronized.

[0024] An example of the configuration of a signal processing circuit according to a first embodiment of the present invention is shown in Fig. 2. The signal processing circuit in Fig. 2 corresponds to one pixel of the image sensor 7, and configures a 1-bit A / D conversion circuit (1-bit ADC).

[0025] The signal processing circuit includes a voltage detection node 11 of a photodiode (PD) 10 and a reset transistor (M R ) 20, an inverter circuit (inverter chain) 30, and a counter circuit (up-down counter) 40. Among these, a voltage detection node 11, a reset transistor (M R The photodiode 20 and the inverter circuit (inverter chain) 30 constitute a pulse generating circuit that generates a pulse corresponding to the amount of photoelectrically converted charge (amount of charge to be detected). Each of the components will be described below.

[0026] The photodiode (PD) 10 functions as a photoelectric conversion element, and its voltage detection node 11 detects a voltage (V PD ) changes. The voltage detection node 11 can use the electrode of the photodiode (PD) 10 as it is, but the electrode of a floating diffusion (capacitor for storing charge) can also be used as the voltage detection node 11, and a transfer transistor can be provided between the photodiode (PD) and the floating diffusion to transfer the photocharge generated in the photodiode (PD) 10 to the floating diffusion, thereby detecting the voltage. The voltage (V PD ) is input to the inverter circuit 30.

[0027] Reset transistor (M R ) 20 is the output voltage (V OUT ), and by turning it on (conducting), a reset voltage (V RST ) is applied. In this way, the reset transistor (M R ) 20 functions as a reset means.

[0028] The inverter circuit 30 is a multi-stage inverter circuit in which inverters (Inv1, Inv2, . . . Inv2n+1) are connected in an odd number of stages. Each inverter is configured, for example, by a CMOS (Complementary Metal Oxide Semiconductor) inverter. The potential (V PD ) is input to the first inverter (Inv1). The first stage of the inverter circuit (inverter chain) 30 may be replaced by a comparator instead of the inverter Inv1. The output of the inverter circuit 30 is the output (V OUT ), which is input to the counter circuit 40 and applied to the gate electrode of the reset transistor 20. The inverters are connected in 2n+1 stages in order to stabilize the circuit operation by utilizing the delay caused by the multiple inverter stages.

[0029] The counter circuit 40 receives the output (V OUT ) and outputs the bit values as counter outputs (O1 to O9) at predetermined detection intervals. The counter circuit 40 is a 9-bit counter. The counter is configured with 1-bit counters 41 to 49 connected in series. An up / down control signal (U / D) is input to each counter. When U / D is at a low level, the counter counts up, and when it is at a high level, the counter counts down. The first-bit counter has an enable terminal EN, which controls the counter so that it operates when EN is at a high level and does not operate when EN is at a low level. If the first-bit counter 41 is not operating, the second and subsequent bits do not change, so the enable terminal EN can control the counters of all bits (counting operation and stopping). Similarly to the up / down control signal (U / D), a counter reset signal (CRST) (not shown) is also input to each counter. When CRST is at a high level, the counter value of each bit is reset to its initial value, and when CRST is at a low level, the counter operates normally.

[0030] The pulse generation operation of the signal processing circuit of FIG. 2 will be described using the timing chart of FIG. 3 showing the relationship between the photodiode voltage (voltage detection node voltage) and the pulse output.

[0031] (1) The timing chart in FIG. 3 starts when the potential (V PD ) is the reset voltage (V RST ), which indicates that the reset of the photodiode has been completed and the reset has been released. The input of the first inverter (Inv1) is High and the output is Low, the output of the second inverter (Inv2) is High, and the output of the final inverter (Inv2n+1) (the output of the inverter circuit 30), i.e., the output of the pulse generating circuit (V OUT ) is Low, and the reset transistor (M R)20 is in the OFF state.

[0032] (2) When light is incident on the photodiode 10, charges (photocurrent) generated by photoelectric conversion are accumulated in the photodiode 10, and the voltage (V PD ) gradually decreases.

[0033] (3) The voltage of the voltage detection node 11 of the photodiode 10 (V PD ) is the inversion threshold voltage (V TH ), the output of the inverter (Inv1) is inverted to High. The inverters are connected in an odd number of stages (2n+1 stages), and the output is inverted and transmitted in sequence until the output of the final stage inverter (Inv2n+1), that is, the output of the pulse generation circuit (V OUT ) becomes High.

[0034] (4) Output of the pulse generator circuit (V OUT ) becomes High, the reset transistor 20 turns on, and a reset voltage (V RST ) is applied, resetting the photodiode 10 (and the voltage detection node 11) again.

[0035] (5) When the photodiode 10 is reset, the input of the first inverter (Inv1) goes High, and the output of the inverter circuit (V OUT ) goes low and returns to the initial state (1).

[0036] (6) Through this process, the output (V OUT ) generates a pulse. After that, the above steps (1) to (5) are repeated to generate multiple pulses. If the amount of light incident on the photodiode 10 is large, the amount of photoelectrically converted charge increases, the potential change of the voltage detection node 11 of the photodiode 10 becomes faster, and the inversion timing of the inverter circuit 30 becomes faster. Therefore, the output (V OUT ) generates pulses proportional to the amount of light.

[0037] In this way, the signal processing circuit of this embodiment converts the amount of photoelectrically converted charge into pulses and counts them, so the amount of light that can be input is not limited by the storage capacity of the photodiode, and has the advantage of being able to expand the dynamic range.

[0038] The counter circuit 40 (counters 41 to 49) counts the number of pulses generated within a predetermined period. However, in the signal processing circuit of this embodiment, as described below, the number of pulses may be counted up (added) or down (decremented) depending on the up / down control signal (U / D). After counting up / down for a predetermined period, the final pulse count (bit value) is read out from the counter circuit 40. The counter value for each pixel is read out, for example, using an XY addressing method. The reset signal (CRST: not shown) for each counter is at a low level during the pulse counting operation described above, but after the counter value is read out, it goes to a high level to reset the counter. Thereafter, it goes to a low level again to resume counting for the next detection period.

[0039] When the counter value of each pixel is read out using the XY address method, scanning is performed in both the X and Y directions, resulting in a time difference in the readout timing between the first pixel to be read out and the last pixel to be read out. If the counter value changes during the counter readout period, the exposure time between pixels will change depending on the pixel readout timing, so it is desirable to stop the counter operation during the counter readout period. Therefore, in this embodiment, by setting the enable terminal EN to Low and stopping the counter operation during the counter readout period, it is possible to prevent the exposure time between pixels from changing.

[0040] Before describing the operation of the signal processing circuit of this embodiment, the functions and effects of the image sensor of the present invention will be described.

[0041] In the holographic imaging device of Figure 1, in the four-step phase shift method, the phase of the reference light (light on the plane mirror side) is changed in four steps, namely, 0, π / 2, π, and 3π / 2, and the holograms (intensity distributions of the interfered light) acquired by the imaging element 7 are designated as I1 to I4, respectively. The complex amplitude distribution u of the object 1 on the imaging plane of the imaging element 7 can be calculated as follows:

[0042] u = 1 / 4×{(I1- I3) + i (I2- I4)} (i is the imaginary unit)

[0043] In conventional imaging devices, the intensity distribution of holograms I1 to I4 is acquired four times and output each time to the outside of the imaging device. However, the imaging device 7 of the present invention acquires a signal corresponding to (I1-I3) and a signal corresponding to (I2-I4) (differential intensity distribution) within each pixel and then outputs them to the outside, thereby reducing the number of outputs to two, half of the conventional number, and enabling a reduction in data volume and readout time.

[0044] Another method for subtracting two signals within a pixel is to provide a capacitor within the pixel to hold the signal from the first exposure as an analog value for one frame period, subtract it from the signal from the second exposure, and output the result; however, providing a capacitor for each pixel increases the pixel size and reduces the resolution. Alternatively, the signal can be converted to digital within the pixel, the digital values from the first and second exposures are stored in memory, and a digital signal subtractor subtracts the result and outputs the result; however, a multi-bit subtractor requires a large-scale circuit, which increases the pixel size and reduces the resolution.

[0045] In contrast to this, the image pickup device of the present invention controls the counter circuit 40 of the signal processing circuit, thereby enabling subtraction processing of signals within the image pickup device without particularly expanding the circuit configuration.

[0046] FIG. 4 is a timing chart illustrating the operation of a signal processing circuit according to one embodiment, showing a case in which an image sensor equipped with the signal processing circuit of this embodiment is used in digital holography. One cycle of the four-step phase shift is defined as 0 to t4, and exposure is performed corresponding to each phase (0, π / 2, π, 3π / 2) in a time divided into four. The phase indicates the phase state of the piezoelectric element 6 (i.e., corresponding to the position of the plane mirror 5) used to perform the phase shift. The phase synchronization controller 8 shown in FIG. 1 controls the timing of the piezoelectric element 6 and the signal processing circuit described below, enabling synchronization between them.

[0047] Here, if the hologram has 8-bit gradations, the range of output values of I1 to I4 is 0 to 255, and therefore the range of (I1-I3) and (I2-I4) that can be output is -255 to 255. In order to shift this range to positive count values, the initial value of counter circuit 40 is set to 256, and the total range counted by the counter is set to 511 (9 bits), thereby changing the range of (I1-I3) and (I2-I4) that can be output to 1 to 511. Note that, as will be described later, the initial value is not limited to 256 and can be set as appropriate based on the characteristics of the hologram, etc.

[0048] At time 0, the reset signal CRST is turned on, resetting the counter to 256. Then, from 0 to t1 (the first detection period), hologram I1, corresponding to phase 0, is exposed. With Phase set to 0, U / D is set to Low and counts up. At time t1, the counter holds the value obtained by adding 256 to the intensity (8-bit gradation output value) corresponding to I1. Next, from t1 to t2 (the second detection period), hologram I3, corresponding to phase π, is exposed. During this period, Phase is set to π, and U / D is set to High and counts down. At time t2, the counter holds the value obtained by adding 256 to the intensity corresponding to (I1-I3), which is then used to output the counter values for all pixels. The output from each pixel is read out by sequentially selecting the counter using, for example, an XY addressing method. When READ in Figure 4 is set to High, it indicates that the counter is outputting. In this way, the subtraction process for the light intensities I1 and I3 can be obtained from each pixel. Thereafter, (I2-I4) also performs up-counting and down-counting in the same manner as (I1-I3), and outputs at t4.

[0049] Through the above operations, (I1-I3) and (I2-I4), which are necessary for calculating the complex amplitude distribution u of the object on the imaging element plane, can be obtained from the imaging element with two outputs, half the number of outputs required in conventional methods. The signal processing circuit of the imaging element 7 of the present invention can detect the difference between two digital signals using a simple circuit configuration consisting of a relatively small number of transistors, namely an inverter chain and a counter (series-connected 1-bit counter). Furthermore, by using a three-dimensional structure (described later), it is possible to prevent an increase in pixel size and enable high-resolution holographic detection.

[0050] Since it takes a certain amount of time for the driving of the piezo element 6 indicated by Phase and for the output from the counter of each pixel, if that time is longer than the period of the pulse output, it is desirable to set the enable terminal EN to low level as shown in Figure 4 and stop the operation of the counter in order to prevent the count value from changing.

[0051] In the timing chart of FIG. 4, the phase is changed in the order of 0(I1), π(I3), π / 2(I2), and 3π / 2(I4). However, the phases of 0(I1) and π(I3) may be reversed, and the detection order of (I1-I3) and (I2-I4) may be reversed.

[0052] An example of the configuration of a signal processing circuit according to a second embodiment of the present invention is shown in Fig. 5. The signal processing circuit in Fig. 5 corresponds to one pixel of the image sensor 7, and is configured to be provided with a memory capable of holding the same number of bits as the counter, allowing the operation of the counter to be separated from the reading of the counter value. The signal processing circuit in Fig. 5 and its operation will be described below.

[0053] The signal processing circuit of the second embodiment includes a voltage detection node 11 of a photodiode (PD) 10 and a reset transistor (M R The second embodiment is composed of a counter circuit (inverter chain) 20, an inverter circuit (inverter chain) 30, a counter circuit 40, a counter output switch (SW) 50, and a memory 60. Compared to the first embodiment, the counter output switch (SW) 50 and a memory (first memory) 60 have been added, but the rest of the configuration is the same. The added components and their operations will be described below.

[0054] Counter output switches (SW) 50 are provided corresponding to the output of each bit (41 to 49) of the counter circuit 40, and connect the counter circuit 40 to a memory 60. By turning on the counter output switch (SW) 50, the values of all bits can be written to the memory 60. However, it is also possible to output values directly from the counter circuit 40 to outside the pixel.

[0055] The memory 60 corresponds to the number of bits of the counter circuit 40, and is composed of, for example, nine 1-bit memories. Each of the memories 61 to 69 corresponds to the counters 41 to 49, respectively. When the switch 50 is conductive, the output of the counter circuit 40 is written to the memory 60 (61 to 69), and the written bit value is output as a memory output (O1 to O9) at a predetermined read timing. In FIG. 5, nine 1-bit memories are used as an example, but any memory with an arbitrary bit input may be used as long as it can simultaneously write the 9-bit counter circuit output. The memory configuration may be DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory).

[0056] The operation of the second embodiment will be described. As in the first embodiment, the signal processing circuit of the second embodiment includes a voltage detection node 11 of a photodiode (PD) 10, a reset transistor (M R ) 20 and an inverter circuit (inverter chain) 30 function as a pulse generating circuit that generates pulses corresponding to the amount of photoelectrically converted charge (the amount of charge to be detected), and output V OUT The number of pulse signals proportional to the amount of light or the light intensity (amount of photocharge) appears on the pixel 41. The counter circuit 40 counts the number of pulse signals that have been generated.

[0057] 4, instead of outputting all pixels, the counter output switch 50 is turned on to store count values corresponding to (I1-I3) in the memory 60, and then the counter output switch 50 is turned off. CRST immediately goes High, the counter circuit 40 is reset, CRST goes Low, and EN goes High, restarting the pulse count operation. Then, at time t4, the counter circuit 40 stores count values corresponding to (I2-I4), so at this point, (I1-I3) and (I2-I4) can be output simultaneously from the memory 60 and the counter circuit 40 of all pixels, respectively, and the number of outputs from the image sensor can be reduced to one.

[0058] By adopting this output method, when reading out the signal values (I1-I3) and (I2-I4) of each pixel using the XY addressing method, only one scan is required in each of the X and Y directions, which leads to a reduction in power consumption during signal readout.

[0059] Alternatively, similarly with the configuration of Fig. 5, at time t2 in Fig. 4, the counter output switch 50 is turned on to store the count value corresponding to (I1-I3) in the memory 60. Thereafter, after time t2, while the exposures I2 and I4 are being performed, values are read from the memory 60 for each pixel, and then at time t4, the count value corresponding to (I2-I4) is stored in the memory, and after time t4, values are read from the memory for each pixel while the next exposures I1 and I3 are being performed.

[0060] By adopting this output method, the presence of memory 60 allows the counter circuit 40 to count pulses for the next period during the readout period of the memory value of each pixel, making it possible to count pulses continuously. In the first embodiment, the signal from the enable terminal EN stops counting pulses during the counter readout period, but in this embodiment, after the counter value is written to memory 60, the counter circuit 40 can be reset and counting can be resumed immediately, making it possible to minimize the time between detection periods. Even when readout is performed twice, compared to the first embodiment, the next pulse can be counted (exposed) immediately without waiting for the output time from each pixel, eliminating the loss of detectable light and enabling more accurate detection of the amount of light or light intensity (amount of charge).

[0061] An example of the configuration of a signal processing circuit according to a third embodiment of the present invention is shown in Fig. 6. The signal processing circuit in Fig. 6 corresponds to one pixel of the image sensor 7, and is configured to have two sets of memory that can hold the same number of bits as the counter, so that the counter value can be output only once. The signal processing circuit in Fig. 6 and its operation will be described below.

[0062] The signal processing circuit of the third embodiment includes a voltage detection node 11 of a photodiode (PD) 10 and a reset transistor (MR ) 20, an inverter circuit (inverter chain) 30, a counter circuit 40, a counter output switch (SW) 50, a memory A (first memory) 60, a memory output switch (SW) 70, and a memory B (second memory) 80. Compared to the second embodiment, the memory output switch (SW) 70 and memory B 80 are further added, but the rest of the configuration is the same. The added components and their operations will be described below.

[0063] Memory output switches (SW) 70 are provided corresponding to the output of each bit (61 to 69) of memory A 60, and connect memory A 60 to memory B 80. By turning on memory output switch (SW) 70, the values of all bits of memory A 60 can be written to memory B 80. However, it is also possible to output values directly from memory A 60 to outside the pixel.

[0064] Memory B80 corresponds to the number of bits of memory A60, and is composed of, for example, nine 1-bit memories. Memories 81 to 89 correspond to memories 61 to 69, respectively. When switch 70 is conductive, memory B80 (81 to 89) is written with the output of memory A60, and outputs the written bit value as memory output (O1 to O9) at a predetermined read timing. In FIG. 6, nine 1-bit memories are used as an example, but any memory with any bit input may be used as long as it can simultaneously write the 9-bit counter circuit output. Like memory A60, memory B80 may be configured as DRAM or SRAM.

[0065] The operation of the third embodiment will be described. The signal processing circuit of the third embodiment uses the output V OUT The process of counting the number of pulse signals generated by the counter circuit 40 is the same as in the first and second embodiments.

[0066] 4, the counter output switch 50 and the memory output switch 70 are turned on, and the count value corresponding to (I1-I3) is stored in memory B80 via memory A60. Then, the switches (SW) 50 and 70 are turned off, and the counting operation resumes. Thereafter, at time t4, the counter output switch 50 is turned on, and the count value corresponding to (I2-I4) is stored in memory A60. With this configuration, (I1-I3) and (I2-I4) can be output simultaneously from memory A60 and memory B80, respectively, while the next exposures I1 and I3 are being performed after time t4. This reduces the number of outputs from the image sensor to one, and the next exposure can be performed immediately without waiting for the output time from each pixel, which has the advantage of eliminating loss of detectable light and saving time.

[0067] The connection between memory A60 and memory B80 from counter circuit 40 does not have to be such that the two memories are connected in series; memory A60 and memory B80 may be connected in parallel to counter circuit 40, and a switch may be used to switch the destination for writing the counter output between memory A60 and memory B80.

[0068] The method of incorporating memory as shown in Figures 5 and 6 has the disadvantage of increasing the circuit scale of the memory portion, so it is desirable to prevent the pixel size from increasing by using a three-dimensional structure, which will be described later.If a three-dimensional structure is not used, it is best to choose between the two, depending on whether you want to reduce the number of outputs and shorten the time by incorporating memory, or whether you want to increase the resolution by reducing the pixel size using the configuration of Figure 2, which does not incorporate memory.

[0069] The signal processing circuit of the present invention also has the effect of reducing the number of output bits. Because I1 to I4 are holograms composed of the same subject, it is thought that the pixel values at the same location between I1 to I4 will not differ significantly from one another. Therefore, the values of (I1-I3) and (I2-I4) will be smaller than the original I1 to I4. Therefore, by reading only bits within the expected range, the amount of data can be further reduced.

[0070] Furthermore, the imaging element of the present invention is an imaging element that includes the signal processing circuit shown in the first to third embodiments in each pixel and performs pixel-parallel processing. When the imaging element of the present invention is used in a holographic imaging device, an image that has been processed in (I1-I3) or (I2-I4) can be output in a single output. As described above, the amount of data output to the outside and the data readout time can be reduced.

[0071] (Verification by simulation) An example of how data volume can be reduced using simulation results is shown. Figure 7 is a diagram illustrating the flow of a simulation of hologram formation and object image reconstruction. (a) is an input object image used as a subject, (b) is holograms (I1 to I4) generated by a holographic imaging device, and (c) is an image corresponding to (I1-I3) and (I2-I4), which are the output of the imaging element of the present invention. (d) shows the complex amplitude on the imaging plane obtained by calculation from the imaging element output (i.e., the hologram). By performing back propagation calculations based on this complex amplitude, the reconstructed image shown in (e) is obtained.

[0072] The holographic imaging device was based on the system shown in Figure 1. The parameters used in the simulation were an object (subject) 1 with a wavelength of 633 nm, a pixel pitch of 6.5 μm, and an image (input object image) with 256 × 256 pixels, a lens 2 with a focal length of 250 mm and an aperture diameter of 12 mm, a concave mirror 4 with a focal length of 250 mm and an aperture of 10.65 mm × 10.65 mm, and a plane mirror 5 with a focal length of 10 8 The dimensions were set to 10.65mm x 10.65mm, and the aperture was set to 10.65mm x 10.65mm. The image sensor 7 had a pixel count of 2048 x 2048, a pixel pitch of 6.5μm, and a size of 13.31mm x 13.31mm, and the bit rate of the image sensor was 16 bits. The distances between the elements were as shown in Figure 1, with Z1 = 250mm between object 1 and lens 2, Z2 = 100mm between lens 2 and mirrors 4 and 5, and Z3 = 500mm between mirrors 4 and 5 and image sensor 7.

[0073] The maximum and minimum values of intensity (pixel value) on the imaging plane of the holograms (I1 to I4) obtained as a result of the simulation are shown in Table 1. The values of I1 to I4 shown here are normalized so that the maximum value of all pixels included in I1 to I4 is 65535 (16 bits). On the other hand, the hologram image shown in Figure 7(b) is displayed normalized so that the maximum value is 255.

[0074] [Table 1]

[0075] Table 1 also shows the maximum and minimum signal values calculated for (I1-I3) and (I2-I4) for the hologram. Because the minimum values are negative, we can see that in order to make the minimum values greater than or equal to 0, we need to set the initial counter value for (I1-I3) to 1948 or greater and for (I2-I4) to 4388 or greater. For example, if we shift the minimum values to 0, the maximum value for (I1-I3) becomes 3674 and the maximum value for (I2-I4) becomes 5323. This means that the output bits can be 13 bits (8191 or less), reducing the data volume from the conventional 16-bit (hologram output) output. In other words, even if the internal processing of the image sensor is 16 bits, the output only needs to output count values ranging from 1 bit to 13 bits, reducing the number of output bits and the output data volume as well as the readout time. FIG. 7(c) shows a hologram image in which the minimum values of (I1-I3) and (I2-I4) are shifted to 0, and the maximum values are normalized to 255.

[0076] It is desirable to set the initial value of the counter in the signal processing circuit with a margin of error so that (I1-I3) and (I2-I4) do not become negative even if the state of the subject changes and the hologram value changes. It is also possible to externally control the initial value of the counter during imaging operation and change it to adapt to the subject. The above example demonstrated the effect of reducing the output bit count from the original 16 bits to 13 bits. However, conversely, by leaving the output at 16 bits and incorporating more in-pixel counters to detect multi-bit (e.g., 19-bit) holograms, and then outputting the (I1-I3) and (I2-I4) information required for reconstruction as 16 bits, higher-resolution holographic imaging can be achieved.

[0077] In the above embodiments and simulations, application to digital holography technology using a four-step phase-shifting method has been described, but the number of steps in phase-shifting digital holography is not limited to this, and the present invention can also be applied to a three-step phase-shifting method or a phase-shifting method with five or more steps. For example, if holograms obtained using a three-step phase-shifting method in which the phase changes in three steps, 0, 2π / 3, and 4π / 3, are denoted as I1 to I3, respectively, the complex amplitude distribution u on the imaging plane can be calculated as follows:

[0078] u = 1 / 6×{(2I1- I2- I3) + √3i (I2- I3)} (i is the imaginary unit)

[0079] In contrast, the complex amplitude distribution u can be obtained by capturing images that acquire two values, (I1 - I2) and (I2 - I3). This can be calculated as (2I1 - I2 - I3) = 2 × (I1 - I2) + (I2 - I3). In this way, the signal processing circuit and image sensor of the present invention can be applied to various digital holography applications.

[0080] Furthermore, although the above embodiments have been described as examples of incoherent holography optical systems, the present invention is not limited to these optical systems and can be applied to digital holography technology using laser light, interference measurement technology, etc. It can also be applied to cameras and microscopes for business, consumer, industrial, and academic use.

[0081] The imaging element of the present invention has a pulse generating circuit, a counter circuit, etc. for each pixel, which raises concerns about an increase in pixel and element area. For this reason, it is conceivable to give the imaging element a three-dimensional structure, thereby reducing the pixel area and improving the resolution.

[0082] Fig. 8 is a diagram showing an example of an image sensor of the present invention configured with a three-dimensional structure. In the image sensor 100 of Fig. 8, each circuit element constituting a pixel having a signal processing circuit is formed on a different substrate, and these are then stacked in a three-dimensional manner.

[0083] The image sensor 100 is composed of, for example, a photoelectric conversion layer (light receiving layer) 110, a pulse generating circuit layer 120, a counter circuit layer 130, and a control signal supply layer 140. Each layer is divided into pixels (the figure shows nine divisions), and each divided section is connected by vertical connection wiring 111, 121, 131, and the image sensor 100 is composed as a whole.

[0084] That is, the photoelectric conversion element (photodiode) 10 is formed in the uppermost light receiving layer 110, the reset means 20 and the inverter circuit 30 are formed in the second pulse generating circuit layer 120, and the counter circuit 40 is formed in the counter circuit layer 130. Then, the control signal supply circuit (circuit for supplying U / D, CRST, SW signals, etc.) and the scanning circuit for reading out the output signal are formed in the lowermost control signal supply layer 140, and the respective circuits are connected vertically, so that the output 141 can be taken out from the lowermost layer 140, for example.

[0085] In this way, by forming the counter circuit and control signal supply circuit on a substrate separate from the 1-bit ADC and three-dimensionally stacking them, and wiring them three-dimensionally for each pixel, it is possible to reduce the pixel area and realize a high-resolution image sensor. Note that the configuration of each layer is not limited to this example and can be set in various ways. For example, the light receiving unit and pulse generating circuit may be formed on the same layer, or a memory layer may be added as in the second and third embodiments. The three-dimensional structure makes it possible to avoid an increase in pixel size and realize high-resolution holographic imaging.

[0086] In the above embodiment, the application of the imaging element and its signal processing circuit of the present invention to digital holography technology has been described. However, the present invention is not limited to this and can be applied to any signal processing technology that performs addition or subtraction processing of multiple images. For example, in recent signal processing, CDS (Correlated Double Sampling) is used to detect the difference in voltage levels before and after the inflow of signal charge, thereby obtaining an output with reduced noise. The signal processing circuit of the present invention can also be applied to such processing. Furthermore, in technology for detecting the movement of an object, the difference between two temporally consecutive images is extracted, and therefore the imaging element and its signal processing circuit of the present invention can be used.

[0087] In the above embodiment, the configuration and operation of the signal processing circuit of the image sensor have been described, but the present invention is not limited to this and may be configured as a signal processing method. For example, the present invention may be configured as a signal processing method that processes pixel signals in accordance with the circuit diagram of Fig. 2 and the timing chart of Fig. 4.

[0088] Although the above-described embodiments have been described as typical examples, it will be apparent to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the present invention. Therefore, the present invention should not be construed as being limited by the above-described embodiments, and various modifications or alterations are possible without departing from the scope of the claims. For example, the functions included in each block, step, etc. described in the embodiments can be rearranged so as not to be logically inconsistent, and multiple constituent blocks, steps, etc. can be combined or divided into one. [Explanation of symbols]

[0089] 1 object 2 lenses 3 Beam splitter 4 concave mirror 5 plane mirror 6 Piezo element 7. Image sensor 8 Phase Synchronization Control Device 9 Processing Unit 10 Photodiode 11 Voltage detection node 20 Reset transistor 30 Inverter circuit 40 Counter Circuit 41~49 Counter 50 Counter output switch (SW) 60~69 memory 70 Memory output switch (SW) 80~89 memory 100 solid-state image sensor 110 Photoelectric conversion layer 111 Connection wiring 120 Pulse generation circuit layer 121 Connection wiring 130 Counter Circuit Layer 131 Connection wiring 140 Control signal supply layer 141 Output

Claims

1. a pulse generating circuit that generates a pulse corresponding to the amount of charge to be detected; a counter circuit that counts the pulses, the counter circuit counts up the number of pulses corresponding to a first amount of charge generated in a first detection period and counts down the number of pulses corresponding to a second amount of charge generated in a second detection period, thereby outputting the number of pulses corresponding to a difference between the first amount of charge and the second amount of charge; a first memory corresponding to the number of bits of the counter circuit; a counter output switch connecting the counter circuit and the first memory; a second memory corresponding to the number of bits of the first memory; a memory output switch connecting the first memory and the second memory; A signal processing circuit further comprising:

2. A pulse generating circuit that generates pulses corresponding to the amount of charge to be detected; a counter circuit that counts the pulses, the counter circuit counts up the number of pulses corresponding to a first amount of charge generated in a first detection period and counts down the number of pulses corresponding to a second amount of charge generated in a second detection period, thereby outputting the number of pulses corresponding to a difference between the first amount of charge and the second amount of charge; A signal processing circuit, characterized in that the number of bits of the number of pulses to be output is smaller than the number of bits of said counter circuit.

3. 3. The signal processing circuit according to claim 1, The charge generated by the photoelectric conversion element is the charge amount to be detected, A signal processing circuit, comprising: a first detection period; a second detection period; a second detection period; and a third detection period;

4. 4. The signal processing circuit according to claim 1, The signal processing circuit is characterized in that the pulse generation circuit comprises a voltage detection node whose voltage changes according to the amount of charge to be detected, reset means for setting the voltage detection node to a reset voltage, and an inverter circuit to which the voltage of the voltage detection node is input.

5. An imaging device, comprising the signal processing circuit according to any one of claims 1 to 4 for each pixel.

6. 6. The imaging device according to claim 5, An imaging device that detects a hologram obtained by a phase shift method.

Images