Image capture and display device and wearable device

Abstract

To display a suitable image corresponding to a wide brightness range of the outside world.SOLUTION: An imaging display device has a plurality of imaging units, a plurality of display units, and a signal processing unit. The imaging units each has a first imaging unit and a second imaging unit capable of outputting a signal corresponding to a higher incident light level than the first imaging unit. The signal processing unit generates one third image signal from a first image signal from the first imaging unit and a second image signal from a second imaging unit, and a plurality of display devices displays an image based on the third image signal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an imaging and display device and a wearable device. [Background technology]

[0002] In recent years, wearable devices such as head-mounted displays (hereinafter referred to as HMDs) and smart glasses have become known. Patent Document 1 discloses an imaging and display device that includes a plurality of photoelectric conversion elements that convert light incident from outside the imaging and display device into charge signals, and a plurality of light-emitting elements that emit light with an intensity corresponding to the charge signals obtained from the plurality of photoelectric conversion elements. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-060980 Summary of the Invention [Problem to be solved by the invention]

[0004] The brightness of the outside world varies widely, from high brightness to low brightness. Patent Document 1 does not provide a detailed consideration on displaying an image that does not look strange in a wide range of brightness in the outside world. Therefore, an object of the present invention is to provide an imaging and display device that can display an appropriate image corresponding to a wide range of brightness in the outside world. [Means for solving the problem]

[0005] One aspect of the present invention relates to an imaging and display device, the imaging and display device having a plurality of imaging units, a plurality of display units, and a signal processing unit, the plurality of imaging units including a first imaging unit and a second imaging unit, the first imaging unit including a plurality of first unit cells, each of the plurality of first unit cells including an avalanche photodiode, the second imaging unit including a plurality of second unit cells, each of the plurality of second unit cells including a photodiode and a transistor for reading out a signal based on charge of the photodiode, the signal processing unit generates a first image signal based on signals from the first imaging unit and the second imaging unit; The plurality of display units 、 at least The first image signal The present invention is characterized in that it displays an image based on the image. [Effects of the Invention]

[0006] The present invention can provide an imaging and displaying device capable of displaying suitable images corresponding to a wide range of external brightness. [Brief explanation of the drawings]

[0007] [Figure 1] 1A and 1B are external views illustrating an image capturing and displaying device according to a first embodiment; [Figure 2] FIG. 1 is a block diagram illustrating an imaging and displaying apparatus according to a first embodiment. [Figure 3] 1A is a block diagram illustrating the image capturing and displaying apparatus according to the first embodiment, and FIG. 1B is a circuit diagram illustrating a unit cell of the image capturing and displaying apparatus according to the first embodiment. [Figure 4] FIG. 2 is a conceptual diagram illustrating an image signal of the imaging and display device according to the first embodiment. [Figure 5] FIG. 10 is a circuit diagram illustrating a unit cell of an imaging section of an imaging display device according to a second embodiment. [Figure 6] 10A and 10B are conceptual diagrams illustrating the operation of the imaging and display device according to the second embodiment; [Figure 7] 10A and 10B are conceptual diagrams illustrating the operation of the imaging and display device according to the second embodiment; [Figure 8] FIG. 10 is a conceptual diagram illustrating an image signal of an imaging and displaying apparatus according to a second embodiment. [Figure 9] 10 is an operational flowchart according to the fourth embodiment. [Figure 10] (a) An external view illustrating an imaging and display device according to a fifth embodiment. (b) An external view illustrating an imaging and display device according to a sixth embodiment. (c) An external view illustrating an imaging and display device according to a seventh embodiment. (d) An external view illustrating an imaging and display device according to an eighth embodiment. [Figure 11] 13 is an operational flowchart according to the seventh embodiment. [Figure 12] 13 is an operational flowchart according to the eighth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] Examples of imaging and display devices of the present invention will be described below with reference to the accompanying drawings. Similar elements throughout various embodiments are designated by the same reference numerals, and redundant explanations will be omitted. Furthermore, each embodiment can be modified and combined as appropriate. The imaging and display device of the present invention can be applied to wearable devices such as smart glasses, HMDs, and smart contact lenses.

[0009] (First embodiment) An imaging and display device according to the first embodiment will be described below with reference to FIGS. 1 to 4. FIGS. 1(a), 1(b), and 1(c) are external views of a wearable device including an imaging and display device 100 according to the first embodiment. The wearable device has a housing 1. The housing 1 is provided with a lens 2, a lens 3, and the imaging and display device 100. The imaging and display device 100 has multiple imaging units and multiple display units. The multiple imaging units include at least an imaging unit 10 and an imaging unit 20. The multiple display units include at least a display unit 50 and a display unit 60. The housing 1 includes the rims and temples of the glasses, and houses a signal processing unit, a control unit, and a communication unit. The imaging unit 10 and the display unit 50 are provided on the lens 2, and the imaging unit 20 and the display unit 60 are provided on the lens 3. The imaging unit 10 is provided on the front surface 2f of the lens 2, the display unit 50 is provided on the back surface 2b of the lens 2, the imaging unit 20 is provided on the front surface 3f of the lens 3, and the display unit 60 is provided on the back surface 3b of the lens 3.

[0010] Here, the positional relationship between the imaging unit 10 and the lens 2 will be described. The imaging unit 10 is provided on the front surface 2f of the lens 2, and the surface 2f of the lens 2 and the surface of the imaging unit 10 may form an uneven surface. The imaging unit 10 may also be embedded in the lens 2, and the surface 2f of the lens 2 and the surface of the imaging unit 10 may form a single curved or flat surface. The positional relationship between the imaging unit 20 and the lens 3 is the same as the positional relationship between the imaging unit 10 and the lens 2. The positional relationship between the display unit 50 and the lens 2 is also the same as the positional relationship between the imaging unit 10 and the lens 2, and the positional relationship between the display unit 60 and the lens 3 is also the same as the positional relationship between the imaging unit 10 and the lens 2.

[0011] 2 is a block diagram illustrating an image capturing and displaying device 100 according to this embodiment. The image capturing and displaying device 100 includes an image capturing unit 10, an image capturing signal processing unit 15, an image capturing unit 20, an image capturing signal processing unit 25, a signal processing unit 30, a display unit 50, a display unit 60, and an input unit 70.

[0012] The lens unit 11 includes optical mechanisms such as a lens for focusing light from a subject onto the imaging unit 10, and an aperture for adjusting the lens focus, changing the imaging magnification, or changing the amount of light, and is driven based on a control signal from the signal processing unit 30. The imaging unit 10 is a CMOS image sensor and outputs an image signal based on the control signal from the signal processing unit 30. The lens unit 11 and the imaging unit 10 together constitute an imaging system 12. The imaging signal processing unit 15 performs image processing such as color correction, white balance correction, and optical shading correction on the image signal from the imaging unit 10 under the control of the signal processing unit 30, and then outputs the image signal and control signal to the signal processing unit 30.

[0013] The lens unit 21 includes optical mechanisms such as a lens for focusing light from a subject onto the imaging unit 20, and an aperture for adjusting the lens focus, changing the imaging magnification, or changing the amount of light, and is driven based on a control signal from the signal processing unit 30. The imaging unit 20 is a CMOS image sensor and outputs an image signal based on the control signal from the signal processing unit 30. The lens unit 21 and the imaging unit 20 together constitute an imaging system 22. The imaging signal processing unit 25 performs image processing such as color correction, white balance correction, and optical shading correction on the image signal from the imaging unit 20 under the control of the signal processing unit 30, and then outputs the image signal and control signal to the signal processing unit 30.

[0014] The signal processing unit 30 comprehensively controls each unit of the imaging and display device 100 and also performs image processing. The signal processing unit 30 is a microcontroller having a signal processing circuit 35 and a signal holding circuit 36. The signal holding circuit 36 ​​is a non-volatile memory or a volatile memory. The signal holding circuit 36 ​​holds the image signals and control signals output from the imaging signal processing unit 15 and the imaging signal processing unit 25. The signal processing unit 30 performs image processing based on the image signals obtained from the imaging unit 10 and the imaging unit 20, and generates image signals for display. The signal processing unit 30 can also determine control signals for each unit based on the results of this image processing. Here, the image signal is also referred to as image information or image data.

[0015] The display unit 50 displays the image signal supplied from the signal processing unit 30. Any display such as a liquid crystal type, a projection type, or an organic light-emitting type may be used for the display unit 50. A display system 52 is configured from the display unit 50 and an interface circuit (not shown). The display unit 60 displays the image signal supplied from the signal processing unit 30. Any display such as a liquid crystal type, a projection type, or an organic light-emitting type may be used for the display unit 50. A display system 62 is configured from the display unit 60 and an interface circuit (not shown).

[0016] The input unit 70 has, for example, a power button and various operation keys, buttons, dials, etc. for adjusting display brightness and zoom. The input unit 70 outputs a control signal according to an input operation by the user to the signal processing unit 30. Note that the input unit 70 is not limited to physical components such as operation keys, and may also be provided with, for example, an imaging unit and various detection units to perform operations by gesture input, input by detecting the operator's line of sight, voice input, etc.

[0017] Next, the imaging units 10 and 20 will be described with reference to FIG. 3. Here, the imaging units 10 and 20 are assumed to be CMOS-type image sensors 300. FIG. 3(a) is a block diagram illustrating the image sensor 300. The image sensor 300 has a unit cell region 310 and other regions 320. The unit cell region 310 is a region in which a plurality of unit cells 301 are arranged in a matrix. The other regions 320 are other than the unit cell region 310, and include a readout circuit 330, a vertical scanning circuit 340, a horizontal scanning circuit 360, and a timing control circuit 370.

[0018] Each of the unit cells 301 is provided with a color filter of, for example, red (R), green (G), or blue (B) in a Bayer array. The number of unit cells 301 corresponding to one color filter is not limited to one, and the color filter arrangement is not limited to a Bayer array. Column signal lines 302 and row signal lines 303 are arranged in the unit cell region 310. At least one column signal line 302 is provided corresponding to each column of the arranged unit cells 301. The column signal line 302 is provided to read out signals obtained in each unit cell 301. The column signal line 302 is electrically connected to a readout circuit 330 arranged in another region. The row signal line 303 is electrically connected to a vertical scanning circuit 340. The vertical scanning circuit 340 outputs a control signal to operate the elements of each unit cell 301, allowing the signals obtained in each unit cell 301 to be read out. In FIG. 3(a), one row signal line 303 is shown, but the detailed configuration of the row signal line 303 will be described with reference to FIG. 3(b).

[0019] The readout circuit 330 is composed of an amplifier circuit, a correlated double sampling circuit (hereinafter referred to as a CDS circuit), and an analog-to-digital conversion circuit (hereinafter referred to as an ADC circuit). The readout circuit 330 may have one readout circuit section for each column signal line 302. One readout circuit section may be composed of the above-mentioned amplifier circuit and the like. Here, it is assumed that the readout circuit 330 has multiple ADC circuits. The digital signal generated by the readout circuit 330 is output to the outside of the image sensor 300 by the horizontal scanning circuit 360. Note that the timing control circuit 370 generates a timing control signal in accordance with a control signal from the signal processing unit 30 of FIG. 2. The control signal from the timing control circuit 370 controls the operations of the vertical scanning circuit 340, the horizontal scanning circuit 360, the readout circuit 330, and the like.

[0020] Fig. 3(b) is a circuit diagram illustrating a unit cell 301 of the image sensor 300 shown in Fig. 3(a). The unit cell 301 includes a photodiode PD, which is a photoelectric conversion element, a transfer transistor TX, an amplification transistor SF, a selection transistor SL, a reset transistor RS, and an additional capacitance transistor AP. Here, each transistor is assumed to be a switching element configured by an n-channel MOSFET, but the conductivity type and structure of the transistor are not particularly limited.

[0021] A row signal line 303 (TX) is connected to the gate of the transfer transistor TX. A row signal line 303 (SL) is connected to the gate of the selection transistor SL. A row signal line 303 (RS) is connected to the gate of the reset transistor RS. A row signal line 303 (AP) is electrically connected to the gate of the storage capacitance transistor AP. These row signal lines 303 extend in the horizontal direction and can simultaneously drive the unit cells 301 included in the same row. With this configuration, it is possible to perform a line-sequential rolling shutter operation and an all-row simultaneous global shutter operation.

[0022] The photodiode PD performs photoelectric conversion and accumulates the generated signal charge. When the transfer transistor TX is turned on, the signal charge accumulated in the photodiode PD is transferred to the floating diffusion region FD and accumulated there. The drain of the amplifier transistor SF is electrically connected to a power supply voltage VDD, and the gate of the amplifier transistor SF is electrically connected to the floating diffusion region FD. The amplifier transistor SF forms a source follower circuit, and the node of the gate of the amplifier transistor SF can also be considered the input node of the source follower circuit. The amplifier transistor SF outputs a signal based on the signal charge held in the floating diffusion region FD. The drain of the select transistor SL is electrically connected to the source of the amplifier transistor SF. The source of the select transistor SL is electrically connected to a column signal line 302. The select transistor SL can select a unit cell from which a signal is read. When the select transistor SL is turned on, the amplifier transistor SF and a constant current source (not shown) form a source follower, and a signal corresponding to the signal charge held in the floating diffusion region FD is output to the column signal line 302.

[0023] The drain of the reset transistor RS is electrically connected to the power supply voltage VDD, and the source of the reset transistor RS is electrically connected to the drain of the additional capacitance transistor AP. The drain of the additional capacitance transistor AP is electrically connected to the source of the reset transistor RS, and the source of the additional capacitance transistor AP is electrically connected to the floating diffusion region FD. When the transfer transistor TX, the additional capacitance transistor AP, and the reset transistor RS are simultaneously turned on, the floating diffusion region FD and the photodiode PD are set to a predetermined voltage based on the power supply voltage VDD. The operation of setting the voltage to a predetermined voltage is also called a reset operation.

[0024] The imaging units 10 and 20 will be described. The imaging unit 20 corresponds to a light intensity range that includes a higher light intensity than the imaging unit 10. The imaging unit 10 corresponds to a first light intensity range, and the imaging unit 20 corresponds to a second light intensity range. The second light intensity range includes a higher light intensity value than the first light intensity range. Here, the corresponding light intensity range is a light intensity range in which the linearity of the output signal relative to the input light intensity is maintained. Alternatively, the corresponding light intensity range may be a light intensity range defined by the light intensity corresponding to the maximum output signal and the light intensity corresponding to the minimum output signal of each imaging unit.

[0025] One method for changing the corresponding light intensity range is to change the conversion gain. In this case, the conversion gain of the imaging unit 10 is higher than that of the imaging unit 20. Here, the conversion gain indicates the relationship between the magnitude of the signal value obtained for a given light intensity. In other words, an imaging unit 10 with a high conversion gain saturates and cannot output a signal when the incident light intensity is high, but can output a highly accurate, multi-tone signal when the incident light intensity is low. On the other hand, an imaging unit 20 with a low conversion gain can output a signal even when the incident light intensity is high. Methods for changing the magnitude of the conversion gain include, for example, switching the capacitance value of a floating diffusion capacitance or switching the gain of an amplifier in the signal path, and these methods can be used in combination. In this embodiment, a method for switching the capacitance value of a floating diffusion capacitance will be described in detail.

[0026] Let Cfd be the capacitance of the floating diffusion region FD, and Afd be the capacitance that can be added by the additional capacitance transistor AP. When the reset transistor RS is off, the capacitance value of the floating diffusion region FD (FD capacitance value) is as follows: When the additional capacitance transistor AP is on, the FD capacitance value is the sum of the capacitances Cfd and Afd. When the additional capacitance transistor AP is off, the FD capacitance value is only the capacitance Cfd. In Figure 3(b), the capacitances Cfd and Afd are shown in a simplified manner, including parasitic capacitances that should be considered during design. The ability to switch the capacitance value of the floating diffusion region FD makes it possible to switch the photoelectric conversion gain, i.e., sensitivity. This is because, as described above, the signal charge transferred to the floating diffusion region FD by the transfer transistor TX is converted into a voltage signal when output from the amplifier transistor SF.

[0027] In this embodiment, the capacitance of the floating diffusion region FD in the imaging unit 10 of FIG. 2 is configured only as capacitance Cfd, and the capacitance of the floating diffusion region FD in the imaging unit 20 of FIG. 2 is configured as the sum of capacitances Cfd and Afd. In FIG. 3(b), the additional capacitance transistor AP can be in an on state or an off state. However, when the image sensor 300 is used as the imaging unit 20, the additional capacitance transistor AP may be configured to be always on, that is, the power supply voltage VDD may be electrically connected to the gate of the additional capacitance transistor AP. Furthermore, when the image sensor 300 is used as the imaging unit 20, the additional capacitance transistor AP may be omitted, and a large capacitance CAfd may be used, which is the sum of capacitance Cfd and capacitance Afd.

[0028] In the imaging and display device 100 shown in FIGS. 1 and 2, image processing is performed by the signal processing unit 30 based on image signals acquired by the imaging units 10 and 20, and images are displayed on the display units 50 and 60 based on the processed image signals. Therefore, to display images that do not cause discomfort to the user, it is desirable for the imaging units 10 and 20 to acquire good image signals. For example, it is important to acquire good image signals having a wide dynamic range. When a display image is generated based on an image signal with a narrow dynamic range, the user may feel overstimulated by the brightness, or the bright areas may be blown out and the dark areas may be crushed and unrecognizable, which may cause discomfort due to a situation different from what the user sees with their own eyes. Therefore, the imaging and display device 100 of this embodiment generates a display image based on image signals acquired using the imaging units 10 and 20 with different sensitivities.

[0029] FIG. 4 is a conceptual diagram illustrating an image signal of an imaging display device. The horizontal axis of the conceptual diagram in FIG. 4 represents the amount of light incident on the image sensor, and the vertical axis of the conceptual diagram in FIG. 4 represents the magnitude of the signal output by the image sensor. As described above, the conversion gain G1 of the imaging unit 10 is higher than the conversion gain G2 of the imaging unit 20. To reduce the conversion gain G2, the additional capacitance transistor AP in FIG. 3(b) is turned on, and the FD capacitance value of the unit cell 301 in the imaging unit 20 is increased to Afd+Cfd. The relationship between the charge amount Q, capacitance value C, and voltage value V is Q=CV. In other words, for the same amount of light (i.e., the same amount of charge), a larger potential change may occur in the floating diffusion layer FD of the unit cell 301 of the imaging unit 10 than in the floating diffusion layer FD of the unit cell 301 of the imaging unit 20. Therefore, assuming that the noise levels of the readout path after the readout circuit 330 are equivalent, the signal amount S1, which is the overall noise level of the image capturing unit 10, is smaller than the signal amount S2, which is the overall noise level of the image capturing unit 20. In other words, in light amount range A, which is low, the image capturing unit 10 can be said to have better image quality than the image capturing unit 20. On the other hand, the light amount (saturation light amount) at which the signal amount S3, which is the saturation level of the image capturing unit, is reached is light amount L3 for the image capturing unit 10, which is smaller than light amount L4 for the image capturing unit 20. Therefore, the image capturing unit 20 has better image quality in light amount range C, which is high. In light amount range B, which is medium, both the image capturing unit 10 and the image capturing unit 20 can obtain good image quality.

[0030] The image processing performed in the signal processing unit 30 in Fig. 2 is, for example, dynamic range expansion processing (hereinafter referred to as HDR processing). In HDR processing, a plurality of image signals acquired under a plurality of imaging conditions with different light intensity ranges are used to generate an image signal with a wide dynamic range. The light intensity range in which the imaging unit 10 can acquire an image signal is range R1, and the light intensity range in which the imaging unit 20 can acquire an image signal is range R2. By combining these plurality of image signals with different light intensity ranges, an image signal corresponding to range R3 from light intensity L1 to light intensity L4 can be generated.

[0031] In HDR processing, when image signals are switched, a difference in the signal-to-noise ratio can occur. Therefore, it is advisable to combine multiple image signals in light intensity range B, where image signals can be acquired from both of the multiple image capture units. In this case, signal processing can be performed to combine the image signals from the two image capture units at a ratio of, for example, α:(1-α), where α is between 0 and 1. Here, α does not necessarily have to be a constant value and can be varied within light intensity range B. For example, α can be set to a value greater than 0.5 on the side closer to light intensity range A and less than 0.5 on the side closer to light intensity range C, so as to minimize the appearance of unnaturalness in the image after HDR processing.

[0032] When image signals are acquired using multiple image capture units, slight differences in brightness, color, white balance, etc., due to variations in the image signals may cause discomfort to the user. Correction processing may be performed by the signal processor 30 using signals in the light intensity range B in which image signals can be acquired by the multiple image capture units.

[0033] Furthermore, when a good image can be obtained using only the image signal from one of the multiple imaging units, such as in light intensity range A or light intensity range C, power consumption can be reduced by temporarily suspending signal readout from the same area of ​​the other imaging unit.

[0034] An image with a wide dynamic range can be obtained by appropriately processing image signals acquired by multiple image capturing units with different conversion gains in the signal processing unit 30. That is, the image capturing and displaying device of this embodiment can display good images with little sense of incongruity, free from crushed shadows and blown-out highlights, on multiple display units.

[0035] As described above, the method for switching the conversion gain is not limited to switching the capacitance value of the floating diffusion capacitance. The imaging units 10 and 20 may each have an amplifier that amplifies a signal from a unit cell. In this case, the amplifier that amplifies a signal from the unit cell of the imaging unit 10 has a higher gain than the amplifier that amplifies a signal from the unit cell of the imaging unit 20. This configuration allows the conversion gain to be switched.

[0036] (Second embodiment) The imaging display device according to this embodiment differs from the first embodiment in that the imaging unit 1 is a SPAD (Single Photon Avalanche Diode) image sensor rather than a CMOS type. In the following description, the contents described in the first embodiment will not be described.

[0037] FIG. 5 is a circuit diagram showing a unit cell 301S included in the imaging unit 10. The unit cell 301S includes an avalanche photodiode APD, a quench resistor QR, a buffer Buf, a control unit DRV, and a counter CT. A reverse bias voltage of potential HVDD is applied to the avalanche photodiode APD via the quench resistor QR. The potential HVDD is set to a voltage equal to or greater than the breakdown voltage to drive the avalanche photodiode APD in Geiger mode. The output of the buffer Buf is input to the counter CT in the control unit DRV. The image signal counted by the counter CT is read out via a column signal line 302. The readout circuitry and other components subsequent to the column signal line 302 can be operated by appropriately modifying FIG. 3(a).

[0038] Here, we will briefly explain the operation of the unit cell 301S when photons are incident using Figure 6. First, Figure 6(a) shows the current-voltage characteristics (IV characteristics) of the avalanche photodiode APD. As described above, the avalanche photodiode APD operates in Geiger mode. When photons are incident on the avalanche photodiode APD, a large current (photocurrent) flows due to avalanche amplification (operation A). At the same time as this current flows, the reverse bias voltage drops due to the quench resistor QR. The reverse bias voltage applied to the avalanche photodiode APD falls below the breakdown voltage, and avalanche amplification stops (operation B). When avalanche amplification stops, the cathode of the avalanche photodiode APD is charged again by the potential HVDD, returning to Geiger mode (operation C). The voltage change at the buffer input terminal due to operations A to C is pulse-shaped by the buffer Buf and measured by the counter CT. By repeating this process, it is possible to count the number of photons incident on the avalanche photodiode APD.

[0039] Figure 6(b) is a schematic diagram showing the relationship between the pulse waveform of the output voltage due to avalanche amplification output from the avalanche photodiode APD when photons are incident, and the decision threshold Vth used to determine the incidence of photons, with the horizontal axis representing time. Assume that a potential HVDD is supplied to the avalanche photodiode APD. At this time, photon A (time t1), photon B (time t2), and photon C (time t3) incident on the avalanche photodiode APD cause avalanche amplification to the extent that a pulse waveform that changes beyond the decision threshold Vth of the counter is output. The number of incident photons can be measured by counting the pulse waveforms.

[0040] The imaging unit 10 includes a unit cell 301S shown in FIG. 5, and the imaging unit 20 includes a unit cell 301 shown in FIG. 3(b). The imaging unit 20 may be a CMOS-type image sensor. The FD capacitance value of the unit cell 301 shown in FIG. 3(b) may be Cfd alone, or the additional capacitance transistor AP may be omitted. These configurations are easily modified. Unlike the unit cell 301, the unit cell 301S does not include a reset transistor RS or an amplification transistor SF. Therefore, the imaging unit 10 does not generate kTC noise or RTS noise due to the individual transistors, and has a superior signal-to-noise ratio (S / N ratio) compared to the imaging unit 20. This makes the imaging unit 20 suitable for capturing still images, where random noise has a significant impact on image quality.

[0041] The imaging unit 10, which is composed of a SPAD-type image sensor, can acquire high-quality image signals with a high conversion gain even under low illumination conditions with a small number of incident photons. However, under high illumination conditions with a large number of incident photons, a problem known as a count error can occur. Figure 7 illustrates a case in which a count error occurs in a SPAD-type image sensor. Figure 7(a) is a schematic diagram showing the relationship between the pulse waveform of the output voltage due to avalanche amplification when a photon is incident and the decision threshold Vth, with the horizontal axis representing time. Unlike the behavior described in Figure 6(b), after a voltage fluctuation exceeding the decision threshold Vth due to photon D (time t4) is input, photon E (time t5) is incident before the avalanche amplification operation in operation B described in Figure 6(a) stops. At this time, photon E is incident at time t5 before the waveform fluctuation due to avalanche amplification that occurred at time t4 exceeds the decision threshold Vth, preventing the count operation for photon E. Furthermore, when photon F (time t6) is incident, the same state occurs as between time t4 and time t5, so photon F is also not counted. In this way, when the brightness of the subject is high, photons are incident continuously before exceeding the judgment threshold Vth, so the count value is smaller than the number of photons actually incident, resulting in a count error (count saturation). Note that, between time t6 and time t7, no photons are incident and the voltage temporarily exceeds the judgment threshold Vth, so the voltage pulse waveform for photon G (time t7) that is incident thereafter is counted.

[0042] Figure 7(b) shows the relationship between illuminance and count value in a SPAD-type image sensor. As the illuminance increases, the number of photons increases, and so the count value counted by the SPAD-type image sensor also increases proportionally. However, when the illuminance exceeds M, the state from time t4 to time t6 in Figure 7(a) occurs, and a count error (count saturation) occurs. As the illuminance increases further, the number of simultaneously incident photons increases even more, and the count error state continues, so the actual count value (solid line) is inversely proportional to the ideal count value (dashed line) at illuminance N.

[0043] In the image capturing and displaying device 100 of this embodiment, image signals acquired by the image capturing unit 10 and the image capturing unit 20, which have different conversion gains and operating modes, are processed by the signal processing unit 30, thereby enabling high-quality images to be displayed on multiple display units. FIG. 8 is a conceptual diagram illustrating the image signals of the image capturing and displaying device 100 of this embodiment. The horizontal axis of the conceptual diagram of FIG. 8 represents the amount of light incident on the image sensor, and the vertical axis of the conceptual diagram of FIG. 8 represents the magnitude of the signal output by the image sensor. The image capturing unit 20 corresponds to a light intensity range that includes a higher amount of light than the image capturing unit 10. The image capturing unit 10 corresponds to light intensity range R1, and the image capturing unit 20 corresponds to light intensity range R2. The image capturing unit 10 can output a signal corresponding to light in light intensity range R1. The image capturing unit 20 can output a signal corresponding to light in light intensity range R2. Light intensity range R2 includes a higher amount of light than light intensity range R1. In this embodiment, different types of image sensors are used for the image capturing unit 10 and the image capturing unit 20 to change the corresponding light intensity ranges.

[0044] The imaging unit 10 can handle low-brightness images ranging from light intensity L1 to light intensity L3. In the imaging unit 20, signals with light intensities smaller than light intensity L2 are buried in signal S2, which is a noise signal, and cannot be read out. On the other hand, signals with light intensities larger than light intensity L3 can be read out by the imaging unit 20. By using such imaging units 10 and 20, it is possible to obtain image signals corresponding to a wide range of light intensities.

[0045] The image processing performed by the signal processing unit 30 is, for example, dynamic range expansion processing (hereinafter referred to as HDR processing). In HDR processing, a plurality of image signals acquired under a plurality of imaging conditions with different light intensity ranges are used to generate an image signal with a wide dynamic range. The light intensity range in which the imaging unit 10 can acquire an image signal is range R1, and the light intensity range in which the imaging unit 20 can acquire an image signal is range R2. By combining these plurality of image signals with different light intensity ranges, an image signal corresponding to a light intensity range R3 from light intensity L1 to light intensity L4 can be generated.

[0046] While the imaging unit 10 can obtain high-quality image signals even in low illuminance, a large current due to avalanche amplification can result in large power consumption, so power saving may be achieved by temporarily pausing signal readout in high illuminance areas in light intensity range C shown in Fig. 8. Furthermore, the light intensity range in which the imaging unit 10 is paused may be expanded to include part or all of light intensity range B in addition to light intensity range C.

[0047] In this embodiment, the imaging unit 10 is a SPAD type image sensor, but other types of image sensors can also be used.

[0048] (Third embodiment) In this embodiment, a method of operating an imaging and display device will be described. In the imaging and display device 100 according to the first and second embodiments, the signal processing unit 30 can expand the dynamic range of image signals displayed on multiple display units. Here, the brightness that causes discomfort to different users may vary from person to person. The imaging and display device 100 has an input unit 70 as shown in FIG. 2. The user can adjust the brightness by operating the input unit 70. By inputting an operation signal from the input unit 70 to the signal processing unit 30, the width of the dynamic range used by the signal processing unit 30 can be adjusted within a range in which the imaging and display device 100 can operate correctly. Furthermore, the brightness of the display units can be adjusted by the operation signal from the input unit 70.

[0049] The operation of the input unit 70 is not limited to a button, switch, touch panel, or the like. It may also be gesture input, which detects the movement of an indicator (motion detection) based on the image signals of at least one of the imaging units 10 and 20 and accepts an operation based on the detected movement of the indicator. Furthermore, the input unit 70 may have a sound detection unit, and the operation of the input unit 70 may be voice input, which is performed by detecting sound. Furthermore, the input unit 70 may have a gaze detection unit that detects the user's gaze and pupils, and the operation of the input unit 70 may be gaze input by detecting the movement of the gaze or blink input by detecting the number of blinks. Furthermore, an operation such as adjusting the brightness of the display image described above may be performed by detecting a change in pupil size. To prevent malfunction, a combination of the above input methods may be used. For example, a gesture can be activated by pressing a button while making a gesture. By enabling the above operations, an imaging and display device that is more personalized to the user and less uncomfortable can be provided.

[0050] (Fourth embodiment) In this embodiment, the display of the imaging and display device 100 will be described. The imaging and display device 100 described above can position display units corresponding to the left and right eyes of a user. When a person views an object, parallax occurs because the left and right eyes are physically separated. This parallax allows the user to perceive the object as three-dimensional and as having depth relative to other objects and the background. Therefore, if the same image signal is displayed on multiple display units in an HMD or smart glasses, the user may not perceive the three-dimensionality or depth, which can cause discomfort. However, according to the imaging and display device described above, by arranging multiple imaging units in physically separate locations, images corresponding to the parallax of both eyes can be acquired by the multiple imaging units and image signals corresponding to different parallaxes can be displayed on the multiple display units. Specifically, when the distance between the imaging unit 10 and the display unit 50 is equal to the distance between the imaging unit 20 and the display unit 60, an image signal is generated using the distance between the imaging unit 10 and the display unit 50. The distance between the imaging unit 10 and the display unit 50 can be used as so-called parallax information. For example, the distance between the imaging unit 10 and the display unit 50 is the distance between the optical center of the imaging unit 10 and the optical center of the display unit 50 .

[0051] FIG. 9 is an operational flowchart illustrating the operation of the imaging and display device 100. First, the imaging units 10 and 20 each acquire an image signal. The image signals are then processed by the imaging signal processing units 15 and 25 and the signal processing unit 30 shown in FIG. 2. In this processing, corrections are made to output signals within a common light intensity range to correct for variations in brightness and color. The image signals are then combined to generate an image signal with an expanded dynamic range. The image signal with the expanded dynamic range is then redistributed based on the above-described parallax information to generate display images tailored to the respective display units 50 and 60.

[0052] Furthermore, when only one image signal from multiple imaging units is used as an image signal to be displayed on multiple display units, as in light intensity range A and light intensity range C shown in FIG. 4, displaying the same image signal on multiple display units can cause discomfort, as described above. Therefore, adding pseudo-parallax using the signal processing unit 30 and displaying different image signals corresponding to the parallax on the multiple display units can reduce the discomfort felt by the user. When adding this parallax, parallax information can be obtained based on unused image signals. The parallax information may be obtained using image signals that can be acquired from multiple imaging units, as in light intensity range B, or the signal processing unit 30 may recognize an object based on deep learning or the like to add pseudo-parallax.

[0053] In addition to multiple image capture units, a ToF (Time of Flight) sensor or LiDAR may be used to measure distance. Measuring the distance to the subject allows for accurate calculation of parallax information to be added to image signals displayed on multiple display systems, making it less likely for the user to feel uncomfortable. Furthermore, by using a gaze detection sensor or the like, it is possible to achieve a more three-dimensional effect by lowering the resolution of areas other than the point the user is focusing on, making it less likely for the user to feel uncomfortable and also reducing the power consumption of the image capture units and display units.

[0054] (Fifth embodiment) The imaging and display device of this embodiment will be described with reference to FIGS. 10(a) and 9. In the imaging and display device of this embodiment, imaging units 10A and 10B are arranged in portions corresponding to the left and right lenses as shown in FIG. 10(a). Another imaging unit 20 is arranged between imaging units 10A and 10B. This embodiment differs from the first embodiment in that the imaging unit 10 of the first embodiment is divided into two and the imaging unit 20 of the first embodiment is positioned between them. In the following description, the contents described in the previous embodiment will be omitted.

[0055] To display images that do not cause discomfort to the wearer, it is desirable to add appropriate parallax information to each display image on the multiple display units 50 and 60. However, the previous embodiment used a method for estimating left and right parallax information using imaging units 10 and 20 with different configurations. In this case, because imaging units with different configurations are used, there may be significant variations in brightness and color balance in the imaging results, even when the same subject is being imaged. This may result in a sense of discomfort in the displayed image. Therefore, as shown in FIG. 10(a), imaging units 10A and 10B with identical configurations are arranged approximately symmetrically on the left and right as imaging units for estimating parallax information. This configuration suppresses variations and makes it less likely that insufficient parallax information will occur.

[0056] The processing flow of the image signals is the same as the processing flow described in Fig. 9. That is, the image signals and disparity information acquired by the imaging units 10A, 10B, and 20 are processed by the imaging signal processing units 15 and 25 and the signal processing unit 30, thereby generating display images suitable for the respective display units 50 and 60. Since more base image signals can be acquired than in the previous embodiment, the accuracy of the variation correction process and the generation of disparity information is further improved, and the sense of discomfort felt by the user can be further reduced.

[0057] (Sixth embodiment) The imaging and display device of this embodiment will be described with reference to Fig. 10(b) and Fig. 9. In the imaging and display device of this embodiment, as shown in Fig. 10(b), a set of imaging units 10A and 20A is arranged in one lens, and a set of imaging units 10B and 20B is arranged in the other lens. This embodiment differs from the first embodiment in that the imaging unit 10 of the first embodiment is divided into two and the imaging unit 20 of the first embodiment is divided into two. In the following description, the contents described in the previous embodiment will be omitted.

[0058] In order to display an image that does not cause discomfort to the wearer, it is desirable to add appropriate parallax information to each display image of the multiple display units 50, 60. Therefore, similar to the fifth embodiment, in this embodiment, the imaging units 10A and 10B having the same configuration are arranged in a generally symmetrical manner on the left and right as shown in FIG. 10(b). Furthermore, the imaging units 20A and 20B having the same configuration are arranged in a generally symmetrical manner on the left and right as shown in FIG. 10(b). In this embodiment, at least one of the pair of imaging units 10A and 10B and the pair of imaging units 20A and 20B is used as the imaging unit for estimating the parallax information. This configuration suppresses variations and makes it less likely that a lack of parallax information will occur.

[0059] The image signal processing flow is the same as the processing flow described in Fig. 9. That is, the image signals acquired by the imaging units 10A, 10B, 20A, and 20B are processed by the imaging signal processing units 15 and 25 and the signal processing unit 30, thereby generating display images suitable for the respective display units 50 and 60. Since more base image signals can be acquired than in the previous embodiment, the accuracy of the variation correction process and the generation of parallax information is further improved, and the sense of discomfort felt by the user can be further reduced.

[0060] In FIG. 10(b), the imaging units 10A and 20A are arranged at different positions in a plan view. The imaging units 10B and 20B are also arranged at different positions in a plan view. However, the imaging units of the pair of imaging units 10A and 20A, or the pair of imaging units 10B and 20B, can be formed within the same chip for a common optical system, for example. That is, the imaging units 10A and 20A can be stacked within the same chip, or can be combined in a checkerboard pattern, for example. The imaging units 10B and 20B can be stacked within the same chip, or can be combined in a checkerboard pattern, for example.

[0061] (Seventh embodiment) The imaging and display device of this embodiment will be described with reference to Fig. 10(c) and Fig. 11. In the imaging and display device of this embodiment, as shown in Fig. 10(c), an imaging unit 10 is arranged in one lens, an imaging unit 20 is arranged in the other lens, and a distance measurement optical system 40 is arranged between them. This embodiment differs from the first embodiment in that the distance measurement optical system 40 is provided between the imaging units 10 and 20 of the first embodiment. In the following description, the contents described in the previous embodiment will be omitted.

[0062] An imaging and display device according to embodiment E will be described with reference to Fig. 10(c) and Fig. B1. The imaging and display device according to this embodiment differs from the first embodiment in that it additionally includes a distance measurement optical system 40. In the following description, the details described in the previous embodiment will not be repeated.

[0063] As mentioned above, in order to display images that do not feel strange to the wearer, it is desirable to add appropriate parallax information to each display image on the multiple display units 50, 60. However, in the above embodiment, left and right parallax information is estimated using deep learning or other methods based on image signals and an estimate of the distance to the subject, which raises concerns that the result may differ from actual distance information or that the accuracy may be low. Therefore, as shown in FIG. 10(c), a distance measurement optical system 40 is provided to accurately measure distance information. This configuration can improve the accuracy of the parallax information used to redistribute image signals to the display units 50, 60.

[0064] For example, an autofocus sensor module can be used for the distance measurement optical system 40. Alternatively, a laser light source for irradiating a semiconductor laser can be arranged in the distance measurement optical system 40, and distance measurement can be performed using a Time of Flight method (hereinafter referred to as the ToF method) or the like in at least one of the image capture units 10 and 20. Furthermore, a technology called image capture surface phase difference autofocus can be introduced into the pixels of the image capture units 10 and 20, and used in combination with phase difference information acquired by the image capture units 10 and 20.

[0065] The image signal processing flow is shown in FIG. 11. The imaging units 10 and 20 each acquire an image signal. The image signal is then processed by the imaging signal processing units 15 and 25 and the signal processing unit 30, as in the other embodiments. Image information is then generated. In parallel, the signal processing unit 30 processes the signal obtained from the distance measurement optical system 40 to generate distance information. The distance measurement optical system 40 can acquire a distance measurement signal, i.e., a distance signal. This distance information is then used to generate display parallax information. The signal processing unit 30 redistributes the image information and display parallax information according to each display unit 50 and 60, generating a display image. Since more accurate distance information can be acquired than in the previous embodiment, the accuracy of the variation correction process and the generation of parallax information is further improved, making it less likely that the user will experience discomfort.

[0066] In this embodiment, by acquiring parallax information with higher accuracy, it is possible to provide a display image that reduces the sense of discomfort felt by the user.

[0067] (Eighth embodiment) The imaging and display device of this embodiment will be described with reference to FIGS. 10(d) and 12. In the imaging and display device of this embodiment, as shown in FIG. 10(d), a pair of imaging units 10A and 20A is arranged in one lens, and a pair of imaging units 10B and 20B is arranged in the other lens. In the imaging and display device of this embodiment, a distance measurement optical system 40 is provided between the two lenses. This embodiment differs from the first embodiment in that the imaging unit 10 of the first embodiment is divided into two and the imaging unit 20 of the first embodiment is divided into two and has a distance measurement optical system 40. In the following description, the content described in the previous embodiment will be omitted.

[0068] For example, the distance measurement optical system 40 in this embodiment is a laser light source for emitting a semiconductor laser. Then, distance measurement using a ToF method or the like is performed in at least one of the image capture units 10 and 20. A technology called image capture plane phase contrast can also be introduced into the pixels of the image capture unit 10 or 20. Image capture plane phase contrast is also called image capture plane phase contrast autofocus. In this embodiment, phase difference information acquired by the ToF method and that acquired by the image capture plane phase contrast method are used in combination. For example, a SPAD sensor is used as the image capture unit 10, and the image capture unit 10 acquires distance information using the ToF method. Then, a CMOS sensor is used as the image capture unit 20, and the image capture unit 20 acquires distance information using the image capture plane phase contrast method. In this embodiment, distance information can be acquired from each of the left and right image capture units, thereby enabling more accurate parallax information to be acquired compared to the previous example. Therefore, the configuration of this embodiment makes it possible to generate a display image with highly accurate parallax information added.

[0069] The signal flow of the image signal is shown in FIG. 12. The imaging units 10 and 20 each acquire an image signal. Then, the imaging units 10 and 20 each acquire a distance signal. The image signal undergoes the same processing as in other embodiments in the imaging signal processing units 15 and 25 and the signal processing unit 30, generating image information. In parallel, the distance signal is processed in the imaging signal processing units 15 and 25 and the signal processing unit 30 to generate distance information. Display parallax information is further generated using the distance information. The signal processing unit 30 redistributes the image information and display parallax information according to each display unit 50 and 60 to generate a display image. Since more accurate distance information can be acquired than in the previous embodiment, the accuracy of the variation correction process and the generation of parallax information is further improved. This makes it possible to provide a display image that further reduces the sense of discomfort felt by the user.

[0070] It is desirable to appropriately correct the distance information, since there may be misalignment of the image capture unit or display unit, or the user may be wearing the device at an angle. Also, it is desirable to make the parallax information adjustable as appropriate, since the degree of discomfort experienced by users varies.

[0071] In each embodiment, the imaging and display device has been described as having two imaging units and two display units, but the present invention also includes a case where the imaging and display device has three or more imaging units and three or more display units. When three or more imaging units are used, for example, a display image with a three-dimensional effect can be generated. Furthermore, while the present invention has been described with respect to visible light, the present invention is applicable to any wavelength. Furthermore, in each embodiment, an example has been given in which up to two imaging units are used on each side, but more than two imaging units may be used. The number of imaging units may be determined based on the balance with the load on signal processing. [Explanation of symbols]

[0072] 10. Imaging unit 20 Imaging unit 30 Signal Processing Unit 50 Display section 60 Display section 100 Image capture and display device

Claims

1. An imaging and display device having a plurality of imaging units, a plurality of display units, and a signal processing unit, the plurality of imaging units include a first imaging unit and a second imaging unit, the first imaging unit has a plurality of first unit cells, each of the plurality of first unit cells includes an avalanche photodiode; the second imaging unit has a plurality of second unit cells, each of the plurality of second unit cells includes a photodiode and a transistor for reading out a signal based on a charge of the photodiode; the signal processing unit generates a first image signal based on signals from the first imaging unit and the second imaging unit; The imaging and display device, wherein the plurality of display sections display at least an image based on the first image signal.

2. the plurality of display units include a first display unit and a second display unit, a distance between the first imaging unit and the first display unit is equal to a distance between the second imaging unit and the second display unit; 2. The image capturing and displaying device according to claim 1, wherein the first image signal is generated using a distance between the first image capturing section and the first display section.

3. The imaging display device according to claim 1 or 2, characterized in that the signal processing unit performs at least one of a process of detecting a moving object based on a signal from at least one of the first imaging unit or the second imaging unit, and a process of adjusting the brightness of the multiple display units.

4. The device has at least one of a voice detection unit that detects the voice of an operator and a gaze detection unit that detects the gaze of the operator, and an input unit that outputs an operation signal to the signal processing unit, 4. The image pickup display device according to claim 1, wherein the signal processing unit generates the first image signal in response to the operation signal.

5. the plurality of display units include a first display unit and a second display unit, 5. The imaging display device according to claim 1, wherein the signal processing unit generates the first image signal for the first display unit and another first image signal for the second display unit in accordance with a distance signal from the first imaging unit and the second imaging unit to a subject.

6. the imaging and display device has a light source including a semiconductor laser, 6. The image capturing and displaying device according to claim 1, wherein the signal processing unit performs distance measurement processing using a time-of-flight method to calculate the distance to the subject based on the light emitted from the light source.

7. 7. The image capturing and displaying device according to claim 1, wherein the second image capturing section performs a global shutter operation.

8. 8. The image pickup display device according to claim 1, wherein the first image pickup section and the second image pickup section are arranged in a checkerboard pattern.

9. An imaging display device as described in any one of claims 1 to 8, characterized in that the signal processing unit suspends operation of the first imaging unit in a high-brightness area.

10. The imaging and display device according to any one of claims 1 to 9, A wearable device having a housing in which the imaging and display device is placed.

11. The housing has a first surface and a second surface opposite the first surface, the first imaging unit is disposed on the first surface side, The wearable device according to claim 10 , wherein the plurality of display units are arranged on the second surface side.

Images