Signal generation device and signal generation system

Abstract

To provide a signal generation device capable of ensuring appropriate distance resolution without increasing the distance measurement time.SOLUTION: A signal generation device includes a plurality of photoelectric conversion elements and generates a plurality of subframes by using signals from the photoelectric conversion elements. A distance measurement frame is generated by using the signals from the plurality of subframes. The plurality of subframes includes at least: a first subframe in which the exposure period starts after a first period from the light emission from a light-emitting element; and a second subframe in which the exposure period starts after a second period longer than the first period from the light emission from the light-emitting element. The first subframe includes at least a first exposure period and a second exposure period. The timing at which the first exposure period starts differs from the timing at which the second exposure period starts.SELECTED DRAWING: Figure 9

Description

Technical Field

[0001] The present invention relates to a signal generation device and a signal generation system.

Background Art

[0002] Patent Document 1 discloses a distance measuring device that measures the distance to an object by emitting light from a light source and receiving the light including the reflected light from the object with a light receiving element. In the distance measuring device of Patent Document 1, a SPAD (Single Photon Avalanche Diode) element that multiplies electrons generated by photoelectric conversion to acquire a signal is used as the light receiving element. Patent Document 1 discloses a distance measuring method in which measurement is repeatedly performed while changing the start timing of the exposure period (Gating interval) for each sub-frame in which photon detection is performed in the SPAD element.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the distance measuring method as disclosed in Patent Document 1, there is a trade-off relationship between the distance resolution and the shortening of the distance measuring time. That is, when the exposure period in the sub-frame is lengthened, the distance measuring time is shortened, but the distance resolution is decreased. Conversely, when the exposure period in the sub-frame is shortened, the distance resolution is improved, but the distance measuring time is lengthened. However, in order to improve the distance measuring performance, it may be required to ensure an appropriate distance resolution without lengthening the distance measuring time.

[0005] Therefore, an object of the present invention is to provide a signal generation device that ensures an appropriate distance resolution without lengthening the distance measuring time.

Means for Solving the Problem

[0006] A signal generation device according to one aspect of the present invention has a plurality of photoelectric conversion elements, and is a signal generation device that generates a plurality of sub-frames using signals from the plurality of photoelectric conversion elements, and generates a ranging frame using the signals of the plurality of sub-frames. The plurality of sub-frames include at least a first sub-frame that starts an exposure period after a first period from the light emission from the light-emitting element, and a second sub-frame that starts an exposure period after a second period longer than the first period from the light emission from the light-emitting element. The first sub-frame has at least a first exposure period and a second exposure period, and the timing at which the first exposure period starts is different from the timing at which the second exposure period starts.

Effect of the Invention

[0007] According to the present invention, a signal generation device that ensures an appropriate distance resolution without increasing the ranging time is provided.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Embodiments for Carrying Out the Invention

[0009] The following embodiments are for embodying the technical idea of the present invention and do not limit the present invention. The sizes and positional relationships of the members shown in each drawing may be exaggerated for clarity of explanation. In the following description, the same configuration may be denoted by the same number and the description thereof may be omitted.

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the following description, terms indicating a specific direction or position (for example, "up", "down", "right", "left", and other terms including those terms) are used as necessary. The use of those terms is for facilitating the understanding of the embodiments with reference to the drawings, and the technical scope of the present invention is not limited by the meanings of those terms.

[0011] In this specification, a plane refers to a plane viewed from a direction perpendicular to the light incident surface of the semiconductor layer. Also, a cross-section refers to a plane in a direction perpendicular to the light incident surface of the semiconductor layer. When the light incident surface of the semiconductor layer is a rough surface when viewed microscopically, the plane is defined based on the light incident surface of the semiconductor layer when viewed macroscopically.

[0012] In the following description, the anode of an avalanche photodiode (APD) is set to a fixed potential, and the signal is taken out from the cathode side. Therefore, the semiconductor region of the first conductivity type having charges of the same polarity as the signal charge as majority carriers is an N-type semiconductor region, and the semiconductor region of the second conductivity type having charges of a polarity different from the signal charge as majority carriers is a P-type semiconductor region. Note that the present invention is also applicable when the cathode of the APD is set to a fixed potential and the signal is taken out from the anode side. In this case, the semiconductor region of the first conductivity type having charges of the same polarity as the signal charge as majority carriers is a P-type semiconductor region, and the semiconductor region of the second conductivity type having charges of a polarity different from the signal charge as majority carriers is an N-type semiconductor region. Hereinafter, the case where one node of the APD is set to a fixed potential will be described, but the potentials of both nodes may vary.

[0013] In the following embodiments, the connection between the elements of the circuit may be described. In this case, even if another element is interposed between the elements of interest, unless otherwise specified, the elements of interest are treated as being connected. For example, assume that element A is connected to one node of a capacitive element C having a plurality of nodes, and element B is connected to the other node. Even in such a case, element A and element B are treated as being connected unless otherwise specified.

[0014] FIG. 1 is a hardware block diagram showing a schematic configuration example of a distance information generation device 30 according to an embodiment. The distance information generation device 30 includes a light emitting device 31, a signal generation device 32, and a signal processing circuit 33. The signal generation device 32 may include a photoelectric conversion device 100 and a signal processing circuit 33. Note that the configuration of the distance information generation device 30 shown in FIG. 1 is an example, and is not limited to the illustrated configuration. For example, the first frame generation unit 37 and the second frame generation unit 38 included in the signal processing circuit 33 may be included in the photoelectric conversion device 100.

[0015] The distance information generation device 30 is a device that measures the distance to the object X to be measured using technologies such as LiDAR (Light Detection And Ranging). The distance information generation device 30 measures the distance from the distance information generation device 30 to the object X based on the time difference from when the light emitted from the light emitting device 31 is reflected by the object X until it is received by the photoelectric conversion device 100. Further, the distance information generation device 30 can measure distances at a plurality of points in a two-dimensional manner by emitting laser light into a predetermined distance measurement range including the object X and receiving the reflected light by a pixel array. Thereby, the distance information generation device 30 can output distance information. Also, it may output image information based on the distance information (image information in which color differences and contrast differences are provided according to the distance information).

[0016] The light received by the signal generation device 32 includes ambient light such as sunlight in addition to the reflected light from the object X. Therefore, the distance information generation device 30 measures the amount of light incident in each of a plurality of periods (bin periods), and performs distance measurement with reduced influence of ambient light using a method of determining that the reflected light is incident in the period when the light amount peaks.

[0017] The light emitting device 31 is a device that emits light such as laser light outside the distance information generation device 30. For example, the laser light may use a VCSEL (Vertical Cavity Surface Emitting Laser) that is easy to form into a two-dimensional array.

[0018] The signal processing circuit 33 may include a processor that performs arithmetic processing of digital signals, a memory that stores digital signals, and the like. As the memory, for example, a semiconductor memory can be used. Note that the distance information generation device 30 may not have the signal processing circuit 33. In this case, at least a part of the configuration included in the signal processing circuit 33 is arranged inside the photoelectric conversion device 100. In this case, the signal generation device 32 and the photoelectric conversion device 100 are the same.

[0019] The signal generation device 32 generates a pulse signal including pulses based on the incident light. In the present embodiment, the pulse signal is generated by the photoelectric conversion device 100 included in the signal generation device 32. For example, a photoelectric conversion device including an APD as a photoelectric conversion element can be used. In this case, when one photon is incident on the APD and charges are generated, one pulse is generated by avalanche multiplication. The photoelectric conversion device 100 included in the signal generation device 32 is not limited to the one using an APD as the photoelectric conversion element, and may be the one using other photodiode-based photoelectric conversion elements.

[0020] In the present embodiment, the photoelectric conversion device 100 includes a pixel array in which a plurality of photoelectric conversion elements (pixels) are arranged over a plurality of rows and a plurality of columns. Here, the photoelectric conversion device 100 will be described with reference to FIGS. 2 to 6. The configuration example of the photoelectric conversion device described below is an example. The photoelectric conversion device is not limited thereto, and any device capable of realizing the functions of each of the embodiments described later may be used.

[0021] FIG. 2 is a diagram showing the configuration of the stacked photoelectric conversion device 100 included in the distance information generation device 30. The photoelectric conversion device 100 is configured by laminating two substrates, a sensor substrate 11 (first substrate) and a circuit substrate 21 (second substrate), and electrically connecting them. The sensor substrate 11 has a first semiconductor layer having a photoelectric conversion unit 102 to be described later and a first wiring structure. The circuit substrate 21 has a second semiconductor layer having a signal detection circuit such as a signal processing unit 103 to be described later and a second wiring structure. The photoelectric conversion device 100 is configured by laminating the second semiconductor layer, the second wiring structure, the first wiring structure, and the first semiconductor layer in this order. In the photoelectric conversion device 100 described in each embodiment, light is incident from the side of the first surface of the first semiconductor layer of the sensor substrate 11, and the circuit substrate is disposed on the second surface of the first semiconductor layer of the sensor substrate 11 facing the first surface, which is a back-illuminated type photoelectric conversion device.

[0022] Hereinafter, the sensor substrate 11 and the circuit substrate 21 will be described in terms of diced chips, but are not limited to chips. For example, each substrate may be a wafer. Also, each substrate may be diced after being stacked in a wafer state, or each chip may be stacked and joined after being formed into chips from the wafer state.

[0023] The sensor substrate 11 is provided with a photoelectric conversion region 12 in which a plurality of photoelectric conversion elements are arranged in a two-dimensional array, and the circuit substrate 21 is provided with a circuit region 22 that processes signals detected in the photoelectric conversion region 12.

[0024] FIG. 3 is a diagram showing an arrangement example of the sensor substrate 11. The photoelectric conversion elements 101 having the photoelectric conversion unit 102 including APDs are arranged in a two-dimensional array in plan view to form the photoelectric conversion region 12.

[0025] The photoelectric conversion element 101 may be any as long as it can measure the arrival time and amount of light of light. For example, when used for TOF (Time of Flight), it is not necessarily required to form an image. However, the photoelectric conversion element 101 may be a pixel for forming an image.

[0026] FIG. 4 is a configuration diagram of the circuit substrate 21. It has a signal processing unit 103 that processes the charges photoelectrically converted by the photoelectric conversion unit 102 in FIG. 2, a readout circuit 112, a control pulse generation unit 115, a horizontal scanning circuit unit 111, signal lines 113, a vertical scanning circuit unit 110, an output circuit 114, and drive lines 116.

[0027] The photoelectric conversion unit 102 in FIG. 3 and the signal processing unit 103 in FIG. 4 are electrically connected via connection wirings provided for each photoelectric conversion element.

[0028] The vertical scanning circuit unit 110 receives the control pulse supplied from the control pulse generation unit 115 and supplies the control pulse to each photoelectric conversion element via the drive lines 116. Logic circuits such as a shift register and an address decoder are used for the vertical scanning circuit unit 110.

[0029] The signal output from the photoelectric conversion unit 102 of the photoelectric conversion element is processed by the signal processing unit 103. The signal processing unit 103 is provided with a counter, a memory, etc., and digital values are held in the memory.

[0030] The horizontal scanning circuit unit 111 inputs control pulses for sequentially selecting each column to the signal processing unit 103 in order to read signals from the memories of the photoelectric conversion elements holding digital signals.

[0031] Signals are output from the signal processing unit 103 of the photoelectric conversion elements selected by the vertical scanning circuit unit 110 to the signal line 113 for the selected columns.

[0032] The signals output to the signal line 113 are output to a recording unit or a signal processing unit outside the photoelectric conversion device 100 via the output circuit 114.

[0033] In FIG. 2, the arrangement of the photoelectric conversion units in the photoelectric conversion region may be arranged in a one-dimensional manner. The functions of the signal processing unit do not necessarily have to be provided one by one for all the photoelectric conversion units. For example, one signal processing unit may be shared by a plurality of photoelectric conversion units, and signal processing may be performed sequentially.

[0034] As shown in FIGS. 3 and 4, a plurality of signal processing units 103 are arranged in a region overlapping the photoelectric conversion region 12 in a plan view. And in a plan view, the vertical scanning circuit unit 110, the horizontal scanning circuit unit 111, the readout circuit 112, the output circuit 114, and the control pulse generation unit 115 are arranged so as to overlap between the end of the sensor substrate 11 and the end of the photoelectric conversion region 12. In other words, the sensor substrate 11 has the photoelectric conversion region 12 and a non-photoelectric conversion region arranged around the photoelectric conversion region 12. And in a region overlapping the non-photoelectric conversion region in a plan view, the vertical scanning circuit unit 110, the horizontal scanning circuit unit 111, the readout circuit 112, the output circuit 114, and the control pulse generation unit 115 are arranged.

[0035] FIG. 5 is an example of a block diagram including the equivalent circuits of FIGS. 3 and 4. FIG. 5 shows a block diagram of a photoelectric conversion element having a general APD.

[0036] In FIG. 5, the photoelectric conversion unit 102 having the APD 201 is provided on the sensor substrate 11, and the other members are provided on the circuit substrate 21.

[0037] The APD 201 generates charge pairs corresponding to incident light by photoelectric conversion. A voltage VL (first voltage) is supplied to the anode of the APD 201. Also, a voltage VH (second voltage) higher than the voltage VL supplied to the anode is supplied to the cathode of the APD 201. A reverse bias voltage is supplied to the anode and the cathode such that the APD 201 performs an avalanche multiplication operation. By supplying such a voltage, the charges generated by the incident light cause avalanche multiplication, and an avalanche current is generated.

[0038] Note that when a reverse bias voltage is supplied, there are a Geiger mode in which the potential difference between the anode and the cathode operates with a potential difference greater than the breakdown voltage, and a linear mode in which the potential difference between the anode and the cathode operates with a voltage difference near or below the breakdown voltage.

[0039] An APD operating in the Geiger mode is called a SPAD (Single Photon Avalanche Diode). For example, the voltage VL is -30 V and the voltage VH is 1 V. The APD 201 may operate in the linear mode or the Geiger mode.

[0040] The quench element 202 is connected to the power supply that supplies the voltage VH and the APD 201. The quench element 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, and has the function of suppressing the voltage supplied to the APD 201 to suppress avalanche multiplication (quench operation). Also, the quench element 202 has the function of returning the voltage supplied to the APD 201 to the voltage VH by flowing a current corresponding to the voltage drop during the quench operation (recharge operation).

[0041] In FIG. 5, the quench element 202 is constituted by a transistor, and a potential is supplied to the gate of the quench element via the drive line VR. The potential supplied from the drive line VR is a reset signal that resets the cathode potential nodeA of the APD 201 by switching the resistance value of the quench element 202.

[0042] The signal processing unit 103 includes a waveform shaping unit 210, a gate circuit 321 (first selection circuit), a counter 211, and an output circuit 306 (second selection circuit). In this specification, the signal processing unit 103 may include any one of the waveform shaping unit 210, the gate circuit 321, the counter 211, and the output circuit 306.

[0043] The waveform shaping unit 210 shapes the potential change of the cathode of the APD 201 obtained at the time of photon detection and outputs a pulse signal. As the waveform shaping unit 210, for example, an inverter circuit is used. In FIG. 5, an example in which one inverter is used as the waveform shaping unit 210 is shown, but a circuit in which a plurality of inverters are connected in series may also be used, or other circuits having a waveform shaping effect may also be used.

[0044] The gate circuit 321 can be constituted by, for example, an AND circuit. One input terminal of the AND circuit is connected to the waveform shaping unit 210, and the other input terminal is connected to the drive line GATE. The exposure period described later is set by controlling the signal supply to the gate circuit 321. The gate circuit 321 outputs the output signal of the waveform shaping unit 210 to the counter 211 during the period when the gate signal input from the outside of the photoelectric conversion element via the drive line GATE is at the High (H) level. On the other hand, the gate circuit 321 does not output the output signal of the waveform shaping unit 210 to the counter 211 during the period when the gate signal is at the Low (L) level. As the gate signal, for example, by inputting a pulse at the nanosecond to picosecond level during the High (H) level period, only the photon signal incident during the sub-frame period, which is the period of interest, can be selectively detected.

[0045] Counter 211 counts the number (times) of the pulse signals output from the waveform shaping unit 210 and holds the count value. Counter 211 measures the amount of light incident on the photoelectric conversion element. Counter 211 switches whether to perform the counting operation of Counter 211 or to stop the counting according to a control signal input from the outside of the photoelectric conversion element via the drive line CTRL. As an example of Counter 211, a multi-bit digital counter or the like can be mentioned, but a 1-bit digital memory, an analog memory using a capacitive element, or the like may also be used. Counter 211 and output circuit 306 are connected by a number of wirings corresponding to the number of bits of Counter 211. Also, when the control pulse pRES is supplied via the drive line RES, the signal held in Counter 211 is reset.

[0046] Output circuit 306 receives a selection signal input from the outside of the photoelectric conversion element via the drive line SEL, and outputs the signal output from counter 211 to signal line 113. In the present embodiment, a control pulse pSEL is supplied from the vertical scanning circuit unit 110 in FIG. 4 via the drive line SEL in FIG. 5 to switch the electrical connection and disconnection between counter 211 and signal line 113. Output circuit 306 includes, for example, a buffer circuit for outputting a signal, and for example, a 3-state buffer can be used.

[0047] A switch such as a transistor may be arranged between the quench element 202 and the APD 201, or between the photoelectric conversion unit 102 and the signal processing unit 103 to switch the electrical connection. Also, the supply of the voltage VH or the voltage VL supplied to the photoelectric conversion unit 102 may be electrically switched using a switch such as a transistor. Furthermore, without providing an additional switch, the above electrical connection may be switched by switching the voltage input to the gate of the transistor constituting the quench element 202.

[0048] FIG. 6 is a diagram schematically showing the relationship between the operation of the APD and the output signal.

[0049] FIG. 6(a) is a diagram excerpting the APD 201, quenching element 202, and waveform shaping unit 210 of FIG. 5. Here, the input side of the waveform shaping unit 210 is node A, and the output side is node B. FIG. 6(b) shows the waveform change of node A in FIG. 6(a), and FIG. 6(c) shows the waveform change of node B in FIG. 6(a).

[0050] Between time t0 and time t1, a potential difference of VH - VL is applied to the APD 201 in FIG. 6(a). When photons are incident on the APD 201 at time t1, avalanche multiplication occurs in the APD 201, an avalanche multiplication current flows through the quenching element 202, and the voltage at node A drops. When the voltage drop amount further increases and the potential difference applied to the APD 201 decreases, the avalanche multiplication of the APD 201 stops like at time t2, and the voltage level at node A no longer drops below a certain value. Thereafter, between time t2 and time t3, a current that compensates for the voltage drop flows from voltage VL to node A, and at time t3, node A stabilizes at the original potential level. At this time, the portion where the output waveform at node A exceeds a certain threshold is waveform-shaped by the waveform shaping unit 210 and output as a signal at node B.

[0051] Note that the arrangement of the signal line 113, the readout circuit 112, and the output circuit 114 is not limited to FIG. 5. For example, the signal line 113 may be arranged to extend in the row direction, and the readout circuit 112 may be arranged at the end where the signal line 113 extends.

[0052] (Embodiment 1) The photoelectric conversion device according to Embodiment 1 will be described with reference to FIGS. 9 and 10. Here, with FIGS. 7 and 8 as comparative forms for reference, the principle of ensuring an appropriate distance resolution without increasing the ranging time, which is an effect of the present invention, will be described using FIGS. 9 and 10.

[0053] FIG. 7 is a driving timing diagram according to a comparative form, and FIG. 8 is a graph showing the driving of gates per unit time in the driving according to the comparative form.

[0054] In FIG. 7, the emission light indicates the timing at which light from the light-emitting element is emitted. Further, the reflected light indicates the timing at which the light emitted from the light-emitting element irradiates an object and is reflected and detected by a photoelectric conversion device which is a signal detection device.

[0055] One distance measurement frame is generated using signals of N sub-frames (N is an integer of 2 or more) including a first sub-frame and a second sub-frame. In each sub-frame, the period of the H level is a period during which light from the light-emitting element can be detected. Also, in each sub-frame, the period of the L level is a period during which light from the light-emitting element cannot be detected. In the present embodiment, the period during which the signal is at the H level corresponds to an exposure period during which light can be detected by a plurality of photoelectric conversion elements arranged in the photoelectric conversion region, and the period during which the signal is at the L level corresponds to a non-exposure period during which light cannot be detected by the plurality of photoelectric conversion elements. The exposure period refers to, for example, a period during which the photoelectric conversion unit 102 is active and a signal from the photoelectric conversion unit 102 is read out by a signal detection circuit such as a counter circuit. Also, the non-exposure period refers to a period during which a signal from the photoelectric conversion unit 102 is not read out in a signal detection circuit such as a counter circuit.

[0056] For example, the exposure period is a period during which a gate signal input to the gate circuit via the drive line GATE is at the H level. Also, for example, the non-exposure period is a period during which a gate signal input to the gate circuit via the drive line GATE is at the L level. In the drive for resetting the quench element 202 by putting a periodic pulse into VR, the exposure period may be defined as the period from when VR switches to the H level until the gate signal switches to the L level.

[0057] The exposure period and the non-exposure period are not limited to the above examples. A reverse bias potential that can be multiplied by avalanche is applied to the APD, and the quenching element is in a non-quenching state. The period during which a signal detection circuit such as a counter can read the signal from the APD may be defined as the exposure period. And the period during which the quenching element is in the quenching state and the signal from the APD cannot be read through the quenching element may be defined as the non-exposure period. As another example, the period during which the potential difference applied to the APD is reduced so that avalanche multiplication does not occur in the APD may be defined as the non-exposure period, and the period during which a potential difference that allows avalanche multiplication in the APD is applied may be defined as the exposure period. Also, the period during which control is performed so that a signal detection circuit such as a counter is not driven may be defined as the non-exposure period, and the period during which control is performed so that a signal detection circuit such as a counter is driven may be defined as the exposure period.

[0058] The plurality of sub-frames are generated, for example, by the first frame generation unit 37 shown in FIG. 1, and the ranging frame is generated by the second frame generation unit 38. Note that the first frame generation unit 37 and the second frame generation unit 38 may be arranged in the photoelectric conversion device 100. In this case, for example, the first frame generation unit 37 and the second frame generation unit 38 are arranged in the readout circuit 112 shown in FIG. 4, and distance information and the like can be calculated based on the signal output from the APD. Also in this case, the frame timing generation can be performed by the control pulse generation unit 115.

[0059] Note that in FIG. 7, for easy understanding of the explanation, the sub-frames are arranged and shown from the emitted light and the reflected light. Although there are multiple timings of the emitted light, only one emission timing is described for each sub-frame in the drawing. Specifically, in actual driving, the first light quantity measurement of the first sub-frame is performed with the first emitted light. Also, the second light quantity measurement of the first sub-frame is performed with the second emitted light. The m-th (m is an integer of 2 or more) emitted light is used to measure the m-th light quantity of the first sub-frame, and the (m + 1)-th emitted light is used to measure the first light quantity of the second sub-frame. Thereafter, the light quantity measurement is similarly performed up to the N-th sub-frame.

[0060] As shown in FIG. 7, during the period of generating the first sub-frame, the light quantity measurement of the first sub-frame including multiple repeated emissions is performed without changing the timing from the emission to the start of the exposure period of the first sub-frame. Then, the light quantity measurement of the second sub-frame is performed. The period from the emission to the start of the exposure period in the light quantity measurement of the second sub-frame is set longer than the period from the emission to the start of the exposure period in the light quantity measurement of the first sub-frame. Similar to the light quantity measurement of the first sub-frame, the light quantity measurement of the second sub-frame including multiple repeated emissions is performed without changing the timing from the emission to the start of the exposure period of the second sub-frame. Then, the light quantity measurement of the Nth sub-frame is performed, and in the Nth sub-frame as well, the light quantity measurement of the sub-frame is performed multiple times without changing the timing from the emission to the start of the exposure period. Based on the light quantity measurement results of the multiple sub-frames including the first to Nth sub-frames, the histogram information of the reflected light is generated. The distance to the object is calculated based on the time information corresponding to the class with the largest number of photon detections (frequency).

[0061] In the comparative form, in the same sub-frame, the start timing and the end timing of the exposure period are detected multiple times in the same state, and in the next sub-frame, the relative timing of the start timing and the end timing of the exposure period is shifted with respect to the previous sub-frame.

[0062] Fig. 8(a) shows the gate profile in the comparative form and the residence time distribution in each sub-frame. The gate profile is a function that represents the temporal change in sensitivity to incident photons in one light quantity measurement of a predetermined sub-frame, and in the comparative form, it corresponds to the period during which the gate is on. In Fig. 8(a), for simplicity of explanation, the gate profile is represented by a rectangular function, but in reality, due to pulse delay and finite reset time in the sensor, there may be waveform droop, overshoot, ringing, etc. in the rise and fall. The residence time distribution is a distribution that represents how many times each light reception timing is integrated when light emission and light reception are repeatedly measured in a certain sub-frame. In the comparative form, in a sub-frame, the reflected light is measured and integrated multiple times with the period from light emission to the start timing of the exposure period being the same, resulting in a delta function.

[0063] Also, Fig. 8(b) shows the effective gate profile obtained by the convolution of the gate profile and the residence time distribution in the comparative form.

[0064] Next, the present embodiment will be described. Fig. 9 is a timing diagram of the drive according to the present embodiment, and Fig. 10 is a graph showing the drive of the gate per unit time in the drive according to the present embodiment.

[0065] In Fig. 9, since the objects shown by the emitted light, the reflected light, each sub-frame, and the histogram are the same as those in Fig. 7, the description thereof is omitted.

[0066] In FIG. 9, in the light quantity measurement of each sub-frame, the exposure periods of a plurality of array-shaped photoelectric conversion elements arranged in a plurality of matrices included in the signal generation device are simultaneously controlled. For example, in the photoelectric conversion region, the exposure periods of a plurality of array-shaped photoelectric conversion elements arranged in a matrix are simultaneously controlled. Specifically, in at least two or more of the plurality of photoelectric conversion elements, the first exposure period is simultaneously started, and then the second exposure period is simultaneously started. The plurality of photoelectric conversion elements simultaneously control the exposure periods of the plurality of photoelectric conversion elements arranged in the region where signal generation is desired.

[0067] In this embodiment, one sub-frame has a first exposure period P1 and a second exposure period P2, and the timing for starting the first exposure period P1 and the timing for starting the second exposure period P2 are different. The first sub-frame has a first exposure period that starts exposure after a predetermined period from the light emission from the light-emitting element, and a second exposure period that starts exposure after a period longer than the predetermined period from the light emission from the light-emitting element. Here, the predetermined period includes 0. In this embodiment, the period from the light emission from the light-emitting element until the start of the second exposure period is shorter than the period from the light emission from the light-emitting element until the end of the first exposure period. That is, within one sub-frame, the second exposure period has both a period that overlaps with the first exposure period and a period that does not overlap with the first exposure period. Thereby, since the exposure periods can be overlapped within one sub-frame, it becomes possible to perform ranging with high resolution when generating a histogram.

[0068] In FIG. 9, in one sub-frame, the start timing of the exposure period is shifted at a constant speed with respect to the light emission timing of the light source, but it is not limited to this. For example, the start timing of the exposure period may be shifted irregularly with respect to the light emission timing of the photoelectricity.

[0069] For example, the first sub-frame further has a third exposure period and a fourth exposure period, and the period differences in the start timings of the first to fourth exposure periods may be the same or different. Let the period difference between the timing of starting the first exposure period P1 and the timing of starting the second exposure period P2 be the first period difference D1. Let the period difference between the timing of starting the second exposure period P2 and the timing of starting the third exposure period be the second period difference. Let the period difference between the timing of starting the third exposure period and the timing of starting the fourth exposure period be the third period difference. In this embodiment, the first period difference, the second period difference, and the third period difference are the same.

[0070] The light amount values obtained in each exposure period in each sub-frame are integrated, the integrated signal amount obtained by integrating the total signal amount is counted, and a histogram is generated. In this embodiment, information such as the rise, fall, mode value, or centroid of the histogram is used to calculate the time difference from light emission to light reception. Thereby, it becomes possible to measure the distance to the object.

[0071] As shown in FIG. 9, in the present embodiment, the first sub-frame includes a first sub-frame that starts an exposure period after a first period from the light emission from the light-emitting element, and a second sub-frame that starts an exposure period after a second period longer than the first period from the light emission from the light-emitting element. And, any one of the plurality of exposure periods in the first sub-frame and any one of the plurality of exposure periods in the second sub-frame are partially overlapped. The period from the light emission from the light-emitting element in the first sub-frame to at least one end timing of the plurality of exposure periods is longer than the period from the light emission from the light-emitting element in the second sub-frame to at least one start timing of the plurality of exposure periods. Thereby, in the generation of the histogram, the integrated signal amount of the first sub-frame and the integrated signal amount in the second sub-frame can be superimposed. The first period is an integer including 0, and the second period is an integer not including 0. That is, between sub-frames, while shifting the start timing of the exposure period with respect to the light emission timing of the light source, the light amount measurement is performed. Therefore, the distance resolution can be increased as compared with the case where they are not superimposed.

[0072] FIG. 10(a) shows the gate profile in the present embodiment and the residence time distribution in each sub-frame. The gate profile is a function that represents the time change of the sensitivity to incident photons in one light amount measurement of a predetermined sub-frame, and in the present embodiment, it corresponds to the period during which the gate is on. In FIG. 10(a), for the sake of simplicity of explanation, the gate profile is represented by a rectangular function, but actually, due to pulse delay and finite reset time in the sensor, there may be waveform droop, overshoot, ringing, etc. at the rise and fall. As shown in FIG. 10(a), according to the present embodiment, since the integration is performed while shifting the light reception timing at a constant speed within the sub-frame, the integrated residence time distribution becomes a rectangular function. According to the present embodiment, the width of the residence time distribution can be widened as compared with the comparative form.

[0073] Further, FIG. 10(b) shows the effective gate profile obtained by the convolution of the gate profile and the residence time distribution in the present embodiment. In the present embodiment, since the width of the residence time distribution can be widened as shown in FIG. 9(a), the effective gate profile obtained by convolution has a trapezoidal shape.

[0074] According to the present embodiment, the effective gate profile can take an intermediate value instead of being binary, and the period of taking the intermediate value can be overlapped with a part of the effective gate profile of the immediately preceding subframe or the immediately following subframe. As a result, based on the internal ratio of the integrated output in the immediately preceding subframe or the immediately following subframe, it is possible to obtain a resolution finer than the gate shift interval. Therefore, it is possible to provide a signal generation device that ensures an appropriate distance resolution without increasing the ranging time.

[0075] (Embodiment 2) This embodiment will be described with reference to FIG. 11. FIG. 11 is a graph showing the driving of the gate per unit time and the effective gate profile in the driving according to this embodiment. FIG. 11(a) shows the gate profile and the residence time distribution in each subframe, and FIG. 11(b) shows the effective gate profile.

[0076] In Embodiment 1, the start times of the exposure periods were shifted at a constant speed within one subframe. However, in this embodiment, the exposure periods are shifted binary at different timings within one subframe, and the point that exposure is performed multiple times at each timing is different. Since the configuration is substantially the same as that of Embodiment 1 except for this point and the points described below, the description will be omitted.

[0077] In this embodiment, within one subframe, a first exposure period in which exposure is started after a first period has elapsed since the emission from the light-emitting element is repeated a plurality of times, and a second exposure period in which exposure is started after a second exposure period has elapsed since the emission from the light-emitting element is repeated a plurality of times. Then, an integrated signal amount is calculated using the signals obtained in the plurality of first exposure periods and the signals obtained in the plurality of second exposure periods.

[0078] In this embodiment, the number of times the first exposure period is repeated is the same as the number of times the second exposure period is repeated. For example, the first exposure period is repeated 10 times, and the second exposure period is repeated 10 times. Note that the present invention is not limited to this, and the number of times the first exposure period is repeated and the number of times the second exposure period is repeated may be different. In this case, when measuring the distance to the object based on the histogram, it may be performed taking into account the different number of repetitions. In this embodiment, as shown in FIG. 11(b), the effective gate profile has an inverted T shape. That is, the effective gate profile has a convex shape.

[0079] According to this embodiment, similarly to Embodiment 1, it is possible to provide a signal generation device that ensures an appropriate distance resolution without increasing the ranging time. In addition, compared to Embodiment 1, a simple pulse generator can be used. Therefore, it is possible to reduce the cost of the distance information generation device.

[0080] Further, in this embodiment, as a modification, the lengths of the first exposure period and the second exposure period may be changed. FIG. 12(a) shows the gate profile 1 in the first exposure period, and FIG. 12(b) shows the gate profile 2 in the second exposure period. In the modification, the first exposure period is longer than the second exposure period.

[0081] FIG. 12(c) shows the gate profile 1, the gate profile 2, and the effective gate profile obtained by the composite product (convolution) of the residence time distribution. Thus, even when the gate profiles 1 and 2 with different exposure period widths and exposure period start timings are integrated a plurality of times respectively, an effective gate profile similar to FIG. 11(b) can be obtained. Therefore, also in the modification example, it is possible to provide a signal generation device that ensures an appropriate distance resolution without increasing the ranging time.

[0082] (Embodiment 3) This embodiment will be described with reference to FIG. 13. FIG. 13 is a graph showing the driving of the gate per time unit and the effective gate profile in the driving according to this embodiment. FIG. 13(a) shows the gate profile and the residence time distribution in each sub-frame, and FIG. 13(b) shows the effective gate profile.

[0083] In this embodiment, the plurality of exposure periods in the first sub-frame and the plurality of exposure periods in the second sub-frame are shifted. Since it is substantially the same as Embodiment 1 except for this point and the points described below, the description is omitted.

[0084] In this embodiment, the end timing of the plurality of exposure periods in the first sub-frame is controlled to be the same as or later than the start timing of the plurality of exposure periods in the second sub-frame. Also, the timing difference between sub-frames is made the same as or smaller than the width of the gate profile. As a result, compared with Embodiment 1, it is possible to cover the ranging range of interest with a smaller number of sub-frames. Therefore, it is possible to shorten the ranging time compared with Embodiment 1. In particular, by setting the timing difference ≒ the width of the gate profile, it is possible to reverse the increase / decrease relationship of the effective gate profile in the immediately preceding sub-frame or the immediately following sub-frame. More specifically, as shown in Fig. 13(b), in the region where the slope of the effective gate profile is negative in the first sub-frame, the slope of the effective gate profile can be made a positive region in the second sub-frame.

[0085] In this embodiment, similar to Embodiment 1, it is possible to provide a signal generation device that ensures an appropriate distance resolution without increasing the ranging time. Also, by acquiring data in a plurality of sub-frames while maintaining the relationship in which the increase / decrease relationship of the effective gate profile is reversed, it becomes possible to obtain a high resolution regardless of the distance to the object.

[0086] (Embodiment 4) This embodiment will be described with reference to Fig. 14. Fig. 14 is a graph showing the driving of the gate per unit time and the effective gate profile in the driving according to this embodiment. Fig. 14(a) shows the gate profile and the residence period distribution in each sub-frame, and Fig. 14(b) shows the effective gate profile.

[0087] In this embodiment, within one sub-frame, the shift amount of the start timing of the exposure period is gradually decreased with the passage of time and then gradually increased with the passage of time, which is different from Embodiment 1. Since it is substantially the same as Embodiment 1 except for this point and the points described below, the description will be omitted.

[0088] In this embodiment, the residence time distribution in each sub-frame is not rectangular but a symmetric triangle. For example, the first time difference and the third time difference are the same, and the second time difference is smaller than the first time difference. Therefore, as shown in FIG. 14(b), the effective gate profile has a steeper slope than in the first embodiment. Therefore, by causing the light reflected when hitting the object to span the immediately preceding sub-frame or the immediately following sub-frame, the ranging accuracy can be improved compared to the first embodiment.

[0089] In this embodiment, similar to the first embodiment, it is possible to provide a signal generation device that ensures an appropriate distance resolution without increasing the ranging time. Also, the ranging accuracy can be improved compared to the first embodiment.

[0090] Also, in this embodiment, as a modification 1, as shown in FIG. 15(a), the residence time distribution in each sub-frame may be an asymmetric triangle. For example, the shift amount of the start timing of the exposure period may be gradually decreased with the passage of time and then integrated. That is, in one sub-frame, the overlap of a plurality of exposure periods may be increased as the latter half of the sub-frame is reached. The third time difference is smaller than the second time difference, and the second time difference is smaller than the first time difference. Also in this case, as shown in FIG. 15(b), the slope of the effective gate profile becomes steeper than in the first embodiment. Therefore, in modification 1, it is possible to further improve the ranging accuracy. Note that instead of decreasing the shift amount of the start timing of the exposure period with the passage of time, a similar residence time distribution may be realized by changing the number of integrations for each start timing of the exposure period while keeping the shift amount constant. In this case, the number of integrations for the first exposure period and the number of integrations for the second exposure period are different. For example, the number of integrations for the second exposure period can be made larger than the number of integrations for the first exposure period.

[0091] In addition, in the present embodiment, as a modification 2, as shown in FIG. 16(a), the residence time distribution in each sub-frame may be M-shaped. For example, in one sub-frame, the shift amount of the start timing of the exposure period may be gradually increased little by little as the period elapses, and from the middle, the shift amount of the start timing of the exposure period may be gradually decreased little by little as the period elapses. That is, in one sub-frame, the overlap of a plurality of exposure periods may be increased in the first half and the second half of the sub-frame, and the overlap of a plurality of exposure periods may be decreased near the center of the sub-frame. Also in Modification 2, as shown in FIG. 15(b), the slope of the effective gate profile becomes steeper than that in Embodiment 1. Therefore, in Modification 2, it is possible to further improve the ranging accuracy.

[0092] In addition, in the present embodiment, as a modification 3, as shown in FIG. 17(a), the residence time distribution in each sub-frame may be parabolic. For example, in one sub-frame, the overlap of a plurality of exposure periods may be increased near the center of the sub-frame, and the overlap of a plurality of exposure periods may be decreased in the first half and the second half of the sub-frame. For example, the second period difference is smaller than the first period difference, and the third period difference is smaller than the second period difference. Also in this case, as shown in FIG. 17(b), the slope of the effective gate profile becomes steeper than that in Embodiment 1. Therefore, in Modification 3, it is possible to further improve the ranging accuracy.

[0093] (Embodiment 5) The present embodiment will be described with reference to FIG. 18. FIG. 18 is a graph showing the driving of the gate per unit time in the driving according to the present embodiment. FIG. 18(a) shows the gate profile and the residence time distribution in each sub-frame, and FIG. 18(b) shows the effective gate profile.

[0094] In the present embodiment, it is different from Embodiment 1 in that the counter included in the photoelectric conversion element is an up-down counter. Since it is substantially the same as Embodiment 1 except for this point and the points described below, the description is omitted.

[0095] In this embodiment, an up-down counter is used as a counter. As a result, it becomes possible to switch between counting up and counting down within one sub-frame. Therefore, as shown in Fig. 18(a), in the residence time distribution, it becomes possible to effectively take negative values. Also in the folded effective gate profile, negative values can be realized. Therefore, highly accurate distance measurement is possible while canceling the influence of external light.

[0096] In this embodiment, similar to Embodiment 1, it is possible to provide a signal generation device that ensures an appropriate distance resolution without increasing the distance measurement time. Also, compared to Embodiment 1, the influence of external light can be reduced.

[0097] Also, in this embodiment, as a modification, as shown in Fig. 19(a), the residence time distribution in each sub-frame may be triangular instead of rectangular. For example, in one sub-frame, after gradually decreasing the shift amount of the start timing of the exposure period with the passage of time, the shift amount of the start timing of the exposure period may be gradually increased with the passage of time. In this case, as shown in Fig. 19(b), the effective gate profile can take a shape close to a sine wave. Therefore, when performing processing using principles such as Fourier transform, the subsequent arithmetic processing can be simplified.

[0098] (Embodiment 6) The signal generation system according to this embodiment will be described with reference to Fig. 20. Fig. 20 is a block diagram showing the schematic configuration of the signal generation system according to this embodiment.

[0099] The signal generation device (photoelectric conversion device) described in the above embodiment is applicable to various signal generation systems. Examples of applicable signal generation systems (photoelectric conversion systems) include digital still cameras, digital camcorders, surveillance cameras, copiers, fax machines, mobile phones, in-vehicle cameras, observation satellites, etc. In addition, a camera module including an optical system such as a lens and an imaging device is also included in the signal generation system. FIG. 20 illustrates a block diagram of a digital still camera as an example of these.

[0100] The signal generation system illustrated in FIG. 20 includes an imaging device 1004 which is an example of a signal generation device, and a lens 1002 that forms an optical image of a subject on the imaging device 1004. Further, it has a diaphragm 1003 for variably controlling the amount of light passing through the lens 1002, and a barrier 1001 for protecting the lens 1002. The lens 1002 and the diaphragm 1003 are an optical system that condenses light onto the imaging device 1004. The imaging device 1004 is a signal generation device (imaging device) of any of the above embodiments, and converts the optical image formed by the lens 1002 into an electrical signal.

[0101] The signal generation system also has a signal processing unit 1007 which is an image generation unit that generates an image by processing the output signal output from the imaging device 1004. The signal processing unit 1007 performs operations of outputting image data by performing various corrections and compressions as necessary. The signal processing unit 1007 may be formed on the semiconductor substrate on which the imaging device 1004 is provided, or may be formed on a semiconductor substrate different from the imaging device 1004. Further, the imaging device 1004 and the signal processing unit 1007 may be formed on the same semiconductor substrate.

[0102] The signal generation system further includes a memory unit 1010 for temporarily storing image data and an external interface unit (external I / F unit) 1013 for communicating with an external computer or the like. The signal generation system further includes a recording medium 1012 such as a semiconductor memory for recording or reading imaging data, and a recording medium control interface unit (recording medium control I / F unit) 1011 for recording or reading from the recording medium 1012. Note that the recording medium 1012 may be built into the signal generation system or may be detachable.

[0103] The signal generation system further includes an overall control and arithmetic unit 1009 that performs various operations and controls the entire digital still camera, and a timing generation unit 1008 that outputs various timing signals to the imaging device 1004 and the signal processing unit 1007. Here, the timing signals and the like may be input from the outside, and the signal generation system may have at least the imaging device 1004 and a signal processing unit 1007 that processes the output signal output from the imaging device 1004.

[0104] The imaging device 1004 outputs an imaging signal to the signal processing unit 1007. The signal processing unit 1007 performs predetermined signal processing on the imaging signal output from the imaging device 1004 and outputs image data. The signal processing unit 1007 generates an image using the imaging signal.

[0105] As described above, according to the present embodiment, a signal generation system to which the signal generation device (imaging device) of any of the above embodiments is applied can be realized.

[0106] (Embodiment 7) The signal generation system and the moving body of the present embodiment will be described with reference to FIG. 21. FIG. 21 is a diagram showing the configuration of the signal generation system and the moving body of the present embodiment.

[0107] FIG. 21(a) shows an example of a signal generation system related to an in-vehicle camera. The signal generation system 1300 includes a signal generation device 1310. The signal generation device 1310 is the signal generation device described in any of the above embodiments. The signal generation system 1300 includes an image processing unit 1312 that performs image processing on a plurality of pieces of image data acquired by the signal generation device 1310. The signal generation system 1300 also includes a distance acquisition unit 1316 that calculates the distance to an object, and a collision determination unit 1318 that determines whether there is a possibility of collision based on the calculated distance. Here, the distance acquisition unit 1316 may acquire distance information to a ToF (Time Of Flight) object, or may acquire distance information using parallax information or the like. That is, the distance information is information related to parallax, defocus amount, distance to an object, and the like. The collision determination unit 1318 may determine the possibility of collision using any of these distance information. The distance information acquisition means may be realized by dedicatedly designed hardware, or may be realized by a software module. It may also be realized by an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or the like, or may be realized by a combination of these.

[0108] The signal generation system 1300 is connected to a vehicle information acquisition device 1320 and can acquire vehicle information such as vehicle speed, yaw rate, and steering angle. Further, the signal generation system 1300 is connected to a control ECU 1330 which is a control device that outputs a control signal for generating a braking force for the vehicle based on the determination result of the collision determination unit 1318. Further, the signal generation system 1300 is also connected to an alarm device 1340 that issues an alarm to the driver based on the determination result of the collision determination unit 1318. For example, when the collision determination unit 1318 determines that there is a high possibility of collision, the control ECU 1330 performs vehicle control to avoid collision and reduce damage, such as applying brakes, returning the accelerator, and suppressing engine output. The alarm device 1340 warns the user by sounding an alarm such as a sound, displaying alarm information on a screen of a car navigation system, or applying vibration to a seat belt or a steering wheel.

[0109] In the present embodiment, the signal generation system 1300 images the surroundings of the vehicle, for example, the front or the rear. FIG. 21(b) shows the signal generation system when imaging the front of the vehicle (imaging range 1350). The vehicle information acquisition device 1320 sends an instruction to the signal generation system 1300 or the signal generation device 1310. With such a configuration, the ranging accuracy can be further improved.

[0110] In the above, an example of controlling so as not to collide with other vehicles has been described, but it is also applicable to control for automatically driving while following other vehicles and control for automatically driving so as not to deviate from the lane. Further, the signal generation system is not limited to vehicles such as automobiles, and can be applied to moving bodies (moving devices) such as ships, airplanes, or industrial robots. This moving body mainly includes a driving force generation unit that generates a driving force used for the movement of the moving body, and one or both of rotating bodies mainly used for the movement of the moving body. The driving force generation unit can be an engine, a motor, or the like. The rotating body can be a tire, a wheel, a screw of a ship, a propeller of an aircraft, or the like. In addition, it can be applied not only to moving bodies but also to devices that widely use object recognition, such as an advanced road traffic system (ITS).

[0111] (Embodiment 8) The signal generation system of this embodiment will be described with reference to FIGS. 23(a) and (b). FIG. 23(a) illustrates glasses 1600 (smart glasses) which are the signal generation system of this embodiment. The glasses 1600 include a signal generation device 1602. The signal generation device 1602 is the signal generation device described in each of the above embodiments. Also, a display device including a light-emitting device such as an OLED or an LED may be provided on the back side of the lens 1601. There may be one or a plurality of signal generation devices 1602. Also, a combination of multiple types of signal generation devices may be used. The arrangement position of the signal generation device 1602 is not limited to FIG. 23(a).

[0112] The glasses 1600 further include a control device 1603. The control device 1603 functions as a power supply that supplies power to the signal generation device 1602 and the above display device. Also, the control device 1603 controls the operations of the signal generation device 1602 and the display device. An optical system for condensing light onto the signal generation device 1602 is formed in the lens 1601.

[0113] FIG. 23(b) illustrates glasses 1610 (smart glasses) according to one application example. The glasses 1610 have a control device 1612, and a signal generation device corresponding to the signal generation device 1602 and a display device are mounted on the control device 1612. An optical system for projecting the light emitted from the signal generation device and the display device within the control device 1612 is formed in the lens 1611, and an image is projected onto the lens 1611. The control device 1612 functions as a power supply that supplies power to the signal generation device and the display device, and controls the operations of the signal generation device and the display device. The control device may have a gaze detection unit that detects the wearer's gaze. Infrared rays may be used for gaze detection. The infrared light emitting unit emits infrared light toward the eyeball of the user who is gazing at the display image. An imaging image of the eyeball is obtained by the imaging unit having a light receiving element detecting the reflected light of the emitted infrared light from the eyeball. By having a reduction means for reducing the light from the infrared light emitting unit to the display unit in a plan view, a reduction in image quality is reduced.

[0114] The user's line of sight with respect to the display image is detected from the captured image of the eyeball obtained by infrared imaging. Any known method can be applied to the line-of-sight detection using the captured image of the eyeball. As an example, a line-of-sight detection method based on the Purkinje image by the reflection of the irradiation light on the cornea can be used.

[0115] More specifically, a line-of-sight detection process based on the pupil corneal reflection method is performed. Using the pupil corneal reflection method, a line-of-sight vector representing the orientation (rotation angle) of the eyeball is calculated based on the image of the pupil and the Purkinje image included in the captured image of the eyeball, thereby detecting the user's line of sight.

[0116] The display device of the present embodiment may include a signal generation device having a light receiving element, and control the display image of the display device based on the user's line-of-sight information from the signal generation device.

[0117] Specifically, the display device determines a first visual field region that the user is gazing at and a second visual field region other than the first visual field region based on the line-of-sight information. The first visual field region and the second visual field region may be determined by the control device of the display device, or the display device may receive those determined by an external control device. In the display area of the display device, the display resolution of the first visual field region may be controlled to be higher than that of the second visual field region. That is, the resolution of the second visual field region may be made lower than that of the first visual field region.

[0118] Also, the display area has a first display area and a second display area different from the first display area, and based on the line-of-sight information, a region with a higher priority may be determined from the first display area and the second display area. The first visual field region and the second visual field region may be determined by the control device of the display device, or the display device may receive those determined by an external control device. The resolution of the region with a higher priority may be controlled to be higher than that of the region other than the region with a higher priority. That is, the resolution of the region with a relatively lower priority may be made lower.

[0119] Note that AI may be used to determine the first visual field area or the area with high priority. The AI may be a model configured to estimate the angle of the line of sight and the distance to the object at the tip of the line of sight from the eye image, using the eye image and the direction in which the eye in the image is actually looking as teacher data. The AI program may be possessed by the display device, the signal generation device, or an external device. When possessed by an external device, it is transmitted to the display device via communication.

[0120] When performing display control based on visual recognition detection, it can be preferably applied to smart glasses that further have a signal generation device for imaging the outside. The smart glasses can display the captured external information in real time.

[0121] (Embodiment 9) The above-described signal generation device and signal generation system may be applied to electronic devices such as so-called smartphones and tablets.

[0122] FIGS. 23(a) and 23(b) are diagrams showing an example of an electronic device 1500 equipped with a signal generation device. FIG. 23(a) shows the front side of the electronic device 1500, and FIG. 23(b) shows the back side of the electronic device 1500.

[0123] As shown in FIG. 23(a), a display 1510 for displaying an image is arranged at the center of the surface of the electronic device 1500. Along the upper side of the surface of the electronic device 1500, front cameras 1521, 1522 used by the signal generation device, an IR light source 1530 that emits infrared light, and a visible light source 1540 that emits visible light are arranged.

[0124] Also, as shown in FIG. 23(b), along the upper side of the back surface of the electronic device 1500, rear cameras 1551, 1552 used by the signal generation device, an IR light source 1560 that emits infrared light, and a visible light source 1570 that emits visible light are arranged.

[0125] In the electronic device 1500 configured as described above, by applying the signal generation device described above, for example, it is possible to capture a higher-quality image taking into account the distance to the object. Note that the signal generation device can also be applied to other electronic devices such as an infrared sensor, a distance measurement sensor using an active infrared light source, a security camera, and a personal or biometric authentication camera. Thereby, it is possible to improve the accuracy and performance of these electronic devices.

[0126] In this specification, the expressions "A or B" and "at least one of A and B" can include all possible combinations of the listed items unless otherwise explicitly defined. Also, expressions such as "at least one of A or / and B" and "one or more of A or / and B" can include all possible combinations of the listed items unless otherwise explicitly defined. That is, the above expressions are understood to disclose all cases where at least one A is included, at least one B is included, and both at least one A and at least one B are included. This also applies similarly to combinations of three or more elements.

[0127] As described above, the embodiments can be appropriately changed without departing from the technical idea. Note that the disclosure of this specification includes not only what is described in this specification but also all matters that can be grasped from this specification and the drawings attached hereto. Also, the disclosure of this specification includes the complementary set of the concepts described in this specification. That is, for example, if this specification describes that "A is larger than B", even if the description that "A is not larger than B" is omitted, it can be said that this specification discloses that "A is not larger than B". This is because when the description that "A is larger than B" is given, it is premised that the case where "A is not larger than B" is considered.

[0128] The disclosure of this embodiment includes the following configurations and methods.

[0129] (Configuration 1) having a plurality of photoelectric conversion elements, A signal generation device that generates a plurality of sub-frames using signals from the plurality of photoelectric conversion elements, and generates a distance measurement frame using the signals of the plurality of sub-frames. The plurality of sub-frames include at least a first sub-frame that starts an exposure period after a first period from the light emission of the light emitting element, and a second sub-frame that starts an exposure period after a second period longer than the first period from the light emission of the light emitting element. The first sub-frame has at least a first exposure period and a second exposure period. A signal generation device characterized in that the timing at which the first exposure period starts is different from the timing at which the second exposure period starts.

[0130] (Configuration 2) The signal generation device according to Configuration 1, characterized in that the period from the light emission of the light emitting element until the second exposure period starts is shorter than the period from the light emission of the light emitting element until the first exposure period ends.

[0131] (Configuration 3) The plurality of sub-frames are generated by a first frame generation unit. The signal generation device according to Configuration 1 or 2, characterized in that the distance measurement frame is generated by a second frame generation unit.

[0132] (Configuration 4) The first sub-frame has a plurality of exposure periods including the first exposure period and the second exposure period. The signal generation device according to any one of Configurations 1 to 3, characterized in that in the first sub-frame, the timings at which the plurality of exposure periods start shift at a constant speed.

[0133] (Configuration 5) The signal generation device according to any one of Configurations 1 to 3, characterized in that in the first sub-frame, the first exposure period and the second exposure period are repeated a plurality of times.

[0134] (Configuration 6) The signal generation device according to Configuration 5, characterized in that the total signal amount is counted by integrating the light amount values obtained in the plurality of first exposure periods and the light amount values obtained in the plurality of second exposure periods.

[0135] (Configuration 7) The signal generation device according to any one of Configurations 1 to 6, characterized in that the period from the light emission from the light emitting element in the first sub-frame to the end timing of the exposure period is longer than the period from the light emission from the light emitting element in the second sub-frame to the start timing of the exposure period.

[0136] (Configuration 8) The signal generation device according to any one of Configurations 1 to 7, characterized in that at least two or more of the plurality of photoelectric conversion elements start the second exposure period simultaneously.

[0137] (Configuration 9) The first sub-frame has a third exposure period and a fourth exposure period. There is a first period difference between the timing of starting the first exposure period and the timing of starting the second exposure period, a second period difference between the timing of starting the second exposure period and the timing of starting the third exposure period, and a third period difference between the timing of starting the third exposure period and the timing of starting the fourth exposure period. The signal generation device according to any one of Configurations 1 to 8, characterized in that the first period difference and the third period difference are the same, and the second period difference is smaller than the first period difference.

[0138] (Configuration 10) The first sub-frame has a third exposure period and a fourth exposure period. a first time difference between a timing at which the first exposure period starts and a timing at which the second exposure period starts, a second time difference between a timing at which the second exposure period starts and a timing at which the third exposure period starts, and a third time difference between a timing at which the third exposure period starts and a timing at which the fourth exposure period starts, The signal generation device according to any one of Configurations 1 to 8, wherein the first time difference is greater than the second time difference, and the second time difference is greater than the third time difference.

[0139] (Configuration 11) The first sub-frame has a third exposure period and a fourth exposure period. a first time difference between a timing at which the first exposure period starts and a timing at which the second exposure period starts, a second time difference between a timing at which the second exposure period starts and a timing at which the third exposure period starts, and a third time difference between a timing at which the third exposure period starts and a timing at which the fourth exposure period starts, The signal generation device according to any one of Configurations 1 to 8, wherein the third time difference is smaller than the second time difference, and the second time difference is smaller than the first time difference.

[0140] (Configuration 12) The photoelectric conversion element includes an avalanche photodiode and a gate circuit that controls whether to output a signal from the avalanche photodiode. The signal generation device according to any one of Configurations 1 to 11, wherein the first exposure period and the second exposure period are set by controlling signal supply to the gate circuit.

[0141] (Configuration 13) The gate circuit is an AND circuit. The signal generation device according to Configuration 12, wherein a signal from the avalanche photodiode and a signal from a driving line are input to the AND circuit.

[0142] (Configuration 14) The photoelectric conversion element has a counter, and the signal generation device according to Configuration 12 or 13 is characterized thereby.

[0143] (Configuration 15) The counter is an up-down counter, and the signal generation device according to Configuration 14 is characterized thereby.

[0144] (Configuration 16) Integrating the plurality of first exposure periods and integrating the plurality of second exposure periods, The signal generation device according to any one of Configurations 1 to 15, wherein the number of integrations of the first exposure period and the number of integrations of the second exposure period are different.

[0145] (Configuration 17) The signal generation device according to Configuration 16, wherein the number of integrations of the second exposure period is larger than the number of integrations of the first exposure period.

[0146] (Configuration 18) A light emitting device including the light emitting element, a signal generation device according to any one of Configurations 1 to 17, and a signal generation system that detects reflected light irradiated from the light emitting device and reflected from an object by the signal generation device.

Explanation of Reference Numerals

[0147] 101 Photoelectric conversion element P1 First exposure period P2 Second exposure period

Claims

1. having a plurality of photoelectric conversion elements, a signal generation device that generates a plurality of sub-frames using signals from the plurality of photoelectric conversion elements, and generates a ranging frame using signals of the plurality of sub-frames, the plurality of sub-frames include at least a first sub-frame that starts an exposure period after a first period from light emission from a light-emitting element, and a second sub-frame that starts an exposure period after a second period longer than the first period from light emission from the light-emitting element, the first sub-frame has at least a first exposure period and a second exposure period, a signal generation device characterized in that a timing at which the first exposure period starts is different from a timing at which the second exposure period starts.

2. The signal generation device according to claim 1, characterized in that a period from light emission from the light-emitting element until the second exposure period starts is shorter than a period from light emission from the light-emitting element until the first exposure period ends.

3. the plurality of sub-frames are generated by a first frame generation unit, The signal generation device according to claim 1, characterized in that the ranging frame is generated by a second frame generation unit.

4. the first sub-frame has a plurality of exposure periods including the first exposure period and the second exposure period, The signal generation device according to claim 1, characterized in that timings at which the plurality of exposure periods start shift at a constant speed in the first sub-frame.

5. The signal generation device according to claim 1, characterized in that the first exposure period and the second exposure period are each repeated a plurality of times in the first sub-frame.

6. The signal generation device according to claim 5, wherein an integrated signal amount obtained by integrating a light amount value obtained during a plurality of the first exposure periods and a light amount value obtained during a plurality of the second exposure periods is counted.

7. The signal generation device according to claim 1, wherein a period from light emission from a light emitting element in the first sub-frame to an end timing of the exposure period is longer than a period from light emission from the light emitting element in the second sub-frame to a start timing of the exposure period.

8. The signal generation device according to claim 7, wherein in at least two or more of the plurality of photoelectric conversion elements, the second exposure period starts simultaneously.

9. The first sub-frame has a third exposure period and a fourth exposure period. There is a first period difference between a timing at which the first exposure period starts and a timing at which the second exposure period starts, a second period difference between a timing at which the second exposure period starts and a timing at which the third exposure period starts, and a third period difference between a timing at which the third exposure period starts and a timing at which the fourth exposure period starts. The signal generation device according to claim 1, wherein the first period difference and the third period difference are the same, and the second period difference is smaller than the first period difference.

10. The first sub-frame has a third exposure period and a fourth exposure period. There is a first period difference between a timing at which the first exposure period starts and a timing at which the second exposure period starts, a second period difference between a timing at which the second exposure period starts and a timing at which the third exposure period starts, and a third period difference between a timing at which the third exposure period starts and a timing at which the fourth exposure period starts. The signal generation device according to claim 1, wherein the third period difference is smaller than the second period difference, and the second period difference is smaller than the first period difference.

11. The first sub-frame has a third exposure period and a fourth exposure period. There are a first time difference between the timing at which the first exposure period starts and the timing at which the second exposure period starts, a second time difference between the timing at which the second exposure period starts and the timing at which the third exposure period starts, and a third time difference between the timing at which the third exposure period starts and the timing at which the fourth exposure period starts. The signal generation device according to claim 1, wherein the first time difference is greater than the second time difference, and the second time difference is smaller than the third time difference.

12. The photoelectric conversion element has an avalanche photodiode and a gate circuit that controls whether to output a signal from the avalanche photodiode. The signal generation device according to claim 1, wherein the first exposure period and the second exposure period are set by controlling the supply of a signal to the gate circuit.

13. The gate circuit is an AND circuit. The signal generation device according to claim 12, wherein a signal from the avalanche photodiode and a signal from a drive line are input to the AND circuit.

14. The signal generation device according to claim 12, wherein the photoelectric conversion element has a counter.

15. The signal generation device according to claim 14, wherein the counter is an up-down counter.

16. The first exposure periods are integrated a plurality of times, and the second exposure periods are integrated a plurality of times. The signal generation device according to claim 1, wherein the number of integration times of the first exposure period is different from the number of integration times of the second exposure period.

17. The signal generation device according to claim 16, wherein the integrated number of times of the second exposure period is greater than the integrated number of times of the first exposure period.

18. A signal generation system including a light emitting device including the light emitting element, the signal generation device according to any one of claims 1 to 17, and reflected light that is irradiated from the light emitting device and reflected from an object and is detected by the signal generation device.

Images