Photoelectric conversion device, mobile device, photoelectric conversion method, storage medium, and computer program product

Abstract

The invention discloses a photoelectric conversion device, a mobile device, a photoelectric conversion method, a storage medium and a computer program product. The photoelectric conversion device includes: a plurality of pixels each including a sensor unit for emitting pulses according to photons, a counter for counting the number of pulses, and a memory for storing a count value of the counter; and a control section for generating an image signal based on a difference between count values of the counter at a start time and an end time of the accumulation period, and performing a previous reading operation of reading the count values midway in the full frame period, and image signal reading processing is performed such that a time period for a previous reading operation is shorter than a time period for reading a main image signal constituted by the count values of the plurality of pixels stored in the full frame period.

Description

Technical Field

[0001] The present invention relates to a photoelectric conversion device, a movable device, a photoelectric conversion method, a storage medium, and the like. Background Art

[0002] In recent years, photoelectric conversion devices have been developed that digitally count the number of photons incident on an avalanche photodiode (APD) and output the count value from a pixel as a photoelectrically converted digital signal. Furthermore, Japanese Patent No. 7223070 describes a configuration of a photoelectric conversion device including an APD that can output multiple videos with overlapping accumulation periods and enable continuous shooting even under low illumination.

[0003] However, in the case of an imaging element such as an in-vehicle camera installed in a mobile device, recognition processing is performed in units of video frames during normal sensor operation. Therefore, for example, it is not possible to execute recognition processing only every 33.3 ms at 30 fps. Therefore, in an in-vehicle camera, if an object appears immediately after a frame is read, it takes approximately one frame (e.g., 33.3 ms) to start recognition processing.

[0004] Since the accumulation period (exposure period) typically shortens when the frame period is shortened, the image captured by the camera may have a poor signal-to-noise (S / N) ratio, making it unsuitable for display purposes, for example. In particular, in low-light environments such as at night and in tunnels, it is desirable to capture images with satisfactory visibility by extending the frame period (accumulation period). In this way, the required characteristics for the purpose of identifying objects in an image may differ from those for the purpose of displaying the image. Summary of the Invention

[0005] According to one aspect of the present invention, a photoelectric conversion device includes: a plurality of pixels, each of which includes a sensor unit for emitting pulses according to photons, a counter for counting the number of pulses, and a memory for storing the count value of the counter; one or more memories for storing instructions; and one or more processors for executing the instructions to: generate an image signal based on the difference between the count value of the counter at the beginning and end of an accumulation period; and perform a previous reading operation of reading the count value midway through a full frame period, and perform image signal reading processing so that the time period used for the previous reading operation is shorter than the time period used for reading a main image signal composed of the count values ​​of a plurality of pixels stored in the full frame period.

[0006] Further features of the present invention will become apparent from the following description of embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a diagram showing a configuration example of a photoelectric conversion element according to an embodiment of the present invention.

[0008] Figure 2 is a diagram showing a configuration example of the sensor substrate 11 .

[0009] Figure 3 2 is a diagram showing a configuration example of the circuit substrate 21 .

[0010] Figure 4 It shows Figure 2 and Figure 3 FIG. 1 is a diagram showing an equivalent circuit of a pixel 101 and a signal processing circuit 103 corresponding to the pixel 101.

[0011] Figure 5 is a diagram schematically showing the relationship between the operation of the APD 201 and the output signal.

[0012] Figure 6 is a functional block diagram of a photoelectric conversion apparatus 600 and a movable device 700 according to this embodiment.

[0013] Figure 7 1 is a diagram for explaining processing from photoelectric conversion to accumulation in a memory performed by the camera control unit 605 according to the present embodiment.

[0014] Figure 8 3 is a diagram for explaining the processing of image output and recognition result output performed by the camera control unit 605 according to the present embodiment.

[0015] Figure 9 is a diagram showing an example of images of a plurality of divided frames.

[0016] Figure 10 is a diagram showing the relationship between the memory circuit and the buffer according to the present embodiment.

[0017] Figure 11 is a flowchart showing a part of an example of driving of a photoelectric conversion element according to the present embodiment.

[0018] Figure 12 It is a continuation Figure 11 Flowchart of the process.

[0019] Figure 13 It is a continuation Figure 11 Another flow chart of .

[0020] Figure 14 It is a continuation Figure 13 Flowchart of the process. DETAILED DESCRIPTION

[0021] Hereinafter, with reference to the accompanying drawings, an advantageous mode of the present invention will be described using embodiments. In each figure, the same reference numerals are applied to the same members or elements, and repeated descriptions will be omitted or simplified.

[0022] Figure 1 is a diagram showing an example configuration of a photoelectric conversion element according to an embodiment of the present invention. Hereinafter, a photoelectric conversion device having a so-called stacked structure will be described as an example, in which the photoelectric conversion element 100 is composed of two substrates (i.e., a sensor substrate 11 and a circuit substrate 21 that are stacked and electrically connected to each other).

[0023] However, a so-called non-stacked structure may be employed in which the configuration included in the sensor substrate and the configuration included in the circuit substrate are arranged on a common semiconductor layer. The sensor substrate 11 includes a pixel region 12. The circuit substrate 21 includes a circuit region 22 in which signals detected in the pixel region 12 are processed.

[0024] Figure 2 is a diagram showing a configuration example of the sensor substrate 11. The pixel region 12 on the sensor substrate 11 includes a plurality of pixels 101 arranged two-dimensionally in a plurality of rows and columns. The pixels 101 include a photoelectric conversion unit 102 including an avalanche photodiode (hereinafter, APD).

[0025] Here, the photoelectric conversion unit 102 functions as a sensor unit that emits pulses at a frequency corresponding to the photon reception frequency. Figure 2 An example of a pixel arrangement of three rows and four columns is shown, but the number of rows and columns of the pixel array forming the pixel region 12 is not particularly limited.

[0026] Figure 3 2 is a diagram showing an example of the configuration of the circuit substrate 21. The circuit substrate 21 includes a processing Figure 2 The signal processing circuit 103 of the charge photoelectrically converted by each photoelectric conversion unit 102, the reading circuit 112, the control pulse generating unit 115, the horizontal scanning circuit 111, the vertical signal line 113, the vertical scanning circuit 110 and the output circuit 114.

[0027] The vertical scanning circuit 110 receives the control pulse supplied from the control pulse generating unit 115 and sequentially supplies the control pulse to a plurality of pixels arranged in the row direction. A logic circuit such as a shift register or an address decoder is used as the vertical scanning circuit 110 .

[0028] The signals output from the photoelectric conversion unit 102 of each pixel are processed by each signal processing circuit 103. The signal processing circuit 103 is provided with a counter and a memory, and the memory stores digital values. The horizontal scanning circuit 111 inputs a control pulse for sequentially selecting each column to the signal processing circuit 103 so that the signal is read from the memory of each pixel that stores the digital signal.

[0029] Signals are output from the signal processing circuit 103 of pixels in a row selected by the vertical scanning circuit 110 to the vertical signal line 113. The signals output to the vertical signal line 113 are output to the outside of the photoelectric conversion element 100 via the reading circuit 112 and the output circuit 114.

[0030] A plurality of buffers connected to the vertical signal line 113 are incorporated in the reading circuit 112. Furthermore, the control pulse generating unit 115 can control the vertical scanning circuit 110 to perform non-addition reading for selecting all rows.

[0031] Furthermore, when the amount of data of the signal to be read is reduced, control can be performed so that thinning reading is performed, selecting rows to be read from every few rows. In other words, the amount of data can be reduced by performing a previous reading operation to read the previous image signal midway through a full frame period and thinning out and reading pixels in rows or columns when generating the previous image signal.

[0032] Alternatively, control can be performed so that a plurality of adjacent pixel signals equivalent to one pixel signal are read by performing processing such as addition and averaging to perform addition (averaging) reading. In other words, when generating a previous image signal, the amount of data can be reduced by adding the count values ​​of a plurality of pixels. In this case, the number of pixels to be added and averaged can be freely changed to, for example, 2×2 pixels or 4×4 pixels in both the row and column directions.

[0033] Alternatively, control can be performed so that crop reading is performed to select only a portion of the image area (image angle) in order to reduce the amount of signal data to be read. In other words, when generating a previous image signal, the amount of data can be reduced by reading the count values ​​of pixels in a partial area from a plurality of pixels.

[0034] like Figure 2 and Figure 3 As shown, the plurality of signal processing circuits 103 are arranged in a region that vertically overlaps with the pixel region 12. Furthermore, the vertical scanning circuit 110, the horizontal scanning circuit 111, the reading circuit 112, the output circuit 114, and the control pulse generating unit 115 are arranged in a region that vertically overlaps with a peripheral region (non-pixel region) of the pixel region 12.

[0035] Note that the arrangement of the vertical signal line 113 and the arrangement of the reading circuit 112 and the output circuit 114 are not limited to Figure 3 For example, the vertical signal line 113 may be arranged to extend in the row direction, and the reading circuit 112 may be arranged in front of the extension of the vertical signal line 113. In addition, it is not always necessary to provide one signal processing circuit 103 for each of all photoelectric conversion units, and a configuration may be adopted in which a plurality of photoelectric conversion units share one signal processing circuit and perform signal processing sequentially.

[0036] Figure 4 It shows Figure 2 and Figure 3 FIG. 1 is a diagram showing an equivalent circuit of a pixel 101 and a signal processing circuit 103 corresponding to the pixel 101.

[0037] The APD 201 included in the photoelectric conversion unit 102 generates charge pairs corresponding to incident light through photoelectric conversion. One of the two nodes of the APD 201 is connected to a power supply line supplied with a drive voltage VL (first voltage). In addition, the other of the two nodes of the APD 201 is connected to a power supply line supplied with a drive voltage VH (second voltage) higher than the drive voltage VL.

[0038] exist Figure 4 In the embodiment, one node of the APD 201 serves as an anode, and the other node of the APD serves as a cathode. A reverse bias voltage for causing the APD 201 to perform an avalanche multiplication operation is supplied to the anode and cathode of the APD 201. With these voltages supplied, the charge generated by incident light causes avalanche multiplication, and an avalanche current is generated.

[0039] Note that when a reverse bias voltage is supplied, there are Geiger mode and linear mode. In Geiger mode, operation is performed when the voltage difference between the anode and cathode is greater than the breakdown voltage, and in linear mode, operation is performed when the voltage difference between the anode and cathode is close to, equal to, or less than the breakdown voltage. An APD operating in Geiger mode is referred to as a SPAD. For example, in the case of a SPAD, the drive voltage VL (first voltage) is -30V, and the drive voltage VH (second voltage) is 1V.

[0040] The signal processing circuit 103 includes a quenching element 202, a waveform shaping unit 210, a counter circuit 211, and a memory circuit 212. The quenching element 202 is connected to a power supply line to which the driving voltage VH is supplied and one of the anode and cathode of the APD 201.

[0041] The quenching element 202 functions as a load circuit (quenching circuit) when avalanche multiplication is performed on the signal, and operates to reduce the voltage supplied to the APD 201 and reduce avalanche multiplication (quenching operation). Furthermore, the quenching element 202 operates to return the voltage supplied to the APD 201 to the driving voltage VH (recharging operation) by flowing a current corresponding to the voltage drop caused by the quenching operation.

[0042] exist Figure 4 In the illustrated configuration, an example is shown in which the signal processing circuit 103 includes a waveform shaping unit 210 , a counter circuit 211 , and a memory circuit 212 in addition to the quenching element 202 .

[0043] The waveform shaping unit 210 shapes the voltage change of the cathode of the APD 201 obtained when the photon is detected, and outputs a pulse signal. As the waveform shaping unit 210, for example, an inverter circuit is used. Figure 4 An example in which one inverter is used as the waveform shaping unit 210 is shown, but a circuit in which a plurality of inverters are connected in series may be used, or other circuits having a waveform shaping effect may be used.

[0044] The counter circuit 211 counts the number of pulses output from the waveform shaping unit 210 and holds the count value. Furthermore, when a control pulse RES is supplied via the drive line 213, the signal held by the counter circuit 211 is reset. The counter circuit 211 generates a signal based on the difference between the count values ​​at the start and end of the accumulation period.

[0045] The control pulse SEL is from Figure 3 The vertical scanning circuit 110 in Figure 4 The driving line 214 ( Figure 3 ) is supplied to the memory circuit 212, and switches the electrical connection and disconnection between the counter circuit 211 and the vertical signal line 113.

[0046] The memory circuit 212 serves as a memory for temporarily storing the count value of the counter, and outputs the output signal from the counter circuit 211 of the pixel to the vertical signal line 113. Note that the memory circuit 212 may be configured so that it can be used with the same Figure 4 The signal processing circuit 103 shown is adjacent to the signal processing circuit 103 ( Figure 4 An electrical connection is established with the counter circuit inside (not shown), and processing such as addition (averaging) can be performed.

[0047] In this manner, a plurality of pixel signals generated by a plurality of photoelectric conversion units can be added (averaged) or the like and output as one pixel signal, thereby reducing the data amount of the pixel output signal.

[0048] Alternatively, the memory circuit 212 may be capable of selectively switching between a non-addition processing mode in which connection is established to only one counter circuit 211 and an addition and averaging processing mode in which connection is established to a plurality of counter circuits.

[0049] Note that a switch such as a transistor may be arranged between the quenching element 202 and the APD 201 or between the photoelectric conversion unit 102 and the signal processing circuit 103 to switch the electrical connection. Similarly, a switch such as a transistor may be used to electrically switch the supply of the drive voltage VH or the drive voltage VL to the photoelectric conversion unit 102.

[0050] Figure 5 is a diagram schematically illustrating the relationship between the operation and output signal of APD 201. The input side of waveform shaping unit 210 is referred to as node A, and the output side is referred to as node B. Between clock time t0 and clock time t1, a voltage difference of VH − VL is applied to APD 201. When a photon strikes APD 201 at clock time t1, avalanche multiplication occurs in APD 201, an avalanche multiplication current flows through quenching element 202, and the voltage at node A drops.

[0051] If the amount of voltage drop further increases and the potential difference applied to the APD 201 decreases, avalanche multiplication of the APD 201 stops as shown at clock time t2, and the voltage level at the node A does not drop below a certain value.

[0052] Thereafter, from clock time t2 to clock time t3, a current for compensating for the voltage drop from voltage VL flows through node A, and node A stabilizes at the original potential level at clock time t3. At this time, the waveform of the portion exceeding the threshold value in the output waveform of node A is shaped by waveform shaping unit 210, and this portion is thereby outputted as a pulse signal from node B.

[0053] Next, the photoelectric conversion apparatus 600 and the movable device 700 according to the present embodiment will be described. Figure 6 600 and the movable device 700 according to this embodiment. Note that the present invention is realized by causing a computer (not shown) included in the photoelectric conversion device 600 and the movable device 700 to execute a computer program stored in a memory (not shown) as a storage medium. Figure 6 Some functional boxes are shown.

[0054] However, part or all of the functional blocks may be implemented by hardware. As hardware, an application-specific integrated circuit (ASIC) or a processor (reconfiguration processor or DSP) or the like may be used. Figure 6The functional blocks shown in FIG. 1 may not be incorporated in the same housing and may be composed of different devices connected to each other via signal paths.

[0055] The photoelectric conversion device 600 includes Figures 1 to 5 The photoelectric conversion element 100, the image forming optical system 601, the image processing unit 603, the recognition unit 604, the camera control unit 605, the storage unit 606, and the communication unit 607 described in the embodiment of the present invention are shown in FIG. Figures 1 to 5 The avalanche photodiode structure for photoelectric conversion of optical images is described in detail.

[0056] The photoelectric conversion element 100 can selectively switch a plurality of reading methods such as a non-addition reading mode, an addition (averaging) reading mode, and a thinning reading mode based on a control signal from the camera control unit 605 .

[0057] Note that the non-addition reading mode is a mode in which all pixels are read as they are, the addition (averaging) reading mode is a mode in which adjacent pixels are added, averaged, and read, and the thinning reading mode is a mode in which pixels are thinned out and read.

[0058] In this embodiment, the previous image for recognition is read in a 2×2 pixel addition (averaging) reading mode, while the main image for visual recognition is read in a non-additive reading mode. In other words, since the amount of data read for the previous image is one-fourth that of the main image, the time taken to read the previous image is also one-fourth that of the main image. In other words, the pixel signals of the previous image can be read at four times the speed of the main image.

[0059] The photoelectric conversion device in this embodiment is mounted on the mobile device 700, and the camera unit including the image forming optical system 601 and the photoelectric conversion element 100 is configured to capture images in at least one of the front, rear, and lateral directions of the mobile device. Note that a plurality of camera units may be provided in the mobile device 700.

[0060] The image processing unit 603 performs image processing such as black level correction, gamma curve adjustment, noise reduction, digital gain adjustment, demosaicing, and data compression on the image signal acquired by the photoelectric conversion element 100 and generates a final image signal.

[0061] Note that, when the photoelectric conversion element 100 includes an on-chip color filter such as RGB, it is desirable that the image processing unit 603 performs processing such as white balance correction and color conversion. Furthermore, when the read image is used for multiple purposes (such as visual recognition and recognition), it is desirable to retain image processing parameters that differ for each image utilization purpose in advance and switch the image processing parameters at the timing when the image utilization purpose is switched.

[0062] The output of the image processing unit 603 is supplied to the recognition unit 604, the electronic control unit (ECU) 701 of the mobile device 700, and the camera control unit 605. The recognition unit 604 recognizes the subject by performing image recognition based on the image signal. Specifically, the type of subject around the mobile device 700 is identified, such as a person, an animal, a vehicle, a traffic light, or a traffic sign.

[0063] In addition, the reliability indicating how correct the recognition result is and the coordinate information indicating the position and size of the subject in the image are also acquired. Specifically, the offset and size of the Z coordinate and the offset and size of the Y coordinate are acquired as coordinate information.

[0064] Although in this embodiment, for example, three types of information (i.e., subject type, reliability, and coordinate information) are obtained as recognition results, other information may also be obtained. The recognition results obtained by the recognition unit 604 are output to the ECU 701. Note that although this embodiment will be described using an example in which the movable device 700 is, for example, a car, the movable device may be any movable device such as an airplane, train, ship, drone, AGV, or robot, as long as it is movable.

[0065] The camera control unit 605 includes a CPU as a computer and a memory storing computer programs, and controls the respective components of the photoelectric conversion apparatus 600 by the CPU executing the computer programs stored in the memory.

[0066] Note that the camera control unit 605 functions as a control section and controls, for example, the length of the accumulation period of each frame of the photoelectric conversion element 100 and the timing of the control signal CLK via the control pulse generating unit of the photoelectric conversion element 100. Here, one frame period at the frame rate of a video displayed on the display unit 703 to be described later will be described as one full frame or a full frame period for explanation.

[0067] Furthermore, when pixel signals are read from the photoelectric conversion element 100, the camera control unit 605 transmits a control signal to switch the reading mode (reading method). In this embodiment, for example, two reading methods, namely, the addition (average) reading mode and the non-addition reading mode, are selectively switched within a full frame.

[0068] The camera control unit 605 also has a function of setting parameters for various image processing in the image processing unit 603 and controlling the recognition unit 604, as well as a function of acquiring recognition results. The storage unit 606 includes a recording medium such as a memory card or a hard disk, for example, and can store and read image signals.

[0069] The communication unit 607 includes a wireless or wired interface, outputs the generated image signal or recognition result to the outside of the photoelectric conversion device 600, and receives various signals from the outside. Note that as an interface, for example, wired communication based on a standard such as SPI or I2C or a wireless LAN scheme such as Wi-Fi or Bluetooth (registered trademark) can be used.

[0070] The ECU 701 includes a CPU as a computer and a memory storing a computer program, and controls the respective components of the movable device 700 by the CPU executing the computer program stored in the memory.

[0071] The output of the ECU 701 is supplied to a vehicle control unit 702 and a display unit 703. The vehicle control unit 702 functions as a movement control unit that performs driving, stopping, and direction control of a vehicle as a movable device based on the output of the ECU 701. Here, the vehicle control unit 702 as the movement control unit controls the operation of the movable device based on the result of recognition processing using previous image signals.

[0072] Furthermore, the display unit 703 serves as a display portion and includes, for example, a display element such as a liquid crystal device or an organic EL, and is mounted in the mobile device 700. The display unit 703 displays, for example, based on the output of the ECU 701, an image acquired by the photoelectric conversion element 100, the recognition result acquired by the recognition unit 604, and various information related to the vehicle's driving state, etc., to the driver of the mobile device 700 using a GUI.

[0073] Notice, Figure 6 The image processing unit 603 and the recognition unit 604 may not be mounted in the mobile device 700, and may be set in an external terminal that is set separately from the mobile device 700 and is used to remotely control the mobile device 700 or monitor the travel of the mobile device.

[0074] Next, we will use Figure 7 and Figure 8 A photoelectric conversion method according to the present embodiment is described. Figure 7 is a diagram for explaining processing from photoelectric conversion to accumulation in a memory performed by the camera control unit 605 according to the present embodiment, and Figure 83 is a diagram for explaining the processing of image output and recognition result output performed by the camera control unit 605 according to the present embodiment.

[0075] In this embodiment, a full frame of 33.3 milliseconds is divided into thirteen parts. In other words, the photoelectric conversion element 100 sequentially generates twelve frames (frame 1_1, frame 1_2, ..., frame 1_12) as the previous image for recognition, and generates one full frame (frame 1_13) as the main image for visual recognition.

[0076] Here, a frame simply described as a “frame” is an expression including both a previous image reading period and a main image reading period, unlike a full frame.

[0077] Note that in the additive (averaging) reading mode, twelve frames are read as the previous image, while in the non-additive reading mode, one full frame is read as the main image. As described above, since the amount of data in the previous image used for recognition is one-fourth the amount of data in the main image used for recognition, the image reading time is also one-fourth the length. In other words, the time period for reading the main image is 8.33 ms, and the time period for reading the previous image is 2.08 ms, allowing for a rapid reading process.

[0078] Furthermore, the main image is output from the image processing unit 603 to the ECU 701, the previous image is subjected to recognition processing by the recognition unit 604, and then the recognition result is output from the recognition unit 604 to the ECU 701. Although the display section (display unit) displays the main image of frames 1 to 13 accumulated for the longest period for visual recognition in this embodiment, the images of frames 1 to 13 can be used not only for visual recognition but also for recognition.

[0079] However, since the image of frames 1 to 13 is an output signal in the non-additive reading mode and has a large amount of data, it may take time to perform recognition processing. Therefore, recognition processing may be performed on the image of frames 1 to 13 as needed. If recognition processing is performed and there is insufficient time to perform the recognition processing as it is, the image processing unit 603 can be used to reduce the amount of data through compression processing, etc., and then the reduced data can be used for recognition processing.

[0080] Furthermore, frame 1_1 has an accumulation period from the start clock time T0 of frame 1 to clock time T1, and frame 1_2 has an accumulation period from clock time T0 to clock time T2. The same applies to subsequent frames, and frame 1_N (N is a natural number equal to or less than thirteen) has an accumulation period from clock time T0 to clock time TN. In other words, frames 1_1 to 1_13 have the first to thirteenth accumulation periods, respectively.

[0081] First, the processing flow of frame 1_1 (previous image) for recognition purposes will be described. The counter circuit 211 is reset at clock time T0 and acquires a count value C1_1 in addition (average) reading mode at clock time T1.

[0082] When the pixel 101 is a monochrome pixel, a count value C1_1 calculated by adding and averaging the count values ​​of four pixels adjacent to the counter circuit 211 is temporarily stored in the memory circuit 212 .

[0083] For example, in the case where the pixel 101 is a color filter having a Bayer layout of R, Gr, Gb, and B, the count value C1_1 is calculated by adding and averaging the count values ​​of the closest pixels in the same color. Figure 8 As shown, the count value C1_1 temporarily stored in the memory circuit 212 is then output from the photoelectric conversion element to the image processing unit 603 via the buffer of the reading circuit 112 .

[0084] The image processing unit 603 performs the above-mentioned various image processing and outputs the processing results to the recognition unit 604 as they are. Figure 8 As shown, in the period from clock time T1 to clock time T2, the image of frame 1_1 is output to the recognition unit 604, and the output of image signals corresponding to all lines is completed at the timing of clock time T2.

[0085] Furthermore, the subject recognition processing is started at the timing of clock time T2 by the recognition unit 604. Note that although the example in which the recognition processing is started at the timing of clock time T2 when all image signals of frame 1_1 are prepared has been described in this embodiment, the recognition processing may be performed sequentially for each line or each pixel.

[0086] By reducing the processing units for identification, the time required to obtain the identification result can be shortened. When the identification process is completed, the identification result is quickly output to the ECU 701. Figure 8 As shown, the shortest time required for the identification process and result output will be referred to as Td.

[0087] At this time, the output of the recognition result based on the image of frame 1_1 can be transmitted to the ECU 701 of the movable device 700 at the timing of clock time T2+Td in the fastest case. The processing flow for frame 1_1 has been described so far.

[0088] Hereinafter, similar to frame 1_1, previous images of frame 1_2, frame 1_3, ..., frame 1_12 are sequentially read in the periods from clock time T2 to clock time T3, from clock time T3 to clock time T4, ..., and from clock time T12 to clock time T13. Then, recognition processing is sequentially performed at the timings of clock time T3, clock time T4, ..., and clock time T13, and recognition results are sequentially output from recognition unit 604 to ECU 701 at the timings of clock time T3+Td, clock time T4+Td, ..., and clock time T13+Td, respectively.

[0089] Subsequently, the processing flow of frame 1_13 (main image) for visual recognition purposes will be described. At clock time T0 of full frame 2, the count value C1_13 is read and acquired from the counter circuit 211 in the non-addition mode.

[0090] The acquired count value C1_13 is temporarily stored in the memory circuit 212. The count value C1_13 temporarily stored in the memory circuit 212 is output from the photoelectric conversion element to the image processing unit 603 via the buffer of the reading circuit 112.

[0091] The image processing unit 603 performs the above-mentioned various image processing and outputs the processing results to the recognition unit 604 as they are. Figure 8 As shown, in the period from clock time T1 to clock time T5 of the full frame 2, the image of frame 1_13 is quickly output to the ECU 701 in the movable device 700, and the output of the image signals corresponding to all lines is completed at the timing of clock time T5.

[0092] In this embodiment, although recognition processing using the recognition unit 604 is not performed considering the large amount of data in the image signal of frame 1_13, recognition processing may be performed as needed. The image signal of frame 1_13 has high contrast, low noise, and reduced LED flicker, and is used for visual recognition with the highest priority.

[0093] However, by performing recognition processing in parallel, the time required to recognize the subject when switching full frames can be shortened. Figure 8 In the "Recognition Result Output" at the bottom, although C1_12 to C2_1 is a blank period where no recognition result is obtained, the blank period can be shortened by performing recognition processing on the image of C1_13. The processing flow of frame 1_13 has been described so far.

[0094] Figure 9 is a diagram showing an example of images of a plurality of divided frames. Figure 9Four frames of the main image of frame 1_1, frame 1_5, and frame 1_9, three previous images, and frame 1_13 are shown among a total of thirteen images acquired within one full frame (33.3 ms).

[0095] Despite Figure 9 As shown, the image in frame 1_1 is dark due to the short accumulation period, but the subject blur of the person who has jumped out is small. On the other hand, as the accumulation period increases in frames 1_5, 1_9, and 1_13, the subject blur is more likely to appear. Note that the stopped vehicle and the white line are less likely to appear blurred, and the contrast is more likely to increase as the accumulation period increases.

[0096] In this manner, in this embodiment, the first to thirteenth reading timings (T1 to T13) are included in one full-frame period. The camera control unit 605 controls so that the first to twelfth reading timings (T1 to T12) occur earlier than the thirteenth reading timing (T13).

[0097] Furthermore, control is performed so that a period of time (a period required for reading) for reading the first to twelfth image signals as previous image signals is shorter than a period of time for reading the thirteenth image signal as the main image signal.

[0098] In other words, the previous reading operation for reading the count value is performed during the idle period of the full frame period, and the period used for the previous reading operation is set to be shorter than the period used for reading the main image signal composed of the count values ​​of the plurality of pixels stored in the full frame period. In addition, other image signals are further read between the previous reading operation (for example, the reading operation at T1) and the main image signal reading operation (the reading operation at T13).

[0099] In addition, in this embodiment, the first to twelfth accumulation periods overlap with the thirteenth accumulation period, and the first to twelfth accumulation periods and the thirteenth accumulation period start at the same time. In addition, the thirteenth accumulation period ends at the timing when the full frame ends.

[0100] In other words, an image with a short accumulation period and an image with a long accumulation period are created, the short accumulation period is set to end earlier than the long accumulation period, and upon completion of the short accumulation period, the image is output and sent to the recognition unit at the subsequent stage. Furthermore, the subject is recognized based on at least the signal generated during the first accumulation period. Recognition unit 604, as a recognition unit, recognizes the subject based on at least the previous image signal generated during the first accumulation period.

[0101] Therefore, in this embodiment, image recognition can be performed immediately after the time period for reading the thirteenth image has elapsed, in the shortest possible case. In the prior art, image recognition cannot be performed before a full frame has elapsed. In other words, when the movable device is moving at high speed, obstacles, etc., can be quickly identified. Furthermore, by outputting the recognition results to ECU 701 with minimal delay, the brakes can be quickly applied. Alternatively, obstacles can be avoided at an early stage.

[0102] Note that the contrast of an image with a long accumulation period can be increased, allowing it to be used as a display image. In other words, an image with a short accumulation period is suitable for rapid subject recognition, while an image with a long accumulation period is suitable for display. In this way, the display device in this embodiment displays at least the signal generated during the thirteenth accumulation period as an image.

[0103] Furthermore, since this embodiment uses an APD, unlike a CMOS sensor, the accumulated charge does not degrade during reading, allowing the accumulation periods to overlap. Furthermore, because there is no read noise, the original signal does not degrade regardless of the number of times the signal is read during a single accumulation.

[0104] Figure 10 is a diagram showing the relationship between the memory circuit and the buffer in the embodiment. Figure 10 Shown Figure 3 The memory circuits 212 in the signal processing circuit 103 are arranged in a state of N rows and M columns, and the respective memory circuits are represented as memory 1-1 to memory NM.

[0105] also, Figure 10 Buffer 1 to buffer M in the Figure 3 The buffer included in the read circuit 112. Figure 10 The output circuit 114 and Figure 3 Corresponding to the output circuit 114 in.

[0106] Figure 11 is a flowchart showing a part of an example of driving of a photoelectric conversion element according to the present embodiment. Figure 12 It is a continuation Figure 11 Flowchart, Figure 13 It is a continuation Figure 11 Another flow chart of , and Figure 14 It is a continuation Figure 13 In other words, by Figures 11 to 14 The flowchart in shows an example of driving of the photoelectric conversion element according to the present embodiment as a whole.

[0107] Notice, Figures 11 to 14The operations in the respective steps of the flowchart in FIG. 6 are sequentially performed by the CPU as a computer in the camera control unit 605 executing a computer program stored in a memory.

[0108] exist Figure 11 In step S101, the CPU in the camera control unit 605 sets i = 1. Here, i corresponds to frame 1_i. Next, in step S102, the CPU in the camera control unit 605 determines whether the current frame is a frame to be added when the count value is output to the memory, that is, a frame for recognition processing.

[0109] In this embodiment, "yes" is determined when i = 1 to 12, and "no" is determined when i = 13. The determination in step S102 can be made by setting an ID indicating the addition (average) reading mode or the non-addition reading mode, etc., for each frame in advance and checking the ID, or by checking the count value while counting the number of frames.

[0110] In the case where it is desired to selectively change the frames used for addition (average) reading and non-addition reading in one full frame using IDs, the same algorithm can be used for processing by simply reallocating the IDs of the respective frames. When the judgment result in step S102 is "Yes", the processing proceeds to step S103, or if the judgment result is "No", the processing proceeds to step S104 via A. Figure 13 Step S112 in .

[0111] Next, in step S103, the CPU in the camera control unit 605 outputs the result obtained by adding and averaging the count value Count of the counter circuit 211 at the clock time Ti for a total of four pixels (i.e., two pixels in both the horizontal and vertical directions) to the memory circuit 212. Since a monochrome pixel 101 is assumed in this embodiment, the addition and averaging result of the four pixels is output only to the memory circuit 212 in N rows and M columns, where both N and M are odd numbers.

[0112] In other words, there is no count value for the memory circuits 212 in N rows and M columns, where at least either N or M is an even number. For example, in the case of a color filter having a Bayer layout of R, Gr, Gb, and B instead of monochrome, there is one count value for four pixels in both the horizontal and vertical directions for each color. In addition, when the count value is output to the memory circuits 212, the count value is output to all the memory circuits at the same time. This operation is the same as Figure 7 Corresponding to the operation at clock time T1 in .

[0113] Next, in step S104, the CPU in the camera control unit 605 sets j = 1, and in step S105, the CPU in the camera control unit 605 outputs the count value Count(j - k - i) of the memory circuit j - k in Figure 10 to the buffer. At this time, columns 1 to M simultaneously provide outputs to the buffer. However, since the count values only exist in the odd - numbered columns, the number of data items of the count value Count(j - k - i) actually output to the buffer is M / 2.

[0114] In other words, the count values of the even - numbered columns may not be output to the buffer. This operation means the operation of taking the count value of the first row in Figure 10 .

[0115] Next, in step S106, the CPU in the camera control unit 605 sets k = 1, and in step S107, the CPU in the camera control unit 605 outputs the count value Count(j - k - i) of buffer k to the output unit 114. This operation corresponds to the operation of reading the signal of the buffer in the left - most column from the output circuit in Figure 10 .

[0116] Next, in Figure 12 step S108, the CPU in the camera control unit 605 determines whether k < M - 1 holds. In the case of "yes", in step S109, the CPU in the camera control unit 605 adds 2 to k as k = k + 2 and returns to step S107 to perform the operation in step S​​​​​​​​​​​​​​​If the determination in step S110 is "No," this means that reading of all rows has ended, so the process proceeds to step S121, and the CPU in the camera control unit 605 determines whether i < 13. If the determination in step S121 is "Yes," the process proceeds to step S122, and the CPU in the camera control unit 605 increments i by 1 as i = i + 1, and returns to step S102. This operation corresponds to the operation of starting the next reading at clock time T2.

[0121] In this embodiment, the process determined as "yes" in step S102 is repeated twelve times. In other words, the process in steps S103 to S121 is repeated twelve times. This operation corresponds to the operation of reading at clock time T1 to clock time T12. Thereafter, in the state where i=13 is achieved in step S122, the process returns to step S102. In step S102, the CPU in the camera control unit 605 determines "no" and proceeds to step S103. Figure 13 Step S112 in .

[0122] Next, in step S112, the CPU in the camera control unit 605 outputs the count value Count of the counter circuit 211 at the clock time Ti as it is (without performing addition and averaging processing) to the memory circuit 212. At this time, output is provided to all memory circuits at the same time. This operation is the same as Figure 7 The operation at the middle clock time T13 corresponds.

[0123] Next, in step S113, the CPU in the camera control unit 605 sets j=1, and in step S114, the CPU in the camera control unit 605 sets Figure 10 The count value Count(jki) of the memory circuit j–k in is output to the buffer.

[0124] At this time, the output is provided to the buffers of the first column to the Mth column at the same time. Step S114 is different from step S105 in that the count value exists in both the even column and the odd column, and the number of data items is M. This operation means taking Figure 10 The count value of the first row in the operation.

[0125] Next, in step S115, the CPU in the camera control unit 605 sets k=1, and in step S116, the CPU in the camera control unit 605 outputs the count value Count(jki) of the buffer k to the output circuit 114. This operation is the same as reading the count value from the output circuit 114. Figure 10 The operation corresponds to the signal of the buffer in the leftmost column.

[0126] Next, in Figure 14 In step S117, the CPU in the camera control unit 605 determines whether k < M holds. Then, if the result is "yes", in step S118, the CPU in the camera control unit 605 adds 1 to k as k = k + 1, and returns to step S116 to perform the operation in step S116.

[0127] The reason for adding 1 to k in step S118 and adding 2 to k in step S109 is that as described above, the count values exist in both even columns and odd columns. This operation corresponds to the operation of reading the signal of the buffer in the second column from the left in Figure 10 the output circuit.

[0128] In the case where the result of step S117 is "no", that is, when k = M holds, this means that the reading of the signal of the buffer in the M-th column in Figure 10 the output circuit has ended, and then the process proceeds to step S119, and the CPU in the camera control unit 605 determines whether j < N holds.

[0129] In the case where it is determined as "yes" in step S119, in step S120, the CPU in the camera control unit 605 adds 1 to j as j = j + 1, and returns to step S114. This corresponds to the operation of starting to read the next line.

[0130] In the case where it is determined as "no" in step S119, this means that the reading of all lines has ended, so the process proceeds to step S121, and the CPU in the camera control unit 605 determines whether i < 13 holds. Since the thirteenth frame enters the state of i = 13, it is determined as "no" in step S121.

[0131] In the case where it is determined as "no" in step S121, this means that the reading at the clock time T13 has been completed, so this means that the reading of all the image signals accumulated in the first full frame has been completed. As described above, the signals accumulated in the photoelectric conversion element 100 can be read sequentially.

[0132] Note that, for example, an example of adding and averaging 4 pixels (i.e., 2 pixels in the horizontal direction and 2 pixels in the vertical direction) of the previous image for identification has been described in the above embodiment, and the number of pixels to be added and averaged can be arbitrarily set in both the horizontal direction and the vertical direction.

[0133] In addition, the same applies to the case of thinning reading, and the number of pixels to be thinned and the number of rows (thinning rate) can be arbitrarily set. Although the example of the monochrome pixel 101 has been described in this embodiment, a Bayer layout color filter of R, Gr, Gb, or B, etc. can be set, and when adding and averaging processing and thinning processing are performed, processing such as adding and averaging, thinning, etc. can be performed on pixels of the same color.

[0134] Alternatively, the recognition unit 604 may perform subject recognition based on previous image signals, calculate recognition accuracy based on the results, and determine the number of pixels to be added and averaged in the pixel addition (average) reading mode and the interval elimination rate in the interval elimination reading mode according to the recognition accuracy.

[0135] In other words, while the recognition unit 604 can output a high-precision recognition result regardless of the low resolution of the acquired image, the number of pixels to be added and the thinning rate are increased to reduce the amount of image data.

[0136] In addition, which method to use between the addition (average) reading mode and the thinning reading mode can be determined according to the brightness of the ambient light. In other words, the reduction rate of the data amount can be controlled based on the brightness of the ambient light.

[0137] For example, when ambient light is equal to or greater than a predetermined value, the perception of noise does not significantly deteriorate even when the signal level decreases. Therefore, selecting a thinning-out reading mode (i.e., a mode in which pixels in rows or columns are thinned out) for simple reading processing can reduce the amount of data. On the other hand, when ambient light is less than a predetermined value, selecting an addition (average) reading mode to improve the perception of noise and adding (averaging) the count values ​​of multiple pixels can reduce the amount of data.

[0138] As described above, in this embodiment, in addition to the image acquired in full-frame (33.3ms) units (hereinafter referred to as the main image), an image read in the accumulated mid-process state (hereinafter referred to as the previous image) is also acquired and recognition processing is performed on it. Therefore, the time period required to switch from full-frame to recognition can be reduced.

[0139] Furthermore, because the accumulation period for previous images is shorter than that for the main image, and the previous images have relatively little subject blur, the recognition rate can be improved even for subjects that cannot be identified from the main image alone. Furthermore, by increasing the frequency of acquiring previous images, the time required to switch from full-frame to recognition can be further shortened.

[0140] Furthermore, in this embodiment, the photoelectric conversion device is incorporated into a mobile device and used for visual recognition purposes, such as displaying the surrounding environment as an image to the driver using a display unit (such as an electronic mirror), and for subject recognition for avoiding obstacles and applying the brakes in autonomous driving or driving assistance. In this case, because the main image is used as the image for visual recognition, high image quality, such as high contrast and low noise, can be achieved.

[0141] On the other hand, the previous image used for recognition can be transmitted with minimal delay because the resolution and image data size are reduced to achieve faster processing. Furthermore, since the image resolution is optimized based on the subject recognition accuracy, a resolution sufficient for subject recognition and a minimum data size can be achieved.

[0142] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

[0143] In addition, as part or all of the control according to the embodiment, a computer program for realizing the functions of the above-described embodiment may be supplied to the photoelectric conversion device, etc., via a network or various storage media. Then, the computer (or CPU, MPU, etc.) of the photoelectric conversion device, etc. may be configured to read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention.

[0144] Furthermore, the present invention includes those implemented using at least one processor or circuit configured to perform the functions of the above-described embodiments. For example, a plurality of processors may be used for distributed processing to perform the functions of the above-described embodiments.

[0145] This application claims the benefit of Japanese Patent Application No. 2023-106949, filed on June 29, 2023, which is hereby incorporated by reference herein in its entirety.

Claims

1. A photoelectric conversion device, comprising: A plurality of pixels, each comprising: A sensor unit that emits pulses based on photons. a counter for counting the number of pulses, and A memory, used to store the count value of the counter; and a control section for generating an image signal based on a difference between count values ​​of the counter at the start and end of an accumulation period, In which, the control unit performs a previous reading operation of reading the count value in the middle of a full frame period, and performs image signal reading processing so that the time period used for the previous reading operation is shorter than the time period used to read the main image signal composed of the count values ​​of multiple pixels stored in the full frame period.

2. The photoelectric conversion device according to claim 1, further comprising: An identification unit, configured to perform identification processing for identifying a subject based on a previous image signal; as well as The display unit is used to display a main image signal for visual recognition.

3. The photoelectric conversion device according to claim 1, wherein The control section further reads other image signals between the previous reading operation and the reading operation of the main image signal.

4. The photoelectric conversion device according to claim 1, wherein The control section reduces the amount of data when generating the previous image signal.

5. The photoelectric conversion device according to claim 4, wherein: When generating the previous image signal, the control section reduces the amount of data by adding count values ​​of the plurality of pixels.

6. The photoelectric conversion device according to claim 4, wherein: When generating the previous image signal, the control section reduces the amount of data by thinning out and reading pixels in a row or a column.

7. The photoelectric conversion device according to claim 4, wherein: When generating the previous image signal, the control section reduces the amount of data by reading count values ​​from pixels in a partial area among the plurality of pixels.

8. The photoelectric conversion device according to claim 4, wherein The control section determines a reduction rate of the data amount based on accuracy of subject recognition from the previous image signal.

9. The photoelectric conversion device according to claim 4, wherein: The control section controls a reduction rate of the data amount based on the brightness of the ambient light.

10. The photoelectric conversion device according to claim 9, wherein The control unit reduces the amount of data by thinning out and reading pixels in a row or column when the amount of ambient light is equal to or greater than a predetermined value, and reduces the amount of data by adding count values ​​of a plurality of pixels when the amount of ambient light is less than the predetermined value.

11. The photoelectric conversion device according to claim 1, wherein The sensor unit includes an avalanche photodiode.

12. A photoelectric conversion device comprising: A plurality of pixels, each comprising: A sensor unit that emits pulses based on photons. a counter for counting the number of pulses, and A memory, used to store the count value of the counter; and a control section for generating an image signal based on a difference between count values ​​of the counter at the start and end of an accumulation period, In which, the control unit performs a previous reading operation of reading the count value in the middle of a full frame period, and performs image signal reading processing so that the resolution of the previous image signal read in the previous reading operation is lower than the resolution of the main image signal composed of the count values ​​of multiple pixels stored in the full frame period.

13. A movable device comprising: A plurality of pixels, each comprising: A sensor unit that emits pulses based on photons. a counter for counting the number of pulses, and A memory, used for storing the count value of the counter; a control section for generating an image signal based on a difference between count values ​​of the counter at the start and end of an accumulation period, wherein the control section performs a previous reading operation of reading the count value in the middle of a full frame period, and performs image signal reading processing so that a resolution of the previous image signal read in the previous reading operation is lower than a resolution of a main image signal composed of count values ​​of a plurality of pixels stored in the full frame period; and A movement control section is configured to control the operation of the movable device based on a result of recognition processing using the previous image signal.

14. A photoelectric conversion method for performing photoelectric conversion by a plurality of pixels, each pixel comprising a sensor unit for emitting a pulse according to a photon, a counter for counting the number of pulses, and a memory for storing a count value of the counter, the photoelectric conversion method comprising: generating an image signal based on a difference between count values ​​of the counter at the beginning and the end of an accumulation period; as well as A previous reading operation of reading the count value midway through a full frame period is performed, and image signal reading processing is performed so that a time period used for the previous reading operation is shorter than a time period used for reading a main image signal composed of count values ​​of a plurality of pixels stored in the full frame period.

15. A non-transitory computer-readable storage medium configured to store a computer program for controlling a photoelectric conversion device, the photoelectric conversion device being configured to have photoelectric conversion performed by a plurality of pixels, each pixel comprising a sensor unit for emitting a pulse according to a photon, a counter for counting the number of pulses, and a memory for storing a count value of the counter, in, The computer program includes instructions for performing the following processes: generating an image signal based on a difference between count values ​​of the counter at the beginning and the end of an accumulation period; as well as A previous reading operation of reading the count value midway through a full frame period is performed, and image signal reading processing is performed so that a time period used for the previous reading operation is shorter than a time period used for reading a main image signal composed of count values ​​of a plurality of pixels stored in the full frame period.

16. A computer program product comprising a computer program for controlling a photoelectric conversion device, the photoelectric conversion device being configured to have photoelectric conversion performed by a plurality of pixels, each pixel comprising a sensor unit for emitting a pulse according to a photon, a counter for counting the number of pulses, and a memory for storing a count value of the counter, in, The computer program includes instructions for performing the following processes: generating an image signal based on a difference between count values ​​of the counter at the beginning and the end of an accumulation period; as well as A previous reading operation of reading the count value midway through a full frame period is performed, and image signal reading processing is performed so that a time period used for the previous reading operation is shorter than a time period used for reading a main image signal composed of count values ​​of a plurality of pixels stored in the full frame period.

Images