Photoelectric conversion apparatus, mobile device, control method, storage medium, and computer program product

Abstract

The invention provides a photoelectric conversion apparatus, a mobile device, a control method, a storage medium, and a computer program product. The photoelectric conversion device includes: a plurality of pixels each including a photoelectric conversion unit for emitting pulses corresponding to photons, a counter for counting the number of pulses, and a memory for storing a count value of the counter; and a control means for generating a signal on the basis of a difference between a count value of the counter at the start of the accumulation period and a count value at the end of the accumulation period, and for performing control such that the signal generated in the first accumulation period is output between the end of the first accumulation period and the end of the second accumulation period, one full frame period comprises a first accumulation period and a second accumulation period longer than the first accumulation period; and a recognition processing means for performing a recognition process based on the image signal and for changing a recognition frequency within one full frame period based on the subject information in the recognition process.

Description

Technical Field

[0001] The present invention relates to a photoelectric conversion device, a movable device, a control method, a storage medium and the like. Background Art

[0002] In recent years, a photoelectric conversion device has been developed that digitally counts the number of photons incident on an avalanche photodiode (APD) and outputs the count value from a pixel as a photoelectric conversion digital signal. For example, Japanese Patent No. 7223070 discloses a structure in which a photoelectric conversion device including an APD can output a plurality of images whose accumulation periods overlap each other, thereby enabling continuous imaging even under low illumination.

[0003] However, for example, assuming that the image pickup element used for the vehicle-mounted camera installed in the mobile device performs recognition processing in units of frames in normal sensor driving, recognition processing can only be performed every 33.3ms at 30fps. Therefore, even when an object appears in the vehicle-mounted camera immediately after the frame change, recognition processing cannot be performed until the frame period ends.

[0004] In addition, in a vehicle-mounted camera, in order to suppress flickering caused by traffic lights, the accumulation period is often set to a specific period (11ms) or more, and in particular, a long accumulation period is set to capture a bright image under low illumination. However, due to the long accumulation period, subject blurring may occur depending on the condition of the vehicle or the condition of the subject, and it is considered that the recognition rate is reduced.

[0005] In addition, if there is an approaching subject in the moving direction of the vehicle, it is necessary to pay attention to the subject because it is dangerous. In this case, the recognition processing in frames may not be frequent enough to confirm the status of the subject, which may cause delays in responding to emergencies. Summary of the invention

[0006] A photoelectric conversion device according to one aspect of the present invention comprises:

[0007] a plurality of pixels, each pixel including a photoelectric conversion unit for emitting a pulse corresponding to a photon, a counter for counting the number of the pulses, and a memory for storing a count value of the counter;

[0008] a control section for generating a signal based on a difference between a count value of the counter at the start of an accumulation period and a count value of the counter at the end of the accumulation period, and performing control so that the signal generated in the first accumulation period is output between the end of the first accumulation period and the end of a second accumulation period, wherein one full frame period includes the first accumulation period and the second accumulation period longer than the first accumulation period; and

[0009] The recognition processing component is used to perform recognition processing based on the image signal and to change the recognition frequency within the one full frame period based on the object information.

[0010] Further features of the present invention will become apparent from the following description of embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a diagram showing a structural example of a photoelectric conversion element according to an embodiment of the present invention.

[0012] Figure 2 is a diagram showing a structural example of the sensor substrate 11 .

[0013] Figure 3 is a diagram showing a structural example of the circuit board 21 .

[0014] Figure 4 It is shown Figure 2 and Figure 3 1 is a diagram of an equivalent circuit of a pixel 101 and a signal processing circuit 103 corresponding to the pixel 101.

[0015] Figure 5 2 is a diagram schematically showing the relationship between the operation of the APD 201 and the output signal.

[0016] Figure 6 is a functional block diagram of a photoelectric conversion apparatus 600 and a movable device 700 according to an embodiment.

[0017] Figure 7 : is a diagram illustrating a photoelectric conversion method performed by the camera control unit 605 according to the embodiment.

[0018] Figure 8 is a diagram showing an example of images of a plurality of divided frames.

[0019] Fig. 9 is a diagram showing the relationship between the memory circuit and the buffer in the embodiment.

[0020] Fig.10 is a flowchart showing details of an example of driving of a photoelectric conversion element in the embodiment.

[0021] Fig.11 yes Fig.10 Continuation of the flowchart.

[0022] Fig.12 is a graph showing the recognition frequency during normal travel in the embodiment.

[0023] Fig.13 This is a diagram showing the recognition frequency when a subject approaching in the moving direction of the vehicle is detected in the embodiment.

[0024] Fig.14 This is a flowchart for explaining changes in the recognition frequency when a subject approaching in the moving direction of the vehicle is detected in the embodiment. DETAILED DESCRIPTION

[0025] Hereinafter, an advantageous mode of the present invention will be described using embodiments with reference to the accompanying drawings. In each of the drawings, the same reference numerals are applied to the same components or elements, and repeated descriptions will be omitted or simplified.

[0026] Figure 1 1 is a diagram showing a structural example of a photoelectric conversion element according to an embodiment of the present invention. Hereinafter, a description will be given of an example of a photoelectric conversion device having a so-called stacked structure in which a photoelectric conversion element 100 is composed of two substrates (i.e., a sensor substrate 11 and a circuit board 21) stacked and electrically connected.

[0027] However, the photoelectric conversion device may have a so-called non-stacked structure in which components included in the sensor substrate and components included in the circuit board are arranged in a common semiconductor layer. The sensor substrate 11 includes a pixel area 12. The circuit board 21 includes a circuit area 22 for processing a signal detected in the pixel area 12.

[0028] Figure 2 1 is a diagram showing a structural example of the sensor substrate 11. The pixel region 12 of the sensor substrate 11 includes a plurality of pixels 101 arranged two-dimensionally across a plurality of rows and columns. Each pixel 101 includes a photoelectric conversion unit 102 including an avalanche photodiode (hereinafter referred to as APD), and the photoelectric conversion unit 102 emits pulses at a frequency corresponding to a reception frequency of photons. The number of rows and columns of the pixel array forming the pixel region 12 is not particularly limited.

[0029] Figure 3 2 is a diagram showing an example of the structure of the circuit board 21. The circuit board 21 includes Figure 2 A signal processing circuit 103, a readout circuit 112, a control pulse generating circuit 115, a horizontal scanning circuit 111, a vertical signal line 113, a vertical scanning circuit 110 and an output circuit 114 are provided for processing the charges photoelectrically converted by each photoelectric conversion unit 102.

[0030] The vertical scanning circuit 110 receives the control pulse supplied from the control pulse generating unit 115 and sequentially supplies the control pulse to a plurality of pixels arranged in the row direction. As the vertical scanning circuit 110, a logic circuit such as a shift register and an address decoder is used.

[0031] The signal output from the photoelectric conversion unit 102 of each pixel is processed by each signal processing circuit 103. The signal processing circuit 103 is provided with a counter and a memory, etc., and stores digital values ​​in the memory. The horizontal scanning circuit 111 inputs a control pulse for sequentially selecting a column to the signal processing circuit 103 to read a signal from the memory of each pixel storing the digital signal.

[0032] The signal processing circuit 103 of the pixel of the row selected by the vertical scanning circuit 110 outputs a signal to the vertical signal line 113. The signal output to the vertical signal line 113 is output to the outside of the photoelectric conversion element 100 via the readout circuit 112 and the output circuit 114. The readout circuit 112 has a plurality of buffers connected to the vertical signal line 113 built therein.

[0033] like Figure 2 and Figure 3 As shown in FIG. 1 , a plurality of signal processing circuits 103 are arranged in a region overlapping with the pixel region 12 in a plan view. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged so that an edge of the sensor substrate 11 and an edge of the pixel region 12 overlap with each other in a plan view.

[0034] That is, the sensor substrate 11 has a pixel region 12 and a non-pixel region arranged around the pixel region 12. In addition, a vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating circuit 115 are arranged in a region overlapping the non-pixel region in a plan view.

[0035] The configuration of the vertical signal line 113, the readout circuit 112, and the output circuit 114 is not limited to Figure 3 For example, the vertical signal line 113 may be arranged to extend in the row direction, and the readout circuit 112 may be arranged at the end where the vertical signal line 113 extends. In addition, the signal processing circuit 103 does not necessarily need to be provided for each photoelectric conversion unit, and one signal processing unit may be shared by a plurality of photoelectric conversion units to perform signal processing in sequence.

[0036] Figure 4 It is shown Figure 2 and Figure 3 1 is a diagram of an equivalent circuit of a pixel 101 and a signal processing circuit 103 corresponding to the pixel 101.

[0037] The APD 201 included in the photoelectric conversion unit 102 generates a charge pair corresponding to the incident light by photoelectric conversion. One of the two nodes of the APD 201 is connected to a power supply line that supplies a driving voltage VL (first voltage). The other of the two nodes of the APD 201 is connected to a power supply line that supplies a driving voltage VH (second voltage) higher than the voltage VL.

[0038] exist Figure 4 In the embodiment, one node of the APD 201 is an anode and the other node of the APD 201 is a cathode. A reverse bias voltage is supplied to the anode and cathode of the APD 201 so that the APD 201 performs an avalanche multiplication operation. By supplying such a voltage, the charge generated by the incident light causes avalanche multiplication and generates an avalanche current.

[0039] When a reverse bias voltage is supplied, there is a Geiger mode in which the anode and cathode operate with a voltage difference greater than the breakdown voltage, and a linear mode in which the anode and cathode operate with a voltage difference close to or equal to or less than the breakdown voltage. An APD operating in the Geiger mode is called a SPAD. In the case of a SPAD, for example, the drive voltage VL (first voltage) is -30V and the drive voltage VH (second voltage) is 1V.

[0040] The signal processing circuit 103 includes a quenching element 202, a waveform shaping unit 210, a counter circuit 211, and a memory circuit 212. The quenching element 202 is connected to a power supply line supplying a driving voltage VH and one of an anode node and a cathode node of the APD 201.

[0041] The quenching element 202 serves as a load circuit (quenching circuit) during signal multiplication by avalanche multiplication, and has a function of suppressing the voltage to be supplied to the APD 201 and suppressing avalanche multiplication (quenching operation). The quenching element 202 also has a function of restoring the voltage to be supplied to the APD 201 to the driving voltage VH by applying a current equivalent to the voltage drop caused by the quenching operation (charging operation).

[0042] Figure 4 An example is shown in which the signal processing circuit 103 includes a waveform shaping unit 210 , a counter circuit 211 , and a memory circuit 212 in addition to the quenching element 202 .

[0043] The waveform shaping unit 210 shapes the voltage change of the cathode of the APD 201 obtained when a photon is detected, and outputs a pulse signal. For example, an inverter circuit is used as the waveform shaping unit 210. Figure 4An example in which one inverter is used as the waveform shaping unit 210 is shown, but a circuit in which a plurality of inverters are connected in series or other circuits having a waveform shaping effect may be used.

[0044] The counter circuit 211 counts the number of pulses output from the waveform shaping unit 210 and stores the count value. In addition, when the control pulse RES is supplied via the drive line 213, the signal stored in the counter circuit 211 is reset. Here, the counter circuit 211 generates a signal based on the difference between the count value at the beginning of the accumulation period and the count value at the end of the accumulation period.

[0045] from Figure 3 The vertical scanning circuit 110 in Figure 4 The driving line 214 ( Figure 3 The memory circuit 212 is supplied with a control pulse SEL (not shown) and switches electrical connection and disconnection between the counter circuit 211 and the vertical signal line 113. The memory circuit 212 serves as a memory for temporarily storing the count value of the counter and outputs the output signal from the counter circuit 211 of the pixel to the vertical signal line 113.

[0046] A switch such as a transistor may be arranged between the quenching element 202 and the APD 201 or between the photoelectric conversion unit 102 and the signal processing circuit 103 to switch the electrical connection. Similarly, a switch such as a transistor may be used to electrically switch the supply of the driving voltage VH or the driving voltage VL supplied to the photoelectric conversion unit 102.

[0047] Figure 5 2 is a diagram showing a schematic diagram of the relationship between the operation and the output signal of the APD 201. It is assumed that the input side of the waveform shaping unit 210 is a node A, and the output side of the waveform shaping unit 210 is a node B. Between time t0 and time t1, a potential difference between VH and VL is applied to the APD 201. When a photon is incident on the APD 201 at time t1, avalanche multiplication occurs in the APD 201, an avalanche multiplication current flows into the quenching element 202, and the voltage of the node A drops.

[0048] When the voltage drop becomes even larger and the potential difference applied to the APD 201 becomes smaller, the avalanche multiplication of the APD 201 stops as at time t2, and the voltage level of the node A does not drop below a certain value.

[0049] Thereafter, between time t2 and time t3, current flows from voltage VL to node A to compensate for the voltage drop, and node A is in a static state of its original potential level at time t3. At this time, a portion of the output waveform of node A exceeding a certain threshold is shaped by the waveform shaping unit 210, and node B outputs a waveform as a pulse signal.

[0050] Next, the photoelectric conversion device 600 and the movable device 700 according to the embodiment will be described. Figure 6 is a functional block diagram of a photoelectric conversion apparatus 600 and a movable device 700 according to an embodiment. Figure 6 Some of the functional blocks shown in FIG. 7 are realized by causing a computer (not shown) included in the photoelectric conversion apparatus 600 and the movable device 700 to execute a computer program stored in a memory serving as a storage medium (not shown).

[0051] However, Figure 6 Some or all of the functional blocks shown in the figure may be implemented by hardware. As hardware, a dedicated circuit (ASIC) or a processor (reconfigurable processor (DSP)) may be used. In addition, Figure 6 The functional blocks shown in the figure may be built in the same housing and may be constituted by separate devices connected to each other via signal paths.

[0052] The photoelectric conversion device 600 includes Figures 1 to 5 The photoelectric conversion element 100, the imaging optical system 601, the image processing unit 603, the recognition unit 604, the camera control unit 605, the storage unit 606, and the communication unit 607 described in the embodiment of the present invention are shown in FIG. The photoelectric conversion unit of the photoelectric conversion element 100 is composed of Figures 1 to 5 The avalanche photodiode structure for photoelectric conversion of optical images described in .

[0053] The photoelectric conversion device according to the embodiment is installed in the movable device 700, and the camera unit including the imaging optical system 601 and the photoelectric conversion element 100 is configured to capture images in at least one direction of the front, rear and side of the movable device. A plurality of camera units may be provided on the movable device 700.

[0054] The image processing unit 603 performs image processing such as black level correction, gamma curve adjustment, noise reduction, digital gain adjustment, demosaicing, and data compression on the image signal acquired by the photoelectric conversion element 100 to generate a final image signal. In the case where the photoelectric conversion element 100 includes an on-chip color filter such as RGB, it is desirable to perform processing such as white balance correction and color conversion in the image processing unit 603.

[0055] Furthermore, the output of the image processing unit 603 is supplied to the recognition unit 604, the electric control unit (ECU) 701 of the movable device 700, and the camera control unit 605. The recognition unit 604 functions as a recognition processing unit for performing recognition processing based on an image signal generated from a signal generated by the photoelectric conversion unit, recognizes a person or vehicle nearby, etc. by performing image recognition based on the image signal, and issues a warning, etc. as necessary.

[0056] In the present embodiment, the movable device 700 is described using an example of a car, but as long as the movable device is movable, the movable device may be any one of an airplane, a train, a ship, a drone, an AGV, or a robot.

[0057] The camera control unit 605 has a built-in CPU as a computer and a memory for storing computer programs, and the CPU executes the computer program stored in the memory to control each unit of the photoelectric conversion apparatus 600 .

[0058] The camera control unit 605 functions as a control section, and controls the length of the exposure period of each frame of the photoelectric conversion element 100 and the timing of the control signal CLK via, for example, a control pulse generating unit of the photoelectric conversion element 100. Although the details will be described later, the camera control unit 605 also controls the recognition processing of the recognition unit 604 based on the output from the ECU 701.

[0059] The storage unit 606 includes a recording medium such as a memory card or a hard disk, and can store and read image signals. The communication unit 607 includes a wireless interface or a wired interface, outputs the generated image signal to the outside of the photoelectric conversion device 600, and receives various signals from the outside.

[0060] The ECU 701 includes a built-in CPU as a computer and a memory for storing a computer program, and the CPU executes the computer program stored in the memory to control each unit of the movable device 700 .

[0061] A subject detection unit 704 is also connected, and detection information about the subject is input to the ECU 701. Examples of information detected by the subject detection unit 704 include a moving direction of the subject, a type of the subject, an amount of change of the subject per unit time, a moving speed of the subject, and a position of the subject, etc. from the detection information of the subject.

[0062] In addition, Figure 6 In the embodiment of the present invention, the subject is detected based on the image acquired by the photoelectric conversion device 600 , but the subject may be detected based on an image signal output from a sensor other than the photoelectric conversion device 600 .

[0063] The output of the ECU 701 is supplied to the vehicle control unit 702, the display unit 703, and the camera control unit 605. The vehicle control unit 702 functions as a movement control unit for performing drive, stop, and direction control of a vehicle as a movable device based on the output of the ECU 701.

[0064] The vehicle control unit 702 is a mobile control unit that controls the operation of the movable device based on the recognition result obtained by performing recognition processing using a signal generated at least during the first accumulation period. The first accumulation period will be described later. The display unit 703 serves as a display component, includes a display element such as a liquid crystal device or an organic EL, and is installed in the movable device 700.

[0065] The display unit 703 displays an image acquired by the photoelectric conversion element 100 and various information on the running state of the vehicle and the like to the driver of the movable device 700 based on the output of the ECU 701 by using, for example, a GUI.

[0066] Furthermore, the camera control unit 605 receives input of subject information detected by the subject detection unit 704 via the ECU 701, and receives output of a control signal for changing the image recognition frequency within one full frame period. The change of the recognition frequency will be described later.

[0067] Figure 6 The image processing unit 603 and the recognition unit 604 etc. may not be installed in the movable device 700, and may be set in, for example, an external terminal which is set separately from the movable device 700 and is used to remotely control the movable device 700 or monitor the travel of the movable device.

[0068] Figure 7 is a diagram showing a photoelectric conversion method performed by the camera control unit 605 according to the embodiment. In the present embodiment, one frame period having a length of 33.3 ms is referred to as a full frame (or main frame), and four parts into which the full frame is divided are referred to as frames (or subframes).

[0069] In the present embodiment, one full frame period having a length of 33.3 ms is divided into four parts.

[0070] That is to say, Figure 7 As shown in , the full frame 1 is divided into frames 1_1, 1_2, 1_3, and 1_4 of equal duration (8.33 ms). Figure 7 In and subsequent figures, frame 0_3, frame 0_4, frame 1_1, frame 1_2, frame 1_3, frame 1_4, frame 2_1 and frame 2_2 are displayed as F0_3, F0_4, F1_1, F1_2, F1_3, F1_4, F2_1 and F2_2.

[0071] Frame 1_1 has an accumulation period from the start time T0 of the full frame 1 to time T1, and frame 1_2 has an accumulation period from time T0 to time T2. Frame 1_3 has an accumulation period from time T0 to time T3, and frame 1_4 has an accumulation period from time T0 to time T4. In the present embodiment, for example, the accumulation period from time T0 to time T1 is referred to as the first accumulation period, and the accumulation period from time T0 to time T4 is referred to as the second accumulation period.

[0072] Then, the counter circuit 211 is reset at time T0 , and count values ​​C1_1 , C1_2 , C1_3 , and C1_4 are obtained from the counter circuit 211 at time T1 to time T4 .

[0073] The count values ​​C1_1, C1_2, C1_3, and C1_4 are temporarily stored in the memory circuit 212. Signals corresponding to one row temporarily stored in the memory circuit 212 are sequentially output from the photoelectric conversion elements via the buffer of the readout circuit 112.

[0074] In this way, according to the present embodiment, the signal accumulated during the period of frame 1_1 is read out from time T1 to time T2, and is quickly processed by the recognition unit 604. Therefore, image recognition can be performed quickly. Similarly, the signals accumulated during the periods of frame 1_2, frame 1_3, and frame 1_4 are sequentially read out from time T2 to time T3, from time T3 to time T4, and from time T4 to time T1, respectively, and image recognition can be repeatedly performed.

[0075] Figure 8 is a diagram showing an example of an image of a plurality of individual frames. Figure 8 As shown in , the image of frame 1_1 is dark due to the short accumulation time, but there is almost no subject blur of a person jumping out. On the other hand, the accumulation time becomes longer in the order of frame 1_2, frame 1_3, and frame 1_4, so subject blur is more likely to occur. Blurring is less likely to occur in stopped vehicles and white lines, and the longer the accumulation time, the more likely the contrast is to increase.

[0076] In this way, in the present embodiment, one frame period has a first accumulation period and a second accumulation period, the first accumulation period is shorter than the second accumulation period, and control is performed so that a signal generated during the first accumulation period is output between the end of the first accumulation period and the end of the second accumulation period.

[0077] In this embodiment, the first accumulation period and the second accumulation period overlap each other, and the first accumulation period and the second accumulation period start at the same time. In addition, the end of the second accumulation period is an interruption of the frame period, and the second accumulation period is an integer multiple of the first accumulation period. That is, the end of the second accumulation period coincides with the end of the full frame period.

[0078] However, the second accumulation period does not need to be an integral multiple of the first accumulation period, and it is sufficient that the second accumulation period is longer than the first accumulation period and the end of the second accumulation period is later than the end of the first accumulation period.

[0079] That is, an image with a short accumulation period and an image with a long accumulation period are created, and the timing of the end of the short accumulation period is set to be earlier than the timing of the end of the long accumulation period. As soon as the short accumulation period ends, the image is output and transmitted to the recognition unit of the subsequent stage. Then, the subject is recognized based on the signal generated during at least the first accumulation period. The recognition unit 604 as the recognition component recognizes the subject based on the signal generated during at least the first accumulation period.

[0080] Therefore, in the prior art, image recognition cannot be performed until one frame has passed, whereas in the present embodiment, image recognition can be performed after at least a quarter of a frame period, and, for example, when the movable device is moving at high speed, an obstacle or the like can be quickly recognized. Therefore, the brakes can be quickly applied. Alternatively, the obstacle can be avoided in advance.

[0081] The contrast of an image with a long accumulation time can be improved, so that the image can be used as a display image. That is, an image with a short accumulation time is suitable for rapid image recognition, and an image with a long accumulation time is suitable for displaying an image. In this way, the display unit in the present embodiment displays the signal generated during at least the second accumulation period as an image. In the present embodiment, the recognition processing unit further recognizes the subject based on the signal generated during the second accumulation period.

[0082] In addition, since APD is used in this embodiment, unlike CMOS sensors, the charge accumulated by reading out does not deteriorate, so the accumulation periods can overlap each other. In addition, since there is no readout noise, the original signal will not deteriorate no matter how many times readout is performed by one accumulation.

[0083] Fig. 9 is a diagram showing the relationship between the storage circuit and the buffer in the embodiment. Fig. 9 middle, Figure 3 The memory circuit 212 in the signal processing circuit 103 shown in FIG. 1 is configured into N rows and M columns, and the memory circuits are shown as memory 1-1 to memory NM. In addition, Fig. 9 Buffer 1 to buffer M in Figure 3 The buffer included in the readout circuit 112. Fig. 9 The output circuit 114 and Figure 3 Corresponding to the output circuit 114 in.

[0084] Fig.10 is a flowchart showing details of an example of driving of a photoelectric conversion element in the embodiment, and Fig.11 yes Fig.10 The following is a continuation of the flowchart of . The following is sequentially performed by causing the CPU or the like as a computer in the camera control unit 605 to execute a computer program stored in the memory. Fig.10 and Fig.11 The operations of the steps in the flowchart.

[0085] exist Fig.10 In step S101, the CPU in the camera control unit 605 sets i=1, and in step S102 outputs the count value Count of the counter circuit 211 at time Ti to the memory circuit 212. At this time, the count value is output to all memory circuits at the same time. This operation is the same as Figure 7 It corresponds to the operation at time T1 in .

[0086] Next, in step S103, the CPU in the camera control unit 605 sets j=1, and in step S104 Fig. 9 The count value Count(jki) of the memory circuit jk in the memory circuit is output to the buffer. At this time, the count value is simultaneously output to the buffers in columns 1 to M. This operation means that Fig. 9 An operation that places the count value in the first row of the buffer.

[0087] Next, in step S105, the CPU in the camera control unit 605 sets k=1, and in step S106 outputs the count value Count(jki) of the buffer k to the output circuit 114. This operation is similar to reading out the count value from the output circuit. Fig. 9 The operation corresponds to the signal in the buffer in the leftmost column.

[0088] Next, the process enters via A Fig.11 , and the CPU in the camera control unit 605 determines in step S107 whether k<M. If the determination result is "yes", the CPU sets k=k+1 in step S108 and increments k by 1. Thereafter, the process returns to step S106 via B, and the operation of step S106 is performed. This operation is similar to the operation of reading out the output circuit. Fig. 9 The operation of the signal of the buffer in the second column from the left corresponds to the operation of the signal of the buffer in the second column from the left.

[0089] If the result of the determination in step S107 is "No", that is, k=M, it means that the output circuit is read out. Fig. 9 The signal of the buffer of the Mth column in the image has ended. Next, the process advances to step S109, and the CPU in the camera control unit 605 determines whether j<N.

[0090] If the determination result in step S109 is YES, the CPU in the camera control unit 605 sets j=j+1 in step S110 , increments j by 1, and returns to step S104 via C. This corresponds to an operation for starting readout of the next line.

[0091] In the case where the determination result in step S109 is "No", it means that the readout of all rows has ended. Therefore, the process proceeds to step S111, and the CPU in the camera control unit 605 determines whether j<4. In the case where the determination result in step S111 is "Yes", the process proceeds to step S112, and the CPU in the camera control unit 605 sets i=i+1, increments i by 1, and returns to step S102 via D. This operation corresponds to the operation of starting the readout at the next time T2.

[0092] If the result of determination in step S111 is "No", it means that the readout at time T4 has been completed. Therefore, the process proceeds to step S113, and the CPU in the camera control unit 605 resets the counter circuit 211 using the reset signal. This operation is similar to Figure 7 This corresponds to the reset operation of the counter circuit 211 at the time T4 in FIG.

[0093] Next, refer to Fig.12 and Fig.13 A method of changing the frequency of image recognition performed by the recognition unit 604 based on the subject information detected by the subject detection unit 704 in the present embodiment is described. The subject information includes information such as the moving direction of the subject.

[0094] Fig.12 : is a graph showing the recognition frequency during normal travel in the embodiment, which is the frequency when the moving direction of the subject is a direction away from the moving direction of the vehicle. Figure 7 As shown, the full frame 1 is divided into frames 1_1, 1_2, 1_3 and 1_4 of equal time period (8.33 ms).

[0095] At this time, the ECU 701 determines whether the subject is moving away from the moving direction of the vehicle based on the position information and moving direction information of the subject detected by the subject detection unit 704. When it is determined that the subject is moving away from the moving direction of the vehicle, the ECU 701 notifies the camera control unit 605 to read the signal of the accumulation period of frame 1_2 and frame 1_4, and performs two image recognitions.

[0096] Regarding the moving direction of the vehicle, the road may be detected based on the video signal acquired by the photoelectric conversion device 600, and the moving direction of the vehicle may be determined based on the detection information, or the moving direction of the vehicle may be determined based on the road detection result obtained from the video signal and the steering angle of the vehicle. The moving direction of the vehicle may also be determined based on map information and GPS information obtained from a car navigation system or the like.

[0097] In this embodiment, even when the frequency of image recognition is reduced, image recognition can be performed every 1 / 2 full frame period, so obstacles can be quickly recognized while the movable device is moving. Therefore, the brakes can be quickly applied or obstacles can be avoided in advance.

[0098] then, Fig.13 : is a diagram showing the recognition frequency when a subject approaching in the moving direction of the vehicle is detected in the embodiment, and the recognition frequency is the recognition frequency when the moving direction of the subject is a direction approaching relative to the moving direction of the vehicle. Figure 7 As shown, the full frame 1 is divided into frames 1_1, 1_2, 1_3 and 1_4 of equal time period (8.33 ms).

[0099] In this embodiment, when the moving direction of the subject detected by the subject detection unit 704 is a direction approaching relative to the moving direction of the vehicle, the signals of the accumulation periods of frame 1_1, frame 1_2, frame 1_3 and frame 1_4 are read out respectively. Then, the ECU 701 notifies the camera control unit 605 to perform four image recognitions.

[0100] In this way, in the present embodiment, the intervals of the timings at which the recognition processing unit starts recognition are set to be equal.Then, the recognition processing unit performs recognition processing based on the signal read out for each first accumulation period within the second accumulation period.

[0101] In the prior art, image recognition cannot be performed until a full frame period has passed, whereas in the present embodiment, image recognition can be performed by reading out the signal multiple times every 1 / 4 frame period, and a more accurate judgment can be made quickly with respect to the movement of obstacles, etc. Alternatively, obstacles can be avoided in advance.

[0102] Fig.14 The flowchart is a flowchart showing the change of the recognition frequency when a subject approaching in the moving direction of the vehicle is detected in the embodiment. The CPU or the like as a computer in the ECU 701 executes a computer program stored in a memory. Fig.14 The operations of the steps in the flowchart.

[0103] exist Fig.14 In step S141, the CPU of the ECU performs initial setting of the frequency of the recognition process. First, before the subject is detected, the frequency of the recognition process is set to the initial setting of every 1 / 2 full frame period. When the initialization of the recognition process is completed, the process proceeds to step S142.

[0104] Next, in step S142 , the CPU of the ECU detects a subject using the subject detection unit 704 , acquires an output through the ECU 701 , and proceeds to step S143 .

[0105] Next, in step S143 , the CPU of the ECU calculates the moving direction of the subject based on the output of the subject detection unit 704 , and proceeds to step S144 .

[0106] Next, in step S144, the CPU of the ECU determines whether the moving direction of the subject is a direction approaching relative to the moving direction of the vehicle, and makes a result. When the moving direction of the subject is a direction approaching relative to the moving direction of the vehicle (yes), the process proceeds to step S145, and when the moving direction of the subject is a direction moving away from the moving direction of the vehicle (no), the process proceeds to step S146.

[0107] When the process enters step S145, the CPU of the ECU sets the recognition process to be performed every 1 / 4 full frame period so that image recognition is performed four times within one full frame period. After setting the recognition process within one frame period for every 1 / 4 full frame period, the process enters step S147.

[0108] In step S147, the CPU of the ECU causes the ECU 701 to notify the camera control unit 605 of the recognition frequency of the recognition processing within one full frame period set in step S145, and proceed to step S142 to continue processing. The camera control unit 605 notified by the CPU of the ECU 701 outputs a control signal for changing the recognition frequency to the recognition unit 604.

[0109] Next, the processing when the processing proceeds to step S146 in step S144 will be described. When the processing proceeds to step S146, the CPU of the ECU sets the recognition processing to be performed every 1 / 2 full frame period so that image recognition is performed twice within one full frame period.

[0110] Then, in step S147, the CPU of the ECU causes the ECU 701 to notify the camera control unit 605 of the recognition frequency of the recognition processing within one full frame period set in step S146, and proceeds to step S142 to continue the processing.

[0111] That is, when the subject is moving away, the support of image recognition is minimized, warnings for recognition processing are not frequently issued, and the processing load is reduced. In addition, when the subject approaches, the frequency of image recognition increases to be able to avoid accidents.

[0112] In this way, in the recognition unit 604 according to the present embodiment, the frequency of image recognition within one full frame period can be changed based on the condition of the subject (subject information), thereby optimizing the recognition accuracy of the subject.

[0113] In the present embodiment, the case where the frequency of image recognition is changed when a predetermined switching condition is satisfied based on the moving direction of the subject indicating whether the subject is approaching as subject information is described, but the subject information is not limited thereto.

[0114] For example, as an example of subject information for changing the image recognition frequency when the car is traveling, the moving speed of the subject can be detected, and the switching condition can be changed according to the moving speed of the subject. For example, control can be performed so that the recognition frequency increases as the moving speed of the subject becomes higher. Alternatively, the recognition frequency can be changed based on both the moving speed and direction of the subject. For example, control can be performed so that the recognition frequency increases as the relative approach speed of the subject to the vehicle becomes higher.

[0115] In addition, in the present embodiment, there is no limitation on the kind of subject, but the subjects to be detected may be limited to animals, cars, bicycles, people, and the like, and the switching condition may be changed according to the kind of subject.

[0116] That is, for example, control can be performed so that the recognition frequency when the detected subject is a person or a vehicle carrying people is higher than the recognition frequency when the detected subject is an animal. In this way, the subject information includes at least one of the moving speed, moving direction and type of the subject.

[0117] In addition, for example, in the above-mentioned embodiment, the accumulation is performed for a minimum of 1 / 4 full frame period, but the accumulation may be performed for 1 / N full frame period (N is an integer). Even when the readout cycle is set to every 1 / N full frame period, the actual accumulation period may be set to be shorter than 1 / N full frame period by resetting the counter circuit midway during the 1 / N full frame period, depending on the brightness of the subject, the accuracy of image recognition, etc.

[0118] Alternatively, you can Figure 7 The counter circuit is reset once at time T1 in the example. Thus, the count value read out at subsequent time T2, time T3, time T4, etc. can be adjusted.

[0119] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments.The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

[0120] In addition, as part or all of the control according to the embodiment, a computer program that implements the functions of the above-described embodiment may be provided to the photoelectric conversion device, etc., through a network or various storage media. Then, the computer (or CPU, or MPU, etc.) of the photoelectric conversion device, etc. may be configured to read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention.

[0121] Furthermore, the present invention includes those implemented using at least one processor or circuit configured to perform the functions of the above-described embodiments. For example, a plurality of processors may be used to share processing to perform the functions of the above-described embodiments.

[0122] This application claims the benefit of priority from Japanese Patent Application No. 2023-147812, filed on September 12, 2023, which is hereby incorporated by reference in its entirety.

Claims

1. A photoelectric conversion device, comprising: a plurality of pixels, each pixel including a photoelectric conversion unit for emitting a pulse corresponding to a photon, a counter for counting the number of the pulses, and a memory for storing a count value of the counter; a control section for generating a signal based on a difference between a count value of the counter at the start of an accumulation period and a count value of the counter at the end of the accumulation period, and for performing control so that the signal generated in the first accumulation period is output between the end of the first accumulation period and the end of a second accumulation period, wherein one full frame period includes the first accumulation period and the second accumulation period which is longer than the first accumulation period; as well as The recognition processing component is used to perform recognition processing based on the image signal, and is used to change the recognition frequency within the one full frame period based on the object information in the recognition processing.

2. The photoelectric conversion device according to claim 1, wherein The intervals of the timings at which the recognition processing sections start recognition are set to be equal.

3. The photoelectric conversion device according to claim 1, wherein The first accumulation period and the second accumulation period overlap with each other.

4. The photoelectric conversion device according to claim 1, wherein The first accumulation period starts simultaneously with the second accumulation period.

5. The photoelectric conversion device according to claim 1, wherein An end of the second accumulation period coincides with an end of the full frame period.

6. The photoelectric conversion device according to claim 1, wherein The recognition processing section recognizes a subject based on a signal generated in at least the first accumulation period in the recognition processing.

7. The photoelectric conversion device according to claim 6, wherein: The recognition processing section further recognizes the subject based on the signal generated in the second accumulation period in the recognition processing.

8. The photoelectric conversion device according to claim 1, wherein The recognition processing section performs the recognition processing based on the signals read out for each first accumulation period in the second accumulation period in the recognition processing. 9 . The photoelectric conversion apparatus according to claim 1 , further comprising a display unit for displaying at least the signal generated in the second accumulation period as an image.

10. The photoelectric conversion device according to claim 1, wherein The photoelectric conversion unit includes an avalanche photodiode.

11. The photoelectric conversion device according to claim 1, wherein The object information includes a moving direction of the object.

12. The photoelectric conversion device according to claim 1, wherein The object information includes the moving speed of the object.

13. The photoelectric conversion device according to claim 1, wherein The object information includes the type of the object.

14. A movable device comprising: a plurality of pixels, each pixel including a photoelectric conversion unit for emitting a pulse corresponding to a photon, a counter for counting the number of the pulses, and a memory for storing a count value of the counter; a control section for generating a signal based on a difference between a count value of the counter at the start of an accumulation period and a count value of the counter at the end of the accumulation period, and for performing control so that the signal generated in the first accumulation period is output between the end of the first accumulation period and the end of a second accumulation period, wherein one full frame period includes the first accumulation period and the second accumulation period which is longer than the first accumulation period; an identification processing component, configured to perform identification processing based on the image signal, and to change the identification frequency within the one full-frame period based on the subject information in the identification processing; and A movement control component is used to control the operation of the movable device based on the recognition result of the recognition process.

15. A control method for a photoelectric conversion device, the photoelectric conversion device comprising a plurality of pixels, each pixel comprising a photoelectric conversion unit for emitting a pulse corresponding to a photon, a counter for counting the number of the pulses, and a memory for storing a count value of the counter, the control method comprising: generating a signal based on a difference between a count value of the counter at the beginning of an accumulation period and a count value of the counter at the end of the accumulation period; performing control so that a signal generated in the first accumulation period is output between the end of the first accumulation period and the end of the second accumulation period, and one full frame period includes the first accumulation period and the second accumulation period which is longer than the first accumulation period; Perform recognition processing based on image signals; as well as The recognition frequency within the one full frame period is changed based on the object information.

16. A computer-readable storage medium configured to store a computer program for a photoelectric conversion device, in, The photoelectric conversion device includes a plurality of pixels, each pixel including a photoelectric conversion unit for emitting a pulse corresponding to a photon, a counter for counting the number of the pulses, and a memory for storing a count value of the counter, The computer program includes instructions for performing the following processes: generating a signal based on a difference between a count value of the counter at the beginning of an accumulation period and a count value of the counter at the end of the accumulation period; performing control so that a signal generated in the first accumulation period is output between the end of the first accumulation period and the end of the second accumulation period, and one full frame period includes the first accumulation period and the second accumulation period which is longer than the first accumulation period; Perform recognition processing based on image signals; as well as The recognition frequency within the one full frame period is changed based on the object information.

17. A computer program product comprising a computer program for controlling a photoelectric conversion device, in, The photoelectric conversion device includes a plurality of pixels, each pixel including a photoelectric conversion unit for emitting a pulse corresponding to a photon, a counter for counting the number of the pulses, and a memory for storing a count value of the counter, The computer program includes instructions for performing the following processes: generating a signal based on a difference between a count value of the counter at the beginning of an accumulation period and a count value of the counter at the end of the accumulation period; performing control so that a signal generated in the first accumulation period is output between the end of the first accumulation period and the end of the second accumulation period, and one full frame period includes the first accumulation period and the second accumulation period which is longer than the first accumulation period; Perform recognition processing based on image signals; as well as The recognition frequency within the one full frame period is changed based on the object information.

Images