Photoelectric conversion device, mobile, control method, and computer program

Abstract

To provide a photoelectric conversion device which seta a region of interest in an imaging range and reads out image data for a relatively short storage period in the middle in the region of interest.SOLUTION: A photoelectric conversion device comprises: a photoelectric conversion section 102 which generates a pulse in response to the incidence of a photon; and a signal processing circuit 103 each including a counter 211 which counts the number of pulses and a memory 212 in which a count value is stored. A signal is generated on the basis of a difference of count values in the start and end of a storage period in the photoelectric conversion section, and there are included at least a first storage period and a second storage period within one full frame. The first storage period is shorter than the second storage period, and control is performed in such a manner that a signal generated during the first storage period is output from the end of the first storage period to the end of the second storage period.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a photoelectric conversion device, a moving object, a control method, a computer program, and the like. [Background technology]

[0002] 2. Description of the Related Art In recent years, photoelectric conversion devices have been developed that digitally count the number of photons incident on an avalanche photodiode (APD) and output the counted value from a pixel as a photoelectrically converted digital signal.

[0003] For example, Patent Document 1 describes a configuration in which a photoelectric conversion device having an APD can output multiple images whose accumulation periods overlap with each other, enabling continuous shooting even in low illuminance. Patent Document 2 describes a configuration in which signals are read out at a high frame rate from a pixel area in which a moving subject is detected. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 7223070 [Patent Document 2] Patent Publication No. 2021-34786 Summary of the Invention [Problem to be solved by the invention]

[0005] However, when considering the image sensor of a camera, for example, because the recognition process is performed on a frame-by-frame basis when the sensor is normally driven, for example, at 30 fps (frames per second), the recognition process can only be performed every 33.3 ms. Therefore, even if an object appears immediately after a frame change in a normal camera, the recognition process cannot be performed until the end of the frame.

[0006] On the other hand, in a monitoring system using an in-vehicle camera, for example, in order to reduce the risk of collisions, etc., it is required to accurately and quickly recognize vehicles or people that suddenly appear in areas that are in the driver's blind spot.

[0007] For this reason, vehicle-mounted cameras are required to capture and recognize images at high resolution in a short time. However, capturing images at high resolution and a short frame rate poses the problem of increased data volume and power consumption.

[0008] Therefore, an object of the present invention is to provide a photoelectric conversion device that can set a region of interest within a shooting range and read out image data having a relatively short accumulation period in the region of interest midway. [Means for solving the problem]

[0009] A photoelectric conversion device according to one aspect of the present invention comprises: An optical system; A plurality of pixels each including a photoelectric conversion unit that emits a pulse in response to an incidence of a photon, a counter that counts the number of the pulses, and a memory that stores the count value of the counter; a control unit that generates a signal based on a difference between count values ​​of the counter at the start and end of an accumulation period in the photoelectric conversion unit, has at least a first accumulation period and a second accumulation period within one full frame, the first accumulation period is shorter than the second accumulation period, and controls so that the signal generated during the first accumulation period is output between the end of the first accumulation period and the end of the second accumulation period; The sensor unit consisting of the plurality of pixels is characterized by having a setting unit capable of setting a first pixel region having at least the first accumulation period and the second accumulation period, and a second pixel region having at least the second accumulation period. Effect of the Invention

[0010] According to the present invention, it is possible to provide a photoelectric conversion device capable of setting a region of interest within a shooting range and reading out image data having a relatively short accumulation period in the middle of the region of interest. [Brief description of the drawings]

[0011] [Figure 1] 1 is a diagram illustrating an example of the configuration of a sensor unit 100 according to a first embodiment. [Diagram 2] 2 is a diagram showing an example of the configuration of a sensor portion substrate 11 according to the first embodiment. FIG. [Diagram 3] 2 is a diagram showing an example of the configuration of a circuit board 21 according to the first embodiment. FIG. [Figure 4] 2 and 3, this is a diagram showing an equivalent circuit of the photoelectric conversion unit 102 and the signal processing circuit 103 corresponding to the photoelectric conversion unit 102. [Diagram 5] 2 is a diagram illustrating a relationship between the operation of an APD 201 and an output signal. FIG. [Figure 6] 1 is a functional block diagram of a photoelectric conversion device 600 according to a first embodiment. [Figure 7] 6 is a diagram for explaining a photoelectric conversion method performed by a control unit 605 according to the first embodiment. FIG. [Figure 8] FIG. 4 is a diagram showing an example of images of a plurality of divided accumulation periods according to the first embodiment. [Figure 9] FIG. 2 is a diagram illustrating the relationship between a memory and a buffer in the first embodiment. [Figure 10] 5 is a flowchart showing details of an example of driving the sensor unit 100 in the first embodiment. [Figure 11] This is a continuation of the flowchart in Figure 10. [Figure 12] FIG. 4 is a diagram illustrating an example of setting a region of interest for a moving object in the first embodiment. [Figure 13] FIG. 11 is a functional block diagram showing an example of the configuration of a photoelectric conversion device 600 and a moving object 700 according to a third embodiment. [Figure 14] 11 is a flowchart illustrating a method for controlling a moving object according to a third embodiment. [Figure 15]13 is a flowchart illustrating a method for controlling a moving object according to a fourth embodiment. [Figure 16] 20 is a flowchart showing details of an example of driving the sensor unit 100 regarding a method of setting an attention area according to a sixth embodiment. [Figure 17] This is a continuation of the flowchart in Figure 16. [Figure 18] FIG. 13 is a diagram showing an example of images taken for a plurality of accumulation periods after a region of interest is set. [Figure 19] 13 is a flowchart showing details of an example in which an accumulation period is set and the sensor unit 100 is driven in the seventh embodiment. [Figure 20] This is a continuation of the flowchart in Figure 19. [Figure 21] 20 is a flowchart showing an example of an operation of a control unit after an image is constructed according to the eighth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In each drawing, the same members or elements are given the same reference numerals, and duplicated descriptions are omitted or simplified.

[0013] <Embodiment 1> 1 is a diagram showing a configuration example of a sensor unit 100 according to embodiment 1. In the following, the sensor unit has a so-called laminated structure in which two substrates, a sensor unit substrate 11 and a circuit substrate 21, are laminated and electrically connected.

[0014] However, a so-called non-laminated structure in which the components included in the sensor substrate 11 and the components included in the circuit substrate 21 are disposed on a common semiconductor layer may be used. The sensor substrate 11 includes a pixel region 12. The circuit substrate 21 includes a circuit region 22 that processes signals detected in the pixel region 12.

[0015] 2 is a diagram showing an example of the configuration of the sensor portion substrate 11 according to embodiment 1. The pixel region 12 of the sensor portion substrate 11 includes a plurality of pixels 101 arranged two-dimensionally across a plurality of rows and a plurality of columns.

[0016] The pixel 101 includes a photoelectric conversion unit 102 including an avalanche photodiode (hereinafter, APD). Here, the photoelectric conversion unit 102 emits a pulse at a frequency corresponding to the frequency of receiving photons. That is, the photoelectric conversion unit 102 emits a pulse in response to the incidence of photons. The number of rows and columns of the pixel array constituting the pixel region 12 is not particularly limited.

[0017] Fig. 3 is a diagram showing an example of the configuration of a circuit board 21 according to embodiment 1. The circuit board 21 has a signal processing circuit 103 that processes charges photoelectrically converted by each photoelectric conversion unit 102 in Fig. 2, a readout circuit 112, a control pulse generation unit 115, a horizontal scanning circuit 111, a vertical signal line 113, a vertical scanning circuit 110, and an output circuit 114.

[0018] The vertical scanning circuit 110 receives a control pulse supplied from a control pulse generating unit 115, and sequentially supplies the control pulse to a plurality of pixels arranged in the row direction. The vertical scanning circuit 110 is composed of logic circuits such as a shift register and an address decoder.

[0019] The signal output from the photoelectric conversion unit 102 of each pixel is processed by each signal processing circuit 103. The signal processing circuit 103 is provided with a counter, a memory, etc., and a digital value is stored in the memory. The horizontal scanning circuit 111 inputs a control pulse that sequentially selects each column to the signal processing circuit 103 in order to read the signal from the memory of each pixel in which the digital signal is stored.

[0020] A signal is output to the vertical signal line 113 from the signal processing circuit 103 of the pixel of the row selected by the vertical scanning circuit 110. The signal output to the vertical signal line 113 is output to the outside of the sensor unit 100 via the readout circuit 112 and the output circuit 114. The readout circuit 112 has a plurality of buffers built therein, each connected to the vertical signal line 113.

[0021] 2 and 3, a plurality of signal processing circuits 103 are arranged in a region overlapping the pixel region 12 in a plan view. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged in a region of the sensor portion substrate 11 outside the pixel region 12 so as to overlap with each other in a plan view.

[0022] That is, the sensor portion substrate 11 has a pixel region 12 and a non-pixel region arranged around the pixel region 12. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged in a region overlapping the non-pixel region in a plan view.

[0023] The arrangement of the vertical signal lines 113, the readout circuits 112, and the output circuits 114 are not limited to the example shown in Fig. 3. For example, the vertical signal lines 113 may be arranged to extend in the row direction, and the readout circuits 112 may be arranged at the ends of the vertical signal lines 113.

[0024] Moreover, it is not necessary to provide one signal processing circuit 103 for each photoelectric conversion unit. One signal processing circuit may be shared by a plurality of photoelectric conversion units, and may be configured to perform signal processing sequentially.

[0025] Fig. 4 is a diagram showing an equivalent circuit of the photoelectric conversion unit 102 and the signal processing circuit 103 corresponding to the photoelectric conversion unit 102 in Fig. 2 and Fig. 3. The photoelectric conversion unit 102 and the signal processing circuit 103 are included in each pixel. The APD 201 included in the photoelectric conversion unit 102 generates a charge pair according to incident light by photoelectric conversion.

[0026] One of the two nodes of the APD 201 is connected to a power supply line to which a driving voltage VL (first voltage) is supplied, and the other of the two nodes of the APD 201 is connected via a quench element 202 to a power supply line to which a driving voltage VH (second voltage) higher than the voltage VL is supplied.

[0027] In Fig. 4, one node of the APD 201 is the anode, and the other node of the APD is the cathode. A reverse bias voltage is supplied to the anode and cathode of the APD 201 so that the APD 201 performs avalanche multiplication. By supplying such a voltage, the charge generated by the incident light undergoes avalanche multiplication, generating an avalanche current.

[0028] When a reverse bias voltage is supplied, there are two modes: Geiger mode, which operates with the voltage difference between the anode and cathode greater than the breakdown voltage, and linear mode, which operates with the voltage difference between the anode and cathode close to or less than the breakdown voltage. An APD that operates in Geiger mode is called a SPAD. In the case of a SPAD, for example, the voltage VL (first voltage) is -30V and the voltage VH (second voltage) is 1V.

[0029] The signal processing circuit 103 includes a quench element 202, a waveform shaping unit 210, a counter 211, and a memory 212. The quench element 202 is connected to a power supply line to which a driving voltage VH is supplied and one of the anode and cathode nodes of the APD 201.

[0030] The quench element 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, and has a function of suppressing avalanche multiplication by suppressing the voltage supplied to the APD 201 (quench operation).The quench element 202 also has a function of returning the voltage supplied to the APD 201 to the drive voltage VH by flowing a current equivalent to the voltage drop caused by the quench operation (recharge operation).

[0031] 4 shows an example in which the signal processing circuit 103 has a waveform shaping unit 210, a counter 211, and a memory 212 in addition to the quench element 202. The waveform shaping unit 210 shapes the voltage change of the cathode of the APD 201 obtained at the time of photon detection, and outputs a pulse signal.

[0032] For example, an inverter circuit is used as the waveform shaping unit 210. In Fig. 4, an example in which one inverter is used as the waveform shaping unit 210 is shown, but a circuit in which a plurality of inverters are connected in series may be used, or another circuit having a waveform shaping effect may be used.

[0033] The counter 211 counts the number of pulses output from the waveform shaping unit 210 and holds the count value. When a control pulse RES is supplied via a drive line 213, the signal held in the counter 211 is reset. Here, the counter 211 generates a signal based on the difference between the count values ​​at the start and end of the accumulation period.

[0034] 3 to the memory 212 via a drive line 214 (not shown in FIG. 3) in FIG. 4, and switches between electrical connection and disconnection between the counter 211 and the vertical signal line 113. The memory 212 functions as a memory that temporarily stores the count value of the counter, and outputs an output signal from the counter 211 of the pixel to the vertical signal line 113.

[0035] Note that electrical connections may be switched by disposing a switch such as a transistor between the quench element 202 and the APD 201 or between the photoelectric conversion unit 102 and the signal processing circuit 103. Similarly, the supply of the voltage VH or the voltage VL to the photoelectric conversion unit 102 may be electrically switched using a switch such as a transistor.

[0036] 5 is a diagram showing a schematic diagram of the relationship between the operation of the APD 201 and the output signal. The input side of the waveform shaping unit 210 is nodeA, and the output side is nodeB. Between time t0 and time t1, a potential difference of VH-VL is applied to the APD 201. When a photon is incident on the APD 201 at time t1, avalanche multiplication occurs in the APD 201, an avalanche multiplication current flows through the quench element 202, and the voltage at nodeA drops.

[0037] When the amount of voltage drop further increases and the potential difference applied to the APD 201 decreases, the avalanche multiplication of the APD 201 stops as at time t2, and the voltage level of nodeA does not drop below a certain value.

[0038] After that, between time t2 and time t3, a current flows through nodeA to compensate for the voltage drop from voltage VL, and at time t3, nodeA is stabilized to its original potential level. At this time, the portion of the output waveform at nodeA that exceeds a certain threshold is shaped by waveform shaping section 210 and is output as a pulse signal at nodeB.

[0039] Next, a photoelectric conversion device 600 according to the first embodiment will be described. Fig. 6 is a functional block diagram of the photoelectric conversion device 600 according to the first embodiment. Some of the functional blocks shown in Fig. 6 are realized by causing a computer (not shown) included in the photoelectric conversion device 600 to execute a computer program stored in a memory serving as a storage medium (not shown).

[0040] However, a part or all of these may be realized by hardware, which may be a dedicated circuit (ASIC) or a processor (a reconfigurable processor, DSP), etc.

[0041] In addition, the functional blocks shown in Fig. 6 do not have to be built in the same housing, and may be configured by separate devices connected to each other via signal paths. The above explanation regarding Fig. 6 also applies to Fig. 13 described later.

[0042] 1 to 5, an optical system 601, a detection unit 602, an image processing unit 603, a recognition unit 604, a control unit 605, a storage unit 606, a communication unit 607, etc. The sensor unit is composed of an avalanche photodiode for photoelectrically converting an optical image, as described in FIGS.

[0043] The camera unit, which is a set of the optical system 601 and the sensor unit 100, is configured to capture an image in at least one direction, for example, in the front, rear, or side of the photoelectric conversion device 600.

[0044] In this embodiment, the optical system 601 is a wide-angle lens (e.g., a fisheye lens) having an angle of view of, for example, 120°, and forms an optical image (subject image) of an object in front of the photoelectric conversion device 600 on the imaging surface of the sensor unit 100. The detection unit 602 detects information (hereinafter, referred to as environmental information) related to the surrounding environment of the photoelectric conversion device 600. In addition, depending on the output of the detection unit 602, the positions and sizes of an attention area and a non-attention area, which will be described later, are changed.

[0045] The image processing unit 603 performs image processing such as black level correction, gamma correction, noise reduction, digital gain adjustment, demosaic processing, data compression, etc. on the image signal acquired by the sensor unit 100 to generate a final image signal. If the sensor unit has an on-chip color filter such as RGB, the image processing unit 603 performs processing such as white balance correction and color conversion.

[0046] Moreover, the output of the image processing unit 603 is supplied to a recognition unit 604 and a control unit 605. The recognition unit 604 recognizes people, vehicles, objects, etc. captured in the image by performing image recognition based on the image signal. The recognition result by the recognition unit 604 is output to the control unit 605 and is reflected in, for example, changing the control mode of the photoelectric conversion device 600. Furthermore, the recognition result is stored in a storage unit 606 and transmitted to the outside via a communication unit 607, etc.

[0047] The control unit 605 has a built-in CPU as a computer and a memory that stores a computer program. The control unit 605 also functions as a setting unit, and sets the length of the exposure period of each frame of the sensor unit 100 and the timing control of the control signal CLK for each region via the control pulse generation unit 115 of the sensor unit 100.

[0048] Furthermore, the control unit 605 also functions as an acquisition unit, and acquires, as characteristic information of the photoelectric conversion device 600, sensor characteristic information such as the size and number of pixels of the sensor unit 100, and optical characteristic information such as the angle of view and resolution of the optical system 601. Furthermore, the control unit 605 acquires information such as the installation height and installation angle as installation information of the photoelectric conversion device 600 from the detection unit 602, and acquires environmental information related to the surrounding environment of the photoelectric conversion device 600 from the detection unit 602.

[0049] Based on the information acquired from the detection unit 602 , the CPU executes a computer program stored in a memory built into the control unit 605 to control each unit of the photoelectric conversion device 600 .

[0050] The storage unit 606 includes a recording medium such as a memory card, a hard disk, etc., and can store and read image signals. The communication unit 607 includes a wireless or wired interface, and outputs the generated image signal to the outside of the photoelectric conversion device 600 and receives various signals from the outside.

[0051] The photoelectric conversion device 600 is used, for example, as a camera, an in-vehicle camera, a pet camera, a surveillance camera, a camera for detection or inspection used in a manufacturing line, a camera for logistics, etc. In addition, the device can be used for various purposes such as a medical endoscope camera, a condition detection camera for nursing care, a camera for infrastructure inspection, an agricultural camera, etc.

[0052] 7 is a diagram for explaining a photoelectric conversion method by the control unit 605 according to the first embodiment. In this embodiment, photoelectric conversion is periodically driven at, for example, 30 full frames / second. Moreover, a frame for one vertical period having a length of 33.3 ms is called a full frame, and each of the full frames divided into four is called a frame.

[0053] That is, as shown in Fig. 7, a full frame 1 is divided into frames 1_1, 1_2, 1_3, and 1_4 each having an equal period (8.33 ms). Note that in Fig. 7 and subsequent figures, a frame may be abbreviated as F.

[0054] Frame 1_1 has an accumulation period from start time T0 of full frame 1 to time T1, frame 1_2 has an accumulation period from time T0 to time T2, frame 1_3 has an accumulation period from time T0 to time T3, and frame 1_4 has an accumulation period from time T0 to time T4.

[0055] At times T1 to T4, count values ​​C1_1, C1_2, C1_3, and C1_4 are obtained from the counter 211. The count values ​​C1_1, C1_2, C1_3, and C1_4 are temporarily stored in the memory 212.

[0056] Then, the signals for one row temporarily stored in the memory 212 are sequentially output from the sensor unit via the buffer of the readout circuit 112. Also, the counter 211 is reset at time T0. Thus, according to this embodiment, the signals accumulated during the period of frame 1_1 are read out from time T1 to time T2 and quickly processed by the recognition unit 604.

[0057] Therefore, image recognition can be performed quickly. Similarly, signals accumulated during the periods of frame 1_2, frame 1_3, and frame 1_4 are sequentially read out from time T2 to T3, T3 to T4, and T4 to T1, respectively, and image recognition can be performed repeatedly. Note that the length of a full frame is not limited to the above example. Also, the number of divisions of a full frame is not limited to 4.

[0058] Fig. 8 is a diagram showing an example of images of a plurality of divided accumulation periods according to the first embodiment. As shown in Fig. 8, the image of frame 1_1 is dark because the accumulation period is short, but there is little subject blur of a person who jumps out. On the other hand, the accumulation periods of frames 1_2, 1_3, and 1_4 are longer in this order, so subject blur is more likely to occur. Note that blur is less likely to occur in stopped vehicles and white lines, and the longer the accumulation period, the more likely the contrast is to be improved.

[0059] In this manner, in this embodiment, one full frame has a first accumulation period (e.g., time T1 to T2) and a second accumulation period (e.g., time T1 to T4), and the first accumulation period is shorter than the second accumulation period. Also, the signal generated during the first accumulation period is controlled to be output between the end of the first accumulation period and the end of the second accumulation period.

[0060] In this embodiment, the first and second accumulation periods overlap, that is, the first and second accumulation periods start at the same time, the end of the second accumulation period coincides with the end of a full frame, and the second accumulation period is an integer multiple of the first accumulation period.

[0061] However, the second accumulation period does not need to be an integer multiple of the first accumulation period. The second accumulation period only needs to be longer than the first accumulation period (the first accumulation period is shorter than the second accumulation period) and the second accumulation period only needs to end after the first accumulation period.

[0062] That is, an image with a short accumulation period and an image with a long accumulation period are created, the timing at which the short accumulation period ends is set to be earlier than the timing at which the long accumulation period ends, and as soon as the short accumulation period ends, the image is output and sent to the downstream recognition unit 604.

[0063] The subject is recognized based on the signal generated at least during the first accumulation period. The recognition unit 604 recognizes the subject based on the signal generated at least during the first accumulation period. Therefore, while image recognition was not possible until one full frame period had elapsed in the past, image recognition can be performed after as little as ¼ full frame period in this embodiment.

[0064] In addition, an image with a long accumulation period can improve the contrast, and can be used as an image for display. That is, an image with a short accumulation period is suitable for quick subject recognition, and an image with a long accumulation period is suitable for an image for display. Therefore, the display unit 703 of this embodiment displays an image with a long accumulation period, that is, a signal generated during the second accumulation time, as an image.

[0065] In addition, because this embodiment uses an APD, unlike a CMOS image sensor, the accumulated charge is not degraded by reading out, and the accumulation periods can be overlapped. Also, since there is no readout noise, the original signal does not degrade even if it is read out multiple times from one accumulation.

[0066] On the other hand, in the case of a CMOS image sensor, it is possible to shorten the accumulation period by increasing the frame frequency, but each time the accumulation period is shortened, readout noise is superimposed, deteriorating the signal-to-noise ratio (S / N) of the output, making it difficult to obtain an image suitable for image recognition.

[0067] In addition, in order to overlap the accumulation periods in a CMOS image sensor, the same charge must be accumulated in the pixels again after reading out, which makes the circuitry more complex and larger in scale.In addition, noise is also superimposed when accumulating charge in the pixels again, which significantly worsens the S / N ratio of the output, making it difficult to obtain an image suitable for image recognition.

[0068] Fig. 9 is a diagram showing the relationship between memories and buffers in embodiment 1. Fig. 9 shows a state in which memories 212 in signal processing circuit 103 in Fig. 3 are arranged in N rows and M columns, and each memory 212 is represented as memory 1-1 to memory NM. Also, buffer 1 to buffer M in Fig. 9 indicate buffers included in read circuit 112 in Fig. 3. Output circuit 114 in Fig. 9 corresponds to output circuit 114 in Fig. 3.

[0069] Fig. 10 is a flowchart showing details of an example of driving the sensor unit 100 in the first embodiment, and Fig. 11 is a flowchart continuing from Fig. 10. Note that the CPU as a computer in the control unit 605 executes a computer program stored in the memory, thereby sequentially performing the operations of the steps in the flowcharts of Fig. 10 and Fig. 11.

[0070] In step S101 of FIG. 10, the CPU of the control unit 605 sets i = 1. Next, in step S102, the CPU of the control unit 605 causes the memory 212 to output the count value Count of the counter 211 at time Ti. At this time, the output is performed simultaneously for all memories. This operation corresponds to the operation at time T1 in FIG. 7.

[0071] Next, in step S103, the CPU of the control unit 605 sets j = 1. Next, in step S104, the CPU of the control unit 605 sets k = 1. In step S105, the CPU of the control unit 605 causes the buffer to output the count value Count(j - k - i) in the memory j - k of FIG. 9. At this time, the output is performed to the buffer for 1 to M columns simultaneously. This operation means the operation of taking in the count value of the first row in FIG. 9 into the buffer.

[0072] In step S106, the CPU of the control unit 605 causes the output circuit 114 to output the count value Count(j - k - i) of the buffer k. This operation corresponds to the operation of reading out the signal of the buffer in the leftmost column in FIG. 9 from the output circuit.

[0073] Next, proceed to step S107 of FIG. 11 via A. In step S107, the CPU of the control unit 605 determines whether k < M. If Yes in step S107, then in step S108, k is incremented by 1 as k = k + 1, and return to step S106 via B, and perform the operation of step S106. This operation corresponds to the operation of reading out the signal of the buffer in the second column from the left in FIG. 9 from the output circuit.

[0074] When it becomes No in step S107, that is, when k = M, it means that the signal of the buffer in the Mth column in FIG. 9 has been read out from the output circuit. Next, proceed to step S109. The CPU of the control unit 605 determines whether j < N. If Yes in step S109, then in step S110, the CPU of the control unit 605 sets j = J + 1, increments j by 1, and returns to step S104 via C. This corresponds to the operation for starting the reading of the next row.

[0075] If the determination in step S109 is No, this means that reading of all rows has been completed, so the process proceeds to step S111, where the CPU of the control unit 605 determines whether i<4. If the determination in step S111 is Yes, in step S112, the CPU of the control unit 605 sets i=i+1, increments i by 1, and returns to step S102 via D. This operation corresponds to the operation of starting reading at the next time T2.

[0076] If the result of the determination in step S111 is No, this means that the readout at time T4 is completed, so the process proceeds to step S113, where the CPU of the control unit 605 resets the counter 211 with a reset signal. This operation corresponds to the reset operation of the counter 211 at time T4 in Fig. 7. The flows in Figs. 10 and 11 are repeated periodically. In this manner, the signals accumulated in the sensor unit can be read out sequentially.

[0077] In the sensor section, when the circuit configuration is as shown in FIG. 9, it becomes possible to change the setting of the accumulation period for each column.

[0078] For example, in the first column (k=1), four pieces of data are read out by changing the accumulation period of the full frame 1 to frame 1_1, frame 1_2, frame 1_3, and frame 1_4, as shown in Fig. 7. In other words, the first column is set as the first image area with a short accumulation period.

[0079] On the other hand, in the second column (k=2), it is possible to divide the period of full frame 1 into two and read out only the data of the accumulation periods of frames 1_2 and 1_4. In other words, the second column is set as the second image area with a longer accumulation period.

[0080] In this way, according to this embodiment, the accumulation period and the number of times can be changed arbitrarily, and by changing the accumulation period and the number of times, it is possible to change the intensity of the accumulated signal and increase or decrease the number of accumulations. Also, it is possible to increase or decrease the amount of data according to the number of accumulations. However, if the amount of data increases, the power consumption of the sensor unit and the photoelectric conversion device 600 also increases accordingly.

[0081] The method of changing the accumulation period setting is not limited to changing it for each column, but it is also possible to change the accumulation period setting for each row or for each pixel by changing the signal line of the sensor unit 100. The setting of the accumulation period for each region or pixel is performed by the control unit 605.

[0082] As described above, the control unit 605 has a built-in CPU as a computer and a memory that stores a computer program, and acquires information from the detection unit 602, and the CPU executes the computer program stored in the memory based on the information. This makes it possible to set the accumulation period for each region or pixel.

[0083] In this way, the control unit 605 can set the accumulation period for each region or pixel of the sensor unit 100, so that the timing at which a short accumulation period ends can be made earlier than the timing at which a long accumulation period ends. Therefore, as soon as the short accumulation period ends, it becomes possible to output the image and send it to the downstream recognition unit 604.

[0084] That is, based on the signal generated in the first accumulation period, the recognition unit 604 enables object recognition in an image area with a short accumulation period at time T1 when frame 1_1 ends, which is earlier than time T4 when full frame 1 ends in the first row (k=1). Therefore, whereas image recognition was not possible until one full frame period had elapsed in the past, this embodiment makes it possible to set an area where image recognition can be performed in a shorter time.

[0085] Also, because images can be captured over multiple integration periods, properly exposed images can be generated for both light and dark areas, maximizing the signal-to-noise ratio (S / N) in each area. In other words, in dark areas, a long integration period increases the signal, maximizing the S / N ratio.

[0086] On the other hand, in bright areas, a short integration period allows signals to be acquired without saturating the signal, maximizing the S / N ratio. Therefore, by applying an integration period appropriate for each area, it is possible to acquire images with maximized S / N ratios without saturating the signal for each area and suppressing noise.

[0087] In addition, by applying an appropriate accumulation period to each region, blurring due to the movement of the subject can be suppressed. A long accumulation period may result in a blurred image (blur) of the subject. On the other hand, a short accumulation period can reduce blurring of the subject. Therefore, by applying an appropriate accumulation period, it is possible to obtain an image with reduced blurring due to the movement of the subject.

[0088] Fig. 12 is a diagram for explaining an example of setting an attention area for a moving object in the first embodiment. For example, first, a moving object in a shooting area is recognized, and its surroundings are set as an attention area. Fig. 12 shows an image captured by a photoelectric conversion device 600. The photoelectric conversion device 600 has an optical system 601 with a horizontal angle of view of 120 degrees and a vertical angle of view of 30 degrees. The number of pixels of the sensor unit 100 is 1920 pixels in the M column direction and 1080 pixels in the N row direction.

[0089] 12, the recognition unit 604 performs image recognition to determine whether a moving object is captured based on signals read from the pixel area of ​​the sensor unit 100 and image-processed by the image processing unit 603. Specifically, images captured by the sensor unit 100, for example, full frames, are compared, and an area where the image changes over time is recognized as a moving object.

[0090] When a moving object is recognized, a first pixel region is set as a region of interest around the moving object and in the moving direction, as shown in gray in Fig. 12. The first pixel region has at least a first accumulation period and a second accumulation period.

[0091] Here, the control unit 605 functions as a setting unit capable of setting a first pixel region having at least a first accumulation period and a second accumulation period, and a second pixel region having at least a second accumulation period, in a sensor unit consisting of a plurality of pixels. In this embodiment, the control unit 605 sets the first pixel region and the second pixel region based on an image read out from the sensor unit.

[0092] In this manner, in this embodiment, the periphery of the moving object is set as the first pixel region as the region of interest, and an image is acquired in the first accumulation period that is shorter than the full frame period, thereby making it possible to acquire an image of the moving object with a high S / N ratio and with little blurring due to motion. Furthermore, by acquiring such an image, when recognizing a moving object and tracking it as a shooting target, it becomes possible to recognize the moving object with high accuracy, and the moving object can be appropriately photographed and recorded as a shooting target.

[0093] On the other hand, it is sufficient to capture areas other than the periphery of a moving object as a background image, and in many cases, images with a high S / N ratio and little blur are not required. Therefore, by setting these areas as non-attention areas in the second pixel area and acquiring images during the second accumulation period, which is longer than the first accumulation period, during the full frame period, it is possible to reduce the amount of data in the captured image.

[0094] In this manner, in this embodiment, a determination is made as to whether or not an object is moving based on an image captured and generated by the photoelectric conversion device 600, and a first pixel region and a second pixel region are set according to the determination result, and image capturing is performed for each region with different accumulation periods. This allows images of moving objects to be recognized earlier than the full frame period. Furthermore, it is possible to reduce the number of data items and power consumption while achieving a high S / N ratio and reducing pixel blurring due to motion.

[0095] <Embodiment 2> Next, a second embodiment will be described in which a specific person, animal, etc. within a shooting area is recognized and the surrounding area is set as a region of interest.

[0096] In the photographing area, the recognition unit 604 recognizes whether a specific person or animal that has been registered in advance is photographed from an image generated using a signal read from the pixel area of ​​the sensor unit 100. The recognition method is to evaluate the characteristic points of the face of the person or animal based on the positions and sizes of the eyes, nose, mouth, etc. Then, when a specific person or animal is recognized, a first pixel area is set around the person or animal as a region of interest.

[0097] For this purpose, in the case of a general camera, the faces of family and friends, and in the case of a pet camera, the faces of pets, etc. are registered in advance. Also, in the case of a surveillance camera, etc., the features of the person to be monitored, such as the face and physique, are registered in advance. Alternatively, the features of the person to be monitored, such as the face and physique, which have been registered in advance, may be acquired from an external server, etc.

[0098] In this way, by setting the area around a particular person, animal, etc. as the area of ​​interest in the first pixel area and acquiring an image during a first accumulation period that is shorter than the full frame period, it is possible to acquire an image with a high S / N ratio for moving objects and little blurring due to motion.

[0099] Furthermore, by acquiring such images, it becomes possible to recognize specific people, animals, etc. with high accuracy when automatically tracking them as subjects for photography, and they can be appropriately photographed and recorded as subjects.

[0100] On the other hand, it is sufficient to capture the area outside the region of interest as a background image, and therefore, by setting these areas as non-regions of interest in the second pixel area and acquiring an image during the second accumulation period during the full frame period, it is possible to reduce the amount of data in the captured image.

[0101] In this way, in the second embodiment, a specific image is recognized based on an image captured and generated by the photoelectric conversion device 600, and a first pixel region and a second pixel region are set based on the result, and imaging is performed with different accumulation periods in each region. This allows an image of the region of interest to be acquired earlier than a full frame period, and since the S / N ratio is high and pixel blurring due to motion is small, highly accurate image recognition is possible. Furthermore, it is possible to reduce the amount of data and power consumption.

[0102] In the above embodiment, a method for registering people and animals in advance has been described. However, it is also possible to register parts, units, packaging labels, etc. in a factory production line, tumors and abnormal areas in an endoscope camera, monitoring targets in an infrastructure camera, crops in an agricultural camera, etc.

[0103] Then, by setting a pre-registered subject as the area of ​​interest and reading out the image with a short accumulation period and early timing, it is possible to achieve a high S / N ratio and reduce pixel blurring caused by motion while reducing the amount of data and power consumption.

[0104] <Embodiment 3> Next, as a third embodiment, an example will be described in which a traffic object is recognized and an attention area is set based on the recognition result when the photoelectric conversion device 600 is attached to a moving body 700 and used. In this embodiment, the traffic object refers to, for example, people or moving objects on the road, such as pedestrians, motorbikes, and bicycles, around the moving body.

[0105] Fig. 13 is a functional block diagram showing a configuration example of a photoelectric conversion device 600 and a moving body 700 according to the third embodiment. The photoelectric conversion device 600 according to the third embodiment is mounted on the moving body 700, and sets an attention area based on at least one of the positions, speeds, traveling directions, distances, etc. of people, cars, etc., which are traffic participants. In Fig. 13, the same reference numerals as those in Fig. 6 denote the same functional blocks, and therefore the description thereof will be omitted.

[0106] In the third embodiment, the output of the image processing unit 603 is supplied to a recognition unit 604, a control unit 605, and an ECU (Electric Control Unit) 701 serving as a movement control unit of a moving object 700. The recognition unit 604 recognizes surrounding people, vehicles, objects, etc. by performing image recognition based on the image signal, and outputs the recognition result to the ECU 701.

[0107] The control unit 605 also acquires information such as the installation height and installation angle as installation information of the photoelectric conversion device 600 on the moving object 700 from the ECU 701. Furthermore, the control unit 605 acquires environmental information of the photoelectric conversion device 600 from the detection unit 602, and acquires information on the moving object 700 and environmental information from the ECU 701.

[0108] The information of the moving object acquired from the ECU 701 includes at least one of vehicle speed information, acceleration information, steering angle information, brake information, engine information, etc. The environmental information includes at least one of information on the presence or absence, position, speed, acceleration, distance, etc. of traffic participants, map information, GPS information, road conditions, road surface conditions, weather information, surrounding brightness, time, etc.

[0109] Then, based on the acquired information, the CPU executes a computer program stored in a memory built into the control unit 605 to control each unit of the photoelectric conversion device 600 .

[0110] The ECU 701 incorporates a CPU as a computer and a memory that stores a computer program, and controls each part of the moving body 700 by the CPU executing the computer program stored in the memory.

[0111] The output of the ECU 701 is also supplied to a vehicle control unit 702 and a display unit 703. The vehicle control unit 702 functions as a driving control unit that performs operations such as driving, stopping, and directional control of the vehicle as a moving body based on the output of the ECU 701.

[0112] The display unit 703 includes a display element such as a liquid crystal device or an organic EL device, and is mounted on the moving body 700. The display unit 703 displays images acquired by a sensor unit and various information related to the vehicle's running state and the like to the driver of the moving body 700 using, for example, a GUI based on the output of the ECU 701.

[0113] 13 may not be mounted on the moving body 700. For example, they may be provided in an external terminal or an external server provided separately from the moving body 700 for remotely controlling the moving body 700 or monitoring the traveling of the moving body.

[0114] Fig. 14 is a flowchart for explaining a method for controlling a moving object according to embodiment 3. Note that the control unit 605 and a CPU serving as a computer in the ECU 701 execute a computer program stored in the memory, thereby sequentially performing the operations of the steps in the flowchart in Fig. 14.

[0115] First, in step S141, the CPU of the control unit 605 acquires video data from the photoelectric conversion device 600 while driving. Next, in step S1412, the CPU of the control unit 605 performs image recognition based on a signal read from the pixel area of ​​the sensor unit 100 of the photoelectric conversion device 600, thereby recognizing an object that is a traffic participant.

[0116] Then, in step S1413, the CPU of the control unit 605 acquires the position and speed of the recognized object, such as a traffic object, etc. This may be acquired by observing changes in the object in multiple full-frame video data, or by using a millimeter wave radar or LiDAR (Light Detection and Ranging) as another distance measuring device.

[0117] Next, in step S1414, the CPU of the ECU 701 acquires the traveling direction of the object that is a traffic participant and the distance to the object. As in the previous example, this may be acquired by observing the change in the object in multiple full-frame video data, or by using a millimeter-wave radar or LiDAR as another distance measuring device.

[0118] Then, in step S1415, the CPU of the ECU 701 sets an attention area based on at least one of the position, speed, traveling direction, distance, etc. of an object such as a traffic participant object, etc. Next, in step S1416, the braking distance of the host vehicle is calculated.

[0119] The braking distance is not limited to the speed of the moving body 700, but varies depending on the inclination angle of the road surface, the condition of the road surface (asphalt, dirt, gravel, rain, snow, ice, etc.), the degree of tire wear (characteristics of the tire itself, distance used, etc.), and the weight of the moving body 700. Therefore, such information may be used comprehensively to make the determination.

[0120] Next, in step S1417, the CPU of the ECU 701 evaluates the collision risk with a target who is a traffic participant, taking into consideration factors such as the speed, traveling direction, and distance of the target who is a traffic participant. If it is determined in step S1417 that the collision risk is higher than the predetermined value, the CPU of the ECU 701 generates a warning signal in step S1418, and issues a warning to the driver via the display unit 703 or a voice message, and further activates an automatic brake system as necessary.

[0121] If it is determined in step S1417 that the collision risk is equal to or less than the predetermined value, the process returns to step S1411, and the next full frame of video data is obtained.

[0122] After step S1418, the process proceeds to step S1419, where the CPU of ECU 701 determines whether the user has turned off the power to the moving object or performed an operation to end the flow of Fig. 14. If the result in step S1419 is Yes, the flow of Fig. 14 ends. If the result in step S1419 is No, the process returns to step S1411, and the flow of Fig. 14 is repeated.

[0123] In this way, by setting the area around the traffic object as the first pixel area as the region of interest and acquiring an image with a first accumulation period that is shorter than the full frame period, it is possible to acquire an image with a high S / N ratio for moving objects and little blurring due to motion.

[0124] Furthermore, by acquiring such an image, it becomes possible to recognize the traffic object with high accuracy in the function of continuing to photograph it as a photographing target, etc. Also, when there is a high risk of collision between the moving body 700 and the traffic object, a warning signal is generated, a warning is given to the driver through the display unit 703 or a voice, and further, an automatic braking system or the like is activated as necessary, thereby realizing a collision prevention system.

[0125] On the other hand, it is sufficient to capture areas other than the surrounding area of ​​the traffic object as a background image, and in many cases, images with a high S / N ratio and little blur are not required. Therefore, by setting those areas as non-attention areas and acquiring images with a second accumulation time longer than the first accumulation period during the full frame period, it is possible to reduce the amount of data in the captured image.

[0126] In this way, in the third embodiment, a traffic object is recognized based on an image captured and generated by the photoelectric conversion device 600, a first pixel region and a second pixel region are set based on the recognition result, and each region is captured with a different accumulation period. This makes it possible to recognize an image faster than a full frame period, achieve a high S / N ratio, reduce pixel blurring due to motion, and reduce the amount of data and power consumption.

[0127] In addition, in this embodiment, the recognition of the traffic participating object has been described based on the image captured and generated by the photoelectric conversion device 600, but it is not limited to the image, and a distance measuring device such as a millimeter wave radar or LiDAR may be used. By using a millimeter wave radar or LiDAR, the moving direction and moving speed of the traffic participating object can be obtained more accurately, and the surrounding area of ​​the traffic participating object can be set more accurately.

[0128] <Embodiment 4> In the fourth embodiment, an example in which blind spots in a shooting area are recognized and the periphery of each blind spot area and the moving direction of a moving object are set as a region of interest will be described with reference to Fig. 15. Fig. 15 is a flowchart for explaining a method for controlling a moving object according to the fourth embodiment.

[0129] The control unit 605 and a CPU serving as a computer in the ECU 701 execute a computer program stored in the memory, whereby the operations of the steps in the flowchart of FIG. 15 are performed in sequence.

[0130] First, in step S1501, the CPU of the control unit 605 acquires video data from the photoelectric conversion device 600 while driving. Next, in step S1502, the CPU of the control unit 605 performs image recognition based on a signal read from the pixel area of ​​the sensor unit 100 of the photoelectric conversion device 600, thereby recognizing a blind spot area within the shooting area as an object.

[0131] The blind spot area in the fourth embodiment means an area where a traffic object that is not captured at all or hardly captured in an image captured by the photoelectric conversion device 600 may suddenly appear. In other words, it may be an intersection with poor visibility, a wall of a building, a shadow of a car other than the vehicle itself, a signboard, a tree, etc.

[0132] When a blind spot area is recognized within the shooting area, in step S1503, the CPU of the control unit 605 assumes a traffic participant object that will jump out of the blind spot area. Specifically, the traffic participant object is assumed based on the size of an object that may be in the blind spot area, road conditions, etc. For example, if it is a motor vehicle-only road, cars and motorcycles can be assumed as traffic participants that will jump out, and if it is a small tree or signboard, traffic participants other than cars are assumed.

[0133] Furthermore, in step S1504, the CPU of the control unit 605 sets an attention area around the blind spot area according to the assumed traffic object, and sets a first pixel area as the attention area for each blind spot area. That is, if there is a possibility that a car will jump out from the blind spot area, the area is set to be large, and if there is a high possibility that the jumping out traffic object is a pedestrian, the area is set to be small.

[0134] Next, in step S1505, the CPU of the ECU 701 calculates the braking distance of the moving object 700. The braking distance varies depending on the speed of the moving object 700, the inclination angle of the road surface, the condition of the road surface (asphalt, dirt, gravel, rain, snow, ice, etc.), the degree of tire wear (characteristics of the tire itself, distance used, etc.), the weight of the moving object 700, etc. Therefore, the braking distance may be determined by comprehensively using such information.

[0135] Next, in step S1506, the CPU of the ECU 701 evaluates the collision risk with the assumed traffic participant, taking into consideration factors such as the speed, traveling direction, and distance of the traffic participant.

[0136] If it is determined in step S1506 that the collision risk is higher than the predetermined value, the CPU of the ECU 701 generates a warning signal and issues a warning to the driver via the display unit 703 or a voice in step S1507. Furthermore, an automatic brake system is activated as necessary.

[0137] If it is determined in step S1506 that the collision risk is equal to or less than the predetermined value, the process returns to step S1501, and the next full frame of video data is obtained.

[0138] After step S1507, the process proceeds to step S1508, where the CPU of ECU 701 determines whether the user has turned off the power to the moving object or performed an operation to end the flow of Fig. 15. If the answer is Yes in step S1508, the process ends. If the answer is No in step S1508, the process returns to step S1501, and the flow of Fig. 15 is repeated.

[0139] In this way, in the fourth embodiment, the blind spot area is recognized as an object, its surroundings are set as a first pixel area as a region of interest, and an image is acquired in an accumulation period shorter than a full frame period. This makes it possible to recognize an image of the blind spot area earlier than the full frame period, and to acquire an image of the blind spot area with a high S / N ratio and little blurring due to motion.

[0140] Furthermore, by acquiring images of such blind spot areas early, it becomes possible to recognize traffic objects emerging from the blind spot areas early and with high accuracy, enabling early assessment of collision risk and reducing the collision risk.

[0141] On the other hand, since the risk of collision due to a traffic object rushing out is low in areas other than the peripheral areas of the blind spot, it is often not required to recognize an image faster than the full frame period or to obtain an image with a high S / N ratio or little blur. Therefore, by setting these areas as non-attention areas and acquiring an image with a second accumulation time longer than the first accumulation period during the full frame period, it is possible to reduce the amount of data in the captured image.

[0142] In this manner, in the fourth embodiment, a blind spot area is recognized based on an image captured and generated by the photoelectric conversion device 600, a first pixel area and a second pixel area are set based on the recognition result, and image capturing is performed for each area with a different accumulation period. This makes it possible to recognize an image faster than a full frame period, achieve a high S / N ratio, reduce pixel blurring due to motion, and reduce the amount of data and power consumption.

[0143] <Embodiment 5> Next, in a fifth embodiment, an example will be described in which the road conditions within a shooting area are recognized and an attention area is set in accordance with the recognized road conditions.

[0144] In the fifth embodiment, the recognition unit 604 recognizes information about roads with a high collision risk based on the image of the shooting area. The information about roads with a high collision risk here refers to intersections, oncoming lanes, merging lanes, etc. These roads are recognized from the image, and the surrounding areas are set as attention areas.

[0145] Then, when these roads with a high collision risk are recognized within the shooting area, a first pixel area is set as a region of interest for each road. Note that information on roads with a high collision risk may be acquired based on GPS information, map information, etc. Furthermore, as attribute information of the map information, at least one of information on road locations, time periods, weather, dates, etc. where many accidents have occurred in the past, the current road construction status, traffic congestion status, traffic accident occurrence status, etc. may be acquired from an external server.

[0146] As a method for setting the attention area, it is preferable to consider the possible movement direction and speed of traffic participants that may appear from these roads, and set the area range accordingly. This is determined based on the type of traffic participants that are a collision risk, road conditions, etc.

[0147] In other words, at intersections, it is preferable to set an area where vehicles will invade the lane in which the vehicle is traveling by going straight or turning right or left, and in oncoming lanes, to set a large area taking into account the relative speed with respect to the vehicle, and in merging lanes, to set an area where vehicles will invade the lane in which the vehicle is traveling.

[0148] In this manner, in the fifth embodiment, the vicinity of a road where a collision risk is high is set as a first pixel region as a region of interest, and an image is acquired with an accumulation period shorter than a full frame period. Therefore, an image of the vicinity of a road where a collision risk is high can be recognized earlier than a full frame period, and an image with a high S / N ratio and little blurring due to motion can be acquired.

[0149] Furthermore, by acquiring such images at an early stage, it becomes possible to recognize traffic objects at an early stage and to evaluate the collision risk at an early stage with high accuracy, thereby effectively reducing the collision risk.

[0150] On the other hand, in areas other than around roads where the collision risk is high, the risk of collision with traffic objects is low, so there is often no need to recognize images faster than the full frame period or to obtain images with a high S / N ratio or little blur. Therefore, by setting these areas as non-attention areas and acquiring images with a second accumulation time longer than the first accumulation period during the full frame period, it is possible to reduce the amount of data in the captured image.

[0151] In this way, in the fifth embodiment, the first pixel region and the second pixel region are set based on road information, and an image is captured in each region with a different accumulation period, so that an image can be recognized faster than the full frame period. Furthermore, it is possible to reduce the amount of data and power consumption while achieving a high S / N ratio and reducing pixel blur caused by motion.

[0152] In order to improve the accuracy of road condition recognition, map information and GPS location information are utilized. By recognizing road conditions based on a combination of map information, GPS location information, and images captured by the photoelectric conversion device 600, it becomes possible to grasp the situation with higher accuracy.

[0153] Furthermore, when traveling straight ahead, it is desirable to set an attention area based on road information such as traffic objects ahead, traffic lights, speed limit signs, intersections, lanes, shoulders, and crosswalks.

[0154] In addition, at intersections, the vehicle often crosses traffic participants in the direction of travel, including traffic signals, signs, and priority indications. Therefore, it is necessary to set an attention area that is not limited to the movement of recognized traffic participants, but takes into consideration the risk to traffic participants hidden in various blind spots. In other words, it is desirable to set an attention area based on, for example, blind spot information according to the fourth embodiment in addition to road condition information.

[0155] In addition, in parking lots, it is necessary to set attention areas taking into consideration the movements of traffic participants in all directions, including the rear as well as the front, and the risks to traffic participants hidden in various blind spots.

[0156] <Embodiment 6> Next, Fig. 16 is a flowchart showing details of an example of driving the sensor unit 100 regarding a method for setting an attention area according to the sixth embodiment, and Fig. 17 is a flowchart continuing from Fig. 16. Note that the CPU as a computer in the control unit 605 executes a computer program stored in the memory, thereby sequentially performing the operations of the steps in the flowcharts of Figs. 16 and 17.

[0157] In step S1601, the CPU of the control unit 605 of the photoelectric conversion device 600 sets a region of interest and a region of non-interest within the imaging area based on characteristic information relating to the characteristics of the photoelectric conversion device, installation information of the photoelectric conversion device, and environmental information of the photoelectric conversion device.

[0158] In addition, the control unit 605 may set the first pixel region or the second pixel region based on at least one of characteristic information regarding the characteristics of the photoelectric conversion device, installation information regarding the installation state of the photoelectric conversion device, and environmental information regarding the surrounding environment of the photoelectric conversion device.

[0159] Furthermore, at least one of the position, size, and accumulation period of the first pixel region or the second pixel region may be set based on at least one of the above-mentioned characteristic information, installation information, and environment information. Also, the first pixel region or the second pixel region may be set according to distance information of the subject.

[0160] The characteristic information includes optical characteristic information of the optical system, the resolution of the sensor unit, and the pixel size of the sensor unit. The installation information includes at least one of the height, angle, and direction at which the photoelectric conversion device is installed. Information on the moving body as one piece of environmental information includes at least one of vehicle speed information, acceleration information, steering angle information, brake information, engine information, and the like.

[0161] In addition, the environmental information may include at least one of information such as the presence, location, speed, acceleration, and distance of traffic participants (pedestrians, motorbikes, bicycles, etc. around the moving object), map information, GPS information, road conditions, road surface conditions, weather information, surrounding brightness, and time.

[0162] Also, the control unit may set the first pixel area or the second pixel area based on the output of the ECU as a movement control unit. At least one of the positions and sizes of the first pixel area and the second pixel area may be set by the user.

[0163] In this embodiment, a case where there is one region of interest and one region of non-interest will be described. For the sake of explanation, the accumulation period of the first pixel region, which is the region of interest, is divided into four parts of one full frame period, and the accumulation period of the second pixel region, which is the region of non-interest, is not divided and is one full frame period.

[0164] Here, the first pixel area starts from the pixel in row j1, column k1 and extends to row N1, column M1, and imaging is performed over four different accumulation periods, whereas the second pixel area starts from the pixel in row j2, column k2 and extends to row N2, column M2, and imaging is performed over one accumulation period.

[0165] Next, in step S1602, the CPU of the control unit determines whether it is the target area. If Yes, the process proceeds to step S1611 to perform the process of determining the accumulation period. On the other hand, if it is determined as No in step S1602, the process proceeds to step S1701.

[0166] Since the target area is divided into four parts, i starts from 1 and repeats up to 4. Therefore, in step S1611, the CPU of the control unit first sets the value of i to 1. Next, in step S1612, the CPU of the control unit outputs the count value Count of the counter 211 at time Ti to the memory 212. At this time, the output is performed simultaneously for all memories. This operation corresponds to the operation at time T1 in FIG. 7.

[0167] Next, in step S1613, the CPU of the control unit sets j = j1. j1 is the start position of the Nth row of the target area. Next, in step S1614, the CPU of the control unit sets k = k1. k1 is the start position of the Mth column of the target area.

[0168] In step S1615, the CPU of the control unit outputs the count value Count(j - k - i) in the memory j - k in FIG. 9 to the buffer. At this time, the output is performed for columns 1 to M simultaneously to the buffer. This operation means the operation of taking in the count value of the first row in FIG. 9 into the buffer.

[0169] In step S1616, the CPU of the control unit outputs the count value Count(j - k - i) of the buffer k to the output circuit 114. This operation corresponds to the operation of reading out the signal of the buffer in the leftmost column in FIG. 9 from the output circuit.

[0170] Next, via E, the process proceeds to step S1617 in FIG. 17. In step S1617, the CPU of the control unit determines whether k < M1. If Yes, in step S1618, k is incremented by 1 as k = k + 1, and via F, the process returns to step S1616. M1 is the end position of the Mth column of the target area. This operation corresponds to the operation of reading out the signal of the buffer in the second column from the left in FIG. 9 from the output circuit.

[0171] If it is determined as No in step S1617, that is, if k = M1, it means that the signal of the buffer in the M1 column of FIG. 9 has been read out from the output circuit. Next, proceed to step S1619, and the CPU of the control unit determines whether j < N1. N1 is the end position of the Nth row of the target area. If it is determined as Yes in step S1619, then in step S1620, the CPU of the control unit sets j = j + 1, increments j by 1, and returns to step S1614 via G. This corresponds to the operation for starting the reading of the next row.

[0172] If it is determined as No in step S1619, it means that the reading of all rows has been completed. So, proceed to step S1621, and the CPU of the control unit determines whether j < 4. If it is determined as Yes in step S1621, then the CPU of the control unit sets i = i + 1, increments i by 1, and returns to step S1612 via H. This operation corresponds to the operation for starting the reading at the next time T2.

[0173] If it is determined as No in step S1621, it means that the reading at time T4 has been completed. So, proceed to step S1623, and the CPU of the control unit resets the counter 211 with a reset signal. This operation corresponds to the reset operation of the counter 211 at time T4 in FIG. 7. After step S1623, return to step S1601 via M. In the above manner, the signals accumulated in the sensor unit can be sequentially read out.

[0174] On the other hand, since the non-target area does not divide one full frame period, only the flow when i = 4 is looped. Therefore, in step S1701, the CPU of the control unit sets i = 4. Next, in step S1702, the CPU of the control unit outputs the count value Count of the counter 211 at time Ti to the memory 212. At this time, the output is performed simultaneously for all memories. This operation corresponds to the operation at time T1 in FIG. 7.

[0175] Next, in step S1703, the CPU of the control unit sets j = j2. j2 is the starting position of the N rows of the non-target area. Next, in step S1704, the CPU of the control unit sets k = k2. k2 is the starting position of the M columns of the non-target area.

[0176] In step S1705, the CPU of the control unit outputs the count value Count(j - k - i) in the memory j - k in FIG. 9 to the buffer. At this time, it outputs to the buffer simultaneously for columns 1 to M. This operation means the operation of taking in the count value of the first row in FIG. 9 into the buffer.

[0177] In step S1706, the CPU of the control unit outputs the count value Count(j - k - i) of buffer k to the output circuit 114. This operation corresponds to the operation of reading out the signal of the buffer in the leftmost column in FIG. 9 from the output circuit.

[0178] Next, it proceeds to step S1707 in FIG. 17 via I. In step S1707, the CPU of the control unit determines whether k < M2. If it is Yes in step S1707, then in step S1708, the CPU of the control unit increments k by 1 as k = k + 1 and returns to step S1706 via J. M2 is the ending position of the M columns of the non-target area. This operation corresponds to the operation of reading out the signal of the buffer in the second column from the left in FIG. 9 from the output circuit.

[0179] When it becomes No in step S1707, that is, when k = M2, it means that the signal of the buffer in the M2 - th column in FIG. 9 has been read out from the output circuit. Next, it proceeds to step S1709, and the CPU of the control unit determines whether j < N2.

[0180] N2 is the ending position of the N rows of the non-target area. If it is Yes in step S1709, then in step S1710, the CPU of the control unit sets j = j + 1 and increments j by 1, and returns to step S1704 via K. This corresponds to the operation for starting to read the next row.

[0181] If the determination in step S1709 is No, this means that reading of all rows has been completed, so the process proceeds to step S1711, where the control unit CPU determines whether j<4. If the determination in step S1711 is Yes, the control unit CPU increments i by 1, setting i=i+1, and returns to step S1702 via L. This operation corresponds to the operation of starting reading at the next time T2. However, since i=4 was set in step S1701, the result in step S1711 is No.

[0182] If the determination in step S1711 is No, this means that the readout at time T4 is completed, so the process proceeds to step S1713, where the CPU of the control unit resets counter 211 with a reset signal. This operation corresponds to the reset operation of counter 211 at time T4 in Fig. 7. After step S1713, the process returns to step S1601 via M. In this manner, the signals accumulated in the sensor unit can be read out sequentially.

[0183] Therefore, the region of interest starts from the pixel in row j1, column k1 and repeats through row N1, column M1, and is imaged over four different accumulation periods, whereas the region of non-interest starts from the pixel in row j2, column k2 and repeats through row N2, column M2, and is imaged over only one accumulation period.

[0184] Next, a description will be given of the processing of images acquired in the region of interest and the region of non-interest in the photoelectric conversion device 600. The method of acquiring an image of the region of interest is based on the method shown in Fig. 16 and Fig. 17. Also, the region of interest acquires an image in four accumulation periods of frames 1_1, 1_2, 1_3, and 1_4 in Fig. 7. On the other hand, the region of non-interest acquires an image in the accumulation period of only frame 1_4 in Fig. 7.

[0185] An example of an image created according to the flow of Fig. 16 and Fig. 17 is shown in Fig. 18. Fig. 18 is a diagram showing an example of an image for each of a plurality of accumulation periods, with an attention area being set. In each diagram, the range on the lower side where a close distance is shot is set as the attention area, and the range on the upper side where a long distance is shot is set as the non-attention area, and the car on the left side is stopped, and the car on the right side is moving from right to left at a certain speed.

[0186] As shown in Fig. 18, the area of ​​interest at the bottom of the screen is updated in four stages from frame 1_1 to 1_4. Therefore, a stopped car's image becomes gradually brighter as the accumulation period gets longer from frame 1_1 to 1_4. Also, a moving car's image becomes blurred in the front and rear of the moving direction as the accumulation period gets longer from frame 1_1 to 1_4, and the brightness of the overlapping moving parts gets higher.

[0187] On the other hand, the non-attention area, which is the upper part of the screen, is updated only by the output of frame 1_4. Therefore, a stopped car is updated every full frame and photographed during the accumulation period of the full frame period, resulting in an image with high brightness. Similarly, a moving car is updated every full frame and photographed during the accumulation period of the full frame period, so blurring occurs in the front and rear of the moving direction, and the brightness of the part where the movements overlap is high.

[0188] Therefore, in the non-attention area, the images of frames 1_1 to 1_3 are all images captured during the accumulation period of frame 1_4 in the previous full frame period. Also, frame 1_4 is updated to an image captured during the accumulation period of frame 1_4 in the current full frame period.

[0189] 16 and 17, the area of ​​interest is updated four times in a full frame period, making it possible to acquire an image in a shorter time than a full frame period and perform faster recognition. On the other hand, the area of ​​non-interest is updated only once in a full frame period, making it possible to reduce the amount of data and power consumption.

[0190] Based on the image data created according to the flowcharts of FIGS. 16 and 17, subsequent processing is performed according to the characteristics and applications of the photoelectric conversion device 600.

[0191] For example, in the case of a security camera, the control unit 605 monitors, photographs, and records the behavior of a target person. In some cases, it is possible to display a warning, turn on a light installed in the security camera, or emit a warning sound via the communication unit 607.

[0192] In the case of an in-vehicle camera, the control unit 605 can issue a warning or display a warning via the ECU, and the vehicle control unit 702 can activate safety devices to slow down or stop the vehicle, making it possible to reduce or avoid a collision.

[0193] In addition, for example, by using the camera to recognize people or registered faces, it is possible to capture images with an appropriate accumulation time when the person is within the shooting area. Also, as a pet camera, it is possible to recognize pets, remotely check on their condition, and take pictures of them.

[0194] In addition, when used as a surveillance camera, in places where safety management and surveillance are required (e.g. parking lots, public facilities, factories, etc.), specific locations or areas can be set as areas of interest in order to grasp the situation, monitor and prevent fraudulent activities, collect evidence as records, etc. This makes it possible to monitor the areas of interest more frequently and record and display video and audio.

[0195] In addition, when used as a defect detection camera in a factory, it can be used to support quality control and efficiency of the manufacturing process, and to detect and remove defective products early, troubleshoot production lines, and improve quality, etc. In that case, it becomes possible to detect and record product defects and abnormalities with greater accuracy on the factory production line or at the work site.

[0196] In addition, when used as a packaging inspection camera in a factory, it will be possible to inspect the product packaging condition, label accuracy, and defects in order to strengthen product quality control and ensure the integrity of the packaging and accuracy of the label.

[0197] When used as a sorting camera for logistics, it can improve the efficiency and accuracy of logistics operations in logistics warehouses and distribution centers, and realize proper product management and a quick delivery process. It can also recognize the shape, size, barcode, etc. of products and packages, enabling accurate sorting and delivery processing.

[0198] When used as an endoscopic camera, the camera can be incorporated into endoscopes used in the medical field to obtain high-quality images of the inside of the body and organs during endoscopic surgery and diagnosis, allowing for more accurate observation of conditions inside the body and helping to diagnose diseases and develop treatment plans.

[0199] When used as a camera to detect the need for care, it will be possible to monitor the living conditions of elderly people and those requiring care and detect abnormal or dangerous situations in order to provide early detection and support for falls, abnormal behavior, and emergency situations, ensuring the safety and lifestyle support of those in need of care.

[0200] When used as infrastructure surveillance cameras, they can monitor infrastructure such as roads, bridges, railways, and power plants to maintain safety and reliability, and detect abnormalities, damage, and unauthorized activity in order to provide early warning and proper maintenance.

[0201] In addition, when used as infrastructure surveillance cameras, they can monitor public places, facilities, and specific areas to ensure public safety, deter crime, prevent fraud, and collect evidence for criminal investigations, making it possible to prevent, monitor, and record fraudulent and criminal activities.

[0202] In addition, when used as an agricultural camera, it can support appropriate cultivation management, early detection of pests, and effective agricultural measures to improve the productivity and quality of agricultural crops. In that case, it will be possible to monitor the growth status of agricultural crops and the occurrence of pests and diseases, and provide video data that is useful for agricultural production management.

[0203] <Embodiment 7> Next, a description will be given of embodiment 4. In embodiment 7, a method for setting the accumulation period of a region of interest according to the brightness of the region of interest, etc. will be described.

[0204] It has been explained that for surveillance cameras and in-vehicle cameras attached to moving objects, it is effective to capture images of close-up subjects or subjects on the side of a moving object with a short accumulation period at an early timing. However, it is known that a short accumulation period increases the ratio of noise in the signal component, deteriorating the so-called S / N ratio.

[0205] This is particularly noticeable when shooting a dark area. Therefore, if the area of ​​interest is dark, even if an image is captured with a short accumulation period, the image will have a poor S / N ratio, making it difficult to display appropriately or use for subsequent processing.

[0206] Therefore, by performing unnecessary processing, appropriate judgment cannot be performed and power consumption of the device increases, which is a problem. Therefore, in the fourth embodiment, a method of capturing an image with a good S / N ratio by adjusting the accumulation period according to the brightness of the target area will be described.

[0207] Fig. 19 is a flowchart showing details of an example in which an accumulation period is set and the sensor unit 100 is driven in the seventh embodiment, and Fig. 20 is a flowchart continuing from Fig. 19. Note that the CPU as a computer in the control unit 605 executes a computer program stored in the memory, thereby sequentially performing the operations of the steps in the flowcharts of Figs. 19 and 20.

[0208] First, in step S1901, the CPU of the control unit sets an area of ​​interest based on at least one of product information, installation information, and environmental information of the photoelectric conversion device 600 recorded in the control unit 605 of the photoelectric conversion device 600, as in the previous embodiments.

[0209] Next, in step S1902, the CPU of the control unit judges whether the area is a region of interest or a region of non-interest. If it is a region of interest, the process proceeds to step S1911 in Fig. 19, where the CPU of the control unit performs a process of determining an accumulation period. On the other hand, if it is a region of non-interest, the process proceeds to step S1931.

[0210] Next, in step S1903, the CPU of the control unit determines the accumulation period of the region of interest. In this embodiment, for example, the average luminance value of the image acquired in the previous full frame within the region of interest is calculated by the control unit 605, and a look-up table (LUT) or a calculation formula for the accumulation period corresponding to the luminance value is recorded in the memory of the control unit 605, and is used to determine the accumulation period.

[0211] As described above, in this embodiment, the length of at least one of the first and second accumulation periods is set based on the luminance information of a signal generated during at least one of the first and second accumulation periods.

[0212] The luminance value may be calculated based on the peak of a histogram, the deviation of the peak, the spread, or the variance of the peak, instead of the average luminance. The luminance difference between adjacent pixels in the screen may be calculated, and the noise amount may be calculated from the histogram of the difference, and the accumulation period may be calculated from the result. Furthermore, the luminance value may be calculated based on the average value of multiple previous full frames, instead of the previous full frame, or may be calculated based on the average luminance or histogram for each accumulation period in the full frame.

[0213] In this embodiment, flowcharts for the case where the accumulation period is four are shown in FIGS. 19 and 20. However, the accumulation period is not limited to four, and it may be one, three, or four or more. And in step S19011, the CPU of the control unit sets the required accumulation period (Ti).

[0214] In step S1911 of FIG. 19, the CPU of the control unit sets i = 1. Next, in step S1912, the CPU of the control unit outputs the count value Count of the counter 211 at time Ti to the memory 212. At this time, the output is performed simultaneously for all memories. This operation corresponds to the operation at time T1 in FIG. 7.

[0215] Next, in step S1913, the CPU of the control unit sets j = j1. j1 is the start position of the Nth row of the target area. Next, in step S1914, the CPU of the control unit sets k = k1. k1 is the start position of the Mth column of the target area.

[0216] In step S1915, the CPU of the control unit outputs the count value Count(j - k - i) in the memory j - k of FIG. 9 to the buffer. At this time, the output is performed for columns 1 to M simultaneously. This operation means the operation of taking in the count value of the first row in FIG. 9 into the buffer.

[0217] In step S1916, the CPU of the control unit outputs the count value Count(j - k - i) of buffer k to the output circuit 114. This operation corresponds to the operation of reading the signal of the buffer in the leftmost column in FIG. 9 from the output circuit.

[0218] Next, proceed to step S1917 of FIG. 20 via E. In step S1917, the CPU of the control unit determines whether k < M1. If Yes, in step S1918, the CPU of the control unit increments k by 1 as k = k + 1 and returns to step S1916 via F. M1 is the end position of the Mth column of the target area. This operation corresponds to the operation of reading the signal of the buffer in the second column from the left in FIG. 9 from the output circuit.

[0219] If it becomes No in step S1917, that is, if k = M1, it means that the signal of the buffer in the M-th column of FIG. 9 has been read out from the output circuit. Next, it proceeds to step S1919, and the CPU of the control unit determines whether j < N1.

[0220] N1 is the end position of the N-th row of the target area. If it is Yes in step S1919, then in step S1920, the CPU of the control unit sets j = j + 1, increments j by 1, and returns to step S1914 via G. This corresponds to the operation for starting the reading of the next row.

[0221] If it is determined to be No in step S1919, it means that the reading of all rows has been completed. So, it proceeds to step S1921, and the CPU of the control unit determines whether i < 4. If it is determined to be Yes in step S1921, then the CPU of the control unit sets i = i + 1, increments i by 1, and returns to step S1912 via H. This operation corresponds to the operation for starting the reading at the next time T2.

[0222] If it is determined to be No in step S1921, it means that the reading at time T4 has been completed. So, it proceeds to step S1923, and the CPU of the control unit resets the counter 211 with a reset signal.

[0223] This operation corresponds to the reset operation of the counter 211 at time T4 in FIG. 7. After step S1713, it returns to step S1901 via M. In the above manner, the signals accumulated in the sensor unit can be sequentially read out.

[0224] On the other hand, since the accumulation period of the non-target area is equal to the full-frame period, an image is acquired with a determined accumulation period. That is, in step S1931 of FIG. 19, the CPU of the control unit sets i = 4. Next, in step S1932, the CPU of the control unit outputs the count value Count of the counter 211 at time Ti to the memory 212. At this time, the output is performed simultaneously for all memories. This operation corresponds to the operation at time T1 in FIG. 7.

[0225] Next, in step S1933, the CPU of the control unit sets j = j2. j2 is the starting position of the Nth row of the non-target area. Next, in step S1934, the CPU of the control unit sets k = k2. k2 is the starting position of the Mth column of the non-target area.

[0226] In step S1935, the CPU of the control unit outputs the count value Count(j - k - i) in the memory j - k of FIG. 9 to the buffer. At this time, the output is performed for columns 1 to M simultaneously to the buffer. This operation means the operation of taking in the count value of the first row in FIG. 9 into the buffer.

[0227] In step S1936, the CPU of the control unit outputs the count value Count(j - k - i) of buffer k to the output circuit 114. This operation corresponds to the operation of reading out the signal of the buffer in the leftmost column in FIG. 9 from the output circuit.

[0228] Next, proceed to step S1937 of FIG. 20 via I. In step S1937, the CPU of the control unit determines whether k < M2. If Yes, in step S1938, the CPU of the control unit increments k by 1 as k = k + 1 and returns to step S1936 via J. M2 is the end position of the Mth column of the non-target area. This operation corresponds to the operation of reading out the signal of the buffer in the second column from the left in FIG. 9 from the output circuit.

[0229] If the result is No in step S1937, that is, if k = M2, it means that the signal of the buffer in the M-th column in FIG. 9 has been read out from the output circuit. Next, the process proceeds to step S1939, and the CPU of the control unit determines whether j < N2.

[0230] N2 is the end position of the N-th row in the region of interest. If the result is Yes in step S1939, then in step S1940, the CPU of the control unit sets j = J + 1, increments j by 1, and returns to step S1944 via K. This corresponds to the operation for starting to read the next row.

[0231] If it is determined that the result is No in step S1939, it means that the reading of all rows has been completed. Thus, the process proceeds to step S1941, and the CPU of the control unit determines whether i < 4. If it is determined that the result is Yes in step S1941, then the CPU of the control unit sets i = i + 1, increments i by 1, and returns to step S1932 via L. This operation corresponds to the operation for starting to read at the next time T2. However, since i = 4 is set in step S1931, the result will be No in step S1941.

[0232] If it is determined that the result is No in step S1941, it means that the reading at time T4 has been completed. Thus, the process proceeds to step S1943, and the CPU of the control unit resets the counter 211 with a reset signal. This operation corresponds to the reset operation of the counter 211 at time T4 in FIG. 7. After step S1943, the process returns to step S1901 via M.

[0233] In the above manner, the signals accumulated in the sensor unit can be sequentially read out. Note that the non - region of interest may also acquire an image with the accumulation period determined by the control unit 605 in the same way. In that case, a flow for setting the accumulation period before acquiring the image is added.

[0234] For example, in the above embodiment, accumulation is performed for a minimum of 1 / 4 full frame period, but the length of the minimum accumulation period may be changed to, for example, 1 / 5 full frame period or 1 / 3 full frame period depending on the recognition accuracy in the recognition unit 604. Alternatively, the length of the minimum accumulation period may be changed depending on the brightness of the subject. Furthermore, the recognition unit may further recognize the subject based on a signal generated in a second accumulation period.

[0235] Furthermore, even if the readout cycle is set to every 1 / 4 full frame period, the counter may be reset midway through the accumulation period of frame 1_1 in Fig. 7 depending on the brightness of the subject, the image recognition accuracy, etc. As a result, the substantial accumulation period may be made shorter than the 1 / 4 full frame period.

[0236] Alternatively, the counter may be reset once at time T1 in Fig. 7. This may adjust the count value read out at time T4. Note that, although the accumulation periods are uniformly spaced in the above embodiment, the present invention is not limited to this.

[0237] In the above embodiments, various processes are performed on the basis of captured image data using various cameras, devices attached to the cameras, personal computers, and the like.

[0238] For example, in the case of a security camera, the control unit 605 monitors, photographs, and records the behavior of a target person. In some cases, the control unit 605 displays a warning, illuminates a light installed in the security camera, and generates a warning sound via the communication unit 607. In the case of an in-vehicle camera, the control unit 605 issues a warning or displays a warning via the ECU 701, and activates a safety device to slow down or stop the vehicle via the vehicle control unit 702.

[0239] <Embodiment 8> In the eighth embodiment, before various processes are performed, the control unit 605 determines whether the image acquired by the photoelectric conversion device 600 is suitable for the subsequent processes. This further reduces the processing load, and reduces the processing load and power consumption. Furthermore, it is possible to avoid erroneous decisions due to inappropriate images, erroneous warnings in security cameras, and erroneous activation of safety devices in vehicle-mounted cameras.

[0240] In the eighth embodiment, the captured image data is evaluated by the control unit 605. Then, using the average luminance value of each full frame output, it is judged whether the image is appropriate or not, and if it is appropriate, it is judged whether to pass it on to the next process.

[0241] Fig. 21 is a flowchart showing an example of the operation of the control unit after constructing an image according to embodiment 8. Note that the CPU as a computer in the control unit 605 executes a computer program stored in the memory, thereby sequentially performing the operations of the steps in the flowchart in Fig. 21.

[0242] The flow up to acquiring an image uses, for example, the examples shown in Figures 16 and 17. First, in step S2100, the CPU of the control unit composes an image using the count value output in step S106 of Figure 16.

[0243] Then, in step S2101, the CPU of the control unit determines whether the image is suitable for use in the next process. Here, an image that is not suitable refers to an image with a poor S / N ratio and full of noise, or an image that is too dark to be recognized.

[0244] The determination method in step S2101 can be determined from the average brightness data of the acquired image. That is, the control unit 605 calculates the average brightness value, and determines whether the value can be used for the next process. That is, the control unit controls whether to perform a predetermined process in the control unit according to the brightness information of the signal generated in at least one of the first accumulation period and the second accumulation period.

[0245] Also, it is not limited to the average luminance, but may be obtained from the peak of the histogram, the bias of the peak, the spread, the variance, etc. Also, the luminance difference between adjacent pixels in the screen may be obtained, and the amount of noise may be calculated from the histogram of the difference, and from the result, it may be judged whether or not it can be used for the next process.

[0246] Furthermore, the determination may be based not only on the previous full frame, but also on the average value of multiple previous full frames, or on the average luminance for each accumulation period within the full frame, a histogram, etc. If it is determined in step S2101 that the image is appropriate, then in step S2102, the CPU of the control unit executes processing using the computer program of the control unit 605.

[0247] On the other hand, if it is determined that the image is not appropriate, in step S2103, the CPU of the control unit does not process that image using the computer program of the control unit 605, and stops processing of the computer program until an image of the next accumulation period is acquired.

[0248] 21, the number of times that unnecessary processing is performed by the control unit 605 can be reduced, and the product power consumption can be reduced. In addition, it is possible to avoid incorrect judgment due to inappropriate images, false warnings in security cameras, and erroneous activation of safety devices in vehicle-mounted cameras.

[0249] In the above-described embodiment, the moving body 700 has been described using an example of an automobile, but the moving body may be any moving body, such as an airplane, a train, a ship, a drone, an AGV, or a robot.

[0250] Moreover, the photoelectric conversion device 600 of the above embodiment is not limited to a moving body, and may be attached to a wall, a pillar, etc. In this case, it is desirable to store the installation height, installation angle, etc. in the storage unit 606, etc. as installation information when the photoelectric conversion device 600 is attached to a wall, a pillar, etc.

[0251] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to the above-mentioned embodiments, and various modifications are possible based on the spirit of the present invention, and are not excluded from the scope of the present invention. The present invention also includes combinations of the above-mentioned embodiments. The present invention also includes the following configurations, methods, etc.

[0252] (Configuration 1) A photoelectric conversion device comprising: an optical system; a plurality of pixels each including a photoelectric conversion unit that emits a pulse in response to incidence of a photon; a counter that counts the number of the pulses; and a memory that stores the count value of the counter; a control unit that generates a signal based on the difference between the count values ​​of the counter at the start and end of an accumulation period in the photoelectric conversion unit, having at least a first accumulation period and a second accumulation period within one full frame, the first accumulation period being shorter than the second accumulation period, and controlling so that the signal generated during the first accumulation period is output between the end of the first accumulation period and the end of the second accumulation period; and a setting unit that is capable of setting a first pixel region having at least the first accumulation period and the second accumulation period, and a second pixel region having at least the second accumulation period, in a sensor unit consisting of the plurality of pixels.

[0253] (Configuration 2) The photoelectric conversion device according to configuration 1, wherein the setting section sets the first pixel region and the second pixel region based on an image read out from the sensor section.

[0254] (Configuration 3) The photoelectric conversion device according to configuration 1 or 2, wherein the first accumulation period and the second accumulation period overlap each other.

[0255] (Configuration 4) The photoelectric conversion device according to configuration 2, wherein the first accumulation period and the second accumulation period start simultaneously.

[0256] (Configuration 5) The photoelectric conversion device according to any one of configurations 1 to 4, wherein the end of the second accumulation period coincides with the end of a full frame.

[0257] (Configuration 6) The photoelectric conversion device according to any one of configurations 1 to 5, further comprising a recognition section that recognizes a subject based on a signal generated during at least the first accumulation period.

[0258] (Configuration 7) The photoelectric conversion device according to configuration 6, wherein the recognition section further recognizes the subject based on a signal generated during the second accumulation period.

[0259] (Configuration 8) The photoelectric conversion device according to any one of configurations 1 to 7, further comprising a display section that displays a signal generated during the second accumulation period as an image.

[0260] (Configuration 9) The photoelectric conversion device according to any one of configurations 1 to 8, wherein the sensor portion includes an avalanche photodiode.

[0261] (Configuration 10) The photoelectric conversion device described in configuration 2, characterized in that the setting unit sets the first pixel region or the second pixel region based on at least one of characteristic information regarding the characteristics of the photoelectric conversion device, installation information regarding the installation state of the photoelectric conversion device, and environmental information regarding the surrounding environment of the photoelectric conversion device.

[0262] (Configuration 11) The photoelectric conversion device described in Configuration 10, characterized in that the setting unit sets at least one of the position, size, and accumulation period of the first pixel region or the second pixel region based on at least one of the characteristic information, the installation information, and the environmental information.

[0263] (Configuration 12) The photoelectric conversion device according to configuration 10 or 11, wherein the characteristic information includes optical characteristic information of the optical system, a resolution of the sensor unit, and a pixel size of the sensor unit.

[0264] (Configuration 13) The photoelectric conversion device according to any one of configurations 10 to 12, wherein the installation information includes at least one of the height, angle, and direction at which the photoelectric conversion device is installed.

[0265] (Configuration 14) The photoelectric conversion device according to any one of configurations 1 to 13, wherein the setting section sets the first pixel region or the second pixel region according to distance information of a subject.

[0266] (Configuration 15) A photoelectric conversion device according to any one of configurations 1 to 14, characterized in that the length of at least one of the first accumulation period and the second accumulation period is set based on luminance information of a signal generated during at least one of the first accumulation period and the second accumulation period.

[0267] (Configuration 16) A photoelectric conversion device described in any one of configurations 1 to 15, characterized in that the control unit controls whether or not to perform a predetermined processing in the control unit depending on luminance information of a signal generated during at least one of the first accumulation period and the second accumulation period.

[0268] (Configuration 17) A moving body comprising the photoelectric conversion device described in any one of configurations 1 to 16 and a movement control unit that controls the operation of the moving body, wherein the control unit sets the first pixel area or the second pixel area based on the output of the movement control unit.

[0269] (Method) A control method for controlling a photoelectric conversion device having a plurality of pixels, each of which has an optical system, a photoelectric conversion unit that emits a pulse in response to the incidence of a photon, a counter that counts the number of the pulses, and a memory that stores the count value of the counter, the control method comprising: generating a signal based on the difference in the count value of the counter at the start and end of an accumulation period in the photoelectric conversion unit; having at least a first accumulation period and a second accumulation period within one full frame, the first accumulation period being shorter than the second accumulation period; controlling so that the signal generated during the first accumulation period is output between the end of the first accumulation period and the end of the second accumulation period; and being capable of setting a first pixel region having at least the first accumulation period and the second accumulation period, and a second pixel region having at least the second accumulation period, in a sensor unit consisting of the plurality of pixels.

[0270] (Program) A computer program for controlling each part of the photoelectric conversion device according to any one of configurations 1 to 16 or the moving body according to claim 17 by a computer.

[0271] A computer program for implementing the functions of the above-described embodiment, part or all of the control in this embodiment, may be supplied to a photoelectric conversion device or the like via a network or various storage media. A computer (or a CPU, MPU, or the like) in the photoelectric conversion device or the like may then read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0272] 11: Sensor board 12: Pixel area 21: Circuit board 22: Circuit area 100: Sensor section 101: Pixel 102: Photoelectric conversion unit 103: Signal processing circuit 110: Vertical scanning circuit 111: Horizontal scanning circuit 112: Readout circuit 113: Vertical signal line 114: Output circuit 115: Control pulse generation unit 201: Avalanche photodiode 202: Quench element 210: Waveform shaping section 211: Counter 212: Memory 213: Drive line 600: Photoelectric conversion device 601:Optical system 602: Detection unit 603: Image processing unit 604: Recognition part 605: Control unit 606: Storage section 607: Communications Department

Claims

1. An optical system; A plurality of pixels each including a photoelectric conversion unit that emits a pulse in response to an incidence of a photon, a counter that counts the number of the pulses, and a memory that stores the count value of the counter; a control unit that generates a signal based on a difference between count values ​​of the counter at the start and end of an accumulation period in the photoelectric conversion unit, has at least a first accumulation period and a second accumulation period within one full frame, the first accumulation period is shorter than the second accumulation period, and controls so that the signal generated during the first accumulation period is output between the end of the first accumulation period and the end of the second accumulation period; a setting unit capable of setting a first pixel region having at least the first accumulation period and the second accumulation period, and a second pixel region having at least the second accumulation period, in a sensor unit composed of the plurality of pixels; A photoelectric conversion device comprising:

2. The photoelectric conversion device according to claim 1 , wherein the setting section sets the first pixel region and the second pixel region based on an image read from the sensor section.

3. 2. The photoelectric conversion device according to claim 1, wherein the first accumulation period and the second accumulation period overlap each other.

4. 3. The photoelectric conversion device according to claim 2, wherein the first accumulation period and the second accumulation period start simultaneously.

5. 2. The photoelectric conversion device according to claim 1, wherein the end of the second accumulation period coincides with the end of a full frame.

6. 2. The photoelectric conversion device according to claim 1, further comprising a recognition section that recognizes a subject based on a signal generated during at least the first accumulation period.

7. 7. The photoelectric conversion device according to claim 6, wherein the recognition section further recognizes the subject based on a signal generated during the second accumulation period.

8. 2. The photoelectric conversion device according to claim 1, further comprising a display unit that displays the signal generated during the second accumulation period as an image.

9. 2. The photoelectric conversion device according to claim 1, wherein the sensor portion includes an avalanche photodiode.

10. The photoelectric conversion device described in claim 2, characterized in that the setting unit sets the first pixel region or the second pixel region based on at least one of characteristic information regarding the characteristics of the photoelectric conversion device, installation information regarding the installation state of the photoelectric conversion device, and environmental information regarding the surrounding environment of the photoelectric conversion device.

11. The photoelectric conversion device according to claim 10, characterized in that the setting unit sets at least one of the position, size, and accumulation period of the first pixel region or the second pixel region based on at least one of the characteristic information, the installation information, and the environmental information.

12. The photoelectric conversion device according to claim 10 , wherein the characteristic information includes optical characteristic information of the optical system, a resolution of the sensor unit, and a pixel size of the sensor unit.

13. The photoelectric conversion device according to claim 10 , wherein the installation information includes at least one of a height, an angle, and a direction in which the photoelectric conversion device is installed.

14. The photoelectric conversion device according to claim 1 , wherein the setting section sets the first pixel area or the second pixel area in accordance with distance information of a subject.

15. 2. The photoelectric conversion device according to claim 1, wherein the length of at least one of the first accumulation period and the second accumulation period is set based on luminance information of a signal generated during at least one of the first accumulation period and the second accumulation period.

16. The photoelectric conversion device according to claim 1, characterized in that the control unit controls whether or not to perform a predetermined processing in the control unit depending on luminance information of a signal generated during at least one of the first accumulation period and the second accumulation period.

17. The photoelectric conversion device according to any one of claims 1 to 16, A movement control unit that controls the operation of the moving body, The moving body, wherein the control unit sets the first pixel area or the second pixel area based on an output of the movement control unit.

18. A control method for controlling a photoelectric conversion device having a plurality of pixels, each of which includes an optical system, a photoelectric conversion unit that emits a pulse in response to incidence of a photon, a counter that counts the number of the pulses, and a memory that stores a count value of the counter, the method comprising the steps of: a signal is generated based on a difference between count values ​​of the counter at the start and end of an accumulation period in the photoelectric conversion unit, at least a first accumulation period and a second accumulation period are provided within one full frame, the first accumulation period is shorter than the second accumulation period, and the signal generated during the first accumulation period is controlled to be output between the end of the first accumulation period and the end of the second accumulation period; A control method characterized in that, in a sensor section consisting of a plurality of pixels, a first pixel area having at least the first accumulation period and the second accumulation period, and a second pixel area having at least the second accumulation period can be set.

19. A computer program for controlling each part of the photoelectric conversion device according to any one of claims 1 to 16 or the moving body according to claim 17 by a computer.

Images