Photoelectric conversion device, mobile, control method, and computer program

Abstract

To provide a photoelectric conversion device capable of reading out image data for a relatively short storage period in the middle in a region of interest.SOLUTION: A photoelectric conversion device comprises: a photoelectric conversion section 102 which generates a pulse in response to the incidence of a photon; and a signal processing circuit 103 each including a counter 211 which counts the number of pulses and a memory 212 in which a count value of the counter is stored. A signal is generated on the basis of a differential of count values of the counter in the start and end of a storage period in the photoelectric conversion section, and there are included at least a first storage period and a second storage period within one full frame. The first storage period is shorter than the second storage period, and control is performed in such a manner that a signal generated during the first storage period is output from the end of the first storage period to the end of the second storage period. In a sensor section consisting of a plurality of pixels, it is possible to set a first pixel region having at least the first storage period and the second storage period and a second pixel region having at least the second storage period.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a photoelectric conversion device, a moving object, a control method, a computer program, and the like. [Background technology]

[0002] 2. Description of the Related Art In recent years, photoelectric conversion devices have been developed that digitally count the number of photons incident on an avalanche photodiode (APD) and output the counted value from a pixel as a photoelectrically converted digital signal.

[0003] For example, Patent Document 1 describes a configuration in which a photoelectric conversion device having an APD can output multiple images whose accumulation periods overlap with each other, enabling continuous shooting even in low illuminance. Patent Document 2 describes a configuration in which signals are read out at a high frame rate from a pixel area in which a moving subject is detected. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 7223070 [Patent Document 2] Patent Publication No. 2021-34786 Summary of the Invention [Problem to be solved by the invention]

[0005] For example, in the case of a camera's image sensor, because normal sensor operation involves performing recognition processing on a frame-by-frame basis, for example, in the case of 30 fps (frames per second), recognition processing can only be performed every 33.3 ms. Therefore, in a normal camera, even if an object appears immediately after a frame change, recognition processing cannot be performed until the end of the frame.

[0006] On the other hand, a faster frame rate can lead to increased noise in low-light environments, as well as increased data volume and power consumption.

[0007] Therefore, an object of the present invention is to provide a photoelectric conversion device capable of reading out image data having a relatively short accumulation period in a region of interest midway. [Means for solving the problem]

[0008] A photoelectric conversion device according to one aspect of the present invention comprises: An optical system; A plurality of pixels each including a photoelectric conversion unit that emits a pulse in response to an incidence of a photon, a counter that counts the number of the pulses, and a memory that stores the count value of the counter; a control unit that generates a signal based on a difference between count values ​​of the counter at the start and end of an accumulation period in the photoelectric conversion unit, has at least a first accumulation period and a second accumulation period within one full frame, the first accumulation period is shorter than the second accumulation period, and controls so that the signal generated during the first accumulation period is output between the end of the first accumulation period and the end of the second accumulation period; a setting unit capable of setting a first pixel region having at least the first accumulation period and the second accumulation period, and a second pixel region having at least the second accumulation period, in a sensor unit composed of the plurality of pixels; The setting unit is characterized in that it sets the first pixel region and the second pixel region based on at least one of characteristic information regarding the characteristics of the photoelectric conversion device, installation information regarding the installation state of the photoelectric conversion device, and environmental information regarding the surrounding environment of the photoelectric conversion device. Effect of the Invention

[0009] According to the present invention, it is possible to provide a photoelectric conversion device capable of reading out image data having a relatively short accumulation period in a region of interest midway. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating an example of the configuration of a sensor unit 100 according to a first embodiment. [Diagram 2] 2 is a diagram showing an example of the configuration of a sensor portion substrate 11 according to the first embodiment. FIG. [Diagram 3] 2 is a diagram showing an example of the configuration of a circuit board 21 according to the first embodiment. FIG. [Figure 4] 2 and 3, this is a diagram showing an equivalent circuit of the photoelectric conversion unit 102 and the signal processing circuit 103 corresponding to the photoelectric conversion unit 102. [Diagram 5] 2 is a diagram illustrating a relationship between the operation of an APD 201 and an output signal. FIG. [Figure 6] 1 is a functional block diagram of a photoelectric conversion device 600 according to a first embodiment. [Figure 7] 6 is a diagram for explaining a photoelectric conversion method performed by a control unit 605 according to the first embodiment. FIG. [Figure 8] FIG. 4 is a diagram showing an example of images of a plurality of divided accumulation periods according to the first embodiment. [Figure 9] FIG. 2 is a diagram illustrating the relationship between a memory and a buffer in the first embodiment. [Figure 10] 5 is a flowchart showing details of an example of driving the sensor unit 100 in the first embodiment. [Figure 11] This is a continuation of the flowchart in Figure 10. [Figure 12] 4A to 4C are diagrams for explaining an example of setting an attention area in the first embodiment. [Figure 13] 13A to 13C are diagrams illustrating an example of setting an attention area in the second embodiment. [Figure 14] FIG. 11 is a functional block diagram showing an example of the configuration of a photoelectric conversion device 600 and a moving object 700 according to a third embodiment. [Figure 15] 11 is a flowchart illustrating a method for controlling a moving object according to a third embodiment. [Figure 16] 13 is a diagram for explaining an example of setting an attention area in the fourth embodiment. FIG. [Figure 17] 13 is a diagram for explaining an example of setting an attention area in the fifth embodiment. FIG. [Figure 18] 13 is a diagram for explaining an example of setting an attention area in the seventh embodiment. FIG. [Figure 19] 13 is a diagram for explaining an example of setting a region of interest in the eighth embodiment. FIG. [Figure 20] 23 is a flowchart showing details of an example of driving the sensor unit 100 regarding a method of setting an attention area according to a ninth embodiment. [Figure 21] This is a continuation of the flowchart in Figure 20. [Figure 22] FIG. 13 is a diagram showing an example of images taken for a plurality of accumulation periods after a region of interest is set. [Figure 23] 23 is a flowchart showing details of an example in which an accumulation period is set and the sensor unit 100 is driven in the tenth embodiment. [Figure 24] This is a continuation of the flowchart in Figure 23. [Diagram 25] 23 is a flowchart showing an example of the operation of the control unit after an image is constructed according to the eleventh embodiment. [Figure 26] 23A to 23C are diagrams for explaining an example of setting an attention area in the twelfth embodiment. [Figure 27] 23A to 23C are diagrams for explaining an example of setting an attention area in the thirteenth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In each drawing, the same members or elements are given the same reference numerals, and duplicated descriptions are omitted or simplified.

[0012] <Embodiment 1> 1 is a diagram showing a configuration example of a sensor unit 100 according to embodiment 1. In the following, the sensor unit has a so-called laminated structure in which two substrates, a sensor unit substrate 11 and a circuit substrate 21, are laminated and electrically connected.

[0013] However, a so-called non-laminated structure in which the components included in the sensor substrate 11 and the components included in the circuit substrate 21 are disposed on a common semiconductor layer may be used. The sensor substrate 11 includes a pixel region 12. The circuit substrate 21 includes a circuit region 22 that processes signals detected in the pixel region 12.

[0014] 2 is a diagram showing an example of the configuration of the sensor portion substrate 11 according to embodiment 1. The pixel region 12 of the sensor portion substrate 11 includes a plurality of pixels 101 arranged two-dimensionally across a plurality of rows and a plurality of columns.

[0015] The pixel 101 includes a photoelectric conversion unit 102 including an avalanche photodiode (hereinafter, APD). Here, the photoelectric conversion unit 102 emits a pulse at a frequency corresponding to the frequency of receiving photons. That is, the photoelectric conversion unit 102 emits a pulse in response to the incidence of photons. The number of rows and columns of the pixel array constituting the pixel region 12 is not particularly limited.

[0016] Fig. 3 is a diagram showing an example of the configuration of a circuit board 21 according to embodiment 1. The circuit board 21 has a signal processing circuit 103 that processes charges photoelectrically converted by each photoelectric conversion unit 102 in Fig. 2, a readout circuit 112, a control pulse generation unit 115, a horizontal scanning circuit 111, a vertical signal line 113, a vertical scanning circuit 110, and an output circuit 114.

[0017] The vertical scanning circuit 110 receives a control pulse supplied from a control pulse generating unit 115, and sequentially supplies the control pulse to a plurality of pixels arranged in the row direction. The vertical scanning circuit 110 is composed of logic circuits such as a shift register and an address decoder.

[0018] The signal output from the photoelectric conversion unit 102 of each pixel is processed by each signal processing circuit 103. The signal processing circuit 103 is provided with a counter, a memory, etc., and a digital value is stored in the memory. The horizontal scanning circuit 111 inputs a control pulse that sequentially selects each column to the signal processing circuit 103 in order to read the signal from the memory of each pixel in which the digital signal is stored.

[0019] A signal is output to the vertical signal line 113 from the signal processing circuit 103 of the pixel of the row selected by the vertical scanning circuit 110. The signal output to the vertical signal line 113 is output to the outside of the sensor unit 100 via the readout circuit 112 and the output circuit 114. The readout circuit 112 has a plurality of buffers built therein, each connected to the vertical signal line 113.

[0020] 2 and 3, a plurality of signal processing circuits 103 are arranged in a region overlapping the pixel region 12 in a plan view. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged in a region of the sensor portion substrate 11 outside the pixel region 12 so as to overlap with each other in a plan view.

[0021] That is, the sensor portion substrate 11 has a pixel region 12 and a non-pixel region arranged around the pixel region 12. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged in a region overlapping the non-pixel region in a plan view.

[0022] The arrangement of the vertical signal lines 113, the readout circuits 112, and the output circuits 114 are not limited to the example shown in Fig. 3. For example, the vertical signal lines 113 may be arranged to extend in the row direction, and the readout circuits 112 may be arranged at the ends of the vertical signal lines 113.

[0023] Moreover, it is not necessary to provide one signal processing circuit 103 for each photoelectric conversion unit. One signal processing circuit may be shared by a plurality of photoelectric conversion units, and may be configured to perform signal processing sequentially.

[0024] Fig. 4 is a diagram showing an equivalent circuit of the photoelectric conversion unit 102 and the signal processing circuit 103 corresponding to the photoelectric conversion unit 102 in Fig. 2 and Fig. 3. The photoelectric conversion unit 102 and the signal processing circuit 103 are included in each pixel. The APD 201 included in the photoelectric conversion unit 102 generates a charge pair according to incident light by photoelectric conversion.

[0025] One of the two nodes of the APD 201 is connected to a power supply line to which a driving voltage VL (first voltage) is supplied, and the other of the two nodes of the APD 201 is connected via a quench element 202 to a power supply line to which a driving voltage VH (second voltage) higher than the voltage VL is supplied.

[0026] In Fig. 4, one node of the APD 201 is the anode, and the other node of the APD is the cathode. A reverse bias voltage is supplied to the anode and cathode of the APD 201 so that the APD 201 performs avalanche multiplication. By supplying such a voltage, the charge generated by the incident light undergoes avalanche multiplication, generating an avalanche current.

[0027] When a reverse bias voltage is supplied, there are two modes: Geiger mode, which operates with the voltage difference between the anode and cathode greater than the breakdown voltage, and linear mode, which operates with the voltage difference between the anode and cathode close to or less than the breakdown voltage. An APD that operates in Geiger mode is called a SPAD. In the case of a SPAD, for example, the voltage VL (first voltage) is -30V and the voltage VH (second voltage) is 1V.

[0028] The signal processing circuit 103 includes a quench element 202, a waveform shaping unit 210, a counter 211, and a memory 212. The quench element 202 is connected to a power supply line to which a driving voltage VH is supplied and one of the anode and cathode nodes of the APD 201.

[0029] The quench element 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, and has a function of suppressing avalanche multiplication by suppressing the voltage supplied to the APD 201 (quench operation).The quench element 202 also has a function of returning the voltage supplied to the APD 201 to the drive voltage VH by flowing a current equivalent to the voltage drop caused by the quench operation (recharge operation).

[0030] 4 shows an example in which the signal processing circuit 103 has a waveform shaping unit 210, a counter 211, and a memory 212 in addition to the quench element 202. The waveform shaping unit 210 shapes the voltage change of the cathode of the APD 201 obtained at the time of photon detection, and outputs a pulse signal.

[0031] For example, an inverter circuit is used as the waveform shaping unit 210. In Fig. 4, an example in which one inverter is used as the waveform shaping unit 210 is shown, but a circuit in which a plurality of inverters are connected in series may be used, or another circuit having a waveform shaping effect may be used.

[0032] The counter 211 counts the number of pulses output from the waveform shaping unit 210 and holds the count value. When a control pulse RES is supplied via a drive line 213, the signal held in the counter 211 is reset. Here, the counter 211 generates a signal based on the difference between the count values ​​at the start and end of the accumulation period.

[0033] 3 to the memory 212 via a drive line 214 (not shown in FIG. 3) in FIG. 4, and switches between electrical connection and disconnection between the counter 211 and the vertical signal line 113. The memory 212 functions as a memory that temporarily stores the count value of the counter, and outputs an output signal from the counter 211 of the pixel to the vertical signal line 113.

[0034] Note that electrical connections may be switched by disposing a switch such as a transistor between the quench element 202 and the APD 201 or between the photoelectric conversion unit 102 and the signal processing circuit 103. Similarly, the supply of the voltage VH or the voltage VL to the photoelectric conversion unit 102 may be electrically switched using a switch such as a transistor.

[0035] 5 is a diagram showing a schematic diagram of the relationship between the operation of the APD 201 and the output signal. The input side of the waveform shaping unit 210 is nodeA, and the output side is nodeB. Between time t0 and time t1, a potential difference of VH-VL is applied to the APD 201. When a photon is incident on the APD 201 at time t1, avalanche multiplication occurs in the APD 201, an avalanche multiplication current flows through the quench element 202, and the voltage at nodeA drops.

[0036] When the amount of voltage drop further increases and the potential difference applied to the APD 201 decreases, the avalanche multiplication of the APD 201 stops as at time t2, and the voltage level of nodeA does not drop below a certain value.

[0037] After that, between time t2 and time t3, a current flows through nodeA to compensate for the voltage drop from voltage VL, and at time t3, nodeA is stabilized to its original potential level. At this time, the portion of the output waveform at nodeA that exceeds a certain threshold is shaped by waveform shaping section 210 and is output as a pulse signal at nodeB.

[0038] Next, a photoelectric conversion device 600 according to the first embodiment will be described. Fig. 6 is a functional block diagram of the photoelectric conversion device 600 according to the first embodiment. Some of the functional blocks shown in Fig. 6 are realized by causing a computer (not shown) included in the photoelectric conversion device 600 to execute a computer program stored in a memory serving as a storage medium (not shown).

[0039] However, a part or all of these may be realized by hardware, which may be a dedicated circuit (ASIC) or a processor (a reconfigurable processor, DSP), etc.

[0040] In addition, the functional blocks shown in Fig. 6 do not have to be built in the same housing, and may be configured by separate devices connected to each other via signal paths. The above explanation regarding Fig. 6 also applies to Fig. 14 described later.

[0041] 1 to 5, an optical system 601, a detection unit 602, an image processing unit 603, a recognition unit 604, a control unit 605, a storage unit 606, a communication unit 607, etc. The sensor unit is composed of an avalanche photodiode for photoelectrically converting an optical image, as described in FIGS.

[0042] The camera unit, which is a set of the optical system 601 and the sensor unit 100, is configured to capture an image in at least one direction, for example, in the front, rear, or side of the photoelectric conversion device 600.

[0043] In this embodiment, the optical system 601 is a wide-angle lens (e.g., a fisheye lens) having an angle of view of, for example, 120°, and forms an optical image (subject image) of an object in front of the photoelectric conversion device 600 on the imaging surface of the sensor unit 100. The detection unit 602 detects information (hereinafter, referred to as environmental information) related to the surrounding environment of the photoelectric conversion device 600. In addition, depending on the output of the detection unit 602, the positions and sizes of an attention area and a non-attention area, which will be described later, are changed.

[0044] The image processing unit 603 performs image processing such as black level correction, gamma correction, noise reduction, digital gain adjustment, demosaic processing, data compression, etc. on the image signal acquired by the sensor unit 100 to generate a final image signal. If the sensor unit has an on-chip color filter such as RGB, the image processing unit 603 performs processing such as white balance correction and color conversion.

[0045] Moreover, the output of the image processing unit 603 is supplied to a recognition unit 604 and a control unit 605. The recognition unit 604 recognizes people, vehicles, objects, etc. captured in the image by performing image recognition based on the image signal. The recognition result by the recognition unit 604 is output to the control unit 605 and is reflected in, for example, changing the control mode of the photoelectric conversion device 600. Furthermore, the recognition result is stored in a storage unit 606 and transmitted to the outside via a communication unit 607, etc.

[0046] The control unit 605 has a built-in CPU as a computer and a memory that stores a computer program. The control unit 605 also functions as a setting unit, and sets the length of the exposure period of each frame of the sensor unit 100 and the timing control of the control signal CLK for each region via the control pulse generation unit 115 of the sensor unit 100.

[0047] Furthermore, the control unit 605 also functions as an acquisition unit, and acquires, as characteristic information of the photoelectric conversion device 600, sensor characteristic information such as the size and number of pixels of the sensor unit 100, and optical characteristic information such as the angle of view and resolution of the optical system 601. Furthermore, the control unit 605 acquires information such as the installation height and installation angle as installation information of the photoelectric conversion device 600 from the detection unit 602, and acquires environmental information related to the surrounding environment of the photoelectric conversion device 600 from the detection unit 602.

[0048] Based on the information acquired from the detection unit 602 , the CPU executes a computer program stored in a memory built into the control unit 605 to control each unit of the photoelectric conversion device 600 .

[0049] The storage unit 606 includes a recording medium such as a memory card, a hard disk, etc., and can store and read image signals. The communication unit 607 includes a wireless or wired interface, and outputs the generated image signal to the outside of the photoelectric conversion device 600 and receives various signals from the outside.

[0050] The photoelectric conversion device 600 is used, for example, as a camera, an in-vehicle camera, a pet camera, a surveillance camera, a camera for detection or inspection used in a manufacturing line, a camera for logistics, etc. In addition, the device can be used for various purposes such as a medical endoscope camera, a condition detection camera for nursing care, a camera for infrastructure inspection, an agricultural camera, etc.

[0051] 7 is a diagram for explaining a photoelectric conversion method by the control unit 605 according to the first embodiment. In this embodiment, the photoelectric conversion is periodically driven at, for example, 30 full frames / second. In addition, hereinafter, a frame of one vertical period having a length of 33.3 ms is called a full frame, and each of the full frames divided into four is called a frame.

[0052] That is, as shown in Fig. 7, a full frame 1 is divided into frames 1_1, 1_2, 1_3, and 1_4 each having an equal period (8.33 ms). Note that in Fig. 7 and subsequent figures, a frame may be abbreviated as F.

[0053] Frame 1_1 has an accumulation period from start time T0 of full frame 1 to time T1, frame 1_2 has an accumulation period from time T0 to time T2, frame 1_3 has an accumulation period from time T0 to time T3, and frame 1_4 has an accumulation period from time T0 to time T4.

[0054] At times T1 to T4, count values ​​C1_1, C1_2, C1_3, and C1_4 are obtained from the counter 211. The count values ​​C1_1, C1_2, C1_3, and C1_4 are temporarily stored in the memory 212.

[0055] Then, the signals for one row temporarily stored in the memory 212 are sequentially output from the sensor unit via the buffer of the readout circuit 112. Also, the counter 211 is reset at time T0. Thus, according to this embodiment, the signals accumulated during the period of frame 1_1 are read out from time T1 to time T2 and quickly processed by the recognition unit 604.

[0056] Therefore, image recognition can be performed quickly. Similarly, signals accumulated during the periods of frame 1_2, frame 1_3, and frame 1_4 are sequentially read out from time T2 to T3, T3 to T4, and T4 to T1, respectively, and image recognition can be performed repeatedly. Note that the length of a full frame is not limited to the above example. Also, the number of divisions of a full frame is not limited to 4.

[0057] Fig. 8 is a diagram showing an example of images of a plurality of divided accumulation periods according to the first embodiment. As shown in Fig. 8, the image of frame 1_1 is dark because the accumulation period is short, but there is little subject blur of a person who jumps out. On the other hand, the accumulation periods of frames 1_2, 1_3, and 1_4 are longer in this order, so subject blur is more likely to occur. Note that blur is less likely to occur in stopped vehicles and white lines, and the longer the accumulation period, the more likely the contrast is to be improved.

[0058] In this manner, in this embodiment, one full frame has a first accumulation period (e.g., time T1 to T2) and a second accumulation period (e.g., time T1 to T4), and the first accumulation period is shorter than the second accumulation period. Also, the signal generated during the first accumulation period is controlled to be output between the end of the first accumulation period and the end of the second accumulation period.

[0059] In this embodiment, the first and second accumulation periods overlap, that is, the first and second accumulation periods start at the same time, the end of the second accumulation period coincides with the end of a full frame, and the second accumulation period is an integer multiple of the first accumulation period.

[0060] However, the second accumulation period does not need to be an integer multiple of the first accumulation period. The second accumulation period only needs to be longer than the first accumulation period (the first accumulation period is shorter than the second accumulation period) and the second accumulation period only needs to end after the first accumulation period.

[0061] That is, an image with a short accumulation period and an image with a long accumulation period are created, the timing at which the short accumulation period ends is set to be earlier than the timing at which the long accumulation period ends, and as soon as the short accumulation period ends, the image is output and sent to the downstream recognition unit 604.

[0062] The subject is recognized based on the signal generated at least during the first accumulation period. The recognition unit 604 recognizes the subject based on the signal generated at least during the first accumulation period. Therefore, while image recognition was not possible until one full frame period had elapsed in the past, image recognition can be performed after as little as ¼ full frame period in this embodiment.

[0063] In addition, an image with a long accumulation period can improve the contrast, and can be used as an image for display. That is, an image with a short accumulation period is suitable for quick subject recognition, and an image with a long accumulation period is suitable for an image for display. Therefore, the display unit 703 of this embodiment displays an image with a long accumulation period, that is, a signal generated during the second accumulation time, as an image.

[0064] In addition, because this embodiment uses an APD, unlike a CMOS image sensor, the accumulated charge is not degraded by reading out, and the accumulation periods can be overlapped. Also, since there is no readout noise, the original signal does not degrade even if it is read out multiple times from one accumulation.

[0065] On the other hand, in the case of a CMOS image sensor, it is possible to shorten the accumulation period by increasing the frame frequency, but each time the accumulation period is shortened, readout noise is superimposed, deteriorating the signal-to-noise ratio (S / N) of the output, making it difficult to obtain an image suitable for image recognition.

[0066] In addition, in order to overlap the accumulation periods in a CMOS image sensor, the same charge must be accumulated in the pixel again after reading out, which makes the circuit more complex and larger in scale.In addition, noise is also superimposed when accumulating charge in the pixel again, which significantly deteriorates the output S / N ratio and makes it difficult to obtain an image suitable for image recognition.

[0067] Fig. 9 is a diagram showing the relationship between memories and buffers in embodiment 1. Fig. 9 shows a state in which memories 212 in signal processing circuit 103 in Fig. 3 are arranged in N rows and M columns, and each memory 212 is represented as memory 1-1 to memory NM. Also, buffer 1 to buffer M in Fig. 9 indicate buffers included in read circuit 112 in Fig. 3. Output circuit 114 in Fig. 9 corresponds to output circuit 114 in Fig. 3.

[0068] Fig. 10 is a flowchart showing details of an example of driving the sensor unit 100 in the first embodiment, and Fig. 11 is a flowchart continuing from Fig. 10. Note that the CPU as a computer in the control unit 605 executes a computer program stored in the memory, thereby sequentially performing the operations of the steps in the flowcharts of Fig. 10 and Fig. 11.

[0069] 10, the CPU of the control unit 605 sets i to 1. Next, in step S102, the CPU of the control unit 605 causes the count value Count of the counter 211 at time Ti to be output to the memory 212. At this time, output is performed simultaneously to all memories. This operation corresponds to the operation at time T1 in FIG.

[0070] Next, in step S103, the CPU of the control unit 605 sets j=1. Next, in step S104, the CPU of the control unit 605 sets k=1. In step S105, the CPU of the control unit 605 outputs the count value Count(jki) in memory jk in FIG. 9 to the buffer. At this time, columns 1 to M are output to the buffer at the same time. This operation means that the count value of the first row in FIG. 9 is taken into the buffer.

[0071] In step S106, the CPU of the control unit 605 causes the count value Count(jki) of buffer k to be output to the output circuit 114. This operation corresponds to the operation of reading out the signals of the buffers in the leftmost column in FIG.

[0072] Next, it proceeds to step S107 in FIG. 11 via A. In step S107, the CPU of the control unit 605 determines whether k < M. If it is Yes in step S107, then in step S108, k is incremented by 1 such that k = k + 1, and it returns to step S106 via B and performs the operation of step S106. This operation corresponds to the operation of reading the signal of the buffer in the second column from the left in FIG. 9 from the output circuit.

[0073] If it becomes No in step S107, that is, if k = M, it means that the signal of the buffer in the M-th column in FIG. 9 has been read from the output circuit. Next, it proceeds to step S109, and the CPU of the control unit 605 determines whether j < N. If it is Yes in step S109, then in step S110, the CPU of the control unit 605 sets j = J + 1, increments j by 1, and returns to step S104 via C. This corresponds to the operation for starting to read the next line.

[0074] If it is determined to be No in step S109, it means that the reading of all lines has been completed. So, it proceeds to step S111, and the CPU of the control unit 605 determines whether i < 4. If it is determined to be Yes in step S111, then in step S112, the CPU of the control unit 605 sets i = i + 1, increments i by 1, and returns to step S102 via D. This operation corresponds to the operation for starting to read at the next time T2.

[0075] If it is determined to be No in step S111, it means that the reading at time T4 has been completed. So, it proceeds to step S113, and the CPU of the control unit 605 resets the counter 211 with a reset signal. This operation corresponds to the reset operation of the counter 211 at time T4 in FIG. 7. Note that the flows in FIGS. 10 and 11 are periodically repeated. In the above manner, the signals accumulated in the sensor unit can be sequentially read out.

[0076] In the sensor unit, in the case of the circuit configuration as shown in FIG. 9, it is possible to change the setting of the accumulation period for each column.

[0077] For example, in the first column (k=1), four pieces of data are read out by changing the accumulation period of the full frame 1 to frame 1_1, frame 1_2, frame 1_3, and frame 1_4, as shown in Fig. 7. In other words, the first column is set as the first image area with a short accumulation period.

[0078] On the other hand, in the second column (k=2), it is possible to divide the period of full frame 1 into two and read out only the data of the accumulation periods of frames 1_2 and 1_4. In other words, the second column is set as the second image area with a longer accumulation period.

[0079] In this way, according to this embodiment, the accumulation period and the number of times can be changed arbitrarily, and by changing the accumulation period and the number of times, it is possible to change the intensity of the accumulated signal and increase or decrease the number of accumulations. Also, it is possible to increase or decrease the amount of data according to the number of accumulations. However, if the amount of data increases, the power consumption of the sensor unit and the photoelectric conversion device 600 also increases accordingly.

[0080] The method of changing the accumulation period setting is not limited to changing it for each column, but it is also possible to change the accumulation period setting for each row or for each pixel by changing the signal line of the sensor unit 100. The setting of the accumulation period for each region or pixel is performed by the control unit 605.

[0081] As described above, the control unit 605 has a built-in CPU as a computer and a memory that stores a computer program, and acquires information from the detection unit 602, and the CPU executes the computer program stored in the memory based on the information. This makes it possible to set the accumulation period for each region or pixel.

[0082] In this way, the control unit 605 can set the accumulation period for each region or pixel of the sensor unit 100, so that the timing at which a short accumulation period ends can be made earlier than the timing at which a long accumulation period ends. Therefore, as soon as the short accumulation period ends, it becomes possible to output the image and send it to the downstream recognition unit 604.

[0083] That is, based on the signal generated in the first accumulation period, the recognition unit 604 enables object recognition in an image area with a short accumulation period at time T1 when frame 1_1 ends, which is earlier than time T4 when full frame 1 ends in the first row (k=1). Therefore, whereas image recognition was not possible until one full frame period had elapsed in the past, this embodiment makes it possible to set an area where image recognition can be performed in a shorter time.

[0084] Also, because images can be captured over multiple integration periods, properly exposed images can be generated for both light and dark areas, maximizing the signal-to-noise ratio (S / N) in each area. In other words, in dark areas, a long integration period increases the signal, maximizing the S / N ratio.

[0085] On the other hand, in bright areas, a short integration period allows signals to be acquired without saturating the signal, maximizing the S / N ratio. Therefore, by applying an integration period appropriate for each area, it is possible to acquire images with maximized S / N ratios without saturating the signal for each area and suppressing noise.

[0086] In addition, by applying an appropriate accumulation period to each region, blurring due to the movement of the subject can be suppressed. A long accumulation period may result in a blurred image (blur) of the subject. On the other hand, a short accumulation period can reduce blurring of the subject. Therefore, by applying an appropriate accumulation period, it is possible to obtain an image with reduced blurring due to the movement of the subject.

[0087] Next, an example of a method for setting the region of interest will be described with reference to Fig. 12. Fig. 12 is a diagram for explaining an example of setting the region of interest in embodiment 1. In embodiment 1, the region of interest is set by dividing it in the column direction in Fig. 9.

[0088] 12 shows an example of an image captured by a photoelectric conversion device 600. In this photoelectric conversion device 600, the optical system 601 has a horizontal angle of view of 120 degrees and a vertical angle of view of 30 degrees, and the sensor unit 100 has 1920 pixels in the M column direction and 1080 pixels in the N row direction.

[0089] The photoelectric conversion device 600 is installed at a height of 100 mm from the ground and at an angle such that a vanishing point at an infinite distance is 800 pixels from the bottom in the N-row direction of the sensor unit 100. The upper side of the sensor unit 100 in the N-row direction acquires long-distance images, and the lower side acquires close-distance images.

[0090] More specifically, the position of 100 pixels from the bottom captures an object located about 0.4 m away from the photoelectric conversion device 600, the position of 600 pixels from the bottom captures an object located about 1.5 m away, and the position of 780 pixels from the bottom captures an object located about 15 m away.

[0091] When the subject moves in the M-row direction, the closer the subject is, the greater the number of moving pixels becomes compared to a longer distance. Therefore, if the accumulation time is extended, there is a problem that the subject may go outside the shooting area of ​​the photoelectric conversion device 600 or the captured image of the subject may become blurred.

[0092] Therefore, in this embodiment, the short distance range is set as the attention area in the first pixel area, and an image is acquired with an accumulation period shorter than the full frame period. This makes it possible to capture a moving subject within the shooting area, and suppress image blurring of the subject. The first pixel area has at least a first accumulation period and a second accumulation period.

[0093] On the other hand, in the second pixel region, which is the long distance range, even if the subject moves in the M column direction at the same speed, the number of moving pixels is smaller compared to the short distance, so the subject can be captured within the capturing region of the photoelectric conversion device 600, and the image blur of the captured subject is reduced.

[0094] Therefore, even if an image with an accumulation period shorter than a full frame period is acquired, the amount of data with little change, that is, data with little necessity, increases. Therefore, in this embodiment, the accumulation period in the second pixel region is set to a second accumulation time longer than the first accumulation time, thereby making it possible to reduce the number of data and suppress the power consumption of the photoelectric conversion device 600. That is, the second pixel region has at least the second accumulation period.

[0095] Specifically, in this embodiment, a first pixel region is set as a region of interest, for example, 600 pixels or less from the bottom in the column direction, and a second pixel region is set as a region of non-interest, for example, 601 pixels or more from the bottom in the column direction.

[0096] In addition, the first pixel area is photographed using an accumulation period in which one full frame having a length of 33.3 ms is divided into four, and the second pixel area is photographed using the same accumulation period as the first pixel area, without dividing one full frame having a length of 33.3 ms.

[0097] In this case, if a subject is located about 1.5 m away from the photoelectric conversion device 600 and moves in the M-row direction at 4 km / h, an image captured with an accumulation time of 33.3 ms will be blurred by about 14 pixels. On the other hand, in the case of an accumulation period divided into four, capturing an image with the shortest accumulation period of 8.33 ms will result in an image that is blurred by only about 3 pixels, making it possible to obtain a clear image.

[0098] On the other hand, if the subject is located approximately 15 m away from the photoelectric conversion device 600 and moves in the M-row direction at 4 km / h, then when the image is captured with an accumulation time of 33.3 ms, the movement of the subject will be captured in an image that fits within the angle of view of one pixel.

[0099] Therefore, even when the accumulation period is divided into four, even if shooting is done with the shortest accumulation period of 8.33 ms, the movement caused by the movement of the subject will be contained within the angle of view of one pixel, resulting in an image that is not blurred and a clear image can be obtained.

[0100] Therefore, even if the accumulation period is changed, the blur of the subject image does not change, and four almost identical images with different brightness are captured, resulting in an increase in the amount of data. In other words, at a long distance from the photoelectric conversion device 600, capturing many images with different accumulation periods is likely to result in an increase in unnecessary data.

[0101] In this manner, in this embodiment, the first pixel region and the second pixel region are appropriately set based on the device information and installation information of the photoelectric conversion device 600, and images are captured with different accumulation periods for each region. This makes it possible to reduce the amount of data and power consumption while reducing pixel blur caused by subject movement.

[0102] <Embodiment 2> Next, an example of setting an attention region by dividing it in the row direction in Fig. 9 according to the second embodiment will be described. Fig. 13 is a diagram for explaining an example of setting an attention region according to the second embodiment.

[0103] For example, as shown in FIG. 13, the horizontal angle of view of the optical system 601 of the photoelectric conversion device 600 is 120 degrees and the vertical angle of view is 30 degrees, and the number of pixels of the sensor unit 100 is 1920 pixels in the M column direction and 1080 pixels in the N row direction.

[0104] The photoelectric conversion device 600 is installed at a height of 100 mm and at an angle such that a vanishing point at an infinite distance is 800 pixels from the bottom in the N-row direction of the sensor unit 100. The left and right sides of the M-column direction of the sensor unit 100 acquire images on the left and right sides, respectively, of the angle of view of the photoelectric conversion device 600.

[0105] In this embodiment, the left and right ends of the angle of view are set as the first pixel area as the area of ​​interest, and an image is acquired in an accumulation period (first accumulation period) shorter than the full frame period. This reduces the situation where the subject moves beyond the shooting area and cannot be captured.

[0106] In addition, when recognizing a subject that has entered the shooting area from outside and tracking and shooting the subject as a target, it becomes possible to shoot, recognize and record the subject that has entered from the left or right edge more quickly.

[0107] On the other hand, in the second pixel region, which is the range at the center of the angle of view, the subject can be captured within the capture region even if the subject moves. Also, since the subject may already be recognized, even if an image with an accumulation period shorter than the full frame period is acquired, the amount of unnecessary data increases.

[0108] Therefore, in this embodiment, in the second pixel region, an image is acquired over a second accumulation period that is longer than the first accumulation period, thereby reducing the amount of data and suppressing the power consumption of the photoelectric conversion device 600.

[0109] <Embodiment 3> Next, as a third embodiment, an example will be described in which a traffic object is recognized and an attention area is set based on the recognition result when the photoelectric conversion device 600 is attached to a moving body 700 and used. In this embodiment, the traffic object refers to, for example, people or moving objects on the road, such as pedestrians, motorbikes, and bicycles, around the moving body.

[0110] Fig. 14 is a functional block diagram showing a configuration example of a photoelectric conversion device 600 and a moving body 700 according to the third embodiment. The photoelectric conversion device 600 according to the third embodiment is mounted on the moving body 700, and sets an attention area based on at least one of the positions, speeds, traveling directions, distances, etc. of people, cars, etc., which are traffic participants. In Fig. 14, the same reference numerals as those in Fig. 6 denote the same functional blocks, and therefore the description thereof will be omitted.

[0111] In the third embodiment, the output of the image processing unit 603 is supplied to a recognition unit 604, a control unit 605, and an ECU (Electric Control Unit) 701 serving as a movement control unit of a moving object 700. The recognition unit 604 recognizes surrounding people, vehicles, objects, etc. by performing image recognition based on the image signal, and outputs the recognition result to the ECU 701.

[0112] The control unit 605 also acquires information such as the installation height and installation angle as installation information of the photoelectric conversion device 600 on the moving object 700 from the ECU 701. Furthermore, the control unit 605 acquires environmental information of the photoelectric conversion device 600 from the detection unit 602, and acquires information on the moving object 700 and environmental information from the ECU 701.

[0113] The information on the moving object acquired from the ECU 701 includes at least one of vehicle speed information, acceleration information, steering angle information, brake information, engine information, etc. Furthermore, the environmental information includes at least one of information on the presence, position, speed, acceleration, distance, etc. of traffic participants (pedestrians, motorbikes, bicycles, etc. around the moving object), map information, GPS information, road conditions, road surface conditions, weather information, surrounding brightness, time, etc.

[0114] Then, based on the acquired information, the CPU executes a computer program stored in a memory built into the control unit 605 to control each unit of the photoelectric conversion device 600 .

[0115] The ECU 701 incorporates a CPU as a computer and a memory that stores a computer program, and controls each part of the moving body 700 by the CPU executing the computer program stored in the memory.

[0116] The output of the ECU 701 is also supplied to a vehicle control unit 702 and a display unit 703. The vehicle control unit 702 functions as a driving control unit that performs operations such as driving, stopping, and directional control of the vehicle as a moving body based on the output of the ECU 701.

[0117] The display unit 703 includes a display element such as a liquid crystal device or an organic EL device, and is mounted on the moving body 700. The display unit 703 displays images acquired by a sensor unit and various information related to the vehicle's running state and the like to the driver of the moving body 700 using, for example, a GUI based on the output of the ECU 701.

[0118] 14 may not be mounted on the moving body 700. For example, they may be provided in an external terminal or an external server provided separately from the moving body 700 for remotely controlling the moving body 700 or monitoring the traveling of the moving body.

[0119] Fig. 15 is a flowchart for explaining a method for controlling a moving object according to embodiment 3. Note that the control unit 605 and a CPU serving as a computer in the ECU 701 execute a computer program stored in the memory, thereby sequentially performing the operations of the steps in the flowchart in Fig. 15.

[0120] First, in step S1511, the CPU of the control unit 605 acquires video data from the photoelectric conversion device 600 while driving. Next, in step S1512, the CPU of the control unit 605 performs image recognition based on a signal read from the pixel area of ​​the sensor unit 100 of the photoelectric conversion device 600, thereby recognizing an object that is a traffic participant.

[0121] Then, in step S1513, the CPU of the control unit 605 acquires the position and speed of the recognized object, such as a traffic object, by observing changes in the object in multiple full-frame video data, or by using a millimeter-wave radar or LiDAR (Light Detection and Ranging) as another distance measuring device.

[0122] Next, in step S1514, the CPU of the ECU 701 acquires the traveling direction of the object that is a traffic participant and the distance to the object. As in the previous example, this may be acquired by observing the change in the object in multiple full-frame video data, or by using a millimeter-wave radar or LiDAR as another distance measuring device.

[0123] Then, in step S1515, the CPU of ECU 701 sets an attention area based on at least one of the position, speed, traveling direction, distance, etc. of an object such as a traffic participant object, etc. Next, in step S1516, the braking distance of the host vehicle is calculated.

[0124] The braking distance is not limited to the speed of the moving body 700, but varies depending on the inclination angle of the road surface, the condition of the road surface (asphalt, dirt, gravel, rain, snow, ice, etc.), the degree of tire wear (characteristics of the tire itself, distance used, etc.), and the weight of the moving body 700. Therefore, such information may be used comprehensively to make the determination.

[0125] Next, in step S1517, the CPU of the ECU 701 evaluates the collision risk with a target who is a traffic participant target, taking into consideration factors such as the speed, traveling direction, and distance of the target who is a traffic participant target. If it is determined in step S1517 that the collision risk is higher than the predetermined value, the CPU of the ECU 701 generates a warning signal in step S1518, and issues a warning to the driver via the display unit 703 or a voice message, and further activates an automatic brake system as necessary.

[0126] If it is determined in step S1517 that the collision risk is equal to or less than the predetermined value, the process returns to step S1511, and the next full frame of video data is obtained.

[0127] After step S1518, the process proceeds to step S1519, where the CPU of ECU 701 determines whether the user has turned off the power to the moving object or performed an operation to end the flow of Fig. 15. If the answer is Yes in step S1519, the process ends. If the answer is No in step S1519, the process returns to step S1511, and the flow of Fig. 15 is repeated.

[0128] In this way, by setting the area around the traffic object as the first pixel area as the region of interest and acquiring an image during the first accumulation period that is shorter than the full frame period, it is possible to acquire an image with a high S / N ratio for moving objects and little blurring due to motion.

[0129] Furthermore, by acquiring such an image, it becomes possible to recognize a traffic object with high accuracy when tracking and photographing the object as a photographing target, etc. Also, when there is a high risk of collision between the moving body 700 and the traffic object, a warning signal is generated, a warning is given to the driver through the display unit 703 or a voice, and an automatic braking system, etc., is activated as necessary, thereby realizing a collision prevention system.

[0130] On the other hand, it is sufficient to capture areas other than the surrounding area of ​​the traffic object as a background image, and in many cases, images with a high S / N ratio and little blur are not required. Therefore, by setting those areas as non-attention areas and acquiring images with a second accumulation time longer than the first accumulation period during the full frame period, it is possible to reduce the amount of data in the captured image.

[0131] In this way, in the third embodiment, a traffic object is recognized based on an image captured and generated by the photoelectric conversion device 600, a first pixel region and a second pixel region are set based on the recognition result, and each region is captured with a different accumulation period. This makes it possible to recognize an image faster than a full frame period, achieve a high S / N ratio, reduce pixel blurring due to motion, and reduce the amount of data and power consumption.

[0132] In addition, in this embodiment, the recognition of the traffic participating object has been described based on the image captured and generated by the photoelectric conversion device 600, but it is not limited to the image, and a distance measuring device such as a millimeter wave radar or LiDAR may be used. By using a millimeter wave radar or LiDAR, the moving direction and moving speed of the traffic participating object can be obtained more accurately, and the surrounding area of ​​the traffic participating object can be set more accurately.

[0133] <Embodiment 4> Next, another example of a method for setting an attention region by dividing it in the column direction will be described in embodiment 4. Embodiment 4 is a modified example of embodiment 2. Fig. 16 is a diagram for explaining an example of setting an attention region in embodiment 4, and Fig. 16 is a diagram for explaining an example of setting an attention region in a different range from Fig. 12.

[0134] 16, an object located about 17 m away is photographed at a position 782 pixels from the bottom, and in the fourth embodiment, the range of 782 pixels or less from the bottom is set as a first pixel region as a region of interest, and the range of 783 pixels or more from the bottom is set as a second pixel region as a region of non-interest.

[0135] This is because if the moving body 700 is close to other vehicles, pedestrians, or other traffic participants, there is a high risk of collision if the vehicle in front suddenly stops or a pedestrian suddenly jumps out in front of the moving body 700.

[0136] Therefore, by setting the short distance range (range of 782 pixels or less from the bottom) as the first pixel area as the region of interest and acquiring an image with a first accumulation period shorter than the full frame period, traffic participants (traffic participants) can be recognized earlier, making it possible to recognize collision risks earlier.

[0137] On the other hand, in the second pixel region, which is the long distance range, there is a sufficient distance and time until a collision even if a vehicle ahead suddenly stops or a pedestrian suddenly jumps out, etc. Therefore, even if an image with an accumulation period equivalent to a full frame period is acquired, it is possible to sufficiently avoid a collision.

[0138] Therefore, in the second pixel region, by not acquiring an image with a short accumulation period, it is possible to reduce the amount of data, and also possible to suppress power consumption of the sensor unit 100 or the photoelectric conversion device 600.

[0139] That is, the first pixel area is photographed using an accumulation period that divides one full frame having a length of 33.3 ms into four, and the second pixel area is photographed using the same accumulation period as the first pixel area, without dividing one full frame having a length of 33.3 ms.

[0140] If the moving body 700 moves in the M-row direction at a speed of 30 km / h, and a traffic participant (traffic participant target) with a high collision risk is recognized at a location approximately 15 m away from the photoelectric conversion device 600, the braking distance of the car will require approximately 14 m to stop.

[0141] Furthermore, in order for the recognition unit 604 to determine the collision risk from an image, it is necessary to evaluate the collision risk based on an object detection algorithm that identifies the position and area of ​​an object in the image, and the position, speed, vehicle movement, etc. of the detected object.

[0142] In this embodiment, let us consider a case where a judgment is made based on 10 images. If one full frame with a length of 33.3 ms is not divided and is shot with the accumulation period as it is, it will take 333 ms to acquire 10 images, and even if traveling at a speed of 30 km / h, the vehicle will travel about 3 m.

[0143] Therefore, taking into account the braking distance and the distance required for recognition, the distance required from capture to stopping is approximately 17 m. On the other hand, if one full frame of 33.3 ms is captured with an accumulation period divided into four, it will take 83 ms to capture 10 images, and even if traveling at 30 km / h, the vehicle will move about 1 meter.

[0144] Therefore, taking into account the braking distance and the distance required for recognition, a distance of approximately 15 m is required from shooting to stopping, which makes it possible to reduce the distance to stop by 2 m compared to when the full frame is not divided.

[0145] In other words, if the accumulation period is divided into four parts of one full frame, shooting every 8.33 ms will enable the vehicle to stop faster than if it were not divided, thereby reducing the risk of a collision.

[0146] On the other hand, if a traffic participant (traffic participant target) is located at a position more than approximately 17 m away from the photoelectric conversion device 600, if the moving object moves in the M-row direction at a speed of 30 km / h, it will be possible to stop the moving object by capturing an image with an accumulation time of 33.3 ms without dividing one full frame.

[0147] Therefore, for areas farther than this distance, even if one full frame is divided and photographed, the collision risk will not be significantly reduced and the amount of data will increase. In other words, at a long distance from the photoelectric conversion device 600, capturing many images with different accumulation periods is likely to result in an increase in unnecessary data.

[0148] In this manner, also in the fourth embodiment, the first pixel region and the second pixel region are appropriately set and images are captured with different accumulation periods for each region based on the device information of the photoelectric conversion device 600 and the installation information for the moving body 700. Therefore, it is possible to reduce the amount of data while reducing the collision risk, and further to reduce the power consumption of the photoelectric conversion device 600.

[0149] <Embodiment 5> Next, an example of setting an attention area by dividing it in the row direction of Fig. 9 will be described with reference to the fifth embodiment. Fig. 17 is a diagram for explaining an example of setting an attention area in the fifth embodiment. Note that the fifth embodiment shows a specific example of division setting in the second embodiment.

[0150] In the fifth embodiment, the range of 400 pixels or less from the left and the range of 400 pixels or less from the right in Fig. 6 are set as the first pixel region. Also, the range of 401 pixels to 1519 pixels from the left is set as the second pixel region. In terms of the angle of view, the first pixel region is a range of about 25 degrees, and the second pixel region is a range of about 70 degrees.

[0151] In addition, the first pixel area is photographed using an accumulation period in which one full frame having a length of 33.3 ms is divided into four, and the second pixel area is photographed using the same accumulation period as the first pixel area, without dividing one full frame having a length of 33.3 ms.

[0152] In order to determine the risk of collision with a traffic participant (traffic participant target) approaching from the side or jumping out, it is necessary to evaluate the collision risk using an object detection algorithm that identifies the position and area of ​​the object in the image in the recognition unit 604, and the position, speed, vehicle movement, etc. of the detected object.

[0153] In this embodiment, let us assume that a judgment is made based on 10 images. If one full frame with a length of 33.3 ms is not divided and is captured using the same accumulation period, it will take 333 ms to capture 10 images.

[0154] On the other hand, if one full frame of 33.3 ms is shot with an accumulation period divided into four, it will take 83 ms to capture 10 images, resulting in a time difference of 250 ms.

[0155] If a person suddenly appears from the side, 250 ms is the time it takes for the person to move 30 to 80 cm, a distance that can reduce the risk of a collision. Also, if a moving object is moving at 30 km / h, it takes the person to move about 2 m, and at 60 km / h, it takes the person to move about 4 m, which is also a distance that can sufficiently reduce the risk of a collision.

[0156] On the other hand, in the second pixel area, which is the range in the center of the shooting area, the traveling direction of the moving body 700 is shot, and in this area, the risk of unexpected events such as traffic participants (traffic participants) jumping out is low, and if the inter-vehicle distance is maintained, the risk of collision is low. Therefore, even if an image with an accumulation period shorter than the full frame period is acquired, the amount of unnecessary data increases. Therefore, in the second pixel area, by not acquiring an image with a short accumulation period, it is effective in reducing the amount of data and also makes it possible to suppress the power consumption of the photoelectric conversion device 600.

[0157] In this manner, in this embodiment, the first pixel region and the second pixel region are set based on the device information and installation information of the photoelectric conversion device 600, and imaging is performed with different accumulation periods for each region. This reduces the situation where, when the subject moves, it goes outside the imaging region and cannot be captured.

[0158] In addition, when recognizing a subject that has entered the shooting area from outside the shooting area and performing tracking shooting of the subject, the subject that has entered from the left or right edge can be quickly photographed, recognized, and recorded, making it possible to reduce the amount of data and power consumption.

[0159] <Embodiment 6> Next, in the sixth embodiment, for example, an example will be described in which a first pixel region, which is a region of interest, and a second pixel region, which is a region of non-interest, are changed in response to a change in the shooting region captured by the optical system 601 of the photoelectric conversion device 600.

[0160] Specifically, the change of the photographing area here refers to a change of the photographing area by shifting the photographing area in the optical system 601, enlargement or reduction by zooming, etc. In addition, for example, by changing the installation angle of the photoelectric conversion device 600, the relationship of the distance between the photoelectric conversion device 600 and the pixel position in the sensor unit 100 changes.

[0161] Alternatively, the range captured by the photoelectric conversion device 600 may be changed by re-installing the photoelectric conversion device 600, or by providing a movable part at the attachment part of the photoelectric conversion device 600 and moving the movable part. The installation angle may be manually set in the control unit 605, or may be detected by attaching a gyro sensor unit or the like that detects an angle to the detection unit 602 of the photoelectric conversion device 600.

[0162] For example, when the photoelectric conversion device 600 is tilted downward, an image at a closer distance is captured at the same pixel position of the sensor unit 100. Therefore, when tilting downward, it is desirable to set the first pixel region, which is the region of interest, wider than before tilting downward.

[0163] Conversely, when the photoelectric conversion device 600 is tilted upward, an image at a longer distance is captured at the same pixel position of the sensor unit 100. Therefore, when tilting upward, it is desirable to set the first pixel region, which is the region of interest, narrower than before tilting upward.

[0164] As described above, in the photoelectric conversion device 600 of embodiment 6, the first pixel region, which is the region of interest, and the second pixel region, which is the region of non-interest, are changed depending on the installation angle of the photoelectric conversion device 600, thereby setting the close distance range as the first pixel region as the region of interest. This allows the subject to be captured within the capture area by acquiring an image with an accumulation period shorter than the full frame period, making it possible to suppress image blurring of the subject.Furthermore, it is also possible to reduce the risk of collision between a moving object and a nearby traffic participant (traffic participant target).

[0165] On the other hand, in the second pixel region, which is a long distance range, even if the subject moves, the subject can be photographed within the photographing region, and image blurring of the subject is unlikely to occur. Furthermore, since the risk of collision with a long distance traffic participant (traffic participant target) is low even in a moving body, not acquiring an image with a short accumulation period in the second pixel region is effective in reducing the amount of data, and also makes it possible to suppress the power consumption of the photoelectric conversion device 600.

[0166] Furthermore, as an optical characteristic of the optical system 601, the first pixel region, which is the region of interest, and the second pixel region, which is the region of non-interest, may be changed in accordance with a change in the relationship between the pixel position and the distance to the photoelectric conversion device 600 due to the zoom function.

[0167] Alternatively, the first pixel region, which is the region of interest, and the second pixel region, which is the region of non-interest, may be changed depending on a change in the relationship between the pixel position and the distance to the photoelectric conversion device 600 due to the tilt function of the optical system 601.

[0168] These area change parameters due to changes in the installation angle of the photoelectric conversion device 600 and optical characteristics are recorded in a memory in the control unit 605. Then, it becomes possible to change the first pixel area, which is a region of interest, and the second pixel area, which is a region of non-interest, according to these parameters.

[0169] In this manner, in the sixth embodiment, the first pixel region, which is the region of interest, and the second pixel region, which is the region of non-interest, are changed in accordance with a change in the imaging region of the photoelectric conversion device 600. This makes it possible to reduce the amount of data and the power consumption of the photoelectric conversion device 600 while suppressing subject blurring of a moving subject at a close distance and reducing the risk of collision between a moving body and a traffic participant (traffic participant target) at a close distance.

[0170] Note that changing the photographing area includes tilting the photoelectric conversion device 600 to the left or right. When the photoelectric conversion device 600 is tilted to the right, an image on the right side is captured at the same pixel position of the sensor unit 100 compared to before the tilt. Therefore, compared to before the tilt to the right, the area on the right side becomes wider and the area on the left side becomes narrower.

[0171] Conversely, when the photoelectric conversion device 600 is tilted leftward, an image on the left side is captured more than before tilting at the same pixel position of the sensor unit 100. Therefore, compared to before tilting leftward, the area on the left side becomes wider and the area on the right side becomes narrower.

[0172] The photoelectric conversion device 600 is tilted to the left or right, for example, in the case of a surveillance camera, when the camera is tilted left and right to take pictures, or in the case of an in-vehicle camera, when the side is monitored in order to better observe the direction of travel when turning right or left.

[0173] In this way, it is desirable to change the attention area according to the left / right tilt of the photoelectric conversion device 600 and the traveling direction of the moving object. For example, in the sixth embodiment, when the photoelectric conversion device 600 is tilted to the right, the attention area on the right side is set larger than the attention area on the left side, and when the photoelectric conversion device 600 is tilted to the left, the attention area on the left side is set larger than the attention area on the right side. Also, in the case of a moving object, it is desirable to set the area in the traveling direction and its surrounding area as a large attention area.

[0174] As described above, in the sixth embodiment, the first pixel region, which is the region of interest, and the second pixel region, which is the region of non-interest, are changed depending on the installation conditions of the photoelectric conversion device 600. As a result, the first pixel region is set as the region of interest, and an image is acquired with an accumulation period shorter than a full frame period.

[0175] This allows the subject to be photographed within the field of view, suppressing blurring of the subject image, and further reduces the risk of collision with a nearby traffic participant (traffic participant target) or a traffic participant rushing out from the side of the moving body.

[0176] On the other hand, in the second pixel region, even if the subject moves, the subject can be captured within the angle of view and the image of the subject is less likely to be blurred. Furthermore, even in the case of a moving body, the risk of collision with a distant traffic participant (traffic participant target) or a traffic participant jumping out from the side is low, so even if an image with an accumulation period shorter than the full frame period is acquired, the amount of unnecessary data increases.

[0177] Therefore, in the second pixel region, acquiring an image for a second accumulation period longer than the first accumulation period is effective in reducing the amount of data and also makes it possible to suppress power consumption of the photoelectric conversion device 600.

[0178] In addition to the inclination of the photoelectric conversion device 600, the first pixel area, which is the region of interest, and the second pixel area, which is the region of non-interest, may be changed according to a change in the relationship between the pixel position and the shooting range due to a zoom function as an optical characteristic of the optical system 601.

[0179] Alternatively, the first pixel area, which is the region of interest, and the second pixel area, which is the region of non-interest, may be changed in accordance with a change in the relationship between the pixel position and the shooting range in the photoelectric conversion device 600 due to the tilt function of the optical system 601.

[0180] In addition, the installation angle of these photoelectric conversion devices 600 and area change parameters due to changes in optical characteristics may be recorded in a memory in the control unit 605, and the first pixel area, which is the area of ​​interest, and the second pixel area, which is the area of ​​non-interest, may be changed according to these parameters.

[0181] <Embodiment 7> In the seventh embodiment, a first pixel region that is a region of interest and a second pixel region that is a region of non-interest are set according to distance information to a subject in a shooting region captured by an optical system 601 of a photoelectric conversion device 600, for example.

[0182] This distance information may be obtained from a distance measuring device such as a millimeter wave radar or a LiDAR (Light Detection and Ranging) device. Alternatively, distance information obtained from an image obtained by the photoelectric conversion device 600 may be used.

[0183] Alternatively, distance information obtained by stereo distance measurement from the photoelectric conversion device 600 and another photoelectric conversion device or the like, or by stereo distance measurement using a stereo camera or the like other than the photoelectric conversion device 600 may be used.

[0184] This distance measurement information is sent to the control unit 605 via the detection unit 602 and the communication unit 607. Then, using this distance information, the control unit 605 sets a first pixel area as a region of interest and a second pixel area as a region of non-interest in the column direction, for example.

[0185] In this embodiment, an example will be described in which distance information is obtained by using LiDAR to acquire distance measurement data within the angle of view. Distance measurement data acquired by LiDAR can be obtained with high accuracy even for a subject having height.

[0186] Fig. 18 is a diagram for explaining an example of setting an attention area in embodiment 7. As shown in Fig. 18, when a person is standing at a distance of, for example, 17 m, it is desirable to recognize the entire person as a subject of interest.

[0187] However, as shown in Fig. 18, in the case of a recognition-based distance measurement method that calculates distance based on pixel positions in the column direction of the sensor unit 100, an error occurs when calculating the accurate distance to a tall person or object. That is, for example, the upper body of a person standing at a distance of 17 m may be recognized as being at a distance exceeding 17 m.

[0188] However, by using LiDAR to measure distances, such recognition errors can be significantly reduced. Therefore, by setting a first pixel region as a region of interest and a second pixel region as a region of non-interest based on distance measurement information, the accuracy of region setting can be significantly improved.

[0189] In this way, in the seventh embodiment, an attention area and a non-attention area are set according to distance measurement information acquired from the LiDAR, and an image is acquired in the attention area with an accumulation period shorter than a full frame period. This makes it possible to capture an image of the subject within the angle of view, and to suppress image blurring of the subject. This is also effective in reducing the risk of collision between the moving body 700 and a nearby subject.

[0190] Similarly, when the attention area is set in the column direction, it is possible to set the attention area according to the distance from the photoelectric conversion device 600, the distance from the travel lane to the sidewalk, etc.

[0191] In addition, by setting an area of ​​interest and an area of ​​non-interest according to distance measurement information and acquiring an image in the area of ​​interest with an accumulation period shorter than a full frame period, it is possible to reduce the situation in which a subject moves outside the angle of view and cannot be photographed.

[0192] In addition, when a subject captured within the angle of view is recognized and a tracking photograph is performed as the target, it becomes possible to quickly photograph and recognize a subject entering from the left or right edge, and it becomes possible to properly photograph the target more quickly. In addition, it is possible to reduce the risk of collision between the moving body 700 and a traffic participant (traffic participant target) or the like that suddenly appears.

[0193] <Embodiment 8> In embodiment 8, when the photoelectric conversion device 600 is installed on a moving body 700, a first pixel region which is a region of interest and a second pixel region which is a region of non-interest are changed depending on information about the moving body 700 and environmental information about the moving body 700.

[0194] As described above, the information on the moving body includes at least one of vehicle speed information, acceleration information, steering operation information, brake information, engine information, etc. As described above, the environmental information includes at least one of information on traffic participants (traffic participants) around the moving body, map information, GPS information, road conditions, road surface conditions, weather information, surrounding brightness, time, etc.

[0195] In the eighth embodiment, an example will be described in which the speed of the moving body 700 increases. If the speed of the moving body 700 increases, the distance range in which the collision risk increases increases. Therefore, in the eighth embodiment, the first pixel region, which is the region of interest, is set to a wider range as the speed of the moving body 700 increases.

[0196] For example, the speed of the moving object 700 is received by the control unit 605 via the ECU 701. The control unit 605 uses a computer program to calculate and set an appropriate attention area according to the speed of the moving object 700.

[0197] Let us assume that the speed at this time is 60 km / h. The braking distance at 60 km / h is approximately 44 m. Therefore, it is preferable to set the attention area within a range of approximately 44 m from the moving object. Fig. 19 is a diagram for explaining an example of setting the attention area in the eighth embodiment. In Fig. 19, the range of approximately 44 m from the moving object corresponds to the 793rd pixel from the bottom.

[0198] In embodiment 8, for example, when the speed is 60 km / h, this range is set as the area of ​​interest, and this area of ​​interest is photographed and recognized during the first accumulation period, thereby making it possible to reduce the risk of collision between traffic participants (traffic participation targets) and the moving body 700.

[0199] It is more preferable to set the attention area according to not only the speed of the moving body 700 but also the braking distance of the moving body 700. This is because, as the braking distance becomes longer, the distance until the moving body 700 stops also becomes longer, and the distance at which the collision risk increases becomes longer, and it becomes necessary to widen the range for guarding against vehicles rushing out from the side.

[0200] The braking distance varies depending not only on the speed of the moving body 700 but also on the inclination angle of the road surface, the condition of the road surface (asphalt, dirt, gravel, rain, snow, ice, etc.), the degree of tire wear (characteristics of the tire itself, distance used, number of years used, etc.), and the weight of the moving body 700. Therefore, it is desirable to make a judgment by comprehensively using such information.

[0201] A first pixel region that is an attention region and a second pixel region that is a non-attention region may be set according to the information on the braking distance. That is, the information on the braking distance is sent to the control unit 605 via the ECU 701, and the control unit 605 uses a computer program to calculate and set an appropriate attention region according to the information on the braking distance of the moving body 700.

[0202] In addition, map information may be used to obtain the width of lanes, the distance from sidewalks, the location of intersections, etc., and a first pixel area that is an area of ​​interest and a second pixel area that is an area of ​​non-interest may be set based on this information.

[0203] Map information is also collected in the control unit 605 via the ECU 701 and calculated in a computer program, making it possible to set a first pixel region that is a region of interest and a second pixel region that is a region of non-interest.

[0204] <Embodiment 9> Next, Fig. 20 is a flowchart showing details of an example of driving the sensor unit 100 regarding a method for setting an attention area according to the ninth embodiment, and Fig. 21 is a flowchart continuing from Fig. 20. Note that the CPU as a computer in the control unit 605 executes a computer program stored in the memory, thereby sequentially performing the operations of the steps in the flowcharts of Figs. 20 and 21.

[0205] In step S2001, the CPU of the control unit 605 of the photoelectric conversion device 600 functions as a setting unit that sets a region of interest and a region of non-interest within the shooting area based on characteristic information regarding the characteristics of the photoelectric conversion device, installation information of the photoelectric conversion device, and environmental information of the photoelectric conversion device.

[0206] In addition, the control unit 605 may set the first pixel region or the second pixel region based on at least one of characteristic information regarding the characteristics of the photoelectric conversion device, installation information regarding the installation state of the photoelectric conversion device, and environmental information regarding the surrounding environment of the photoelectric conversion device.

[0207] Furthermore, at least one of the position, size, and accumulation period of the first pixel region or the second pixel region may be set based on at least one of the above-mentioned characteristic information, installation information, and environment information. Also, the first pixel region or the second pixel region may be set according to distance information of the subject.

[0208] The characteristic information includes at least one of the optical characteristic information of the optical system, the resolution of the sensor unit, and the pixel size of the sensor unit. The installation information includes at least one of the height, angle, and direction at which the photoelectric conversion device is installed. Information on the moving body as one piece of environmental information includes at least one of vehicle speed information, acceleration information, steering angle information, brake information, engine information, and the like.

[0209] In addition, the environmental information may include at least one of information such as the presence, location, speed, acceleration, and distance of traffic participants (pedestrians, motorbikes, bicycles, etc. around the moving object), map information, GPS information, road conditions, road surface conditions, weather information, surrounding brightness, and time.

[0210] Also, the control unit may set the first pixel area or the second pixel area based on the output of the ECU as a movement control unit. At least one of the positions and sizes of the first pixel area and the second pixel area may be set by the user.

[0211] In this embodiment, a case where there is one region of interest and one region of non-interest will be described. For the sake of explanation, the accumulation period of the first pixel region, which is the region of interest, is divided into four parts of one full frame period, and the accumulation period of the second pixel region, which is the region of non-interest, is not divided and is one full frame period.

[0212] Here, the first pixel area starts from the pixel in row j1, column k1 and extends to row N1, column M1, and imaging is performed over four different accumulation periods, whereas the second pixel area starts from the pixel in row j2, column k2 and extends to row N2, column M2, and imaging is performed over one accumulation period.

[0213] Next, in step S2002, the CPU of the control unit judges whether or not the area is a region of interest. If it is judged as Yes, the process proceeds to step S2011 to perform a process of determining an accumulation period. On the other hand, if it is judged as No in step S2002, the process proceeds to step S2101.

[0214] Since the region of interest is divided into four, i starts at 1 and repeats up to 4. Therefore, in step S2011, the CPU of the control unit first sets the value of i to 1. Next, in step S2012, the CPU of the control unit outputs the count value Count of the counter 211 at time Ti to the memory 212. At this time, output is performed to all memories simultaneously. This operation corresponds to the operation at time T1 in FIG. 7.

[0215] Next, in step S2013, the CPU of the control unit sets j = j1. j1 is the starting position of the Nth row of the target area. Next, in step S2014, the CPU of the control unit sets k = k1. k1 is the starting position of the Mth column of the target area.

[0216] In step S2015, the CPU of the control unit outputs the count value Count(j - k - i) in the memory j - k of FIG. 9 to the buffer. At this time, it outputs to the buffer simultaneously for columns 1 to M. This operation means the operation of taking in the count value of the first row in FIG. 9 into the buffer.

[0217] In step S2016, the CPU of the control unit outputs the count value Count(j - k - i) of buffer k to the output circuit 114. This operation corresponds to the operation of reading out the signal of the buffer in the leftmost column in FIG. 9 from the output circuit.

[0218] Next, it proceeds to step S2017 in FIG. 21 via E. In step S2017, the CPU of the control unit determines whether k < M1. If Yes, in step S2018, k is incremented by 1 as k = k + 1, and it returns to step S2016 via F. M1 is the ending position of the Mth column of the target area. This operation corresponds to the operation of reading out the signal of the buffer in the second column from the left in FIG. 9 from the output circuit.

[0219] If it is determined as No in step S2017, that is, when k = M1, it means that the signal of the buffer in the M1th column in FIG. 9 has been read out from the output circuit. Next, it proceeds to step S2019, and the CPU of the control unit determines whether j < N1. N1 is the ending position of the Nth row of the target area. If it is determined as Yes in step S2019, then in step S2020, the CPU of the control unit sets j = j + 1, increments j by 1, and returns to step S2014 via G. This corresponds to the operation for starting the reading of the next row.

[0220] If the result of the determination in step S2019 is No, this means that reading of all rows has been completed, so the process proceeds to step S2021, where the control unit CPU determines whether j<4. If the result of the determination in step S2021 is Yes, the control unit CPU increments i by 1 by setting i=i+1, and returns to step S2012 via H. This operation corresponds to the operation of starting reading at the next time T2.

[0221] If the result of the determination in step S2021 is No, this means that the readout at time T4 is completed, so the process proceeds to step S2023, where the CPU of the control unit resets the counter 211 with a reset signal. This operation corresponds to the reset operation of the counter 211 at time T4 in Fig. 7. After step S2023, the process returns to step S2001 via M. In this manner, the signals accumulated in the sensor unit can be read out sequentially.

[0222] On the other hand, since one full frame period is not divided for non-attention areas, only the flow when i is 4 is performed. Therefore, in step S2101, the CPU of the control unit sets i=4. Next, in step S2102, the CPU of the control unit outputs the count value Count of the counter 211 at time Ti to the memory 212. At this time, output is performed simultaneously to all memories. This operation corresponds to the operation at time T1 in FIG. 7.

[0223] Next, in step S2103, the CPU of the control unit sets j=j2, where j2 is the start position of row N of the non-target region. Next, in step S2104, the CPU of the control unit sets k=k2, where k2 is the start position of column M of the non-target region.

[0224] In step S2105, the CPU of the control unit outputs the count value Count(jki) in the memory jk in Fig. 9 to the buffer. At this time, columns 1 to M are output to the buffer at the same time. This operation means that the count value of the first row in Fig. 9 is taken into the buffer.

[0225] In step S2106, the CPU of the control unit outputs the count value Count(j - k - i) of buffer k to output circuit 114. This operation corresponds to the operation of reading the signal of the buffer in the leftmost column of FIG. 9 from the output circuit.

[0226] Next, proceed to step S2107 in FIG. 21 via I. In step S2107, the CPU of the control unit determines whether k < M2. If Yes in step S2107, then in step S2108, the CPU of the control unit increments k by 1 as k = k + 1 and returns to step S2106 via J. M2 is the end position of the Mth column of the non - attention area. This operation corresponds to the operation of reading the signal of the buffer in the second column from the left in FIG. 9 from the output circuit.

[0227] When it becomes No in step S2107, that is, when k = M2, it means that the signal of the buffer in the M2th column of FIG. 9 has been read out from the output circuit. Next, proceed to step S2109, and the CPU of the control unit determines whether j < N2.

[0228] N2 is the end position of the Nth row of the non - attention area. If Yes in step S2109, then in step S2110, the CPU of the control unit sets j = J + 1, increments j by 1, and returns to step S2104 via K. This corresponds to the operation for starting the reading of the next row. When it is determined as No in step S2109, it means that the reading of all rows has been completed, so proceed to step S2111, and the CPU of the control unit determines whether j < 4.

[0229] When it is determined as Yes in step S2111, the CPU of the control unit increments i by 1 as i = i + 1 and returns to step S2102 via L. This operation corresponds to the operation of starting the reading at the next time T2. However, since i = 4 is set in step S2101, it becomes No in step S2111.

[0230] If the determination in step S2111 is No, this means that the readout at time T4 is completed, so the process proceeds to step S2113, where the CPU of the control unit resets counter 211 with a reset signal. This operation corresponds to the reset operation of counter 211 at time T4 in Fig. 7. After step S2113, the process returns to step S2001 via M. In this manner, the signals accumulated in the sensor unit can be read out sequentially.

[0231] Therefore, the region of interest starts from the pixel in row j1, column k1 and repeats through row N1, column M1, and is imaged over four different accumulation periods, whereas the region of non-interest starts from the pixel in row j2, column k2 and repeats through row N2, column M2, and is imaged over only one accumulation period.

[0232] Next, a description will be given of the processing of images acquired in the region of interest and the region of non-interest in the photoelectric conversion device 600. The method of acquiring an image of the region of interest is based on the method shown in Fig. 20 and Fig. 21. Also, the region of interest acquires an image in four accumulation periods of frames 1_1, 1_2, 1_3, and 1_4 in Fig. 7. On the other hand, the region of non-interest acquires an image in the accumulation period of only frame 1_4 in Fig. 7.

[0233] An example of an image created according to the flow of Fig. 20 and Fig. 21 is shown in Fig. 22. Fig. 22 is a diagram showing an example of an image for each of a plurality of accumulation periods, with an attention area being set. In each diagram, the range on the lower side capturing a close distance is the attention area, and the range on the upper side capturing a long distance is the non-attention area, and the car on the left side is stopped, and the car on the right side is moving from right to left at a certain speed.

[0234] As shown in Fig. 22, the area of ​​interest at the bottom of the screen is updated in four stages from frame 1_1 to 1_4. Therefore, a stopped car's image becomes gradually brighter as the accumulation period gets longer from frame 1_1 to 1_4. Also, a moving car's image becomes blurred in the forward and backward directions of its travel as the accumulation period gets longer from frame 1_1 to 1_4, and the brightness of the overlapping parts of the movement gets higher.

[0235] On the other hand, the non-attention area, which is the upper part of the screen, is updated only by the output of frame 1_4. Therefore, a stopped car is updated every full frame and photographed during the accumulation period of the full frame period, resulting in an image with high brightness. Similarly, a moving car is updated every full frame and photographed during the accumulation period of the full frame period, so blurring occurs in the front and rear of the moving direction, and the brightness of the part where the movements overlap is high.

[0236] Therefore, in the non-attention area, the images of frames 1_1 to 1_3 are all images captured during the accumulation period of frame 1_4 in the previous full frame period. Also, frame 1_4 is updated to an image captured during the accumulation period of frame 1_4 in the current full frame period.

[0237] 20 and 21, the area of ​​interest is updated four times in a full frame period, making it possible to acquire an image in a shorter time than a full frame period and perform faster recognition. On the other hand, the area of ​​non-interest is updated only once in a full frame period, making it possible to reduce the amount of data and power consumption.

[0238] Based on the image data created according to the flowcharts of FIGS. 20 and 21, subsequent processing is performed according to the characteristics and applications of the photoelectric conversion device 600.

[0239] For example, in the case of a security camera, the control unit 605 monitors, photographs, and records the behavior of a target person. In some cases, it is possible to display a warning, turn on a light installed in the security camera, or emit a warning sound via the communication unit 607.

[0240] In the case of an in-vehicle camera, the control unit 605 can issue a warning or display a warning via the ECU, and the vehicle control unit 702 can activate safety devices to slow down or stop the vehicle, making it possible to reduce or avoid a collision.

[0241] In addition, for example, by using the camera to recognize people or registered faces, it is possible to capture images with an appropriate accumulation time when the person is within the shooting area. Also, as a pet camera, it is possible to recognize pets, remotely check on their condition, and take pictures of them.

[0242] In addition, when used as a surveillance camera, in places where safety management and surveillance are required (e.g. parking lots, public facilities, factories, etc.), specific locations or areas can be set as areas of interest in order to grasp the situation, monitor and prevent fraudulent activities, collect evidence as records, etc. This makes it possible to monitor the areas of interest more frequently and record and display video and audio.

[0243] In addition, when used as a defect detection camera in a factory, it can be used to support quality control and efficiency of the manufacturing process, and to detect and remove defective products early, troubleshoot production lines, and improve quality, etc. In that case, it becomes possible to detect and record product defects and abnormalities with greater accuracy on the factory production line or at the work site.

[0244] In addition, when used as a packaging inspection camera in a factory, it will be possible to inspect the product packaging condition, label accuracy, and defects in order to strengthen product quality control and ensure the integrity of the packaging and accuracy of the label.

[0245] When used as a sorting camera for logistics, it can improve the efficiency and accuracy of logistics operations in logistics warehouses and distribution centers, and realize proper product management and a quick delivery process. It can also recognize the shape, size, barcode, etc. of products and packages, enabling accurate sorting and delivery processing.

[0246] When used as an endoscopic camera, the camera can be incorporated into endoscopes used in the medical field to obtain high-quality images of the inside of the body and organs during endoscopic surgery and diagnosis, allowing for more accurate observation of conditions inside the body and helping to diagnose diseases and develop treatment plans.

[0247] When used as a camera to detect the need for care, it will be possible to monitor the living conditions of elderly people and those requiring care and detect abnormal or dangerous situations in order to provide early detection and support for falls, abnormal behavior, and emergency situations, ensuring the safety and lifestyle support of those in need of care.

[0248] When used as infrastructure surveillance cameras, they can monitor infrastructure such as roads, bridges, railways, and power plants to maintain safety and reliability, and detect abnormalities, damage, and unauthorized activity in order to provide early warning and proper maintenance.

[0249] In addition, when used as infrastructure surveillance cameras, they can monitor public places, facilities, and specific areas to ensure public safety, deter crime, prevent fraud, and collect evidence for criminal investigations, making it possible to prevent, monitor, and record fraudulent and criminal activities.

[0250] In addition, when used as an agricultural camera, it can support appropriate cultivation management, early detection of pests, and effective agricultural measures to improve the productivity and quality of agricultural crops. In that case, it will be possible to monitor the growth status of agricultural crops and the occurrence of pests and diseases, and provide video data that is useful for agricultural production management.

[0251] <Embodiment 10> Next, in the tenth embodiment, a method for setting the accumulation period of the region of interest depending on the brightness of the region of interest, etc. will be described.

[0252] It has been explained that for surveillance cameras and in-vehicle cameras attached to moving objects, it is effective to capture images of close-up subjects or subjects on the side of a moving object with a short accumulation period at an early timing. However, it is known that a short accumulation period increases the ratio of noise in the signal component, deteriorating the so-called S / N ratio.

[0253] This is particularly noticeable when shooting a dark area. Therefore, if the area of ​​interest is dark, even if an image is captured with a short accumulation period, the image will have a poor S / N ratio, making it difficult to display appropriately or use for subsequent processing.

[0254] Therefore, by performing unnecessary processing, appropriate judgment cannot be performed and power consumption of the device increases, which is a problem. Therefore, in the fifth embodiment, a method of capturing an image with a good S / N ratio by adjusting the accumulation period according to the brightness of the region of interest will be described.

[0255] Fig. 23 is a flowchart showing details of an example in which an accumulation period is set and the sensor unit 100 is driven in the tenth embodiment, and Fig. 24 is a flowchart continuing from Fig. 23. Note that the CPU as a computer in the control unit 605 executes a computer program stored in the memory, thereby sequentially performing the operations of the steps in the flowcharts of Figs. 23 and 24.

[0256] First, in step S2301, the CPU of the control unit sets an area of ​​interest based on at least one of product information, installation information, and environmental information of the photoelectric conversion device 600 recorded in the control unit 605 of the photoelectric conversion device 600, as in the previous embodiments.

[0257] Next, in step S2302, the CPU of the control unit judges whether the area is a region of interest or a region of non-interest. If it is a region of interest, the process proceeds to step S2311 in FIG. 23, where the CPU of the control unit performs a process of determining an accumulation period. On the other hand, if it is a region of non-interest, the process proceeds to step S2331.

[0258] Next, in step S2303, the CPU of the control unit determines the accumulation period of the region of interest. In this embodiment, for example, the average luminance value of the image acquired in the previous full frame in the region of interest is calculated by the control unit 605, and a look-up table (LUT) or a calculation formula for the accumulation period corresponding to the luminance value is recorded in the memory of the control unit 605, and is used to determine the accumulation period.

[0259] In this manner, in this embodiment, the length of at least one of the first and second accumulation periods is set based on the luminance information of a signal generated during at least one of the first and second accumulation periods.

[0260] The luminance value may be calculated based on the peak of a histogram, the deviation of the peak, the spread, or the variance of the peak, instead of the average luminance. The luminance difference between adjacent pixels in the screen may be calculated, and the noise amount may be calculated from the histogram of the difference, and the accumulation period may be calculated from the result. Furthermore, the luminance value may be calculated based on the average value of multiple previous full frames, instead of the previous full frame, or may be calculated based on the average luminance or histogram for each accumulation period in the full frame.

[0261] 23 and 24 show flowcharts for the case where there are four accumulation periods in this embodiment, but the number of accumulation periods is not limited to four, and may be one, three, or more than four. Then, in step S23011, the CPU of the control unit sets the determined accumulation period (Ti).

[0262] 23, the CPU of the control unit sets i to 1. Next, in step S2312, the CPU of the control unit outputs the count value Count of the counter 211 at time Ti to the memory 212. At this time, the output is performed simultaneously to all memories. This operation corresponds to the operation at time T1 in FIG.

[0263] Next, in step S2313, the CPU of the control unit sets j = j1. j1 is the starting position of the Nth row of the target area. Next, in step S2314, the CPU of the control unit sets k = k1. k1 is the starting position of the Mth column of the target area.

[0264] In step S2315, the CPU of the control unit outputs the count value Count(j - k - i) in the memory j - k in FIG. 9 to the buffer. At this time, it outputs to the buffer simultaneously for columns 1 to M. This operation means the operation of taking in the count value of the first row in FIG. 9 into the buffer.

[0265] In step S2316, the CPU of the control unit outputs the count value Count(j - k - i) of buffer k to the output circuit 114. This operation corresponds to the operation of reading the signal of the buffer in the leftmost column in FIG. 9 from the output circuit.

[0266] Next, it proceeds to step S2317 in FIG. 24 via E. In step S2317, the CPU of the control unit determines whether k < M1. If Yes, in step S2318, the CPU of the control unit increments k by 1 as k = k + 1, and returns to step S2316 via F. M1 is the ending position of the Mth column of the target area. This operation corresponds to the operation of reading the signal of the buffer in the second column from the left in FIG. 9 from the output circuit.

[0267] When it becomes No in step S2317, that is, when k = M1, it means that the signal of the buffer in the Mth column in FIG. 9 has been read out from the output circuit. Next, it proceeds to step S2319, and the CPU of the control unit determines whether j < N1.

[0268] N1 is the ending position of the Nth row of the target area. If it is Yes in step S2319, then in step S2320, the CPU of the control unit increments j by 1 as j = j + 1, and returns to step S2314 via G. This corresponds to the operation for starting the reading of the next row.

[0269] If the determination in step S2319 is No, this means that reading of all rows has been completed, so the process proceeds to step S2321, where the control unit CPU determines whether i<4. If the determination in step S2321 is Yes, the control unit CPU increments i by 1, setting i=i+1, and returns to step S2312 via H. This operation corresponds to the operation of starting reading at the next time T2.

[0270] If the determination in step S2321 is No, this means that the reading at time T4 is completed, so that the flow proceeds to step S2323, where the CPU of the control unit resets counter 211 with a reset signal.

[0271] This operation corresponds to the reset operation of counter 211 at time T4 in Fig. 7. After step S2113, the process returns to step S2301 via M. In this manner, signals accumulated in the sensor section can be read out sequentially.

[0272] On the other hand, since the accumulation period of the non-attention area is equivalent to the full frame period, an image is acquired in the determined accumulation period. That is, in step S2331 of FIG. 23, the CPU of the control unit sets i=4. Next, in step S2332, the CPU of the control unit outputs the count value Count of the counter 211 at time Ti to the memory 212. At this time, output is performed simultaneously to all memories. This operation corresponds to the operation at time T1 in FIG. 7.

[0273] Next, in step S2333, the CPU of the control unit sets j=j2, where j2 is the start position of row N of the non-target region. Next, in step S2334, the CPU of the control unit sets k=k2, where k2 is the start position of column M of the non-target region.

[0274] In step S2335, the CPU of the control unit outputs the count value Count(jki) in memory jk in Fig. 9 to the buffer. At this time, columns 1 to M are output to the buffer at the same time. This operation means that the count value of the first row in Fig. 9 is taken into the buffer.

[0275] In step S2336, the CPU of the control unit outputs the count value Count(j - k - i) of buffer k to output circuit 114. This operation corresponds to the operation of reading the signal of the buffer in the leftmost column of FIG. 9 from the output circuit.

[0276] Next, it proceeds to step S2337 in FIG. 24 via I. In step S2337, the CPU of the control unit determines whether k < M2. If Yes, in step S2338, the CPU of the control unit increments k by 1 as k = k + 1 and returns to step S2336 via J. M2 is the end position of the M-th column of the non-target area. This operation corresponds to the operation of reading the signal of the buffer in the second column from the left in FIG. 9 from the output circuit.

[0277] If it becomes No in step S2337, that is, if k = M2, it means that the signal of the buffer in the M-th column of FIG. 9 has been read from the output circuit. Next, it proceeds to step S2339, and the CPU of the control unit determines whether j < N2.

[0278] N2 is the end position of the N-th row of the target area. If it is Yes in step S2339, in step S2340, the CPU of the control unit sets j = J + 1, increments j by 1, and returns to step S2344 via K. This corresponds to the operation for starting the reading of the next row.

[0279] If it is determined to be No in step S2339, it means that the reading of all rows has been completed. So, it proceeds to step S2341, and the CPU of the control unit determines whether i < 4. If it is determined to be Yes in step S2341, the CPU of the control unit sets i = i + 1, increments i by 1, and returns to step S2332 via L. This operation corresponds to the operation of starting the reading at the next time T2. However, since i = 4 is set in step S2331, it becomes No in step S2341.

[0280] If the determination in step S2341 is No, this means that reading at time T4 is completed, so the process proceeds to step S2343, where the CPU of the control unit resets counter 211 with a reset signal. This operation corresponds to the reset operation of counter 211 at time T4 in Fig. 7. After step S2343, the process returns to step S2301 via M.

[0281] In this manner, signals accumulated in the sensor unit can be read out sequentially. Images of non-interest areas may also be acquired in the same manner for an accumulation period determined by the control unit 605. In this case, a flow for setting the accumulation period before acquiring an image is added.

[0282] For example, in the above embodiment, accumulation is performed for a minimum of 1 / 4 full frame period, but the length of the minimum accumulation period may be changed to, for example, 1 / 5 full frame period or 1 / 3 full frame period depending on the recognition accuracy in the recognition unit 604. Alternatively, the length of the minimum accumulation period may be changed depending on the brightness of the subject. Furthermore, the recognition unit may further recognize the subject based on a signal generated in a second accumulation period.

[0283] Furthermore, even if the readout cycle is set to every 1 / 4 full frame period, the counter may be reset midway through the accumulation period of frame 1_1 in Fig. 7 depending on the brightness of the subject, the image recognition accuracy, etc. As a result, the substantial accumulation period may be made shorter than the 1 / 4 full frame period.

[0284] Alternatively, the counter may be reset once at time T1 in Fig. 7. This may adjust the count value read out at time T4. Note that, although the accumulation periods are uniformly spaced in the above embodiment, the present invention is not limited to this.

[0285] In the above embodiments, various processes are performed on the basis of captured image data using various cameras, devices attached to the cameras, personal computers, and the like.

[0286] For example, in the case of a security camera, the control unit 605 monitors, photographs, and records the behavior of a target person. In some cases, the control unit 605 displays a warning, illuminates a light installed in the security camera, and generates a warning sound via the communication unit 607. In the case of an in-vehicle camera, the control unit 605 issues a warning or displays a warning via the ECU 701, and activates a safety device to slow down or stop the vehicle via the vehicle control unit 702.

[0287] <Embodiment 11> In the eleventh embodiment, before various processes are performed, the control unit 605 determines whether the image acquired by the photoelectric conversion device 600 is suitable for the subsequent processes. This further reduces the processing load, and can realize a reduction in the processing load and power consumption, etc. Furthermore, it is possible to avoid erroneous decisions due to inappropriate images, erroneous warnings in security cameras, and erroneous activation of safety devices in vehicle-mounted cameras.

[0288] In the eleventh embodiment, the captured image data is evaluated by the control unit 605. Then, using the average luminance value of each full frame output, it is judged whether the image is appropriate or not, and if it is appropriate, it is judged whether to pass it on to the next process.

[0289] Fig. 25 is a flowchart showing an example of the operation of the control unit after constructing an image according to embodiment 11. Note that the CPU as a computer in the control unit 605 executes a computer program stored in the memory, thereby sequentially performing the operations of the steps in the flowchart in Fig. 25.

[0290] The flow up to acquiring an image uses, for example, the examples shown in Figures 10 and 11. First, in step S2500, the CPU of the control unit composes an image using the count value output in step S106 of Figure 10.

[0291] Then, in step S2501, the CPU of the control unit judges whether the image is suitable for use in the next process. Here, an image that is not suitable refers to an image with a poor S / N ratio and covered in noise, or an image that is too dark to be recognized.

[0292] The determination method in step S2501 can be determined from the average brightness data of the acquired image. That is, the value of the average brightness is calculated by the control unit 605, and it is determined whether or not the value can be used for the next process. That is, the control unit controls whether or not to perform a predetermined process in the control unit according to the brightness information of the signal generated in at least one of the first accumulation period and the second accumulation period.

[0293] Also, it is not limited to the average luminance, but may be obtained from the peak of the histogram, the bias of the peak, the spread, the variance, etc. Also, the luminance difference between adjacent pixels in the screen may be obtained, and the amount of noise may be calculated from the histogram of the difference, and from the result, it may be judged whether or not it can be used for the next process.

[0294] Furthermore, the determination may be based not only on the previous full frame, but also on the average value of multiple previous full frames, or on the average luminance for each accumulation period within the full frame, a histogram, etc. If it is determined in step S2501 that the image is appropriate, then in step S2502, the CPU of the control unit executes processing using the computer program of the control unit 605.

[0295] On the other hand, if it is determined that the image is inappropriate, in step S2503, the CPU of the control unit does not process that image in the computer program of the control unit 605, and stops the processing of the computer program until an image of the next accumulation period is acquired.

[0296] 25, the number of times that unnecessary processing is performed by the control unit 605 can be reduced, and the product power consumption can be reduced. In addition, it is possible to avoid incorrect judgment due to inappropriate images, false warnings in security cameras, and erroneous activation of safety devices in vehicle-mounted cameras.

[0297] <Embodiment 12> In the above embodiment, an example has been described in which the region is divided into N rows and M columns, but the effects of the present invention are not limited to this.

[0298] Fig. 26 is a diagram for explaining an example of setting an attention area in the twelfth embodiment. As shown in Fig. 26, attention areas may be set in a combination of both row and column directions. For example, as shown in gray in Fig. 26, by setting an attention area, it is possible to capture images at close range and to the side with different accumulation periods. This is therefore effective for surveillance by security cameras and collision prevention by vehicle-mounted cameras, and is effective in reducing data volume and power consumption.

[0299] <Embodiment 13> Fig. 27 is a diagram for explaining an example of setting a region of interest in embodiment 13. As shown by regions of interest A and B in Fig. 27, a plurality of regions of interest may be set, and the accumulation period may be different for each region of interest.

[0300] Also, for example, the attention area in the column direction may be changed in accordance with steering wheel operation information. Also, in order to prevent collisions when turning right or left, the attention area may be set to the direction of travel and its surroundings, or surroundings that are likely to become blind spots. Furthermore, as described above, when installed on a moving body, it is effective to flexibly change the attention area in accordance with map information, GPS information, road conditions, etc.

[0301] In the above-described embodiment, the moving body 700 has been described using an example of an automobile, but the moving body may be any moving body, such as an airplane, a train, a ship, a drone, an AGV, or a robot.

[0302] Moreover, the photoelectric conversion device 600 of the above embodiment is not limited to a moving body, and may be attached to a wall, a pillar, etc. In this case, it is desirable to store the installation height, installation angle, etc. in the storage unit 606, etc. as installation information when the photoelectric conversion device 600 is attached to a wall, a pillar, etc.

[0303] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to the above-mentioned embodiments, and various modifications are possible based on the spirit of the present invention, and are not excluded from the scope of the present invention. The present invention also includes appropriate combinations of the above-mentioned embodiments. The present invention also includes the following combinations.

[0304] (Configuration 1) A plurality of pixels each including an optical system, a photoelectric conversion unit that emits a pulse in response to the incidence of a photon, a counter that counts the number of the pulses, and a memory that stores the count value of the counter, and a signal is generated based on the difference between the count values ​​of the counter at the start and end of an accumulation period in the photoelectric conversion unit, the signal has at least a first accumulation period and a second accumulation period within one full frame, the first accumulation period is shorter than the second accumulation period, and the signal generated during the first accumulation period is stored as a signal stored in the memory from the end of the first accumulation period to the end of the second accumulation period. and a setting unit capable of setting, in a sensor unit consisting of the plurality of pixels, a first pixel region having at least the first accumulation period and the second accumulation period, and a second pixel region having at least the second accumulation period, wherein the setting unit sets the first pixel region and the second pixel region based on at least one of characteristic information on characteristics of the photoelectric conversion device, installation information on an installation state of the photoelectric conversion device, and environmental information on the surrounding environment of the photoelectric conversion device.

[0305] (Configuration 2) The photoelectric conversion device according to configuration 1, wherein the first accumulation period and the second accumulation period overlap each other.

[0306] (Configuration 3) The photoelectric conversion device according to configuration 2, wherein the first accumulation period and the second accumulation period start simultaneously.

[0307] (Configuration 4) The photoelectric conversion device according to any one of configurations 1 to 3, wherein the end of the second accumulation period coincides with the end of a full frame.

[0308] (Configuration 5) The photoelectric conversion device according to any one of configurations 1 to 4, further comprising a recognition section that recognizes a subject based on a signal generated during at least the first accumulation period.

[0309] (Configuration 6) The photoelectric conversion device according to configuration 5, wherein the recognition section further recognizes the subject based on a signal generated during the second accumulation period.

[0310] (Configuration 7) The photoelectric conversion device according to any one of configurations 1 to 6, further comprising a display section that displays, as an image, a signal generated during at least the second accumulation period.

[0311] (Configuration 8) The photoelectric conversion device according to any one of configurations 1 to 7, wherein the sensor portion includes an avalanche photodiode.

[0312] (Configuration 9) A photoelectric conversion device described in any one of configurations 1 to 8, characterized in that the setting unit sets at least one of the position, size, and accumulation period of the first pixel region or the second pixel region based on at least one of the characteristic information, the installation information, and the environmental information.

[0313] (Configuration 10) A photoelectric conversion device according to any one of configurations 1 to 9, characterized in that the characteristic information includes at least one of optical characteristic information of the optical system, the resolution of the sensor unit, and the pixel size of the sensor unit.

[0314] (Configuration 11) The photoelectric conversion device according to any one of configurations 1 to 10, wherein the installation information includes at least one of the height, angle, and direction at which the photoelectric conversion device is installed.

[0315] (Configuration 12) The photoelectric conversion device according to any one of configurations 1 to 11, wherein the setting section sets the first pixel region or the second pixel region according to distance information of a subject.

[0316] (Configuration 13) The photoelectric conversion device according to any one of configurations 1 to 12, characterized in that the length of at least one of the first accumulation period and the second accumulation period is set based on luminance information of a signal generated during at least one of the first accumulation period and the second accumulation period.

[0317] (Configuration 14) A photoelectric conversion device described in any one of configurations 1 to 13, characterized in that the control unit controls whether or not to perform a predetermined processing in the control unit depending on luminance information of a signal generated during at least one of the first accumulation period and the second accumulation period.

[0318] (Configuration 15) A moving body comprising the photoelectric conversion device described in any one of configurations 1 to 14 and a movement control unit that controls the operation of the moving body, wherein the control unit sets the first pixel region or the second pixel region based on the output of the movement control unit.

[0319] (Method) A control method for controlling a photoelectric conversion device having a plurality of pixels, each of which includes an optical system, a photoelectric conversion unit that emits a pulse in response to incidence of a photon, a counter that counts the number of the pulses, and a memory that stores the count value of the counter, the method generating a signal based on a difference between the count values ​​of the counter at the start and end of an accumulation period in the photoelectric conversion unit, having at least a first accumulation period and a second accumulation period within one full frame, the first accumulation period being shorter than the second accumulation period, and controlling so that a signal generated during the first accumulation period is output between the end of the first accumulation period and the end of the second accumulation period, and making it possible to set a first pixel region having at least the first accumulation period and the second accumulation period, and a second pixel region having at least the second accumulation period, in a sensor unit consisting of the plurality of pixels, and storing at least one of characteristic information relating to characteristics of the photoelectric conversion device, installation information relating to the installation state of the photoelectric conversion device, and environmental information relating to the surrounding environment of the photoelectric conversion device. The first pixel region and the second pixel region are set based on the A control method comprising:

[0320] (Program) A computer program for controlling each part of the photoelectric conversion device according to any one of configurations 1 to 14 or the moving object according to configuration 15 by a computer.

[0321] A computer program for implementing all or part of the control in this embodiment may be supplied to a photoelectric conversion device or the like via a network or various storage media. A computer (or a CPU, MPU, or the like) in the photoelectric conversion device or the like may then read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0322] 11: Sensor board 12: Pixel area 21: Circuit board 22: Circuit area 100: Sensor section 101: Pixel 102: Photoelectric conversion unit 103: Signal processing circuit 110: Vertical scanning circuit 111: Horizontal scanning circuit 112: Readout circuit 113: Vertical signal line 114: Output circuit 115: Control pulse generation unit 201: Avalanche photodiode 202: Quench element 210: Waveform shaping section 211: Counter 212: Memory 213: Drive line 600: Photoelectric conversion device 601:Optical system 602: Detection unit 603: Signal processing unit 604: Recognition part 605: Control unit 606: Storage section 607: Communications Department

Claims

1. An optical system; A plurality of pixels each including a photoelectric conversion unit that emits a pulse in response to an incidence of a photon, a counter that counts the number of the pulses, and a memory that stores the count value of the counter; a control unit that generates a signal based on a difference between count values ​​of the counter at the start and end of an accumulation period in the photoelectric conversion unit, has at least a first accumulation period and a second accumulation period within one full frame, the first accumulation period is shorter than the second accumulation period, and controls so that the signal generated during the first accumulation period is output between the end of the first accumulation period and the end of the second accumulation period; a setting unit capable of setting a first pixel region having at least the first accumulation period and the second accumulation period, and a second pixel region having at least the second accumulation period, in a sensor unit composed of the plurality of pixels; A photoelectric conversion device characterized in that the setting unit sets the first pixel region and the second pixel region based on at least one of the following information: characteristic information regarding the characteristics of the photoelectric conversion device, installation information regarding the installation state of the photoelectric conversion device, and environmental information regarding the surrounding environment of the photoelectric conversion device.

2. 2. The photoelectric conversion device according to claim 1, wherein the first accumulation period and the second accumulation period overlap each other.

3. 3. The photoelectric conversion device according to claim 2, wherein the first accumulation period and the second accumulation period start simultaneously.

4. 2. The photoelectric conversion device according to claim 1, wherein the end of the second accumulation period coincides with the end of a full frame.

5. 2. The photoelectric conversion device according to claim 1, further comprising a recognition section that recognizes a subject based on a signal generated during at least the first accumulation period.

6. 6. The photoelectric conversion device according to claim 5, wherein the recognition section further recognizes the subject based on a signal generated during the second accumulation period.

7. 2. The photoelectric conversion device according to claim 1, further comprising a display unit that displays, as an image, at least a signal generated during the second accumulation period.

8. 2. The photoelectric conversion device according to claim 1, wherein the sensor portion includes an avalanche photodiode.

9. The photoelectric conversion device according to claim 1, characterized in that the setting unit sets at least one of the position, size, and accumulation period of the first pixel region or the second pixel region based on at least one of the characteristic information, the installation information, and the environmental information.

10. The photoelectric conversion device according to claim 1 , wherein the characteristic information includes at least one of optical characteristic information of the optical system, a resolution of the sensor unit, and a pixel size of the sensor unit.

11. The photoelectric conversion device according to claim 1 , wherein the installation information includes at least one of a height, an angle, and a direction in which the photoelectric conversion device is installed.

12. The photoelectric conversion device according to claim 1 , wherein the setting section sets the first pixel area or the second pixel area in accordance with distance information of a subject.

13. 2. The photoelectric conversion device according to claim 1, wherein the length of at least one of the first accumulation period and the second accumulation period is set based on luminance information of a signal generated during at least one of the first accumulation period and the second accumulation period.

14. The photoelectric conversion device according to claim 1, characterized in that the control unit controls whether or not to perform a predetermined processing in the control unit depending on luminance information of a signal generated during at least one of the first accumulation period and the second accumulation period.

15. The photoelectric conversion device according to any one of claims 1 to 14, A movement control unit that controls the operation of the moving body, The moving body, wherein the control unit sets the first pixel area or the second pixel area based on an output of the movement control unit.

16. A control method for controlling a photoelectric conversion device having a plurality of pixels, each of which includes an optical system, a photoelectric conversion unit that emits a pulse in response to incidence of a photon, a counter that counts the number of the pulses, and a memory that stores a count value of the counter, the method comprising: a signal is generated based on a difference between count values ​​of the counter at the start and end of an accumulation period in the photoelectric conversion unit, at least a first accumulation period and a second accumulation period are provided within one full frame, the first accumulation period is shorter than the second accumulation period, and the signal generated during the first accumulation period is controlled to be output between the end of the first accumulation period and the end of the second accumulation period; In the sensor unit composed of the plurality of pixels, a first pixel region having at least the first accumulation period and the second accumulation period, and a second pixel region having at least the second accumulation period can be set; At least one of characteristic information on characteristics of a photoelectric conversion device, installation information on an installation state of the photoelectric conversion device, and environmental information on a surrounding environment of the photoelectric conversion device. The first pixel region and the second pixel region are set based on the A control method comprising:

17. A computer program for controlling each part of the photoelectric conversion device according to any one of claims 1 to 14 or the moving body according to claim 15 by a computer.

Images