Photoelectric conversion device. mobile, control method, and computer program

Abstract

To provide a photoelectric conversion device capable of setting different storage times with respect to regions of different resolutions.SOLUTION: A photoelectric conversion device comprises: a photoelectric conversion section 102 which generates a pulse in response to the incidence of a photon; a plurality of signal processing circuits 103 each including a counter 211 which counts the number of pulses and a memory 212 in which a count value of the counter is stored; an optical system which forms subject images of different resolutions in a first pixel region and a second pixel region of a sensor section consisting of a plurality of pixels; and a control section which generates a signal on the basis of a difference of count values of the counter in the start and end of a storage period, has at least a first storage period and a second storage period within one full frame, the first storage period being shorter than the second storage period, and performs control in such a manner that a signal generated during the first storage period is output from the end of the first storage period to the end of the second storage period.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a photoelectric conversion device, a moving object, a control method, a computer program, and the like. [Background technology]

[0002] 2. Description of the Related Art In recent years, photoelectric conversion devices have been developed that digitally count the number of photons incident on an avalanche photodiode (APD) and output the counted value from a pixel as a photoelectrically converted digital signal.

[0003] For example, Patent Document 1 describes a configuration in which a photoelectric conversion device having an APD can output multiple images whose accumulation periods overlap with each other, enabling continuous shooting even in low illuminance. Patent Document 2 describes a configuration in which signals are read out at a high frame rate from a pixel area in which a moving subject is detected. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 7223070 [Patent Document 2] Patent Publication No. 2021-34786 Summary of the Invention [Problem to be solved by the invention]

[0005] However, when considering the image sensor of a camera, for example, because the recognition process is performed on a frame-by-frame basis when the sensor is normally driven, for example, at 30 fps (frames per second), the recognition process can only be performed every 33.3 ms. Therefore, even if an object appears immediately after a frame change in a normal camera, the recognition process cannot be performed until the end of the frame.

[0006] On the other hand, in a monitoring system using an in-vehicle camera, for example, in order to reduce the risk of collisions, etc., it is required to accurately and quickly recognize vehicles or people that suddenly appear in areas that are in the driver's blind spot.

[0007] For this reason, vehicle-mounted cameras are required to capture and recognize images at high resolution in a short time. However, capturing images at high resolution and a short frame rate poses the problem of increased data volume and power consumption.

[0008] Therefore, an object of the present invention is to provide a photoelectric conversion device capable of setting different accumulation times for areas with different resolutions. [Means for solving the problem]

[0009] A photoelectric conversion device according to one aspect of the present invention comprises: A plurality of pixels each including a photoelectric conversion unit that emits a pulse in response to an incidence of a photon, a counter that counts the number of the pulses, and a memory that stores the count value of the counter; an optical system that forms subject images having different resolutions in a first pixel region and a second pixel region of the sensor unit that is made up of the plurality of pixels; a control unit that generates a signal based on a difference between count values ​​of the counter at the start and end of an accumulation period, has at least a first accumulation period and a second accumulation period within one full frame, the first accumulation period is shorter than the second accumulation period, and controls so that the signal generated during the first accumulation period is output between the end of the first accumulation period and the end of the second accumulation period; The control unit controls the accumulation time in one of the first pixel region and the second pixel region to be one of the first accumulation period and the second accumulation period. Effect of the Invention

[0010] According to the present invention, it is possible to provide a photoelectric conversion device capable of setting different accumulation times for areas with different resolutions. [Brief description of the drawings]

[0011] [Figure 1] 1 is a diagram illustrating an example of the configuration of a sensor unit 100 according to a first embodiment. [Diagram 2] 2 is a diagram showing an example of the configuration of a sensor portion substrate 11 according to the first embodiment. FIG. [Diagram 3] 2 is a diagram showing an example of the configuration of a circuit board 21 according to the first embodiment. FIG. [Figure 4] 2 and 3, this is a diagram showing an equivalent circuit of the photoelectric conversion unit 102 and the signal processing circuit 103 corresponding to the photoelectric conversion unit 102. [Diagram 5] 2 is a diagram illustrating a relationship between the operation of an APD 201 and an output signal. FIG. [Figure 6] 1 is a functional block diagram of a photoelectric conversion device 600 according to a first embodiment. [Figure 7] 6 is a diagram for explaining a photoelectric conversion method performed by a control unit 605 according to the first embodiment. FIG. [Figure 8] FIG. 4 is a diagram showing an example of images of a plurality of divided accumulation periods according to the first embodiment. [Figure 9] FIG. 2 is a diagram illustrating the relationship between a memory and a buffer in the first embodiment. [Figure 10] 5 is a flowchart showing details of an example of driving the sensor unit 100 in the first embodiment. [Figure 11] This is a continuation of the flowchart in Figure 10. [Figure 12] 2 is a diagram illustrating an example of a mounting position of a photoelectric conversion device 600 according to the first embodiment. [Figure 13] 6A and 6B are diagrams illustrating the optical characteristics of the optical system 601 according to the first embodiment. [Figure 14] 11 is a diagram illustrating an example of a mounting position of a photoelectric conversion device 600 according to the second embodiment. FIG. [Figure 15] 6A and 6B are diagrams illustrating the optical characteristics of an optical system 601 according to the second embodiment. [Figure 16] FIG. 13 is a diagram for explaining setting of a region of interest in the third embodiment. [Figure 17] FIG. 13 is a diagram for explaining setting of a region of interest in the fourth embodiment. [Figure 18] FIG. 13 is a diagram illustrating setting of a region of interest in the fifth embodiment. [Figure 19] FIG. 23 is a diagram for explaining setting of a region of interest in the sixth embodiment. [Figure 20] FIG. 23 is a functional block diagram showing an example of the configuration of a photoelectric conversion device 600 and a moving object 700 according to an eleventh embodiment. [Figure 21] 23 is a flowchart illustrating a method for controlling a moving object according to an eleventh embodiment. [Figure 22] 23 is a flowchart showing details of an example of driving the sensor unit 100 in relation to a method of setting an attention area according to a twelfth embodiment. [Figure 23] This is a continuation of the flowchart in Figure 22. [Figure 24] FIG. 13 is a diagram showing an example of images taken for a plurality of accumulation periods after a region of interest is set. [Diagram 25] 23 is a flowchart showing details of an example in which an accumulation period is set and the sensor unit 100 is driven in the thirteenth embodiment. [Figure 26] This is a continuation of the flowchart in Figure 25. [Figure 27] 23 is a flowchart showing an example of the operation of the control unit after constructing an image according to the fourteenth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In each drawing, the same members or elements are given the same reference numerals, and duplicated descriptions are omitted or simplified.

[0013] <Embodiment 1> 1 is a diagram showing a configuration example of a sensor unit 100 according to embodiment 1. In the following, the sensor unit has a so-called laminated structure in which two substrates, a sensor unit substrate 11 and a circuit substrate 21, are laminated and electrically connected.

[0014] However, a so-called non-laminated structure in which the components included in the sensor substrate 11 and the components included in the circuit substrate 21 are disposed on a common semiconductor layer may be used. The sensor substrate 11 includes a pixel region 12. The circuit substrate 21 includes a circuit region 22 that processes signals detected in the pixel region 12.

[0015] 2 is a diagram showing an example of the configuration of the sensor portion substrate 11 according to embodiment 1. The pixel region 12 of the sensor portion substrate 11 includes a plurality of pixels 101 arranged two-dimensionally across a plurality of rows and a plurality of columns.

[0016] The pixel 101 includes a photoelectric conversion unit 102 including an avalanche photodiode (hereinafter, APD). Here, the photoelectric conversion unit 102 emits a pulse at a frequency corresponding to the frequency of receiving photons. That is, the photoelectric conversion unit 102 emits a pulse in response to the incidence of photons. The number of rows and columns of the pixel array constituting the pixel region 12 is not particularly limited.

[0017] Fig. 3 is a diagram showing an example of the configuration of a circuit board 21 according to embodiment 1. The circuit board 21 has a signal processing circuit 103 that processes charges photoelectrically converted by each photoelectric conversion unit 102 in Fig. 2, a readout circuit 112, a control pulse generation unit 115, a horizontal scanning circuit 111, a vertical signal line 113, a vertical scanning circuit 110, and an output circuit 114.

[0018] The vertical scanning circuit 110 receives a control pulse supplied from a control pulse generating unit 115, and sequentially supplies the control pulse to a plurality of pixels arranged in the row direction. The vertical scanning circuit 110 is composed of logic circuits such as a shift register and an address decoder.

[0019] The signal output from the photoelectric conversion unit 102 of each pixel is processed by each signal processing circuit 103. The signal processing circuit 103 is provided with a counter, a memory, etc., and a digital value is stored in the memory. The horizontal scanning circuit 111 inputs a control pulse that sequentially selects each column to the signal processing circuit 103 in order to read the signal from the memory of each pixel in which the digital signal is stored.

[0020] A signal is output to the vertical signal line 113 from the signal processing circuit 103 of the pixel of the row selected by the vertical scanning circuit 110. The signal output to the vertical signal line 113 is output to the outside of the sensor unit 100 via the readout circuit 112 and the output circuit 114. The readout circuit 112 has a plurality of buffers built therein, each connected to the vertical signal line 113.

[0021] 2 and 3, a plurality of signal processing circuits 103 are arranged in a region overlapping the pixel region 12 in a plan view. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged in a region of the sensor portion substrate 11 outside the pixel region 12 so as to overlap with each other in a plan view.

[0022] That is, the sensor portion substrate 11 has a pixel region 12 and a non-pixel region arranged around the pixel region 12. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged in a region overlapping the non-pixel region in a plan view.

[0023] The arrangement of the vertical signal lines 113, the readout circuits 112, and the output circuits 114 are not limited to the example shown in Fig. 3. For example, the vertical signal lines 113 may be arranged to extend in the row direction, and the readout circuits 112 may be arranged at the ends of the vertical signal lines 113.

[0024] Moreover, it is not necessary to provide one signal processing circuit 103 for each photoelectric conversion unit. One signal processing circuit may be shared by a plurality of photoelectric conversion units, and may be configured to perform signal processing sequentially.

[0025] Fig. 4 is a diagram showing an equivalent circuit of the photoelectric conversion unit 102 and the signal processing circuit 103 corresponding to the photoelectric conversion unit 102 in Fig. 2 and Fig. 3. The photoelectric conversion unit 102 and the signal processing circuit 103 are included in each pixel. The APD 201 included in the photoelectric conversion unit 102 generates a charge pair according to incident light by photoelectric conversion.

[0026] One of the two nodes of the APD 201 is connected to a power supply line to which a driving voltage VL (first voltage) is supplied, and the other of the two nodes of the APD 201 is connected via a quench element 202 to a power supply line to which a driving voltage VH (second voltage) higher than the voltage VL is supplied.

[0027] In Fig. 4, one node of the APD 201 is the anode, and the other node of the APD is the cathode. A reverse bias voltage is supplied to the anode and cathode of the APD 201 so that the APD 201 performs avalanche multiplication. By supplying such a voltage, the charge generated by the incident light undergoes avalanche multiplication, generating an avalanche current.

[0028] When a reverse bias voltage is supplied, there are two modes: Geiger mode, which operates with the voltage difference between the anode and cathode greater than the breakdown voltage, and linear mode, which operates with the voltage difference between the anode and cathode close to or less than the breakdown voltage. An APD that operates in Geiger mode is called a SPAD. In the case of a SPAD, for example, the voltage VL (first voltage) is -30V and the voltage VH (second voltage) is 1V.

[0029] The signal processing circuit 103 includes a quench element 202, a waveform shaping unit 210, a counter 211, and a memory 212. The quench element 202 is connected to a power supply line to which a driving voltage VH is supplied and one of the anode and cathode nodes of the APD 201.

[0030] The quench element 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, and has a function of suppressing avalanche multiplication by suppressing the voltage supplied to the APD 201 (quench operation).The quench element 202 also has a function of returning the voltage supplied to the APD 201 to the drive voltage VH by flowing a current equivalent to the voltage drop caused by the quench operation (recharge operation).

[0031] 4 shows an example in which the signal processing circuit 103 has a waveform shaping unit 210, a counter 211, and a memory 212 in addition to the quench element 202. The waveform shaping unit 210 shapes the voltage change of the cathode of the APD 201 obtained at the time of photon detection, and outputs a pulse signal.

[0032] For example, an inverter circuit is used as the waveform shaping unit 210. In Fig. 4, an example in which one inverter is used as the waveform shaping unit 210 is shown, but a circuit in which a plurality of inverters are connected in series may be used, or another circuit having a waveform shaping effect may be used.

[0033] The counter 211 counts the number of pulses output from the waveform shaping unit 210 and holds the count value. When a control pulse RES is supplied via a drive line 213, the signal held in the counter 211 is reset. Here, the counter 211 generates a signal based on the difference between the count values ​​at the start and end of the accumulation period.

[0034] 3 to the memory 212 via a drive line 214 (not shown in FIG. 3) in FIG. 4, and switches between electrical connection and disconnection between the counter 211 and the vertical signal line 113. The memory 212 functions as a memory that temporarily stores the count value of the counter, and outputs an output signal from the counter 211 of the pixel to the vertical signal line 113.

[0035] Note that electrical connections may be switched by disposing a switch such as a transistor between the quench element 202 and the APD 201 or between the photoelectric conversion unit 102 and the signal processing circuit 103. Similarly, the supply of the voltage VH or the voltage VL to the photoelectric conversion unit 102 may be electrically switched using a switch such as a transistor.

[0036] 5 is a diagram showing a schematic diagram of the relationship between the operation of the APD 201 and the output signal. The input side of the waveform shaping unit 210 is nodeA, and the output side is nodeB. Between time t0 and time t1, a potential difference of VH-VL is applied to the APD 201. When a photon is incident on the APD 201 at time t1, avalanche multiplication occurs in the APD 201, an avalanche multiplication current flows through the quench element 202, and the voltage at nodeA drops.

[0037] When the amount of voltage drop further increases and the potential difference applied to the APD 201 decreases, the avalanche multiplication of the APD 201 stops as at time t2, and the voltage level of nodeA does not drop below a certain value.

[0038] After that, between time t2 and time t3, a current flows through nodeA to compensate for the voltage drop from voltage VL, and at time t3, nodeA is stabilized to its original potential level. At this time, the portion of the output waveform at nodeA that exceeds a certain threshold is shaped by waveform shaping section 210 and is output as a pulse signal at nodeB.

[0039] Next, a photoelectric conversion device 600 according to the first embodiment will be described. Fig. 6 is a functional block diagram of the photoelectric conversion device 600 according to the first embodiment. Some of the functional blocks shown in Fig. 6 are realized by causing a computer (not shown) included in the photoelectric conversion device 600 to execute a computer program stored in a memory serving as a storage medium (not shown).

[0040] However, a part or all of these may be realized by hardware, which may be a dedicated circuit (ASIC) or a processor (a reconfigurable processor, DSP), etc.

[0041] In addition, the functional blocks shown in Fig. 6 do not have to be built in the same housing, and may be configured by separate devices connected to each other via signal paths. The above explanation regarding Fig. 6 also applies to Fig. 20 described later.

[0042] 1 to 5, an optical system 601, a detection unit 602, an image processing unit 603, a recognition unit 604, a control unit 605, a storage unit 606, a communication unit 607, etc. The sensor unit is composed of an avalanche photodiode for photoelectrically converting an optical image, as described in FIGS.

[0043] The camera unit, which is a set of the optical system 601 and the sensor unit 100, is configured to capture an image in at least one direction, for example, in the front, rear, or side of the photoelectric conversion device 600.

[0044] In this embodiment, the optical system 601 is a wide-angle lens (e.g., a fisheye lens) having an angle of view of, for example, 120°, and forms an optical image (subject image) of an object in front of the photoelectric conversion device 600 on the imaging surface of the sensor unit 100. The detection unit 602 detects information (hereinafter, referred to as environmental information) related to the surrounding environment of the photoelectric conversion device 600. In addition, depending on the output of the detection unit 602, the positions and sizes of an attention area and a non-attention area, which will be described later, are changed.

[0045] The image processing unit 603 performs image processing such as black level correction, gamma correction, noise reduction, digital gain adjustment, demosaic processing, data compression, etc. on the image signal acquired by the sensor unit 100 to generate a final image signal. If the sensor unit has an on-chip color filter such as RGB, the image processing unit 603 performs processing such as white balance correction and color conversion.

[0046] Moreover, the output of the image processing unit 603 is supplied to a recognition unit 604 and a control unit 605. The recognition unit 604 recognizes people, vehicles, objects, etc. captured in the image by performing image recognition based on the image signal. The recognition result by the recognition unit 604 is output to the control unit 605 and is reflected in, for example, changing the control mode of the photoelectric conversion device 600. Furthermore, the recognition result is stored in a storage unit 606 and transmitted to the outside via a communication unit 607, etc.

[0047] The control unit 605 has a built-in CPU as a computer and a memory that stores a computer program. The control unit 605 also functions as a setting unit, and sets the length of the exposure period of each frame of the sensor unit 100 and the timing control of the control signal CLK for each region via the control pulse generation unit 115 of the sensor unit 100.

[0048] Furthermore, the control unit 605 also functions as an acquisition unit, and acquires, as characteristic information of the photoelectric conversion device 600, sensor characteristic information such as the size and number of pixels of the sensor unit 100, and optical characteristic information such as the angle of view and resolution of the optical system 601. Furthermore, the control unit 605 acquires information such as the installation height and installation angle as installation information of the photoelectric conversion device 600 from the detection unit 602, and acquires environmental information related to the surrounding environment of the photoelectric conversion device 600 from the detection unit 602.

[0049] Based on the information acquired from the detection unit 602 , the CPU executes a computer program stored in a memory built into the control unit 605 to control each unit of the photoelectric conversion device 600 .

[0050] The storage unit 606 includes a recording medium such as a memory card, a hard disk, etc., and can store and read image signals. The communication unit 607 includes a wireless or wired interface, and outputs the generated image signal to the outside of the photoelectric conversion device 600 and receives various signals from the outside.

[0051] The photoelectric conversion device 600 is used, for example, as a camera, an in-vehicle camera, a pet camera, a surveillance camera, a camera for detection or inspection used in a manufacturing line, a camera for logistics, etc. In addition, the device can be used for various purposes such as a medical endoscope camera, a condition detection camera for nursing care, a camera for infrastructure inspection, an agricultural camera, etc.

[0052] 7 is a diagram for explaining a photoelectric conversion method by the control unit 605 according to the first embodiment. In this embodiment, photoelectric conversion is periodically driven at, for example, 30 full frames / second. Moreover, a frame for one vertical period having a length of 33.3 ms is called a full frame, and each of the full frames divided into four is called a frame.

[0053] That is, as shown in Fig. 7, a full frame 1 is divided into frames 1_1, 1_2, 1_3, and 1_4 each having an equal period (8.33 ms). Note that in Fig. 7 and subsequent figures, a frame may be abbreviated as F.

[0054] Frame 1_1 has an accumulation period from start time T0 of full frame 1 to time T1, frame 1_2 has an accumulation period from time T0 to time T2, frame 1_3 has an accumulation period from time T0 to time T3, and frame 1_4 has an accumulation period from time T0 to time T4.

[0055] At times T1 to T4, count values ​​C1_1, C1_2, C1_3, and C1_4 are obtained from the counter 211. The count values ​​C1_1, C1_2, C1_3, and C1_4 are temporarily stored in the memory 212.

[0056] Then, the signals for one row temporarily stored in the memory 212 are sequentially output from the sensor unit via the buffer of the readout circuit 112. Also, the counter 211 is reset at time T0. Thus, according to this embodiment, the signals accumulated during the period of frame 1_1 are read out from time T1 to time T2 and quickly processed by the recognition unit 604.

[0057] Therefore, image recognition can be performed quickly. Similarly, signals accumulated during the periods of frame 1_2, frame 1_3, and frame 1_4 are sequentially read out from time T2 to T3, T3 to T4, and T4 to T1, respectively, and image recognition can be performed repeatedly. Note that the length of a full frame is not limited to the above example. Also, the number of divisions of a full frame is not limited to 4.

[0058] Fig. 8 is a diagram showing an example of images of a plurality of divided accumulation periods according to the first embodiment. As shown in Fig. 8, the image of frame 1_1 is dark because the accumulation period is short, but there is little subject blur of a person who jumps out. On the other hand, the accumulation periods of frames 1_2, 1_3, and 1_4 are longer in this order, so subject blur is more likely to occur. Note that blur is less likely to occur in stopped vehicles and white lines, and the longer the accumulation period, the more likely the contrast is to be improved.

[0059] In this manner, in this embodiment, one full frame has a first accumulation period (e.g., time T1 to T2) and a second accumulation period (e.g., time T1 to T4), and the first accumulation period is shorter than the second accumulation period. Also, the signal generated during the first accumulation period is controlled to be output between the end of the first accumulation period and the end of the second accumulation period.

[0060] In this embodiment, the first and second accumulation periods overlap, that is, the first and second accumulation periods start at the same time, the end of the second accumulation period coincides with the end of a full frame, and the second accumulation period is an integer multiple of the first accumulation period.

[0061] However, the second accumulation period does not need to be an integer multiple of the first accumulation period. The second accumulation period only needs to be longer than the first accumulation period (the first accumulation period is shorter than the second accumulation period) and the second accumulation period only needs to end after the first accumulation period.

[0062] That is, an image with a short accumulation period and an image with a long accumulation period are created, the timing at which the short accumulation period ends is set to be earlier than the timing at which the long accumulation period ends, and as soon as the short accumulation period ends, the image is output and sent to the downstream recognition unit 604.

[0063] The subject is recognized based on the signal generated at least during the first accumulation period. The recognition unit 604 recognizes the subject based on the signal generated at least during the first accumulation period. Therefore, while image recognition was not possible until one full frame period had elapsed in the past, image recognition can be performed after as little as ¼ full frame period in this embodiment.

[0064] In addition, an image with a long accumulation period can improve the contrast, and can be used as an image for display. That is, an image with a short accumulation period is suitable for quick subject recognition, and an image with a long accumulation period is suitable for an image for display. Therefore, the display unit 703 of this embodiment displays an image with a long accumulation period, that is, a signal generated during the second accumulation time, as an image.

[0065] In addition, because this embodiment uses an APD, unlike a CMOS image sensor, the accumulated charge is not degraded by reading out, and the accumulation periods can be overlapped. Also, since there is no readout noise, the original signal does not degrade even if it is read out multiple times from one accumulation.

[0066] On the other hand, in the case of a CMOS image sensor, it is possible to shorten the accumulation period by increasing the frame frequency, but each time the accumulation period is shortened, readout noise is superimposed, deteriorating the signal-to-noise ratio (S / N) of the output, making it difficult to obtain an image suitable for image recognition.

[0067] In addition, in order to overlap the accumulation periods in a CMOS image sensor, the same charge must be accumulated in the pixels again after reading out, which makes the circuitry more complex and larger in scale.In addition, noise is also superimposed when accumulating charge in the pixels again, which significantly worsens the S / N ratio of the output, making it difficult to obtain an image suitable for image recognition.

[0068] Fig. 9 is a diagram showing the relationship between memories and buffers in embodiment 1. Fig. 9 shows a state in which memories 212 in signal processing circuit 103 in Fig. 3 are arranged in N rows and M columns, and each memory 212 is represented as memory 1-1 to memory NM. Also, buffer 1 to buffer M in Fig. 9 indicate buffers included in read circuit 112 in Fig. 3. Output circuit 114 in Fig. 9 corresponds to output circuit 114 in Fig. 3.

[0069] Fig. 10 is a flowchart showing details of an example of driving the sensor unit 100 in the first embodiment, and Fig. 11 is a flowchart continuing from Fig. 10. Note that the CPU as a computer in the control unit 605 executes a computer program stored in the memory, thereby sequentially performing the operations of the steps in the flowcharts of Fig. 10 and Fig. 11.

[0070] 10, the CPU of the control unit 605 sets i to 1. Next, in step S102, the CPU of the control unit 605 causes the count value Count of the counter 211 at time Ti to be output to the memory 212. At this time, output is performed simultaneously to all memories. This operation corresponds to the operation at time T1 in FIG.

[0071] Next, in step S103, the CPU of the control unit 605 sets j=1. Next, in step S104, the CPU of the control unit 605 sets k=1. In step S105, the CPU of the control unit 605 outputs the count value Count(jki) in memory jk in FIG. 9 to the buffer. At this time, columns 1 to M are output to the buffer at the same time. This operation means that the count value of the first row in FIG. 9 is taken into the buffer.

[0072] In step S106, the CPU of the control unit 605 causes the output circuit 114 to output the count value Count(j - k - i) of buffer k. This operation corresponds to the operation of reading the signal of the buffer in the leftmost column of FIG. 9 from the output circuit.

[0073] Next, proceed to step S107 in FIG. 11 via A. In step S107, the CPU of the control unit 605 determines whether k < M. If it is Yes in step S107, then in step S108, k is incremented by 1 as k = k + 1, and return to step S106 via B to perform the operation of step S106. This operation corresponds to the operation of reading the signal of the buffer in the second column from the left in FIG. 9 from the output circuit.

[0074] When it becomes No in step S107, that is, when k = M, it means that the signal of the buffer in the M-th column of FIG. 9 has been read from the output circuit. Next, proceed to step S109, and the CPU of the control unit 605 determines whether j < N. If it is Yes in step S109, then in step S110, the CPU of the control unit 605 sets j = J + 1 to increment j by 1, and returns to step S104 via C. This corresponds to the operation for starting the reading of the next line.

[0075] When it is determined to be No in step S109, it means that the reading of all lines has been completed. So, proceed to step S111, and the CPU of the control unit 605 determines whether i < 4. If it is determined to be Yes in step S111, then in step S112, the CPU of the control unit 605 sets i = i + 1 to increment i by 1 and returns to step S102 via D. This operation corresponds to the operation of starting the reading at the next time T2.

[0076] If the result of the determination in step S111 is No, this means that the readout at time T4 is completed, so the process proceeds to step S113, where the CPU of the control unit 605 resets the counter 211 with a reset signal. This operation corresponds to the reset operation of the counter 211 at time T4 in Fig. 7. The flows in Figs. 10 and 11 are repeated periodically. In this manner, the signals accumulated in the sensor unit can be read out sequentially.

[0077] In the sensor section, when the circuit configuration is as shown in FIG. 9, it becomes possible to change the setting of the accumulation period for each column.

[0078] For example, in the first column (k=1), four pieces of data are read out by changing the accumulation period of the full frame 1 to frame 1_1, frame 1_2, frame 1_3, and frame 1_4, as shown in Fig. 7. In other words, the first column is set as the first image area with a short accumulation period.

[0079] On the other hand, in the second column (k=2), it is possible to divide the period of full frame 1 into two and read out only the data of the accumulation periods of frames 1_2 and 1_4. In other words, the second column is set as the second image area with a longer accumulation period.

[0080] In this way, the accumulation period and the number of times can be changed arbitrarily, and by changing the accumulation period and the number of times, it is possible to change the intensity of the accumulated signal and increase or decrease the number of accumulations. Also, it is possible to increase or decrease the amount of data according to the number of accumulations. However, if the amount of data increases, the power consumption of the sensor unit and the photoelectric conversion device 600 also increases accordingly.

[0081] The method of changing the accumulation period setting is not limited to changing it for each column, but it is also possible to change the accumulation period setting for each row or for each pixel by changing the signal line of the sensor unit 100. The setting of the accumulation period for each region or pixel is performed by the control unit 605.

[0082] As described above, the control unit 605 has a built-in CPU as a computer and a memory that stores a computer program, and acquires information from the detection unit 602, and the CPU executes the computer program stored in the memory based on the information. This makes it possible to set the accumulation period for each region or pixel.

[0083] In this way, the control unit 605 can set the accumulation period for each region or pixel of the sensor unit 100, so that the timing at which a short accumulation period ends can be made earlier than the timing at which a long accumulation period ends. Therefore, as soon as the short accumulation period ends, it becomes possible to output the image and send it to the downstream recognition unit 604.

[0084] That is, based on the signal generated in the first accumulation period, the recognition unit 604 can recognize the subject in the image area with a short accumulation period at time T1 when frame 1_1 ends, which is earlier than time T4 when full frame 1 ends in the first column (k=1). Therefore, while image recognition was not possible in the past until one full frame period had passed, this embodiment makes it possible to set an area where image recognition can be performed in a shorter time.

[0085] Also, because images can be captured over multiple integration periods, properly exposed images can be generated for both light and dark areas, maximizing the signal-to-noise ratio (S / N) in each area. In other words, in dark areas, a long integration period increases the signal, maximizing the S / N ratio.

[0086] On the other hand, in bright areas, a short integration period allows signals to be acquired without saturating the signal, maximizing the S / N ratio. Therefore, by applying an integration period appropriate for each area, it is possible to acquire images with maximized S / N ratios without saturating the signal for each area and suppressing noise.

[0087] In addition, by applying an appropriate accumulation period to each region, blurring due to the movement of the subject can be suppressed. A long accumulation period may result in a blurred image (blur) of the subject. On the other hand, a short accumulation period can reduce blurring of the subject. Therefore, by applying an appropriate accumulation period, it is possible to obtain an image with reduced blurring due to the movement of the subject.

[0088] Next, the characteristics of the optical system 601 will be described in detail. Fig. 12 is a diagram for explaining an example of the mounting position of the photoelectric conversion device 600 in the first embodiment. Fig. 12 shows a case where the moving body 700 is an automobile (vehicle). Also, it shows a case where the photoelectric conversion device 600 is an on-vehicle camera.

[0089] The moving body 700 is equipped with an in-vehicle system (driving assistance device, display system) (not shown) for assisting a user 40 (driver, passenger, etc.) of the moving body 700 using images acquired by the photoelectric conversion device 600.

[0090] In this embodiment, the photoelectric conversion device 600 is installed so as to capture an image of the rear of the moving body 700, but the photoelectric conversion device 600 may be installed so as to capture an image of the front or side of the moving body 700. Furthermore, two or more photoelectric conversion devices 600 may be installed at two or more locations on the moving body 700.

[0091] As described above, the photoelectric conversion device 600 has the optical system 601 and the sensor unit 100. The optical system 601 has a first angle of view (first field of view) 30 and a second angle of view (second field of view) 31 outside the first angle of view 30 (peripheral side).

[0092] Moreover, the optical system 601 is an optical system (different angle of view lens) having different imaging magnifications at the first angle of view 30 and the second angle of view 31. The imaging surface (light receiving surface) of the sensor unit 100 includes a first region for imaging an object included in the first angle of view 30 and a second region for imaging an object included in the second angle of view 31.

[0093] In this case, the number of pixels per unit angle of view in the first region is greater than the number of pixels per unit angle of view in the second region excluding the first region. In other words, the resolution of the photoelectric conversion device 600 in the first angle of view (first region) is higher than the resolution in the second angle of view (second region).

[0094] 13A and 13B are diagrams illustrating the optical characteristics of the optical system 601 in the first embodiment. Fig. 13(A) is a diagram showing, in the form of contour lines, the image height y at each half angle of view on the imaging surface (light receiving surface of the image sensor) of the sensor unit 100. Fig. 13(B) is a diagram showing the relationship between the half angle of view θ and the image height y (projection characteristics of the optical system 601) in the first quadrant of Fig. 13(A).

[0095] 13B, the optical system 601 is configured such that its projection characteristic y(θ) differs between angles of view less than a predetermined half angle of view θa and angles of view equal to or greater than the half angle of view θa. Therefore, the optical system 601 is configured such that, when the increase in image height y per unit of half angle of view θ is defined as the resolution, the resolution differs depending on the angle of view (the area on the light receiving surface of the sensor unit).

[0096] This local resolution can be expressed as the differential value dy(θ) / dθ of the projection characteristic y(θ) at the half angle of view θ. For example, the greater the gradient of the projection characteristic y(θ) in Fig. 13(B), the higher the resolution. Also, Fig. 13(A) shows that the greater the interval between the contour lines of the image height y at each half angle of view, the higher the resolution.

[0097] The optical system 601 of this embodiment has a projection characteristic in which the rate of increase of image height y (the slope of the projection characteristic y(θ) in FIG. 13(B)) is large in the central region near the optical axis, and the rate of increase of image height y becomes smaller in the peripheral region outside the central region as the angle of view becomes larger.

[0098] 13A, a first region 601a including the center corresponds to an angle of view less than the half angle of view θa, and a second region 601b outside the first region corresponds to an angle of view equal to or greater than the half angle of view θa. The angle of view less than the half angle of view θa corresponds to the first angle of view 30 in FIG. 12, and the angle of view equal to or greater than the half angle of view θa corresponds to the second angle of view 31 in FIG. 12.

[0099] As described above, the first region 601a is a relatively high-resolution region, and the second region 601b is a relatively low-resolution region. Also, the first region 601a is a low-distortion region with relatively little distortion, and the second region 601b is a high-distortion region with relatively much distortion. Therefore, in this embodiment, the first region 601a may be called a high-resolution region or a low-distortion region, and the second region 601b may be called a low-resolution region or a high-distortion region.

[0100] In this way, the optical system 601 of this embodiment forms subject images with different resolutions in the first pixel region and the second pixel region of the light receiving surface of the sensor unit consisting of multiple pixels. Note that the characteristics shown in Fig. 13 are only an example and are not limited to this. For example, the low resolution region and the high resolution region of the optical system do not have to be configured in a concentric circle shape, and each region may have a distorted shape.

[0101] The center of gravity of the low resolution region and the center of gravity of the high resolution region do not have to coincide with each other. The center of gravity of the low resolution region and the center of gravity of the high resolution region may be shifted from the center of the light receiving surface of the sensor unit. In the optical system of this embodiment, it is sufficient that the high resolution region is formed near the optical axis, and the low resolution region is formed on the peripheral side (outside the high resolution region) away from the vicinity of the optical axis.

[0102] The projection characteristic y(θ) in the first region 601a (high resolution region) of the optical system 601 is set to be different from the projection characteristic in the second region 601b (low resolution region). Also, the projection characteristic y(θ) in the first region 601a (high resolution region) is configured to be larger than f×θ (f is the focal length of the optical system 601).

[0103] When θmax is the maximum half angle of view of the optical system 601, the ratio θa / θmax of θa to θmax is preferably equal to or greater than a predetermined lower limit, and for example, the predetermined lower limit is preferably 0.15 to 0.16.

[0104] Also, the ratio θa / θmax of θa to θmax is preferably equal to or less than a predetermined upper limit, for example, 0.25 to 0.35. For example, if θmax is 90°, the predetermined lower limit is 0.15, and the predetermined upper limit is 0.35, then θa is preferably determined in the range of 13.5 to 31.5°. However, the above is just one example, and the present invention is not limited to this.

[0105] Furthermore, the optical system 601 is configured so that its projection characteristic y(θ) also satisfies the following formula 1.

number

[0106] Here, f is the focal length of the optical system 601 as described above, and A is a predetermined constant. The constant A can be determined taking into consideration the balance between the resolution of the high-resolution region and the low-resolution region, and is preferably set to 1.4 to 1.9.

[0107] By setting the lower limit to 1, it is possible to increase the central resolution compared to an orthogonal projection type (y=f×sinθ) fisheye lens having the same maximum imaging height, and by setting the upper limit to A, it is possible to obtain an angle of view equivalent to that of a fisheye lens while maintaining good optical performance. However, the above is just one example and is not limiting.

[0108] By configuring the optical system 601 as described above, a high resolution can be obtained in the first region 601a (high resolution region), while the increase in image height y per unit half angle of view θ can be reduced in the second region 601b (low resolution region). Therefore, it is possible to capture a wider angle of view, and high resolution can be obtained in the high resolution region while keeping the imaging range at a wide angle of view equivalent to that of a fisheye lens.

[0109] <Embodiment 2> Next, a description will be given of embodiment 2. Fig. 14 is a diagram for explaining an example of the mounting position of a photoelectric conversion device 610 in embodiment 2, and shows a case where a moving object 700 is an automobile (vehicle) like embodiment 1.

[0110] In addition, the photoelectric conversion device 610 is an in-vehicle camera. The moving body 700 includes an in-vehicle system (driving assistance device) (not shown) for assisting a user (such as a driver or a passenger) (not shown) using an image acquired by the photoelectric conversion device 610.

[0111] In this embodiment, the photoelectric conversion device 610 is installed on the side of the moving body 700, but the photoelectric conversion device 610 may be installed in front or rear of the moving body 700. Furthermore, two or more photoelectric conversion devices 610 may be installed in two or more places on the moving body 700.

[0112] The photoelectric conversion device 610 has an optical system 611 and a sensor unit. The optical system 611 is an optical system (inverse different angle of view lens) in which the imaging magnification differs between a first angle of view (first field of view) 50 and a second angle of view (second field of view) 51 on the peripheral side (outside the first angle of view) of the first angle of view 50.

[0113] The imaging surface (light receiving surface) of the sensor unit 100 includes a first region for capturing an image of an object included in a first angle of view 50, and a second region for capturing an image of an object included in a second angle of view 51. At this time, the number of pixels per unit angle of view in the second region is greater than the number of pixels per unit angle of view in the first region. In other words, the resolution of the photoelectric conversion device 610 at the second angle of view (second region) is higher than the resolution at the first angle of view (first region).

[0114] 15A and 15B are diagrams illustrating the optical characteristics of the optical system 601 in the second embodiment. Fig. 15(A) is a diagram showing, in the form of contour lines, the image height y at each half angle of view on the imaging surface (light receiving surface) of the sensor unit 100. Fig. 15(B) is a diagram showing the relationship between the half angle of view θ and the image height y (projection characteristics of the optical system 611) in the first quadrant of Fig. 15(A).

[0115] 15B, the optical system 611 is configured such that its projection characteristic y(θ) differs between angles of view less than a predetermined half angle of view θa and angles of view equal to or greater than the half angle of view θa. Therefore, the optical system 611 is configured such that, when the increase in image height y per unit of half angle of view θ is defined as the resolution, the resolution differs depending on the angle of view (the area on the light receiving surface of the sensor unit).

[0116] This local resolution can be expressed as the differential value by(θ) / bθ of the projection characteristic y(θ) at the half angle of view θ. For example, the greater the gradient of the projection characteristic y(θ) in FIG. 15B, the higher the resolution. Also, in FIG. 15A, the greater the interval between the contour lines of the image height y at each half angle of view, the higher the resolution.

[0117] The optical system 611 of this embodiment has a projection characteristic in which the rate of increase of image height y (the slope of the projection characteristic y(θ) in FIG. 15(B)) is small in the central region near the optical axis, and the rate of increase of image height y becomes larger as the angle of view becomes larger in the peripheral region outside the central region.

[0118] 15A, a first region 611a including the center corresponds to an angle of view less than the half angle of view θa, and a second region 611b outside the first region corresponds to an angle of view equal to or greater than the half angle of view θa. The angle of view less than the half angle of view θa corresponds to the first angle of view 50 in FIG. 14, and the angle of view equal to or greater than the half angle of view θa corresponds to the second angle of view 51 in FIG. 14.

[0119] As described above, the first region 611a is a relatively low-resolution region, and the second region 611b is a relatively high-resolution region. Also, the first region 611a is a high-distortion region with relatively more distortion, and the second region 611b is a low-distortion region with relatively less distortion. Therefore, in this embodiment, the first region 611a may be called a low-resolution region or a high-distortion region, and the second region 611b may be called a high-resolution region or a low-distortion region.

[0120] Note that the characteristics shown in Fig. 15 are just an example and are not limited to this. For example, the low-resolution area and the high-resolution area of ​​the optical system do not have to be configured as concentric circles, and each area may have a distorted shape.

[0121] The center of gravity of the low-resolution region and the center of gravity of the high-resolution region do not have to coincide with each other. The center of gravity of the low-resolution region and the center of gravity of the high-resolution region may be offset from the center of the light receiving surface of the sensor unit. In the optical system of this embodiment, it is sufficient that the low-resolution region is formed near the optical axis, and the high-resolution region is formed on the peripheral side away from the optical axis (outside the low-resolution region).

[0122] The optical system 611 is configured to satisfy the following formula 2, where f is the focal length, θ is the half angle of view, y is the image height on the image plane, y(θ) is a projection characteristic that indicates the relationship between the image height y and the half angle of view θ, and θmax is the maximum half angle of view that the optical system has. That is, the optical system 611 is configured so that the projection characteristic y(θ) is different from 2ftan(θ / 2) (stereoscopic projection method).

[0123] 0.2<2×f×tan(θmax / 2) / y(θmax)<0.92 (Formula 2) In an optical system with such optical characteristics, the magnification in the radial direction with respect to the optical axis can be adjusted by adjusting the projection characteristic y(θ). This allows the aspect ratio in the radial direction and circumferential direction with respect to the optical axis to be controlled, so that, unlike conventional fisheye lenses, it is possible to obtain a high-resolution image with little distortion in the peripheral area even with a wide angle of view.

[0124] Moreover, by satisfying the above formula 2, the resolution in the second region 611b can be made higher than that of the stereoscopic projection optical system. However, if the upper limit of formula 2 is exceeded, the resolution in the second region 611b becomes low, and the difference with the resolution in the first region 611a becomes small, which is not preferable.

[0125] Moreover, if the lower limit of the formula 2 is not satisfied, it becomes difficult to satisfactorily correct various aberrations such as field curvature, which is not preferable. Note that the above formula 2 is only an example, and the optical system in the second embodiment is not limited to this.

[0126] By configuring the optical system as described above, high resolution can be obtained in the high resolution region, while the increase in image height y per unit half angle of view θ can be reduced in the low resolution region, making it possible to capture a wider angle of view. Therefore, while providing an imaging range with a wide angle of view equivalent to that of a fisheye lens, high resolution can be obtained in the high resolution region.

[0127] In this embodiment, in the high-resolution region (low-distortion region), the projection characteristics are approximate to the central projection method (y=f×tanθ) and the equidistant projection method (y=f×θ), which are the projection characteristics of a general imaging optical system. Therefore, in the high-resolution region (low-distortion region), the optical distortion is small, and it is possible to generate a fine image.

[0128] <Embodiment 3> Next, a method for setting the region of interest in embodiment 3 will be described. Fig. 16 is a diagram for explaining setting of the region of interest in embodiment 3. In Fig. 16, an example will be described in which the photoelectric conversion device 600 according to embodiment 1 using the optical system 601 in which 601a is a high resolution region and 601b is a low resolution region is attached to the front or rear of a moving body 700.

[0129] In this case, the range near the optical axis in front or behind becomes a high resolution area, and the surrounding range away from the optical axis becomes a low resolution area. Fig. 16 shows an image captured by attaching a photoelectric conversion device 600 to the front of a moving body 700. The photoelectric conversion device 600 has an optical system 601 with a horizontal angle of view of 120 degrees and a vertical angle of view of 30 degrees. The number of pixels of the sensor unit 100 is 1920 pixels in the M column direction and 1080 pixels in the N row direction.

[0130] The area around the optical axis, 50 degrees to the left and right of the optical axis, is a high-resolution area, and this area is set as a first pixel area as a region of interest. The surrounding area (outside the first pixel area) farther from the optical axis is a low-resolution area, and this area is set as a second pixel area as a region of non-interest.

[0131] That is, in the third embodiment, when the resolution in the vicinity of the optical axis of the optical system is higher than the resolution on the peripheral side away from the optical axis, the control unit sets the second pixel region on the peripheral side of the light receiving surface. Note that the accumulation time in the second pixel region as a non-target region is set to the second accumulation time.

[0132] For example, when the moving body 700 moves forward or backward, the area near the optical axis captures the moving direction of the moving body 700, and it is important to grasp traffic participants and obstacles that exist in the vicinity of the moving direction in order to reduce the collision risk. In addition, traffic signals and traffic signs are also present in the vicinity of the moving direction, and it is desirable to grasp the signals and signs quickly and accurately.

[0133] Therefore, in the third embodiment, the vicinity of the optical axis is set as the first pixel region as the region of interest, and an image is acquired in the first pixel region with an accumulation period shorter than a full frame period. This makes it possible to recognize an image faster than the full frame period, obtain an image with a high S / N ratio for moving objects, and with little blurring due to motion.

[0134] On the other hand, since the risk of collision is low in the peripheral range away from the optical axis, it is sufficient to capture the image, and in many cases, images with a high S / N ratio or little blur are not required. Therefore, these areas are set as non-attention areas in the second pixel area, and an image is acquired for a relatively long accumulation period (for example, a full frame period) during the full frame period for the second pixel area.

[0135] In this manner, in this embodiment, the control unit controls the accumulation time in either the first pixel region or the second pixel region to be either the first accumulation period or the second accumulation period. That is, based on the projection characteristics of the optical system 601 of the photoelectric conversion device 600, the first pixel region and the second pixel region are appropriately set, and images are captured in different accumulation periods for each region.

[0136] This allows images to be recognized faster than the full frame period, and also enables a higher S / N ratio and reduced pixel blurring caused by motion while reducing data volume and power consumption.

[0137] <Embodiment 4> Next, another method of setting the region of interest in embodiment 4 will be described. Fig. 17 is a diagram for explaining the setting of the region of interest in embodiment 4. In embodiment 4, when an optical system similar to that in embodiment 3 is used, the settings of the region of interest (first region) and the region of non-interest (second region) are reversed.

[0138] That is, when the resolution in the vicinity of the optical axis of the optical system is higher than the resolution on the peripheral side away from the optical axis, the second pixel region is set on the center side of the light receiving surface. Thus, in the fourth embodiment, the control unit sets a pixel region on the light receiving surface having a relatively high resolution as the second pixel region, and sets the accumulation time in the second pixel region as the second accumulation time.

[0139] In the fourth embodiment, similarly to the third embodiment, it is assumed that a photoelectric conversion device 600 using the optical system 601 of the first embodiment, in which 601a is a high resolution region and 601b is a low resolution region, is attached to the front or rear of a moving body 700.

[0140] That is, the area around the optical axis, 50 degrees to the left and right, is the high resolution area, and this area is set as a non-attention area in the second pixel area.The surrounding area outside of this area, away from the optical axis, is the low resolution area, and this low resolution area is set as a attention area in the first pixel area.

[0141] In this manner, in the fourth embodiment, the second pixel region is provided inside the first pixel region, and the second pixel region is disposed so as to be adjacent to the first pixel region.

[0142] For example, when the moving body 700 moves forward or backward, the surrounding area away from the vicinity of the optical axis captures the side of the moving body 700 in the traveling direction. Since there is a high risk of collision with the moving body 700 due to something jumping out from the side in the traveling direction, it is important to grasp traffic participants and obstacles present on the side in order to reduce the collision risk.

[0143] On the other hand, when something jumps out from the side, it is sufficient to quickly detect its presence, and it is not necessary to acquire it as a high-resolution image. This is because when something jumps out from the side, it is sufficient to quickly detect whether or not there is something heading in the direction of travel of the moving body 700, and it is important to stop the object to avoid a collision, regardless of whether it is a car, a person, or an object.

[0144] Therefore, in the fourth embodiment, the periphery away from the optical axis is set as the first pixel area as the region of interest, and an image is acquired with an accumulation period shorter than a full frame period, making it possible to recognize the image earlier than the full frame period.

[0145] On the other hand, since the risk of collision is low in the range near the optical axis, it is sufficient to capture the area in the accumulation period of the full frame period. Therefore, by setting these areas as non-target areas in the second pixel area and acquiring images in the full frame period, it is possible to reduce the amount of data in the captured image.

[0146] In this manner, also in the fourth embodiment, the control unit controls the accumulation time in either the first pixel region or the second pixel region to be either the first accumulation period or the second accumulation period. That is, the first pixel region and the second pixel region are appropriately set based on the projection characteristics of the optical system 601 of the photoelectric conversion device 600, and images are captured in different accumulation periods for each region. This makes it possible to recognize an image earlier than the full frame period, and reduce the amount of data and power consumption.

[0147] <Embodiment 5> A description will now be given of embodiment 5. In embodiment 5, a method for setting an attention area when an optical system having characteristics as shown in FIG.

[0148] 15, a photoelectric conversion device 610 using an optical system 611 in which 611a is a low-resolution region and 611b is a high-resolution region is assumed to be attached to the side of a moving body 700. In this case, the peripheral range away from the vicinity of the optical axis on the side becomes the high-resolution region, and the range near the optical axis becomes the low-resolution region.

[0149] 18 is a diagram for explaining the setting of the attention area in the fifth embodiment, and shows, for example, an image captured by attaching a photoelectric conversion device 610 to the side of a moving body 700. The photoelectric conversion device 610 has an optical system 611 with a horizontal angle of view of, for example, 180 degrees and a vertical angle of view of 30 degrees. The number of pixels of the sensor unit 100 is 1920 pixels in the M column direction and 1080 pixels in the N row direction.

[0150] The area around the optical axis, 120 degrees to the left and right of the optical axis, is a low-resolution area, and this area is set as a non-target area in the second pixel area. The surrounding area outside of this, away from the optical axis, is a high-resolution area, and this area is set as a target area in the first pixel area.

[0151] For example, when the moving body 700 is moving forward, the peripheral range away from the vicinity of the optical axis captures the front and rear sides in the moving direction of the moving body 700. This means that on the front and rear sides in the moving direction, other moving bodies moving in the same direction as the moving body 700 and other moving bodies moving in the opposite direction to the moving body 700 are captured.

[0152] When the moving body 700 changes lanes, it is necessary to quickly determine the presence of another moving body traveling in the same direction as the moving body 700 on the front or rear side of the moving body's direction of travel and evaluate the risk of collision.

[0153] In addition, it is necessary to quickly determine whether another moving body traveling in the opposite direction to the moving body 700 is entering the traveling direction of the moving body 700 on the forward side in the traveling direction, and to evaluate the risk of collision. In this way, it is important to grasp traffic participants and obstacles that exist on the forward and rear sides in the traveling direction of the moving body 700 in order to reduce the collision risk. In addition, traffic signals and traffic signs are also present on the forward side, and it is necessary to quickly and accurately grasp the traffic signals and signs.

[0154] Therefore, in the fifth embodiment, the periphery away from the optical axis is set as the first pixel region as the region of interest, and an image is acquired with an accumulation period shorter than a full frame period. This makes it possible to recognize an image faster than the full frame period, obtain an image with a high S / N ratio for moving objects, and with little blurring due to motion.

[0155] On the other hand, for the area near the optical axis, the risk of collision is low when traveling straight, so capturing an image in the accumulation period of the full frame period is sufficient. Therefore, by setting these areas as non-attention areas in the second pixel area and acquiring an image in the full frame period, it is possible to reduce the amount of data in the captured image.

[0156] In this manner, also in the fifth embodiment, the control unit controls the accumulation time in either the first pixel region or the second pixel region to be either the first accumulation period or the second accumulation period. That is, based on the projection characteristics of the optical system 611 of the photoelectric conversion device 610, the first pixel region and the second pixel region are appropriately set, and images are captured in different accumulation periods for each region.

[0157] This makes it possible to recognize an image faster than the full frame period, achieve a high S / N ratio, and reduce pixel blurring caused by motion while reducing data volume and power consumption.

[0158] <Embodiment 6> Next, a sixth embodiment will be described. In the sixth embodiment, when an optical system similar to that of the fifth embodiment is used, the settings of the target area (first area) and the non-target area (second area) are reversed.

[0159] 19 is a diagram for explaining setting of the region of interest in embodiment 6. A photoelectric conversion device 610 using an optical system 611 in which 611a is a low resolution region and 611b is a high resolution region is attached to the side of a moving body 700. In this case, the peripheral range away from the vicinity of the optical axis on the side becomes the high resolution region, and the range near the optical axis becomes the low resolution region.

[0160] In the photoelectric conversion device 610 according to the sixth embodiment, the horizontal angle of view of the optical system 611 is, for example, 180 degrees and the vertical angle of view is 30 degrees, and the number of pixels of the sensor unit 100 is 1920 pixels in the M column direction and 1080 pixels in the N row direction.

[0161] The area around the optical axis, 120 degrees to the left and right of the optical axis, is a low-resolution area, and this area is set as a first pixel area as a region of interest. The surrounding area outside of this area, away from the optical axis, is a high-resolution area, and this area is set as a second pixel area as a region of non-interest.

[0162] For example, when the moving body 700 is moving forward, the range near the optical axis captures the side of the moving body 700 in the moving direction. As a result, other moving bodies moving in the same direction as the moving body 700 and other moving bodies moving in the opposite direction to the moving body 700 are captured on the side.

[0163] In particular, when the moving body 700 is an automobile, there are cases where small moving bodies such as motorcycles and bicycles, and pedestrians are photographed. There are cases where it is necessary to quickly determine the presence of these small moving bodies and pedestrians, etc., and evaluate the risk of collision, particularly the risk of collision due to being caught in the vehicle when turning right or left.

[0164] In this way, it is important to reduce the risk of collision to recognize obstacles such as motorcycles that exist on the side of the moving body 700. Therefore, in the sixth embodiment, the vicinity of the optical axis is set as the first pixel region as the region of interest, and an image is acquired with an accumulation period shorter than the full frame period, thereby making it possible to acquire an image that can be recognized earlier than the full frame period.

[0165] On the other hand, for the peripheral range away from the optical axis, it is sufficient to capture the image in the accumulation period of the full frame period, especially when turning right or left. Therefore, by setting these areas as non-target areas in the second pixel area and acquiring an image in the full frame period, it is possible to reduce the amount of data in the captured image.

[0166] In this manner, in the sixth embodiment, the first pixel region and the second pixel region are appropriately set based on the projection characteristics of the optical system 611 of the photoelectric conversion device 610, and images are captured with different accumulation periods for each region. This makes it possible to recognize an image faster than the full frame period, achieve a high S / N ratio, reduce pixel blurring due to motion, and reduce the amount of data and power consumption.

[0167] In the above embodiment, an example has been described in which the attention area and the non-attention area are set according to the optical characteristics of the optical system 601 and the optical system 611, but in these embodiments, the ranges of the attention area and the non-attention area may be changed according to, for example, the speed of the moving body 700. This is because it is preferable to vary the range in which the collision risk is evaluated depending on the speed, etc., of the moving body.

[0168] That is, the attention area and the non-attention area may be switched according to the environment information including the information of the moving object 700 obtained from the moving control unit (ECU) of the moving object. Alternatively, the attention area may be enlarged or reduced according to the environment information including the information of the moving object 700.

[0169] That is, as described above, the positions of the attention area and the non-attention area are changed, switched, or the sizes are changed according to the output of the detection unit 602. At least one of the positions and the sizes of the first pixel area and the second pixel area may be set by the user.

[0170] The information on the moving body as one type of environmental information includes at least one of vehicle speed information, acceleration information, steering angle information, brake information, engine information, etc. The environmental information also includes at least one of other information such as the presence, position, speed, acceleration, and distance of traffic participants (pedestrians, motorbikes, bicycles, etc. around the moving body), map information, GPS information, road conditions, road surface conditions, weather information, ambient brightness, and time.

[0171] In this way, the control unit may set the first pixel region or the second pixel region based on at least one of characteristic information on the characteristics of the photoelectric conversion device, installation information on the installation state of the photoelectric conversion device, and environmental information on the surrounding environment of the photoelectric conversion device. Furthermore, at least one of the position, size, and accumulation period of the first pixel region or the second pixel region may be set based on at least one of the characteristic information, installation information, and environmental information. Also, the first pixel region or the second pixel region may be set according to distance information of the subject.

[0172] The characteristic information includes optical characteristic information of the optical system, the resolution of the sensor unit, and the pixel size of the sensor unit, and the installation information includes at least one of the height, angle, and direction at which the photoelectric conversion device is installed.

[0173] For example, as the speed of the moving body 700 increases, the collision risk can be further reduced by setting the attention area to the front and expanding the attention area in front. Also, for example, when traveling at high speed on a highway or the like, the collision risk due to a vehicle jumping out from the side is reduced, so the attention area on the side may be reduced.

[0174] Also, the attention area and the non-attention area may be switched depending on whether the vehicle is going straight or turning right or left. For example, when going straight, the surroundings in the traveling direction may be set as the attention area. When turning right or left, the side of the turning direction may be set as the attention area in order to reduce the risk of collision due to entrapment. That is, the control unit may set the first pixel area or the second pixel area based on the output of the ECU as a movement control unit.

[0175] In this manner, also in the sixth embodiment, the control unit controls the accumulation time in one of the first pixel region and the second pixel region to be either the first accumulation period or the second accumulation period. The first pixel region may be controlled to have at least the first accumulation period and the second accumulation period. The second pixel region may be controlled to have at least the first accumulation period and the second accumulation period.

[0176] <Embodiment 7> A seventh embodiment will be described below. In the seventh embodiment, a photoelectric conversion device 600 using an optical system 601 in which 601a is a high resolution region and 601b is a low resolution region is attached to a production line or an inspection line as a camera for checking processes or inspecting products in a factory or the like.

[0177] In addition, during manufacturing and inspection, in order to evaluate whether the target parts, units, etc. have been manufactured correctly, the target parts, units, etc. are placed so that they flow within the range near the optical axis. Therefore, the range near the optical axis is set as a high-resolution area, and the surrounding range away from the optical axis is set as a low-resolution area.

[0178] For example, cameras are installed near the optical axis to enable evaluation of target parts and units, allowing for more accurate image capture. On the other hand, cameras away from the optical axis capture images of workers performing their work, making it possible to check whether workers are present, what their work status is, and prevent injuries.

[0179] In the seventh embodiment, the vicinity of the optical axis is set as the first pixel region as the region of interest, and an image is acquired in an accumulation period shorter than a full frame period. This makes it possible to recognize an image faster than the full frame period, increase the S / N ratio of an image of an object moving on a belt conveyor, for example, and acquire an image with less blurring due to motion.

[0180] On the other hand, it is sufficient to capture the peripheral range away from the optical axis in order to check the presence or absence of workers, check the work status, prevent injuries, etc. Therefore, by setting these areas as non-attention areas in the second pixel area and acquiring images during the full frame period, it is possible to reduce the amount of data in the captured image.

[0181] In this way, the first pixel region and the second pixel region are appropriately set based on the projection characteristics of the optical system 601 of the photoelectric conversion device 600, and images are captured with different accumulation periods for each region. This makes it possible to recognize an image faster than the full frame period, achieve a high S / N ratio, reduce pixel blurring due to motion, and reduce data volume and power consumption.

[0182] <Embodiment 8> An eighth embodiment will now be described. In the eighth embodiment, a photoelectric conversion device 600 using an optical system 601 in which 601a is a high-resolution region and 601b is a low-resolution region is installed in a logistics line as a logistics camera.

[0183] It is necessary to read and judge the labels on packages and other items that come along the line. In such cases, logistics cameras may be required to perform three evaluations: determining whether or not a package is present, determining the position of the label, and reading the label.

[0184] However, a logistics camera that satisfies all of these requirements simultaneously requires high performance, which makes it expensive. Therefore, in the eighth embodiment, a photoelectric conversion device 600 using an optical system 601 is used as the logistics camera.

[0185] For example, the presence or absence of luggage and the label position are grasped in a peripheral range away from the vicinity of the optical axis, and the label is read in a range near the optical axis. That is, the range near the optical axis where the label is read is a high-resolution area, while the range away from the vicinity of the optical axis where the presence or absence of luggage and the label position are grasped is a low-resolution area.

[0186] In the eighth embodiment, since it is required to quickly recognize when and what shape an object is passing in the peripheral range away from the optical axis where the presence or absence of baggage and the position of the label are detected, images are acquired with an accumulation period shorter than a full frame period, which allows images to be recognized faster than the full frame period.

[0187] On the other hand, the area near the optical axis where the label is read is set as a non-target area in the second pixel area, and an image is acquired during the full frame period, making it possible to reduce the amount of data in the captured image.

[0188] In this way, by appropriately setting the first pixel region and the second pixel region based on the projection characteristics of the optical system 601 of the photoelectric conversion device 600 and capturing images with different accumulation periods for each region, it is possible to recognize an image faster than the full frame period. Furthermore, it is possible to reduce the amount of data and power consumption while achieving a high S / N ratio and reducing pixel blurring due to motion.

[0189] <Embodiment 9> Next, a description will be given of a ninth embodiment. In the ninth embodiment, a photoelectric conversion device 610 using an optical system 611 in which 611a is a low-resolution region and 611b is a high-resolution region is used as a surveillance camera.

[0190] To assess whether a suspicious person or object is entering a building or a specific area, the camera is installed so that it can capture the surveillance area. Therefore, the area near the optical axis, which is the center of the surveillance area, is set as a low-resolution area, and the surrounding area away from the optical axis, which is the periphery of the surveillance area, is set as a high-resolution area. For example, it is important for a surveillance camera to immediately detect the intrusion of a suspicious person or object from outside the surveillance area, so it is important that it can quickly capture high-quality images of the surrounding area away from the optical axis.

[0191] Therefore, in the ninth embodiment, the periphery away from the optical axis is set as the first pixel region as the region of interest, and an image is acquired in the first pixel region with an accumulation period shorter than a full frame period. This makes it possible to recognize an image faster than the full frame period, obtain an image with a high S / N ratio for moving objects, and with little blurring due to motion.

[0192] On the other hand, while it is necessary to monitor the area near the optical axis for suspicious people and objects, it is often more important to detect them as soon as they enter the area. Therefore, by setting the area near the optical axis as a non-target area in the second pixel area and acquiring an image during the full frame period, it is possible to reduce the amount of data in the captured image.

[0193] In this way, by appropriately setting the first pixel region and the second pixel region based on the projection characteristics of the optical system 611 of the photoelectric conversion device 610 and capturing images with different accumulation periods for each region, it is possible to recognize an image faster than the full frame period. Furthermore, it is possible to reduce the amount of data and power consumption while achieving a high S / N ratio and reducing pixel blurring due to motion.

[0194] <Embodiment 10> A description will be given of embodiment 10. In embodiment 10, a photoelectric conversion device 610 using an optical system 611 in which 611a is a low resolution region and 611b is a high resolution region is used as an agricultural camera.

[0195] The agricultural camera according to the tenth embodiment is used to grasp the status of agricultural crops. Crops may be grown in multiple rows, with ridges, etc. Therefore, as an agricultural camera, one camera is used to capture multiple rows, and the surrounding area away from the optical axis of the capture range is set as a high-resolution area.

[0196] On the other hand, in order to prevent damage to crops by animals, etc., the area near the optical axis, which is the center of the monitoring range and corresponds to the spaces between the rows of crops, may be a low-resolution area. Also, since it is important to quickly detect the intrusion of animals, etc., in order to prevent damage to crops, it is necessary to make it possible to quickly capture images of the area near the optical axis.

[0197] Therefore, in the tenth embodiment, the vicinity of the optical axis is set as the first pixel region as the region of interest, and an image is acquired in the first pixel region with an accumulation period shorter than a full frame period, making it possible to recognize the image earlier than the full frame period.

[0198] On the other hand, for the peripheral area away from the optical axis, where it is important to capture the condition and growth of the crops, a high resolution image is used, but this is set as a second pixel area as a non-interest area, and images are acquired during the full frame period, which makes it possible to reduce the amount of data in the captured image.

[0199] In this way, by appropriately setting the first pixel region and the second pixel region based on the projection characteristics of the optical system 611 of the photoelectric conversion device 610 and capturing images with different accumulation periods for each region, it is possible to recognize an image faster than the full frame period. Furthermore, it is possible to reduce the amount of data and power consumption.

[0200] <Embodiment 11> Fig. 20 is a functional block diagram showing a configuration example of a photoelectric conversion device 600 and a moving object 700 according to embodiment 11. The photoelectric conversion device 600 of embodiment 11 is mounted on the moving object 700. In Fig. 20, the same reference numerals as in Fig. 6 denote the same blocks, and therefore description thereof will be omitted.

[0201] In the eleventh embodiment, the output of the image processing unit 603 is supplied to a recognition unit 604, a control unit 605, and an ECU (Electric Control Unit) 701 serving as a movement control unit of a moving object 700. The recognition unit 604 recognizes surrounding people, vehicles, objects, etc. by performing image recognition based on the image signal, and outputs the recognition result to the ECU 701.

[0202] The control unit 605 also acquires information such as the installation height and installation angle as installation information of the photoelectric conversion device 600 on the moving object 700 from the ECU 701. Furthermore, the control unit 605 acquires environmental information of the photoelectric conversion device 600 from the detection unit 602, and acquires information on the moving object 700 and environmental information from the ECU 701.

[0203] The information on the moving object acquired from the ECU 701 includes at least one of vehicle speed information, acceleration information, steering angle information, brake information, engine information, etc. The environmental information includes at least one of information on the presence, position, speed, acceleration, distance, etc. of traffic participants (pedestrians, motorbikes, bicycles, etc. around the moving object), map information, GPS information, road conditions, road surface conditions, weather information, surrounding brightness, time, etc.

[0204] Then, based on the acquired information, the CPU executes a computer program stored in a memory built into the control unit 605 to control each unit of the photoelectric conversion device 600 .

[0205] The ECU 701 incorporates a CPU as a computer and a memory that stores a computer program, and controls each part of the moving body 700 by the CPU executing the computer program stored in the memory.

[0206] The output of the ECU 701 is also supplied to a vehicle control unit 702 and a display unit 703. The vehicle control unit 702 functions as a driving control unit that performs operations such as driving, stopping, and directional control of the vehicle as a moving body based on the output of the ECU 701.

[0207] The display unit 703 includes a display element such as a liquid crystal device or an organic EL device, and is mounted on the moving body 700. The display unit 703 displays images acquired by a sensor unit and various information related to the vehicle's running state and the like to the driver of the moving body 700 using, for example, a GUI based on the output of the ECU 701.

[0208] In addition, the image processing unit 603, the recognition unit 604, etc. in Figure 20 do not have to be mounted on the moving body 700, and may be provided, for example, in an external terminal provided separately from the moving body 700 for remotely controlling the moving body 700 or monitoring the movement of the moving body.

[0209] Fig. 21 is a flowchart for explaining a method for controlling a moving object according to embodiment 11. Note that the control unit 605 and a CPU serving as a computer in the ECU 701 execute a computer program stored in the memory, thereby sequentially performing the operations of the steps in the flowchart in Fig. 21.

[0210] First, in step S211, the CPU of the control unit 605 acquires video data from the photoelectric conversion device 600 during operation. Next, in step S212, the CPU of the control unit 605 recognizes an object by performing image recognition based on a signal read from a pixel region of the sensor unit 100 of the photoelectric conversion device 600.

[0211] Then, in step S213, the CPU of the control unit 605 acquires the position and speed of the recognized object. This may be acquired by observing changes in the object in multiple frames of video data, or by using a millimeter wave radar or LiDAR (Light Detection and Ranging) as another distance measuring device.

[0212] Next, in step S214, the CPU of the ECU 701 acquires the traveling direction of the object and the distance to the object. As in the previous example, this may be acquired by observing changes in the object in multiple frames of video data, or by using a millimeter wave radar or LiDAR as a separate distance measuring device.

[0213] Then, in step S215, the CPU of the ECU 701 calculates the braking distance of the vehicle. Note that the braking distance varies depending not only on the speed of the moving body 700 but also on the inclination angle of the road surface, the condition of the road surface (asphalt, dirt, gravel, rain, snow, ice, etc.), the degree of tire wear (characteristics of the tire itself, distance used, etc.), and the weight of the moving body 700. Therefore, such information may be used comprehensively to make the determination.

[0214] Next, in step S216, the CPU of the ECU 701 evaluates the risk of collision with the target person. At that time, the evaluation is performed taking into consideration factors such as the speed, traveling direction, distance, etc. Then, in step S217, if the collision risk is high, the CPU of the ECU 701 generates a warning signal and warns the driver via the display unit 703 or audio.

[0215] Furthermore, an automatic brake system is activated as necessary. Also, if it is determined in step S216 that the collision risk is not high, the process returns to step S211 and the next frame of video data is obtained.

[0216] After step S217, the process proceeds to step S218, where the CPU of the ECU 701 determines whether the user has turned off the power of the moving object or performed an operation to end the flow of Fig. 21. If the result is Yes, the process ends. If the result is No in step S218, the process returns to step S211, and the operation of Fig. 21 is repeated.

[0217] <Embodiment 12> Next, Fig. 22 is a flowchart showing details of an example of driving the sensor unit 100 regarding a method for setting an attention area according to the twelfth embodiment, and Fig. 23 is a flowchart continuing from Fig. 22. Note that the CPU as a computer in the control unit 605 executes a computer program stored in the memory, thereby sequentially performing the operations of the steps in the flowcharts of Figs. 22 and 23.

[0218] In step S2201, the CPU of the control unit 605 of the photoelectric conversion device 600 sets a region of interest and a region of non-interest within the shooting region in accordance with the optical characteristics of the optical system 601. In this embodiment, a case where there is one region of interest and one region of non-interest will be described. For the sake of explanation, the accumulation period of the first pixel region, which is the region of interest, is divided into four parts of one full frame period, and the accumulation period of the second pixel region, which is the region of non-interest, is not divided and is one full frame period.

[0219] Here, the first pixel area starts from the pixel in row j1, column k1 and extends to row N1, column M1, and imaging is performed over four different accumulation periods, whereas the second pixel area starts from the pixel in row j2, column k2 and extends to row N2, column M2, and imaging is performed over one accumulation period.

[0220] Next, in step S2202, the CPU of the control unit determines whether it is the target area. If Yes, it proceeds to step S2211 to perform the process of determining the accumulation period. On the other hand, if it is determined No in step S2202, it proceeds to step S2301.

[0221] Since the target area is divided into four parts, i starts from 1 and repeats up to 4. Therefore, in step S2211, the CPU of the control unit first sets the value of i to 1. Next, in step S2212, the CPU of the control unit outputs the count value Count of counter 211 at time Ti to memory 212. At this time, the output is performed simultaneously for all memories. This operation corresponds to the operation at time T1 in FIG. 7.

[0222] Next, in step S2213, the CPU of the control unit sets j = j1. j1 is the starting position of the Nth row of the target area. Next, in step S2214, the CPU of the control unit sets k = k1. k1 is the starting position of the Mth column of the target area.

[0223] In step S2215, the CPU of the control unit outputs the count value Count(j - k - i) in memory j - k of FIG. 9 to the buffer. At this time, the output is performed for columns 1 to M simultaneously to the buffer. This operation means the operation of taking in the count value of the first row in FIG. 9 into the buffer.

[0224] In step S2216, the CPU of the control unit outputs the count value Count(j - k - i) of buffer k to output circuit 114. This operation corresponds to the operation of reading out the signal of the buffer in the leftmost column in FIG. 9 from the output circuit.

[0225] Next, it proceeds to step S2217 in FIG. 23 via E. In step S2217, the CPU of the control unit determines whether k < M1. If Yes, in step S2218, k is incremented by 1 as k = k + 1, and it returns to step S2216 via F. M1 is the end position of the Mth column of the target area. This operation corresponds to the operation of reading out the signal of the buffer in the second column from the left in FIG. 9 from the output circuit.

[0226] If it is determined as No in step S2217, that is, if k = M1, it means that the signal of the buffer in the M1 column of FIG. 9 has been read out from the output circuit. Next, it proceeds to step S2219, and the CPU of the control unit determines whether j < N1. N1 is the end position of the Nth row of the target area. If it is determined as Yes in step S2219, then in step S2220, the CPU of the control unit sets j = j + 1, increments j by 1, and returns to step S2214 via G. This corresponds to the operation for starting the reading of the next row.

[0227] If it is determined as No in step S2219, it means that the reading of all rows has been completed. So, it proceeds to step S2221, and the CPU of the control unit determines whether j < 4. If it is determined as Yes in step S2221, then the CPU of the control unit sets i = i + 1, increments i by 1, and returns to step S2212 via H. This operation corresponds to the operation for starting the reading at the next time T2.

[0228] If it is determined as No in step S2221, it means that the reading at time T4 has been completed. So, it proceeds to step S2223, and the CPU of the control unit resets the counter 211 with a reset signal. This operation corresponds to the reset operation of the counter 211 at time T4 in FIG. 7. After step S2223, it returns to step S2201 via M. In the above manner, the signals accumulated in the sensor unit can be sequentially read out.

[0229] On the other hand, since the non-target area does not divide one full frame period, only the flow when i = 4 is looped. Therefore, in step S2301, the CPU of the control unit sets i = 4. Next, in step S2302, the CPU of the control unit outputs the count value Count of the counter 211 at time Ti to the memory 212. At this time, the output is performed simultaneously for all memories. This operation corresponds to the operation at time T1 in FIG. 7.

[0230] Next, in step S2303, the CPU of the control unit sets j = j2. j2 is the starting position of the N rows of the non-target area. Next, in step S2304, the CPU of the control unit sets k = k2. k2 is the starting position of the M columns of the non-target area.

[0231] In step S2305, the CPU of the control unit outputs the count value Count(j - k - i) in the memory j - k in FIG. 9 to the buffer. At this time, it outputs to the buffer simultaneously for columns 1 to M. This operation means the operation of taking in the count value of the first row in FIG. 9 into the buffer.

[0232] In step S2306, the CPU of the control unit outputs the count value Count(j - k - i) of buffer k to the output circuit 114. This operation corresponds to the operation of reading out the signal of the buffer in the leftmost column in FIG. 9 from the output circuit.

[0233] Next, it proceeds to step S2307 in FIG. 23 via I. In step S2307, the CPU of the control unit determines whether k < M2. If it is Yes in step S2307, then in step S2308, the CPU of the control unit increments k by 1 as k = k + 1 and returns to step S2306 via J. M2 is the ending position of the M columns of the non-target area. This operation corresponds to the operation of reading out the signal of the buffer in the second column from the left in FIG. 9 from the output circuit.

[0234] When it becomes No in step S2307, that is, when k = M2, it means that the signal of the buffer in the M2nd column in FIG. 9 has been read out from the output circuit. Next, it proceeds to step S2309, and the CPU of the control unit determines whether j < N2.

[0235] N2 is the ending position of the N rows of the non-target area. If it is Yes in step S2309, then in step S2310, the CPU of the control unit increments j by 1 as j = J + 1 and returns to step S2304 via K. This corresponds to the operation for starting to read the next row.

[0236] If the determination in step S2309 is No, this means that reading of all rows has been completed, so the process proceeds to step S2311, where the control CPU determines whether j<4. If the determination in step S2311 is Yes, the control CPU increments i by 1, setting i=i+1, and returns to step S2302 via L. This operation corresponds to the operation of starting reading at the next time T2. However, since i=4 was set in step S2301, the result in step S2311 is No.

[0237] If the determination in step S2311 is No, this means that the readout at time T4 is completed, so the process proceeds to step S2313, where the CPU of the control unit resets counter 211 with a reset signal. This operation corresponds to the reset operation of counter 211 at time T4 in Fig. 7. After step S2313, the process returns to step S2201 via M. In this manner, the signals accumulated in the sensor unit can be read out sequentially.

[0238] Therefore, the region of interest starts from the pixel in row j1, column k1 and repeats through row N1, column M1, and is imaged over four different accumulation periods, whereas the region of non-interest starts from the pixel in row j2, column k2 and repeats through row N2, column M2, and is imaged over only one accumulation period.

[0239] Next, a description will be given of the processing of images acquired in the region of interest and the region of non-interest in the photoelectric conversion device 600. The method of acquiring an image of the region of interest is based on the method shown in Fig. 22 and Fig. 23. Also, the region of interest acquires an image in four accumulation periods of frames 1_1, 1_2, 1_3, and 1_4 in Fig. 7. On the other hand, the region of non-interest acquires an image in the accumulation period of only frame 1_4 in Fig. 7.

[0240] An example of an image created according to the flows of Fig. 22 and Fig. 23 is shown in Fig. 24. Fig. 24 is a diagram showing an example of an image for each of a plurality of accumulation periods, with an attention area being set. In each diagram, the range on the lower side capturing a close distance is the attention area, and the range on the upper side capturing a long distance is the non-attention area, and the car on the left is stopped, and the car on the right is moving from right to left at a certain speed.

[0241] As shown in Fig. 24, the area of ​​interest at the bottom of the screen is updated in four stages from frame 1_1 to 1_4. Therefore, a stopped car's image becomes gradually brighter as the accumulation period gets longer from frame 1_1 to 1_4. Also, a moving car's image becomes blurred in the forward and backward directions of its travel as the accumulation period gets longer from frame 1_1 to 1_4, and the brightness of the overlapping parts of the movement gets higher.

[0242] On the other hand, the non-attention area, which is the upper part of the screen, is updated only by the output of frame 1_4. Therefore, a stopped car is updated every full frame and photographed during the accumulation period of the full frame period, resulting in an image with high brightness. Similarly, a moving car is updated every full frame and photographed during the accumulation period of the full frame period, so blurring occurs in the front and rear of the moving direction, and the brightness of the part where the movements overlap is high.

[0243] Therefore, in the non-attention area, the images of frames 1_1 to 1_3 are all images captured during the accumulation period of frame 1_4 in the previous full frame period. Also, frame 1_4 is updated to an image captured during the accumulation period of frame 1_4 in the current full frame period.

[0244] 22 and 23, the area of ​​interest is updated four times in a full frame period, making it possible to acquire an image in a shorter time than the full frame period and perform faster recognition. On the other hand, the area of ​​non-interest is updated only once in a full frame period, making it possible to reduce the amount of data and power consumption.

[0245] Based on the image data created according to the flowcharts of FIGS. 22 and 23, subsequent processing is performed in accordance with the characteristics of the photoelectric conversion device 600.

[0246] For example, in the case of a security camera, the control unit 605 monitors, photographs, and records the behavior of a target person. In some cases, it is possible to display a warning, turn on a light installed in the security camera, or emit a warning sound via the communication unit 607.

[0247] In the case of an in-vehicle camera, the control unit 605 can issue a warning or display a warning via the ECU, and the vehicle control unit 702 can activate safety devices to slow down or stop the vehicle, making it possible to reduce or avoid a collision.

[0248] In addition, for example, by using the camera to recognize people or registered faces, it is possible to capture images with an appropriate accumulation time when the person is within the shooting area. Also, as a pet camera, it is possible to recognize pets, remotely check on their condition, and take pictures of them.

[0249] In addition, when used as a surveillance camera, in places where safety management and surveillance are required (e.g. parking lots, public facilities, factories, etc.), specific locations or areas can be set as areas of interest in order to grasp the situation, monitor and prevent fraudulent activities, collect evidence as records, etc. This makes it possible to monitor the areas of interest more frequently and record and display video and audio.

[0250] In addition, when used as a defect detection camera in a factory, it can be used to support quality control and efficiency of the manufacturing process, and to detect and remove defective products early, troubleshoot production lines, and improve quality, etc. In that case, it becomes possible to detect and record product defects and abnormalities with greater accuracy on the factory production line or at the work site.

[0251] In addition, when used as a packaging inspection camera in a factory, it will be possible to inspect the product packaging condition, label accuracy, and defects in order to strengthen product quality control and ensure the integrity of the packaging and accuracy of the label.

[0252] When used as a sorting camera for logistics, it can improve the efficiency and accuracy of logistics operations in logistics warehouses and distribution centers, and realize proper product management and a quick delivery process. It can also recognize the shape, size, barcode, etc. of products and packages, enabling accurate sorting and delivery processing.

[0253] When used as an endoscopic camera, the camera can be incorporated into endoscopes used in the medical field to obtain high-quality images of the inside of the body and organs during endoscopic surgery and diagnosis, allowing for more accurate observation of conditions inside the body and helping to diagnose diseases and develop treatment plans.

[0254] When used as a camera to detect the need for care, it will be possible to monitor the living conditions of elderly people and those requiring care and detect abnormal or dangerous situations in order to provide early detection and support for falls, abnormal behavior, and emergency situations, ensuring the safety and lifestyle support of those in need of care.

[0255] When used as infrastructure surveillance cameras, they can monitor infrastructure such as roads, bridges, railways, and power plants to maintain safety and reliability, and detect abnormalities, damage, and unauthorized activity in order to provide early warning and proper maintenance.

[0256] In addition, when used as infrastructure surveillance cameras, they can monitor public places, facilities, and specific areas to ensure public safety, deter crime, prevent fraud, and collect evidence for criminal investigations, making it possible to prevent, monitor, and record fraudulent and criminal activities.

[0257] In addition, when used as an agricultural camera, it can support appropriate cultivation management, early detection of pests, and effective agricultural measures to improve the productivity and quality of agricultural crops. In that case, it will be possible to monitor the growth status of agricultural crops and the occurrence of pests and diseases, and provide video data that is useful for agricultural production management.

[0258] <Embodiment 13> Next, a thirteenth embodiment will be described. In the thirteenth embodiment, a method for setting the accumulation period of the region of interest according to the brightness of the region of interest, etc. will be described.

[0259] It has been explained that for surveillance cameras and in-vehicle cameras attached to moving objects, it is effective to capture images of close-up subjects or subjects on the side of a moving object with a short accumulation period at an early timing. However, it is known that a short accumulation period increases the ratio of noise in the signal component, deteriorating the so-called S / N ratio.

[0260] This is particularly noticeable when shooting a dark area. Therefore, if the area of ​​interest is dark, even if an image is captured with a short accumulation period, the image will have a poor S / N ratio, making it difficult to display appropriately or use for subsequent processing.

[0261] Therefore, by performing unnecessary processing, appropriate judgment cannot be performed and power consumption of the device increases, which is a problem. Therefore, in the thirteenth embodiment, a method of capturing an image with a good S / N ratio by adjusting the accumulation period according to the brightness of the region of interest will be described.

[0262] Fig. 25 is a flowchart showing details of an example in which an accumulation period is set and the sensor unit 100 is driven in the thirteenth embodiment, and Fig. 26 is a flowchart continuing from Fig. 25. Note that the CPU as a computer in the control unit 605 executes a computer program stored in the memory, thereby sequentially performing the operations of the steps in the flowcharts of Figs. 25 and 26.

[0263] First, in step S2501, the CPU of the control unit sets an area of ​​interest based on at least one of product information, installation information, and environmental information of the photoelectric conversion device 600 recorded in the control unit 605 of the photoelectric conversion device 600, as in the previous embodiments.

[0264] Next, in step S2502, the CPU of the control unit judges whether the area is a region of interest or a non-region of interest. If it is a region of interest, the process proceeds to step S2511 in FIG. 19, where the CPU of the control unit performs a process of determining an accumulation period. On the other hand, if it is a region of non-interest, the process proceeds to step S2531.

[0265] Next, in step S2503, the CPU of the control unit determines the accumulation period of the region of interest. In this embodiment, for example, the average luminance value of the image acquired in the previous frame within the region of interest is calculated by the control unit 605, and a look-up table (LUT) or a calculation formula for the accumulation period corresponding to the luminance value is recorded in the memory of the control unit 605, and is used to determine the accumulation period.

[0266] As described above, in this embodiment, the length of at least one of the first and second accumulation periods is set based on the luminance information of a signal generated during at least one of the first and second accumulation periods.

[0267] The luminance value may be calculated based on the peak of a histogram, the deviation of the peak, the spread, or the variance of the peak, instead of the average luminance. The luminance difference between adjacent pixels in the screen may be calculated, and the noise amount may be calculated from the histogram of the difference, and the accumulation period may be calculated from the result. Furthermore, the luminance value may be calculated based on the average value of the previous frames, instead of the previous frame, or may be calculated based on the average luminance or histogram for each accumulation period in the frame.

[0268] In this embodiment, flowcharts for the case where the accumulation period is four are shown in FIGS. 25 and 26. However, the accumulation period is not limited to four, and it may be one, three, or four or more. Then, in step S25011, the CPU of the control unit sets the required accumulation period (Ti).

[0269] In step S2511 of FIG. 25, the CPU of the control unit sets i = 1. Next, in step S2512, the CPU of the control unit outputs the count value Count of counter 211 at time Ti to memory 212. At this time, the output is performed simultaneously for all memories. This operation corresponds to the operation at time T1 in FIG. 7.

[0270] Next, in step S2513, the CPU of the control unit sets j = j1. j1 is the start position of the Nth row of the region of interest. Next, in step S2514, the CPU of the control unit sets k = k1. k1 is the start position of the Mth column of the region of interest.

[0271] In step S2515, the CPU of the control unit outputs the count value Count(j - k - i) in memory j - k of FIG. 9 to the buffer. At this time, the output is performed for columns 1 to M simultaneously. This operation means the operation of taking in the count value of the first row in FIG. 9 into the buffer.

[0272] In step S2516, the CPU of the control unit outputs the count value Count(j - k - i) of buffer k to output circuit 114. This operation corresponds to the operation of reading the signal of the buffer in the leftmost column in FIG. 9 from the output circuit.

[0273] Next, proceeding to step S2517 of FIG. 26 via E, in step S2517, the CPU of the control unit determines whether k < M1. If Yes, in step S2518, the CPU of the control unit increments k by 1 as k = k + 1 and returns to step S2516 via F. M1 is the end position of the Mth column of the region of interest. This operation corresponds to the operation of reading the signal of the buffer in the second column from the left in FIG. 9 from the output circuit.

[0274] If the result in step S2517 is No, that is, if k = M1, it means that the signal of the buffer in the M-th column in FIG. 9 has been read out from the output circuit. Next, it proceeds to step S2519, and the CPU of the control unit determines whether j < N1.

[0275] N1 is the end position of the N-th row in the attention area. If the result in step S2519 is Yes, then in step S2520, the CPU of the control unit sets j = j + 1, increments j by 1, and returns to step S2514 via G. This corresponds to the operation for starting the reading of the next row.

[0276] If the determination in step S2519 is No, it means that the reading of all rows has been completed. So, it proceeds to step S2521, and the CPU of the control unit determines whether i < 4. If the determination in step S2521 is Yes, then the CPU of the control unit sets i = i + 1, increments i by 1, and returns to step S2512 via H. This operation corresponds to the operation for starting the reading at the next time T2.

[0277] If the determination in step S2521 is No, it means that the reading at time T4 has been completed. So, it proceeds to step S2523, and the CPU of the control unit resets the counter 211 with a reset signal.

[0278] This operation corresponds to the reset operation of the counter 211 at time T4 in FIG. 7. After step S2313, it returns to step S2501 via M. In the above manner, the signals accumulated in the sensor unit can be sequentially read out.

[0279] On the other hand, since the accumulation period of the non-target area is equal to the full frame period, an image is acquired with a defined accumulation period. That is, in step S2531 of FIG. 25, the CPU of the control unit sets i = 4. Next, in step S2532, the CPU of the control unit outputs the count value Count of the counter 211 at time Ti to the memory 212. At this time, the output is performed simultaneously for all memories. This operation corresponds to the operation at time T1 in FIG. 7.

[0280] Next, in step S2533, the CPU of the control unit sets j = j2. j2 is the starting position of the Nth row of the non-target area. Next, in step S2534, the CPU of the control unit sets k = k2. k2 is the starting position of the Mth column of the non-target area.

[0281] In step S2535, the CPU of the control unit outputs the count value Count(j - k - i) in the memory j - k of FIG. 9 to the buffer. At this time, the output is performed for columns 1 to M simultaneously to the buffer. This operation means the operation of taking in the count value of the first row in FIG. 9 into the buffer.

[0282] In step S2536, the CPU of the control unit outputs the count value Count(j - k - i) of the buffer k to the output circuit 114. This operation corresponds to the operation of reading the signal of the buffer in the leftmost column in FIG. 9 from the output circuit.

[0283] Next, it proceeds to step S2537 of FIG. 26 via I. In step S2537, the CPU of the control unit determines whether k < M2. If Yes, in step S2538, the CPU of the control unit increments k by 1 as k = k + 1 and returns to step S2536 via J. M2 is the end position of the Mth column of the non-target area. This operation corresponds to the operation of reading the signal of the buffer in the second column from the left in FIG. 9 from the output circuit.

[0284] If the result is No in step S2537, that is, if k = M2, it means that the signal of the buffer in the M-th column in FIG. 9 has been read out from the output circuit. Next, it proceeds to step S2539, and the CPU of the control unit determines whether j < N2.

[0285] N2 is the end position of the N-th row in the region of interest. If the result is Yes in step S2539, then in step S2540, the CPU of the control unit sets j = J + 1, increments j by 1, and returns to step S2544 via K. This corresponds to the operation for starting the reading of the next row.

[0286] If it is determined to be No in step S2539, it means that the reading of all rows has been completed. So, it proceeds to step S2541, and the CPU of the control unit determines whether i < 4. If it is determined to be Yes in step S2541, the CPU of the control unit sets i = i + 1, increments i by 1, and returns to step S2532 via L. This operation corresponds to the operation for starting the reading at the next time T2. However, since i = 4 is set in step S2531, it will be No in step S2541.

[0287] If it is determined to be No in step S2541, it means that the reading at time T4 has been completed. So, it proceeds to step S2543, and the CPU of the control unit resets the counter 211 with a reset signal. This operation corresponds to the reset operation of the counter 211 at time T4 in FIG. 7. After step S2543, it returns to step S2501 via M.

[0288] In the above manner, the signals accumulated in the sensor unit can be sequentially read out. Note that the non - region of interest may also acquire an image with the accumulation period determined by the control unit 605 in the same way. In that case, a flow for setting the accumulation period is added before acquiring the image.

[0289] For example, in the above embodiment, accumulation is performed for a minimum of 1 / 4 full frame period, but the length of the minimum accumulation period may be changed to, for example, 1 / 5 full frame period or 1 / 3 full frame period depending on the recognition accuracy in the recognition unit 604. Alternatively, the length of the minimum accumulation period may be changed depending on the brightness of the subject. Furthermore, the recognition unit may further recognize the subject based on a signal generated in a second accumulation period.

[0290] Furthermore, even if the readout cycle is set to every 1 / 4 full frame period, the counter may be reset midway through the accumulation period of frame 1_1 in Fig. 7 depending on the brightness of the subject, the image recognition accuracy, etc. As a result, the substantial accumulation period may be made shorter than the 1 / 4 full frame period.

[0291] Alternatively, the counter may be reset once at time T1 in Fig. 7. This may adjust the count value read out at time T4. Note that, although the accumulation periods are uniformly spaced in the above embodiment, the present invention is not limited to this.

[0292] In the above embodiments, various processes are performed on the basis of captured image data using various cameras, devices attached to the cameras, personal computers, and the like.

[0293] For example, in the case of a security camera, the control unit 605 monitors, photographs, and records the behavior of a target person. In some cases, the control unit 605 displays a warning, illuminates a light installed in the security camera, and generates a warning sound via the communication unit 607. In the case of an in-vehicle camera, the control unit 605 issues a warning or displays a warning via the ECU 701, and activates a safety device to slow down or stop the vehicle via the vehicle control unit 702.

[0294] <Embodiment 14> In the fourteenth embodiment, before various processes are performed, the control unit 605 determines whether the image acquired by the photoelectric conversion device 600 is suitable for the subsequent processes. This further reduces the processing load, and can realize a reduction in the processing load and power consumption, etc. Furthermore, it is possible to avoid erroneous decisions due to inappropriate images, erroneous warnings in security cameras, and erroneous activation of safety devices in vehicle-mounted cameras.

[0295] In the fourteenth embodiment, the captured image data is evaluated by the control unit 605. Then, using the average luminance value of the output of each frame, it is judged whether the image is appropriate or not, and if it is appropriate, it is judged whether to pass it on to the next process.

[0296] Fig. 27 is a flowchart showing an example of the operation of the control unit after constructing an image according to embodiment 14. Note that the CPU as a computer in the control unit 605 executes a computer program stored in the memory, thereby sequentially performing the operations of the steps in the flowchart in Fig. 27.

[0297] The flow up to acquiring an image uses the examples shown in Figures 22 and 23. First, in step S2700, the CPU of the control unit composes an image using the count value output in step S106 of Figure 22.

[0298] Then, in step S2701, the CPU of the control unit judges whether the image is suitable for use in the next process. Here, an image that is not suitable refers to an image with a poor S / N ratio and covered in noise, or an image that is too dark to be recognized.

[0299] The determination method in step S2701 can be determined from the average brightness data of the acquired image. That is, the control unit 605 calculates the average brightness value, and determines whether or not the value can be used for the next process. That is, the control unit controls whether or not to perform a predetermined process in the control unit according to the brightness information of the signal generated in at least one of the first accumulation period and the second accumulation period.

[0300] Also, it is not limited to the average luminance, but may be obtained from the peak of the histogram, the bias of the peak, the spread, the variance, etc. Also, the luminance difference between adjacent pixels in the screen may be obtained, and the amount of noise may be calculated from the histogram of the difference, and from the result, it may be judged whether or not it can be used for the next process.

[0301] Furthermore, the determination may be based not only on the previous full frame, but also on the average value of multiple previous full frames, or on the average luminance for each accumulation period within the full frame, a histogram, etc. If it is determined in step S2701 that the image is appropriate, then in step S2702, the CPU of the control unit executes processing using the computer program of the control unit 605.

[0302] On the other hand, if it is determined that the image is inappropriate, in step S2703, the CPU of the control unit does not process that image in the computer program of the control unit 605, and stops the processing of the computer program until an image of the next accumulation period is acquired.

[0303] 25, the number of times that unnecessary processing is performed by the control unit 605 can be reduced, and the product power consumption can be reduced. In addition, it is possible to avoid incorrect judgment due to inappropriate images, false warnings in security cameras, and erroneous activation of safety devices in vehicle-mounted cameras.

[0304] In the above-described embodiment, the moving body 700 has been described using an example of an automobile, but the moving body may be any moving body, such as an airplane, a train, a ship, a drone, an AGV, or a robot.

[0305] Furthermore, the photoelectric conversion device 600 of the present embodiment is not limited to a moving object, and may be attached to a wall, a pillar, etc. In this case, it is desirable to store the installation height, installation angle, etc. in the storage unit 606 as installation information when the photoelectric conversion device 600 is attached to a wall, a pillar, etc.

[0306] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to the above-mentioned embodiments, and various modifications are possible based on the spirit of the present invention, and are not excluded from the scope of the present invention. The present invention also includes combinations of the above-mentioned embodiments, and includes the following configurations, methods, etc.

[0307] (Configuration 1) A photoelectric conversion device comprising: a plurality of pixels, each of which includes a photoelectric conversion unit that emits a pulse in response to incidence of a photon, a counter that counts the number of the pulses, and a memory that stores the count value of the counter; an optical system that forms subject images of different resolutions in a first pixel region and a second pixel region of a sensor unit consisting of the plurality of pixels; and a control unit that generates a signal based on the difference in the count value of the counter at the start and end of an accumulation period, has at least a first accumulation period and a second accumulation period within one full frame, the first accumulation period is shorter than the second accumulation period, and controls so that the signal generated during the first accumulation period is output between the end of the first accumulation period and the end of the second accumulation period, wherein the control unit controls so that the accumulation time in one of the first pixel region and the second pixel region is one of the first accumulation period and the second accumulation period.

[0308] (Configuration 2) The photoelectric conversion device according to configuration 1, wherein the first pixel region is controlled to have at least the first accumulation period and the second accumulation period.

[0309] (Configuration 3) The photoelectric conversion device according to configuration 1 or 2, wherein the second pixel region is controlled to have at least the first accumulation period and the second accumulation period.

[0310] (Configuration 4) The photoelectric conversion device according to any one of configurations 1 to 3, wherein the first accumulation period and the second accumulation period overlap each other.

[0311] (Configuration 5) The photoelectric conversion device according to any one of configurations 1 to 4, wherein the first accumulation period and the second accumulation period start simultaneously.

[0312] (Configuration 6) The photoelectric conversion device according to any one of configurations 1 to 5, wherein the end of the second accumulation period coincides with the end of a full frame.

[0313] (Configuration 7) The photoelectric conversion device according to any one of configurations 1 to 6, further comprising a recognition section that recognizes a subject based on a signal generated during at least the first accumulation period.

[0314] (Configuration 8) The photoelectric conversion device according to configuration 7, wherein the recognition section further recognizes the subject based on a signal generated during the second accumulation period.

[0315] (Configuration 9) The photoelectric conversion device according to any one of configurations 1 to 8, further comprising a display section that displays a signal generated during the second accumulation period as an image.

[0316] (Configuration 10) The photoelectric conversion device according to any one of configurations 1 to 9, wherein the photoelectric conversion section includes an avalanche photodiode.

[0317] (Configuration 11) A photoelectric conversion device described in any one of configurations 1 to 10, characterized in that the control unit sets the second pixel area on the central side of the light receiving surface of the sensor unit when the resolution near the optical axis of the optical system is higher than the resolution on the peripheral side away from the optical axis.

[0318] (Configuration 12) A photoelectric conversion device described in any one of configurations 1 to 10, characterized in that the control unit sets the second pixel area on the peripheral side of the light receiving surface of the sensor unit when the resolution on the peripheral side away from the optical axis of the optical system is higher than the resolution near the optical axis.

[0319] (Configuration 13) The photoelectric conversion device according to any one of configurations 1 to 12, wherein the control unit sets a pixel region having a relatively high resolution on the light receiving surface of the sensor unit as the second pixel region.

[0320] (Configuration 14) The photoelectric conversion device according to any one of configurations 1 to 13, wherein the second pixel region is provided inside the first pixel region.

[0321] (Configuration 15) The photoelectric conversion device according to any one of configurations 1 to 14, wherein the second pixel region is adjacent to the first pixel region.

[0322] (Configuration 16) The photoelectric conversion device according to any one of configurations 1 to 15, wherein at least one of the positions and the sizes of the first pixel region and the second pixel region can be set by a user.

[0323] (Configuration 17) The photoelectric conversion device described in any one of configurations 1 to 16, characterized in that the control unit sets the first pixel region or the second pixel region based on at least one of characteristic information regarding the characteristics of the photoelectric conversion device, installation information regarding the installation state of the photoelectric conversion device, and environmental information regarding the surrounding environment of the photoelectric conversion device.

[0324] (Configuration 18) The photoelectric conversion device described in Configuration 17, characterized in that the control unit sets at least one of the position, size, and accumulation period of the first pixel region or the second pixel region based on at least one of the characteristic information, the installation information, and the environmental information.

[0325] (Configuration 19) The photoelectric conversion device according to configuration 17 or 18, wherein the characteristic information includes optical characteristic information of the optical system, a resolution of the sensor unit, and a pixel size of the sensor unit.

[0326] (Configuration 20) The photoelectric conversion device according to any one of configurations 17 to 19, wherein the installation information includes at least one of the height, angle, and direction at which the photoelectric conversion device is installed.

[0327] (Configuration 21) The photoelectric conversion device according to any one of configurations 1 to 20, wherein the first pixel region or the second pixel region is set in accordance with distance information of a subject.

[0328] (Configuration 22) A photoelectric conversion device according to any one of configurations 1 to 21, characterized in that the length of at least one of the first accumulation period and the second accumulation period is set based on luminance information of a signal generated during at least one of the first accumulation period and the second accumulation period.

[0329] (Configuration 23) A photoelectric conversion device described in any one of configurations 1 to 22, characterized in that the control unit controls whether or not to perform a predetermined processing in the control unit depending on luminance information of a signal generated during at least one of the first accumulation period and the second accumulation period.

[0330] (Configuration 24) A moving body comprising the photoelectric conversion device described in any one of configurations 1 to 23 and a movement control unit that controls the operation of the moving body, wherein the control unit sets the first pixel region or the second pixel region based on the output of the movement control unit.

[0331] (Method) A control method for controlling a photoelectric conversion device having a plurality of pixels, each of which has a photoelectric conversion unit that emits a pulse in response to the incidence of a photon, a counter that counts the number of the pulses, and a memory that stores the count value of the counter, and an optical system that forms subject images of different resolutions in a first pixel region and a second pixel region of a sensor unit consisting of the plurality of pixels, the control method comprising the steps of: generating a signal based on the difference in the count value of the counter at the start and end of an accumulation period; having at least a first accumulation period and a second accumulation period within one full frame, the first accumulation period being shorter than the second accumulation period; controlling so that the signal generated during the first accumulation period is output between the end of the first accumulation period and the end of the second accumulation period, and controlling so that the accumulation time in one of the first pixel region and the second pixel region is one of the first accumulation period and the second accumulation period.

[0332] (program) A computer program for controlling each part of the photoelectric conversion device according to any one of configurations 1 to 23 or the moving body according to claim 24 by a computer.

[0333] A computer program for implementing all or part of the control in this embodiment may be supplied to a photoelectric conversion device or the like via a network or various storage media. A computer (or a CPU, MPU, or the like) in the photoelectric conversion device or the like may then read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0334] 11: Sensor board 12: Pixel area 21: Circuit board 22: Circuit area 100: Sensor section 101: Pixel 102: Photoelectric conversion unit 103: Signal processing circuit 110: Vertical scanning circuit 111: Horizontal scanning circuit 112: Readout circuit 113: Vertical signal line 114: Output circuit 115: Control pulse generation unit 201: Avalanche photodiode 202: Quench element 210: Waveform shaping section 211: Counter 212: Memory 213: Drive wire 600: Photoelectric conversion device 601:Optical system 602: Detection unit 603: Image processing unit 604: Recognition part 605: Control unit 606: Storage section 607: Communications Department

Claims

1. A plurality of pixels each including a photoelectric conversion unit that emits a pulse in response to an incidence of a photon, a counter that counts the number of the pulses, and a memory that stores the count value of the counter; an optical system that forms subject images having different resolutions in a first pixel region and a second pixel region of the sensor unit that is composed of the plurality of pixels; a control unit that generates a signal based on a difference between count values ​​of the counter at the start and end of an accumulation period, has at least a first accumulation period and a second accumulation period within one full frame, the first accumulation period is shorter than the second accumulation period, and controls so that the signal generated during the first accumulation period is output between the end of the first accumulation period and the end of the second accumulation period, The control unit controls an accumulation time in one of the first pixel region and the second pixel region to be one of the first accumulation period and the second accumulation period.

2. 2 . The photoelectric conversion device according to claim 1 , wherein the first pixel region is controlled to have at least the first accumulation period and the second accumulation period.

3. 2 . The photoelectric conversion device according to claim 1 , wherein the second pixel region is controlled to have at least the first accumulation period and the second accumulation period.

4. 2. The photoelectric conversion device according to claim 1, wherein the first accumulation period and the second accumulation period overlap each other.

5. 2. The photoelectric conversion device according to claim 1, wherein the first accumulation period and the second accumulation period start simultaneously.

6. 2. The photoelectric conversion device according to claim 1, wherein the end of the second accumulation period coincides with the end of a full frame.

7. 2. The photoelectric conversion device according to claim 1, further comprising a recognition section that recognizes a subject based on a signal generated during at least the first accumulation period.

8. 8. The photoelectric conversion device according to claim 7, wherein the recognition section further recognizes the subject based on a signal generated during the second accumulation period.

9. 2. The photoelectric conversion device according to claim 1, further comprising a display unit that displays the signal generated during the second accumulation period as an image.

10. The photoelectric conversion device according to claim 1 , wherein the photoelectric conversion section includes an avalanche photodiode.

11. The photoelectric conversion device according to claim 1, characterized in that the control unit sets the second pixel area on the center side of the light receiving surface of the sensor unit when the resolution near the optical axis of the optical system is higher than the resolution on the peripheral side away from the optical axis.

12. The photoelectric conversion device according to claim 1, characterized in that the control unit sets the second pixel area on the peripheral side of the light receiving surface of the sensor unit when the resolution on the peripheral side away from the optical axis of the optical system is higher than the resolution near the optical axis.

13. The photoelectric conversion device according to claim 1 , wherein the control unit sets a pixel area having a relatively high resolution on the light receiving surface of the sensor unit as the second pixel area.

14. The photoelectric conversion device according to claim 1 , wherein the second pixel region is provided inside the first pixel region.

15. The photoelectric conversion device according to claim 1 , wherein the second pixel region is adjacent to the first pixel region.

16. 2. The photoelectric conversion device according to claim 1, wherein at least one of the positions and the sizes of the first pixel region and the second pixel region can be set by a user.

17. The photoelectric conversion device described in claim 1, characterized in that the control unit sets the first pixel region or the second pixel region based on at least one of characteristic information regarding the characteristics of the photoelectric conversion device, installation information regarding the installation state of the photoelectric conversion device, and environmental information regarding the surrounding environment of the photoelectric conversion device.

18. The photoelectric conversion device described in claim 17, characterized in that the control unit sets at least one of the position, size, and accumulation period of the first pixel region or the second pixel region based on at least one of the characteristic information, the installation information, and the environmental information.

19. 18. The photoelectric conversion device according to claim 17, wherein the characteristic information includes optical characteristic information of the optical system, a resolution of the sensor unit, and a pixel size of the sensor unit.

20. 18. The photoelectric conversion device according to claim 17, wherein the installation information includes at least one of a height, an angle, and a direction in which the photoelectric conversion device is installed.

21. 2. The photoelectric conversion device according to claim 1, wherein the first pixel area or the second pixel area is set in accordance with distance information of a subject.

22. 2. The photoelectric conversion device according to claim 1, wherein the length of at least one of the first accumulation period and the second accumulation period is set based on luminance information of a signal generated during at least one of the first accumulation period and the second accumulation period.

23. The photoelectric conversion device according to claim 1, characterized in that the control unit controls whether or not to perform a predetermined processing in the control unit depending on luminance information of a signal generated during at least one of the first accumulation period and the second accumulation period.

24. The photoelectric conversion device according to any one of claims 1 to 23, A movement control unit that controls the operation of the moving body, The moving body, wherein the control unit sets the first pixel area or the second pixel area based on an output of the movement control unit.

25. A plurality of pixels each including a photoelectric conversion unit that emits a pulse in response to an incidence of a photon, a counter that counts the number of the pulses, and a memory that stores the count value of the counter; a control method for controlling a photoelectric conversion device having an optical system for forming subject images having different resolutions in a first pixel region and a second pixel region of a sensor unit including the plurality of pixels, the method comprising: A control method comprising: generating a signal based on the difference between the count values ​​of the counter at the start and end of an accumulation period; having at least a first accumulation period and a second accumulation period within one full frame, the first accumulation period being shorter than the second accumulation period; controlling so that the signal generated during the first accumulation period is output between the end of the first accumulation period and the end of the second accumulation period; and controlling so that the accumulation time in one of the first pixel area and the second pixel area is either the first accumulation period or the second accumulation period.

26. A computer program for controlling each part of the photoelectric conversion device according to any one of claims 1 to 23 or the moving body according to claim 24 by a computer.

Images