Photoelectric conversion apparatus, mobile device, control method, storage medium, and computer program product

Abstract

The invention provides a photoelectric conversion apparatus, a mobile device, a control method, a storage medium, and a computer program product. The photoelectric conversion device includes: a photoelectric conversion element configured to generate an image signal corresponding to an amount of incident light; a visual recognition image processing unit configured to perform visual recognition image processing to generate an image for visual recognition; an image recognition image processing unit configured to perform image recognition image processing to generate an image for image recognition; and a control unit configured to perform control such that image processing is performed on the image signal generated by the photoelectric conversion element using at least one of the visual recognition image processing unit and the image recognition image processing unit.

Description

Technical Field

[0001] The invention relates to a photoelectric conversion device, a movable device, a control method and a storage medium. Background Art

[0002] In recent years, a photoelectric conversion device for digitally counting the number of photons incident on an avalanche photodiode (APD) and outputting the count value from a pixel as a digital signal after photoelectric conversion has been developed. For example, Japanese Patent No. 7223070 discloses a configuration of a photoelectric conversion device including an APD that can output a plurality of images whose accumulation periods overlap each other, thereby enabling continuous imaging even under low illumination.

[0003] However, for example, when assuming an imaging device for a vehicle-mounted camera installed in a mobile device, recognition processing is performed in units of frames, resulting in, for example, recognition processing being performed only every 33.3 ms at 30 fps. Therefore, in the vehicle-mounted camera, if an object jumps in immediately after the frame has changed, recognition processing cannot be performed until the current frame ends.

[0004] Many on-vehicle cameras have an accumulation period equal to or greater than a predetermined period (11ms) to suppress flickering due to traffic signals and to capture bright images by extending the accumulation period, particularly in low illumination. However, due to the long accumulation period, subject blur occurs in fast-moving objects, and thus the recognition rate thereof decreases. Summary of the invention

[0005] According to one aspect of the present invention, a photoelectric conversion device is provided, comprising: a photoelectric conversion element configured to generate an image signal corresponding to the amount of incident light; a visual recognition image processing unit configured to perform image processing to generate an image for visual recognition; an image recognition image processing unit configured to perform image processing to generate an image for image recognition; and a control unit configured to control so that image processing is performed on the image signal generated by the photoelectric conversion element using at least one of the visual recognition image processing unit and the image recognition image processing unit.

[0006] Further features of the present invention will become apparent from the following description of embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a diagram showing a configuration example of a photoelectric conversion element 100 according to the first embodiment.

[0008] Figure 2 is a diagram showing a configuration example of the sensor board 11 according to the first embodiment.

[0009] Figure 3 is a diagram showing a configuration example of a circuit board 21 according to the first embodiment.

[0010] Figure 4 It is shown Figure 2 and Figure 3 1 is a diagram of an equivalent circuit of the photoelectric conversion unit 102 and the signal processing circuit 103 corresponding to the photoelectric conversion unit 102.

[0011] Figure 5 is a diagram schematically showing the relationship between the operation of the APD 201 and the output signal.

[0012] Figure 6 is a functional block diagram of a photoelectric conversion apparatus 600 and a movable device 700 according to the first embodiment.

[0013] Fig. 7A is a functional block diagram showing a configuration example of the visual recognition image processing unit 603 according to the first embodiment, and Figure 7B : is a functional block diagram showing a configuration example of the image recognition image processing unit 604 according to the first embodiment.

[0014] Figure 8 is a diagram showing performance examples such as quality of visual recognition images, recognition processing performance, and power consumption of the photoelectric conversion element 100, the image processing selection unit 602, the visual recognition image processing unit 603, and the image recognition image processing unit 604 under various control conditions.

[0015] Fig. 9 ] is a diagram showing processing from photoelectric conversion until image output and recognition result output performed by the camera control unit 606 according to the first embodiment.

[0016] Fig.10 is a diagram showing an example of images in a plurality of accumulation sub-periods according to the first embodiment.

[0017] Fig.11 is a diagram showing the relationship between a memory and a buffer according to the first embodiment.

[0018] Fig.12 is a flowchart specifically showing a driving example of the photoelectric conversion element according to the first embodiment.

[0019] Fig.13 yes Fig.12 A subsequent flowchart to the flowchart shown.

[0020] Fig.14 yes Fig.13 A subsequent flowchart to the flowchart shown.

[0021] Fig.15 yes Fig.14 A subsequent flowchart to the flowchart shown.

[0022] Fig.16 6 is a functional block diagram of an example in which the visual recognition image processing unit 603 and the image recognition image processing unit 604 according to the second embodiment are partially shared. DETAILED DESCRIPTION

[0023] Hereinafter, with reference to the accompanying drawings, an advantageous mode of the present invention will be described using embodiments. In the various drawings, the same reference numerals are applied to the same members or elements, and repeated descriptions will be omitted or simplified.

[0024] <First Embodiment>

[0025] Figure 1 1 is a diagram showing a configuration example of a photoelectric conversion element 100 according to a first embodiment of the present invention. In the following description, a photoelectric conversion element that generates an image signal corresponding to the amount of incident light has a so-called stacked structure in which two boards including a sensor board 11 and a circuit board 21 are stacked and electrically connected.

[0026] However, a so-called non-stacked structure may be adopted in which the elements included in the sensor board 11 and the elements included in the circuit board 21 are provided in a common semiconductor layer. The sensor board 11 includes a pixel area 12. The circuit board 21 includes a circuit area 22 for processing a signal detected in the pixel area 12.

[0027] Figure 2 1 is a diagram showing a configuration example of the sensor panel 11 according to the first embodiment of the present invention. The pixel region 12 of the sensor panel 11 includes a plurality of pixels 101 arranged two-dimensionally in the direction of a plurality of rows and a plurality of columns.

[0028] Each pixel 101 includes a photoelectric conversion unit 102, and the photoelectric conversion unit 102 includes an avalanche photodiode (hereinafter referred to as APD). Here, the photoelectric conversion unit 102 emits pulses at a frequency corresponding to the photon reception frequency. That is, the photoelectric conversion unit 102 emits pulses at a frequency corresponding to the photon reception frequency. The number of rows and columns in the pixel array constituting the pixel area 12 is not particularly limited.

[0029] Figure 3 1 is a diagram showing an example of the configuration of a circuit board 21 according to a first embodiment of the present invention. The circuit board 21 includes a circuit board for processing Figure 2The signal processing circuit 103 of the charge photoelectrically converted by the corresponding photoelectric conversion unit 102, the reading circuit 112, the control pulse generator 115, the horizontal scanning circuit 111, the vertical signal line 113, the vertical scanning circuit 110 and the output circuit 114.

[0030] The vertical scanning circuit 110 receives the control pulse supplied from the control pulse generator 115 and sequentially supplies the control pulse to a plurality of pixels arranged in the row direction. A logic circuit such as a shift register or an address decoder is used for the vertical scanning circuit 110 .

[0031] The signal output from the photoelectric conversion unit 102 of each pixel is processed by the corresponding signal processing circuit 103. The signal processing circuit 103 is provided with a counter and a memory, etc., and the digital value is stored in the memory. The horizontal scanning circuit 111 inputs a control pulse for sequentially selecting a column to the signal processing circuit 103 to read the signal from the memory for storing the digital signal of the pixel.

[0032] The signal from the signal processing circuit 103 of the pixel in the row selected by the vertical scanning circuit 110 is output to the vertical signal line 113. The signal output to the vertical signal line 113 is output to the outside of the photoelectric conversion element 100 via the reading circuit 112 and the output circuit 114. A plurality of buffers connected to the vertical signal line 113 are included in the reading circuit 112.

[0033] The control pulse generator 115 can control the vertical scanning circuit 110 to perform non-additive reading for sequentially reading all rows, or when it is intended to reduce the amount of data of the signal to be read, it can control the vertical scanning circuit 110 to perform interval elimination reading for selecting the rows to be read every several rows.

[0034] Alternatively, control may be performed so that addition (average) reading is performed for reading pixel signals of a plurality of adjacent rows as one equivalent pixel signal by performing processing such as addition averaging. At this time, the number of pixels to be averaged may be 2×2 pixels or 4×4 pixels, and the number of pixels to be added in both the row direction and the column direction may be freely changed. Alternatively, in order to reduce the data amount of the signal to be read, control may be performed so that clipping reading is performed for selecting and reading only a portion of the image area (viewing angle).

[0035] like Figure 2 and Figure 3As shown, a plurality of signal processing circuits 103 are arranged in a region overlapping with the pixel region 12 in a plan view. A vertical scanning circuit 110, a horizontal scanning circuit 111, a reading circuit 112, an output circuit 114, and a control pulse generator 115 are arranged to overlap with a region outside the pixel region 12 of the sensor panel 11 in a plan view.

[0036] That is, the sensor panel 11 includes a pixel area 12 and a non-pixel area arranged around the pixel area 12. A vertical scanning circuit 110, a horizontal scanning circuit 111, a reading circuit 112, an output circuit 114, and a control pulse generator 115 are arranged in an area overlapping the non-pixel area in a plan view.

[0037] The arrangement of the vertical signal lines 113 and the arrangement of the readout circuits 112 and the output circuits 114 are not limited to Figure 3 For example, the vertical signal line 113 may be arranged to extend in the row direction, and the reading circuit 112 may be arranged in front of the extending direction of the vertical signal line 113 .

[0038] The signal processing circuit 103 does not have to be provided for each photoelectric conversion unit, and a configuration may be adopted in which one signal processing circuit is shared by a plurality of photoelectric conversion units and signal processing is sequentially performed on the signal processing circuit 103 .

[0039] Figure 4 It is shown Figure 2 and Figure 3 2 is a diagram of an equivalent circuit of a photoelectric conversion unit 102 in FIG. 1 and a signal processing circuit 103 corresponding to the photoelectric conversion unit 102. Each pixel includes the photoelectric conversion unit 102 and the signal processing circuit 103. The APD 201 included in each photoelectric conversion unit 102 generates a charge pair corresponding to incident light by photoelectric conversion.

[0040] One of the two nodes of the APD 201 is connected to a power supply line for supplying a driving voltage VL (first voltage). The other of the two nodes of the APD 201 is connected via a quenching element 202 to a power supply line for supplying a driving voltage VH (second voltage) higher than the voltage VL.

[0041] exist Figure 4 In the embodiment, one node of the APD 201 is an anode and the other node of the APD 201 is a cathode. The anode and cathode of the APD 201 are supplied with a reverse bias voltage for allowing the APD 201 to perform an avalanche multiplication operation. By supplying the reverse bias voltage, the charge generated by the incident light causes avalanche multiplication and generates an avalanche current.

[0042] When a reverse bias voltage is supplied, there is a Geiger mode and a linear mode, in which the voltage difference between the anode and the cathode is greater than the breakdown voltage, and in which the voltage difference between the anode and the cathode is a voltage difference close to the breakdown voltage or equal to or less than the breakdown voltage. An APD operating in Geiger mode is called a SPAD. For example, in a SPAD, the voltage VL (first voltage) is -30V, and the voltage VH (second voltage) is 1V.

[0043] Each signal processing circuit 103 includes a quenching element 202, a waveform shaping circuit 210, a counter circuit 211, and a memory circuit 212. The quenching element 202 is connected to a power supply line for supplying a driving voltage VH and one node of an anode and a cathode of the APD 201.

[0044] The quenching element 202 serves as a load circuit (quenching circuit) when performing signal multiplication by avalanche multiplication, and has a function of suppressing the voltage supplied to the APD 201 to suppress avalanche multiplication (quenching operation). The quenching element 202 also has a function of returning the voltage supplied to the APD 201 to the driving voltage VH by causing a current corresponding to the voltage drop caused by the quenching operation to flow in the quenching element 202 (recharging operation).

[0045] exist Figure 4 In the example shown, the signal processing circuit 103 includes a waveform shaping circuit 210, a counter circuit 211, and a memory circuit 212 in addition to the quenching element 202. The waveform shaping circuit 210 shapes the voltage change of the cathode of the APD 201 acquired when detecting photons, and outputs a pulse signal.

[0046] For example, an inverter circuit is used as the waveform shaping circuit 210. Figure 4 In the example shown, one inverter is used as the waveform shaping circuit 210, but a circuit in which a plurality of inverters are connected in series may be used, or other circuits having a waveform shaping effect may be used.

[0047] The counter circuit 211 counts the number of pulses output from the waveform shaping circuit 210 and stores the count value. When the control pulse RES is supplied via the drive line 213, the signal stored in the counter circuit 211 is reset. Here, the counter circuit 211 generates a signal based on the difference in the count value between the start timing and the end timing of the accumulation period.

[0048] via Figure 4 The driving line 214 ( Figure 3 (not shown) from Figure 3The vertical scanning circuit 110 shown in FIG. 1 supplies a control pulse SEL to the memory circuit 212 to switch between electrical connection and disconnection between the counter circuit 211 and the vertical signal line 113. The memory circuit 212 serves as a memory for temporarily storing a count value of the counter, and outputs an output signal from the counter circuit 211 of the corresponding pixel to the vertical signal line 113.

[0049] The memory circuit 212 may be electrically connected to Figure 4 Other signal processing circuits adjacent to the signal processing circuit 103 shown in FIG. Figure 4 The counter circuit in (not shown) is used and performs processing such as addition averaging. With this configuration, multiple pixel signals generated by multiple photoelectric conversion units can be generated as outputs corresponding to one pixel signal, and the data amount of the pixel output signal can be reduced.

[0050] Alternatively, switching may be performed between a case where the memory circuit 212 is connected to only one counter circuit 211 and performs non-addition processing and a case where the memory circuit 212 is connected to a plurality of counter circuits and performs addition averaging.

[0051] A switch such as a transistor may be provided between the quenching element 202 and the APD 201 or between the photoelectric conversion unit 102 and the signal processing circuit 103 to switch the electrical connection therebetween. Similarly, the supply of the voltage VH or the voltage VL to the photoelectric conversion unit 102 may be electrically switched using a switch such as a transistor.

[0052] Figure 5 2 is a diagram schematically showing the relationship between the operation and output signal of the APD 201. The input side of the waveform shaping circuit 210 is defined as a node A and the output side of the waveform shaping circuit 210 is defined as a node B. Between time t0 and time t1, a potential difference VH-VL is applied to the APD 201. When a photon is incident on the APD 201 at time t1, avalanche multiplication occurs in the APD 201, an avalanche multiplication current flows in the quenching element 202, and the voltage of the node A drops.

[0053] When the voltage drop increases and the potential difference applied to the APD 201 decreases, the avalanche multiplication of the APD 201 stops at time t2 and the voltage level of the node A does not drop to a predetermined value or lower.

[0054] Thereafter, between time t2 and time t3, a current for compensating the voltage drop from the voltage VL flows in the node A, and the node A is statically determined to have the original potential level at time t3. At this time, the portion of the output waveform at the node A that exceeds the predetermined threshold value is shaped by the waveform shaping circuit 210 and output to the node B as a pulse signal.

[0055] A photoelectric conversion apparatus 600 and a movable device 700 according to the first embodiment will be described below. Figure 6 1 is a functional block diagram of the photoelectric conversion device 600 and the movable device 700 according to the first embodiment. The photoelectric conversion device 600 and the movable device 700 are implemented by causing a computer (not shown) included in the photoelectric conversion device 600 and the movable device 700 to execute a computer program stored in a memory as a storage medium not shown. Figure 6 Some functional boxes are shown.

[0056] However, part or all of the functional blocks may be implemented by hardware. An application specific circuit (ASIC) or a processor (such as a reconfigurable processor or DSP) or the like may be used as hardware.

[0057] Figure 6 The functional blocks shown in FIG. 1 may not be included in the same housing and may be implemented by different devices connected to each other via signal lines. Figure 6 Description applies to Fig. 7A , Figure 7B and Fig.16 Function box shown.

[0058] The photoelectric conversion device 600 includes the above-mentioned Figures 1 to 5 The photoelectric conversion element 100, the imaging optical system 601, the image processing selection unit 602, the visual recognition image processing unit 603, the image recognition image processing unit 604, the recognition unit 605, the camera control unit 606, the storage unit 607, and the communication unit 608 are described.

[0059] The photoelectric conversion element 100 includes the above-mentioned Figures 1 to 5 The avalanche photodiode described is used for photoelectric conversion of an optical image. The photoelectric conversion element 100 can be switched between a plurality of reading methods such as non-additive reading for reading all pixels without any change based on a control signal from a camera control unit, additive (average) reading for adding and averaging adjacent pixels and reading, and thinning-out reading for thinning out pixels and reading.

[0060] It is preferable that the image output from the photoelectric conversion element 100 is subjected to non-additive reading when high resolution is required, and the image is subjected to additive (averaging) reading or thinning reading when high image output speed rather than resolution is required.

[0061] For example, when performing addition (average) reading of 2×2 pixels, the pixel count values ​​corresponding to four pixels are artificially converted to count values ​​corresponding to one pixel. Therefore, the amount of data to be read is reduced to 1 / 4, and the time required to read the image signal is reduced to 1 / 4.

[0062] That is, when addition (average) reading of 2×2 pixels is performed, image signals can be read at a speed four times that of when non-addition reading is performed.

[0063] In this embodiment, the photoelectric conversion device 600 is mounted in a mobile device, and the image output from the photoelectric conversion element 100 is used for two applications, namely, visual recognition and image recognition. The image acquired for visual recognition is used for the driver to visually recognize the image in various environments using a display component, etc., or to recognize traffic signals, signs, or lane markings, etc. Therefore, high image quality such as high resolution, low noise, and high contrast is required.

[0064] On the other hand, images acquired for image recognition are used to recognize the rush of a person or vehicle, etc. as early as possible. Therefore, it is first necessary to reduce the delay time (the speed of image output, image processing, and recognition processing) from the time when the subject rushes out until the subject is recognized.

[0065] That is, when an image is output from the photoelectric conversion element 100, it is preferred that an image for image recognition is read using an addition (average) read of 2×2 pixels, and an image for visual recognition is read using a non-addition read. In the present embodiment, an addition (average) read is performed to read an image for image recognition, and a non-addition read is performed to read a visual recognition image. The specific reading method will be described later.

[0066] The photoelectric conversion element according to the embodiment is mounted in the movable device 700, and the camera unit including the set of the imaging optical system 601 and the photoelectric conversion element 100 is configured to capture at least one of the front view, the rear view, and the side view of the movable device, for example. A plurality of camera units may be provided at a plurality of locations of the movable device 700.

[0067] The image processing selection unit 602 is used to control whether the image signal output from the photoelectric conversion element 100 is to be output to the visual recognition image processing unit 603 or the image recognition image processing unit 604 .

[0068] In the present embodiment, when the image signal output from the photoelectric conversion element 100 is used for visual recognition, the camera control unit 606 enables only the visual recognition image processing unit 603, and when the image signal is used for image recognition, enables only the image recognition image processing unit 604. Both the visual recognition image processing unit 603 and the image recognition image processing unit 604 can be enabled or disabled as needed.

[0069] In this embodiment, "valid" means turning on the circuit operation, and "invalid" means turning off the circuit operation. Even in the case of "invalid", power supply from the power supply unit can be performed. The camera control unit controls so that the image signal generated by the photoelectric conversion unit is image processed using at least one of the visual recognition image processing unit 603 and the image recognition image processing unit 604.

[0070] When both the visual recognition image processing unit 603 and the image recognition image processing unit 604 are used, the full frame image may be processed by the visual recognition image processing unit 603 and the full frame image may also be subjected to recognition processing by the image recognition image processing unit 604 in parallel.

[0071] Therefore, for example, when the processing load of the visual recognition image generation processing is large and the display is delayed, the recognition processing is performed by the image recognition image processing unit 604, so the control of the vehicle is less likely to be affected than in the case where the generation and recognition processing of the visual recognition image are performed by one image processing unit.

[0072] The following will refer to Fig. 7A The visual recognition image processing unit 603 is described. Fig. 7A : is a functional block diagram showing a configuration example of the visual recognition image processing unit 603 according to the first embodiment. The visual recognition image processing unit 603 is a processing unit configured to perform image processing to generate an image suitable for visual recognition.

[0073] Therefore, the visual recognition image processing unit 603 performs image processing such as black level correction 6031, defective pixel correction 6032, digital gain correction 6033, noise reduction (NR) correction 6034, development processing 6035, and compression processing 6036 on the image signal. Then, the visual recognition image processing unit 603 generates a final image signal. In this way, the visual recognition image processing unit 603 has a function of correcting defective pixels.

[0074] In the present embodiment, the photoelectric conversion element 100 is monochrome (without a color filter), but when an on-chip color filter of RGB or the like is provided, it is preferable that the visual recognition image processing unit 603 performs processing such as white balance correction and color conversion.

[0075] Since the visual recognition image processing unit 603 requires image processing for enhancing image quality, other image processing can be performed as needed. The images subjected to various types of image processing by the visual recognition image processing unit 603 are supplied to the electronic control unit (ECU) 701 of the mobile device 700 and the camera control unit 606.

[0076] The following will refer to Figure 7BThe image recognition image processing unit 604 is described. Figure 7B 6 is a functional block diagram showing a configuration example of the image recognition image processing unit 604 according to the first embodiment. The image recognition image processing unit 604 is a processing unit configured to perform image processing to generate an image suitable for image recognition, and a high processing speed (short delay time) of the image processing is required. Therefore, it is preferable to perform compression processing for minimizing the data amount of the image signal at the front stage as much as possible.

[0077] For example, by adding (averaging) or thinning out pixels, the resolution can be lowered or the bit depth can be reduced. When the number of image processing functions increases, the delay time also increases, so it is necessary to minimize the number of image processing functions.

[0078] In the present embodiment, two types of image processing including compression processing 6041 and black level correction 6042 are performed to generate an image signal to be output to the recognition unit 605. That is, the image recognition image processing unit 604 includes at least a compression unit configured to reduce the data amount of the image signal generated by the photoelectric conversion element. The image recognition image processing unit 604 also has a function of correcting the black level.

[0079] The compression unit reduces the amount of data by, for example, adding or averaging the pixel values ​​of a plurality of pixels. Alternatively, the compression unit reduces the amount of data by thinning out the pixels in the rows or columns of the photoelectric conversion element. The image recognition image processing unit 604 can reduce the amount of data by cutting out the pixels in a partial area of ​​the photoelectric conversion element. Other image processing is determined based on the specifications of the recognition unit 605 located at the subsequent stage.

[0080] For example, when a convolutional neural network (CNN) is used for the recognition unit 605, the optimal image processing method other than the compression processing 6041 needs to be changed depending on what image data is used for the recognition unit 605 to learn.

[0081] For example, when all image data used for learning by the recognition unit 605 has been subjected to black level correction, the black level correction 6042 is preferably performed after the compression processing 6041 in the image recognition image processing unit 604 .

[0082] Without performing the black level correction 6042, the recognition accuracy decreases when the black level of the image output from the photoelectric conversion element 100 has changed due to a change in the ambient temperature, etc. When the recognition unit 605 performs learning in consideration of the difference in black level, the image recognition image processing unit 604 does not need to perform the black level correction 6042, but the circuit scale of the recognition unit 605 may increase.

[0083] That is, it is preferable to determine the specifications of the image recognition image processing unit 604 in consideration of a balance such as the overall circuit scale of the photoelectric conversion device 600. In the present embodiment, the image recognition image processing unit 604 reduces the amount of data to, for example, 1 / 4 using compression processing 6041, corrects the black level using black level correction 6042, and then outputs the resulting image to the recognition unit 605.

[0084] The recognition unit 605 is used to quickly recognize the rushing out of a person, animal, vehicle, or the like located near the movable device 700 by performing image recognition based on the image signal output from the image recognition image processing unit 604. The recognition unit 605 according to the present embodiment uses CNN as the recognition method as described above, and the objects of the image data to be learned are limited to the subject that may rush out suddenly.

[0085] This is to optimize the image processing using the image recognition image processing unit 604 or the recognition processing using the recognition unit 605 for the subject that rushes out, and to improve the speed and accuracy of the recognition processing. When a fixed subject that only needs to be recognized in units of frames, such as a traffic signal, a sign, or a lane marking, is recognized, a recognition unit for recognizing the fixed subject may be separately provided, for example, in the subsequent stage of the visual recognition image processing unit 603 or in the movable device 700.

[0086] Since distant traffic signals and sign characters can be recognized in a timely manner in units of frames, visual recognition images that have undergone image processing with high image quality can be appropriately used for them. That is, the recognition unit 605 according to this embodiment only recognizes subjects such as movable objects that are not timely at the recognition rate in units of frames as recognition targets.

[0087] Here, the recognition object does not have to be distinguished as described above, and as necessary, an object other than the punched-out subject may be recognized using the recognition unit 605, or a recognition unit (not shown) other than the recognition unit 605 may be used to recognize the punched-out subject.

[0088] The recognition result obtained by the recognition unit 605 is quickly output to the ECU 701. In the present embodiment, the movable device 700 is, for example, a vehicle, but the movable device 700 is not particularly limited as long as it is a movable object such as an airplane, subway, ship, drone, AGV, or robot.

[0089] The camera control unit 606 includes a CPU as a computer and a memory storing a computer program, and controls the components of the photoelectric conversion apparatus 600 by causing the CPU to execute the computer program stored in the memory.

[0090] The camera control unit 606 functions as a control means, and controls, for example, the length of the exposure period of each frame of the photoelectric conversion element 100 , the timing of the control signal CLK, or the like via a control pulse generator of the photoelectric conversion element 100 .

[0091] The camera control unit 606 transmits a control signal for switching the reading method when reading pixel signals to the photoelectric conversion element 100. In this embodiment, the reading method can be switched between two reading methods including addition (average) reading and non-addition reading in one frame.

[0092] The camera control unit 606 has a function of controlling the image processing selection unit 602, the visual recognition image processing unit 603, the image recognition image processing unit 604, and the recognition unit 605. The camera control unit 606 also has a function of acquiring an image signal output from the visual recognition image processing unit 603 or a recognition result when the recognition unit 605 has performed recognition processing.

[0093] The storage unit 607 includes a recording medium such as a memory card or a hard disk, and can store and read image signals. The communication unit 608 includes a wireless or wired interface, outputs the generated image signal or recognition result to the outside of the photoelectric conversion device 600, and receives various types of signals from the outside. For example, the interface can perform wired communication based on a standard such as SPI or I2C, or use a wireless LAN system such as Wi-Fi or Bluetooth (registered trademark).

[0094] The ECU 701 includes a CPU as a computer and a memory storing a computer program, and controls the components of the movable device 700 by causing the CPU to execute the computer program stored in the memory.

[0095] The output of ECU 701 is supplied to vehicle control unit 702 and display unit 703. Vehicle control unit 702 functions as a mobility control unit configured to control operations such as driving, stopping, and direction control of a vehicle as movable device 700 based on the output of ECU 701.

[0096] The display unit 703 functions as a display part, includes a display device such as a liquid crystal device or an organic EL display, and is mounted in the movable device 700. The display unit 703 displays an image acquired by the photoelectric conversion element 100, a recognition result acquired by the recognition unit 605, and various types of information such as a vehicle running state to a driver of the movable device 700 using, for example, a GUI based on the output of the ECU 701.

[0097] Figure 6The image processing selection unit 602, the visual recognition image processing unit 603, the image recognition image processing unit 604, and the recognition unit 605 shown may not be mounted in the mobile device 700. For example, these component units may be provided in an external terminal provided separately from the mobile device 700 for controlling the mobile device 700 or monitoring the travel of the mobile device.

[0098] The following will refer to Figure 8 Describe the control specifications of the functional blocks in the system. Figure 8 is a diagram showing examples of performance such as quality of visual recognition images, recognition processing performance, and power consumption of the photoelectric conversion element 100, the image processing selection unit 602, the visual recognition image processing unit 603, and the image recognition image processing unit 604 under various control conditions.

[0099] The reading method is described in the entry column of “photoelectric conversion element 100.” Specifically, one of non-addition reading and 1 / 4 addition (average) reading is described.

[0100] The state of the image processing selection unit 602 and the state of the visual recognition image processing unit 603 are described in the input column of "Visual recognition image". Specifically, the column of the image processing selection unit 602 indicates whether the output from the visual recognition image processing unit 603 is valid, and the column of the image quality enhancement processing (6031 to 6036) indicates whether the image processing for enhancing the image quality using the visual recognition image processing unit 603 is valid.

[0101] In this embodiment, when the visual recognition image processing unit 603 is effective, it is necessary to perform the image quality enhancement processing (6031 to 6036).

[0102] In the input column of "Image Recognition Image", the state of the image processing selection unit 602 and the state of the acceleration processing (×4) (6041 to 6042) are described. Specifically, the column of the image processing selection unit 602 indicates whether the output of the visual recognition image processing unit 603 is valid, and the column of the acceleration processing (×4) (6041 to 6042) indicates whether the image processing for acceleration using the image recognition image processing unit 604 is valid.

[0103] In this embodiment, in a state where the image processing selection unit 602 is effective, a case where the acceleration processing (×4) (6041 to 6042) is performed and a case where the acceleration processing (×4) (6041 to 6042) is not performed are provided. The reading method using the photoelectric conversion element 100 and the data compression method using the image recognition image processing unit 604 are appropriately utilized according to the system specifications. Ideally, it is preferable to perform data compression at the front stage as much as possible.

[0104] The image quality, recognition speed, and power consumption of the visual recognition image in the control state of the aforementioned 12 patterns are described in the input column of "Performance Comparison".

[0105] In the present embodiment, the system control based on the No. 3 control in which the image quality is the highest and the power consumption is suppressed is performed on the frame for generating the visual recognition image, and the system control based on the No. 10 control in which the recognition processing is performed at the highest speed and the power consumption is suppressed is performed on the frame for performing the recognition processing. In the present embodiment, even when the image data amount of the subject for performing the recognition processing is reduced to 1 / 16 of the image data amount at the time of non-addition reading (normal reading), the rushing out of a person or a vehicle, etc. can be recognized satisfactorily.

[0106] Fig. 9 1 is a diagram showing the processing from photoelectric conversion to image output and recognition result output performed by the camera control unit 606 according to the first embodiment. In this embodiment, the photoelectric conversion is periodically driven at, for example, 30 full frames / second.

[0107] In this embodiment, a frame corresponding to one vertical period of 33.3 ms in length is referred to as a full frame, and a division of the full frame is referred to as a frame. The period length of the full frame is not limited to the above example. The number of divisions of the full frame period is not limited to 4.

[0108] In the present embodiment, a period of one full frame having a length of 33.3 ms is divided into four, and the photoelectric conversion element 100 sequentially generates three frames (frame 1_1, frame 1_2, and frame 1_3) for image recognition images and one frame (frame 1_4) for visual recognition images.

[0109] The system control is performed using the method based on control No. 10 for three frames for the image recognition image, and using the method based on control No. 3 for one frame for the visual recognition image. As described above, the amount of image data read from the photoelectric conversion element 100 in the image recognition image using addition (average) reading is 1 / 4 of the amount of image data read from the photoelectric conversion element 100 in the visual recognition image using non-addition reading.

[0110] Therefore, the length of the image reading time is 1 / 4. That is, the reading period of the vision recognition image is 8.33 ms, and the reading period of the image recognition image is 2.08 ms.

[0111] Frame 1_1 includes an accumulation period from time T0 to time T1 (as the start timing of the full frame 1), and frame 1_2 includes an accumulation period from time T0 to time T2. Frame 1_3 includes an accumulation period from time T0 to time T3, and frame 1_4 includes an accumulation period from time T0 to time T4.

[0112] The following will first describe the processing flow from photoelectric conversion to output of the recognition result for the image of frame 1_1 for image recognition. The counter circuit 211 is reset at time T0 and acquires a count value C1_1 using a method based on addition (average) reading at time T1.

[0113] In the present embodiment, assuming a monochrome pixel 101 , a count value C1_1 obtained by calculating count values ​​of four pixels adjacent to the counter circuit 211 using addition averaging is temporarily stored in the memory circuit 212 .

[0114] For example, when a color filter of a Bayer array having R (red), Gr (green), Gb (green), and B (blue) is arranged on the light receiving surface of the pixel 101, the count value C1_1 is calculated by adding and averaging the count values ​​of the closest pixels of the same color. Then, the count value temporarily stored in the memory circuit 212 is output from the photoelectric conversion element to the image processing selection unit 602 via the buffer of the reading circuit 112.

[0115] Since the visual recognition image processing unit 603 is invalid and the image recognition image processing unit 604 is valid, the image processing selection unit 602 outputs only the image signal of the frame 1_1 (count value C1_1 ) to the image recognition image processing unit 604 .

[0116] Image recognition image processing unit 604 performs the above reference Figure 7B The image processing unit 604 performs the various types of image processing described above and outputs the processing results to the recognition unit 605 without any change. The image recognition image processing unit 604 performs data compression processing using the compression processing 6041 so as to obtain the minimum amount of data required by the recognition unit 605. At this time, the resolution may be reduced or the bit depth may be reduced.

[0117] Therefore, the processing speed of the recognition processing in the subsequent image processing or recognition unit 605 can be improved. In this embodiment, it is assumed that compression processing is performed to obtain 1 / 4 of the data amount. After the image data has been compressed using compression processing 6041, the black level correction 6042 is used to correct the black level.

[0118] Specifically, the count value of the light-shielded pixel is detected, the black level is corrected by subtracting the black level from the count value of the effective pixel, and C1_1 is output to the identification unit 605 .

[0119] In the period from time T1 to time T1+2.08ms, the image of frame 1_1 is output to the recognition unit 605, and the output of the image signal corresponding to one frame is completed at time T1+2.08ms. Then, the recognition unit 605 starts the subject recognition process at time T1+2.08ms.

[0120] In this embodiment, the recognition process starts at the timing of time T2 when the image signal corresponding to one frame is aligned, but the recognition process can be performed sequentially for each line or each pixel. In this way, by reducing the processing units for image recognition, the time until the recognition result is obtained can be shortened.

[0121] Then, at the same time as the recognition process is completed, the recognition result is quickly output to the ECU 701. In this embodiment, the shortest time required for the recognition process and the output of its result is defined as Td. At this time, the output of the recognition result based on the image of frame 1_1 can be transmitted to the mobile device 700 at the maximum speed at time T1+Td. So far, the processing flow of frame 1_1 has been described.

[0122] Thereafter, similar to frame 1_1, the image recognition images of frame 1_2 and frame 1_3 are also sequentially read in the period from time T2 to time T2+2.08ms and in the period from time T3 to time T3+2.08ms. Then, the recognition processing starts sequentially at time T2+2.08ms and time T3+2.08ms, and the output of the recognition result from the recognition unit 605 to the ECU 701 is completed at time T2+Td and time T3+Td.

[0123] Subsequently, the processing flow of the frame 1_4 for visual recognition will be described below. At time T0 of the full frame 2, the count value C1_4 is acquired from the counter circuit 211 using non-additive reading. The acquired count value C1_4 is temporarily stored in the memory circuit 212. Then, the count value temporarily stored in the memory circuit 212 is output from the photoelectric conversion element to the image processing selection unit 602 via the buffer of the reading circuit 112.

[0124] At this time, since the visual recognition image processing unit 603 is valid and the image recognition image processing unit 604 is invalid, the image processing selection unit 602 outputs only the image signal of the frame 1_4 (count value C1_4) to the visual recognition image processing unit 603.

[0125] The visual recognition image processing unit 603 performs the above-mentioned various types of image processing and stores the finally generated image in the storage unit 607. When the various types of image processing are completed, in the period from time T1 to time T2 of the full frame 2, the image of the frame 1-4 is output to the ECU 701 of the movable device 700, and the output of the image signal corresponding to all the lines is completed at time T2.

[0126] As described above, in the present embodiment, one full frame period includes an image recognition accumulation period for generating an image recognition image and a visual recognition accumulation period for generating a visual recognition image. The image recognition accumulation period is shorter than the visual recognition accumulation period, and control is performed so that the image signal generated in the image recognition accumulation period is output between the end of the image recognition accumulation period and the end of the visual recognition accumulation period.

[0127] The visual recognition image processing unit generates an image with higher image quality than the image recognition image processing unit, and generates an image at a higher speed than the visual recognition image processing unit.

[0128] In this embodiment, the recognition processing using the recognition unit 605 is not performed considering that the data amount of the image signal of frame 1_4 is large, but the image signal of frame 1_4 can be recognized as needed. Since the image signal of frame 1_4 has high contrast, low noise, and suppressed LED flicker, the image signal of frame 1_4 is first used for visual recognition.

[0129] However, as described above, by performing recognition processing before reading the image signal of frame 1_4, the time from the switching of the full frame period until the first recognition of the object can be shortened. A fast-moving subject that does not need to be acquired in full frame units can be recognized. The processing flow of frame 1_4 has been described so far.

[0130] In the present embodiment, addition (average) reading is performed by both the inside of the photoelectric conversion element 100 and the compression processing 6041 in the image recognition image processing unit 604, but addition (average) reading may be performed by only one of them. When the pixel addition processing is performed in the photoelectric conversion element 100, the period of reading the image signal from the photoelectric conversion element 100 can be shortened from 8.33 ms to 2.08 ms.

[0131] On the other hand, when pixel addition processing is performed in the compression processing 6041, the period of reading the image signal cannot be shortened, but Fig. 9The period Td in can be shortened. In this way, according to the system specifications of the photoelectric conversion device 600, one or both of the photoelectric conversion element 100 and the compression process 6041 are used to optimize the process for reducing the amount of image data. Therefore, an increase in the recognition processing speed can be achieved. Preferably, the amount of data is reduced as much as possible in the front stage.

[0132] Fig.10 is a diagram showing an example of images in a plurality of accumulation sub-periods according to the first embodiment. Fig.10 , four frames of a total of four images acquired in one full frame period (33.3 ms) are shown: three image recognition images of frame 1_1, frame 1_2, and frame 1_3, and a visual recognition image of frame 1_4.

[0133] like Fig.10 As shown, the image of frame 1_1 is dark because the accumulation period of the image is short, but the subject blur of the person rushing out is small. On the other hand, the accumulation period increases in the order of frame 1_2, frame 1_3, and frame 1_4, so subject blur is more likely to occur. Blurring is less likely to occur in a stopped vehicle or a white line, and as the accumulation period becomes longer, the contrast is more likely to increase.

[0134] Fig.11 is a diagram showing the relationship between the memory circuit and the buffer according to the first embodiment. Fig.11 middle, Figure 3 The memory circuit 212 in the signal processing circuit 103 shown is arranged in N rows and M columns, and the memory circuit is defined as a memory 1-1 To Storage N-M .

[0135] Fig.11 Buffer 1 to Buffer M instruct Figure 3 The read circuit 112 is shown to include a buffer. Fig.11 The output circuit 114 shown is Figure 3 The output circuit 114 shown corresponds.

[0136] Fig.12 is a flowchart specifically showing a driving example of the photoelectric conversion element according to the first embodiment. Fig.13 yes Fig.12 A subsequent flowchart of the flowchart shown, Fig.14 yes Fig.13 A subsequent flowchart of the flowchart shown, and Fig.15 yes Fig.14 The following flowchart is a subsequent flowchart of the flowchart shown in FIG. 6. The following flowchart is sequentially performed by causing the CPU as a computer in the camera control unit 606 to execute a computer program stored in the memory. Figures 12 to 15The operations are as shown in the steps of the flowchart.

[0137] exist Fig.12 In step S1201, the CPU of the camera control unit sets i=1, and in step S1202, when the count value is output to the memory, it is determined whether the current frame is a frame to be added. In this embodiment, it is assumed that frames 1_1 to 1_4 are read, and the description is Fig. 9 The period from time T1 of full frame 1 to time T1 of full frame 2.

[0138] That is, when i=1 to 3, the judgment result of step S1202 is "yes", and when i=4, the judgment result of step S1202 is "no". The judgment of step S1202 may be performed in consideration of a preset ID indicating addition (average) reading or non-addition reading or the like for each frame, or may be performed in consideration of the count value while counting the number of frames.

[0139] When the frame is intended to be dynamically switched between addition (average) reading and non-addition reading in a full frame period using ID, the same algorithm can be used for processing by assigning ID to the frame again. When the judgment result of step S1202 is "yes", the processing flow proceeds to step S1203, and when the judgment result of step S1202 is "no", the processing flow proceeds to step S1212.

[0140] Then, in step S1203, the CPU of the camera control unit outputs the result of adding and averaging a total of four pixels including two pixels in each of the horizontal and vertical directions for the count value Count of the counter circuit 211 at the time Ti to the memory circuit 212. In the present embodiment, since a monochrome pixel 101 is assumed, the result of adding and averaging the four pixels is output only to the memory circuit 212 of the odd-numbered rows in the N rows and the odd-numbered columns in the M columns.

[0141] That is, the count value does not exist in the memory circuit 212 of the odd-numbered rows in the N rows or the odd-numbered columns in the M columns. For example, when a color filter of the Bayer array of R, Gr, Gb, and B is used instead of a monochrome filter, one count value exists for each of four pixels in the horizontal and vertical directions for each color. When the count value is output to the memory circuit 212, the count value is output to all the memory circuits at the same time. This operation is the same as Fig. 9 This corresponds to the operation during time T1.

[0142] Then, in step S1204, the CPU of the camera control unit sets j=1, and in step S1205 outputs the image to the buffer. Fig.11 The count value Count in the memory circuit jkj-k-i At this time, the count values in rows 1 to M are output to the buffer simultaneously.

[0143] Since the count values only exist in the odd columns, the actual number of count values Count output to the buffer j-k-i is M / 2. That is to say, the count values in the even columns may not be output to the buffer. This operation corresponds to the operation of outputting the count values to the Fig.11 buffer in the first row.

[0144] Then, in step S1206, the CPU of the camera control unit sets k = 1, and in step S1207, it outputs the count value Count in the buffer k to the output circuit 114. j-k-i This operation corresponds to the operation of reading the signal in the buffer of the leftmost column from the output circuit Fig.11 .

[0145] Then, in step S1208, the CPU of the camera control unit determines whether k < M - 1 is satisfied. When the determination result is "yes", in step S1209, k is incremented by 2 to k = k + 2, and the operation of step S1207 is performed again in step S1207.

[0146] The reason why k is not incremented by 1 is that the count values do not exist in the even columns as described above. This operation corresponds to the operation of reading the signal in the third buffer from the leftmost in the Fig.11 output circuit.

[0147] When the determination result of step S1208 is "no", that is, when k = M - 1 is satisfied, this means reading all the signals in the (M - 1)th buffer in the Fig.11 output circuit. Then, the processing flow proceeds to step S1210, and the CPU of the camera control unit determines whether j < N - 1 is satisfied. When the determination result of step S1210 is "yes", in step S1211, the CPU of the camera control unit increments j by 2 to j = j + 2, and the processing flow returns to step S1205.

[0148] When the determination result of step S1210 is "no", this means that the reading of all rows has ended. Therefore, in step S1221, the CPU of the camera control unit determines whether i < 4 is satisfied. When the determination result of step S1221 is "yes", in step S1222, the CPU of the camera control unit increments i by 1 to i = i + 1, and the processing flow returns to step S1202. This operation corresponds to the operation of starting to read the next time T2.

[0149] In this embodiment, the process with the judgment result of "Yes" in step S1202 is repeated three times. That is, the processes from step S1203 to S1221 are repeated three times. This operation corresponds to the reading operations at times T1 to T3. Thereafter, when the processing flow returns to step S1202 with the state of i = 4 set in step S1222, the CPU of the camera control unit determines that the judgment result of step S1202 is "No", and the processing flow proceeds to step S1212.

[0150] Then, in step S1212, the CPU of the camera control unit outputs the count value Count in the counter circuit 211 at time Ti (without performing addition averaging processing) to the memory circuit 212 without any change. At this time, the count value is output to all memory circuits simultaneously. This operation corresponds to Fig. 9 the operation at time T4 in

[0151] Then, in step S1213, the CPU of the camera control unit sets j = 1, and in step S1214, outputs the count value Count in the memory circuit j-k shown in Fig.11 to the buffer. j-k-i At this time, the count value is output to the buffers in columns 1 to M simultaneously. In step S1214, different from step S1205, the count value exists in all even columns and odd columns, and the number of data is M. This operation corresponds to the operation of outputting the count value in the first row in Fig.11 to the buffer.

[0152] Then, in step S1215, the CPU of the camera control unit sets k = 1, and in step S1216, outputs the count value Count in the buffer in k to the output circuit 114. j-k-i This operation corresponds to the operation of reading the signal in the leftmost buffer in Fig.11 from the output circuit.

[0153] Then, in step S1217, the CPU of the camera control unit determines whether k < M is satisfied. When the judgment result is "Yes", in step S1218, k is incremented by 1 to k = k + 1, and the operation of step S1216 is performed again.

[0154] Here, k is incremented by 2 in step S1209, but k is incremented by 1 in step S1216 because, as described above, the count value exists in all even columns and odd columns. This operation corresponds to the operation of reading the signal in the second buffer from the leftmost in Fig.11 from the output circuit.

[0155] When the determination result in step S1217 is "No", that is, when k = M is satisfied, this means that all signals in the M-th buffer in Fig.11 are read from the output circuit, and in step S1219, the CPU of the camera control unit determines whether j < N is satisfied. When the determination result in step S1219 is "Yes", in step S1220, the CPU of the camera control unit increments j by 1 to j = j + 1, and the processing flow returns to step S1214. This operation corresponds to the operation of starting to read the next line. Fig.11 When the determination result in step S1219 is "No", this means that the reading of all lines has ended. Therefore, in step S1221, the CPU of the camera control unit determines whether i < 4 is satisfied. Since i = 4 is satisfied in the fourth frame, the determination result in step S1221 is "No".

[0156] When the determination result in step S1221 is "No", this means that the reading at time T4 has been completed. Therefore, the reading of all image signals stored in the first full frame is completed, and

[0157] the processing flow shown ends. As described above, the signals stored in the photoelectric conversion element 100 can be read sequentially. Fig.13 Although the present invention has been described above with reference to the exemplary embodiments, the present invention is not limited to the embodiments, and various forms of modifications can be made without departing from the gist of the present invention, and the modifications are not excluded from the scope of the present invention.

[0158] For example, in the foregoing embodiments, an SPAD sensor is used, but other sensors such as a CMOS sensor can also be used. When an SPAD sensor is used, the emission crosstalk in which signals from adjacent pixels are accidentally detected due to light emission in a region with a high electric field can be corrected. At this time, as a method for correcting the emission crosstalk, an existing correction method based on theoretical values can be used, or other methods based on measurement results or the like can be used.

[0159] <Second Embodiment>

[0160] <Second Embodiment>

[0161] In the first embodiment, the image processing performed by the visual recognition image processing unit 603 and the image processing performed by the image recognition image processing unit 604 are completely different, but a part can be shared.

[0162] That is, in the second embodiment, a part of the visual recognition image processing unit and the image recognition image processing unit is shared. For example, Fig.16 is a functional block diagram showing an example in which a part of the visual recognition image processing unit 603 and the image recognition image processing unit 604 according to the second embodiment is shared.

[0163] Through Fig.16 As shown, the circuit scale can be reduced by using the compression process 1601 and the black level correction 1602 in common. At this time, it is preferable to appropriately change the setting parameters of the compression process 1601 in the visual recognition image and the image recognition image.

[0164] Specifically, for example, "no compression processing" is set for the frame of the visual recognition image, and "compression processing" is set for the frame of the image recognition image. In this way, the common part is set so that the correction parameters are different when visual recognition is performed and when image recognition is performed.

[0165] In the aforementioned embodiment, four pixels including two pixels in the horizontal direction and two pixels in the vertical direction are averaged in the image recognition image, but the number of pixels to be averaged can be freely changed in both the horizontal direction and the vertical direction.

[0166] When performing thinning reading, the number of pixels to be thinned out or the number of rows (thinning rate) can be freely changed similarly. In this embodiment, a monochrome pixel 101 is assumed, but a color filter of a Bayer array of R, Gr, Gb, and B, etc. can be provided, and when performing addition averaging or thinning out, processing such as addition averaging or thinning out of pixels of the same color can be performed.

[0167] Alternatively, the number of averaged pixels added during addition (average) reading and the thinning rate during thinning reading can be determined based on the recognition accuracy in the recognition unit 605. When the resolution of the acquired image is low but the recognition unit 605 can output the recognition result with high accuracy, the speed can be increased by increasing the number of pixels to be added and increasing the thinning rate to reduce the amount of image data. That is, for example, as the recognition accuracy in the recognition unit 605 becomes higher, the number of pixels to be added and the thinning rate can be increased.

[0168] It is possible to determine which of the addition (average) reading and the thinning reading to use based on the brightness of the ambient light. For example, when the brightness of the ambient light is equal to or greater than a predetermined value, the noise sensitivity does not deteriorate significantly even if the signal amount decreases. Therefore, when a simple thinning reading is performed as a reading process and the brightness of the ambient light is less than a predetermined value, control can be performed so that the thinning reading is changed to the addition (average) reading to enhance the noise sensitivity.

[0169] The photoelectric conversion device 600 according to this embodiment is not limited to being mounted in a movable device, but can be attached to, for example, a wall or a pillar. At this time, preferably, when the photoelectric conversion device 600 is attached to a wall or a pillar, etc., the installation height or installation angle, etc. is stored in the storage unit 607, etc. as installation information.

[0170] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

[0171] In addition, as part or all of the control according to the embodiment, a computer program for realizing the functions of the above-described embodiment may be supplied to the photoelectric conversion device, etc., through a network or various storage media. Then, the computer (or CPU or MPU, etc.) of the photoelectric conversion device, etc. may be configured to read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention.

[0172] In addition, the present invention includes those implemented using at least one processor or circuit configured to perform the functions of the above-described embodiments. For example, multiple processors may be used to share processing to perform the functions of the above-described embodiments.

[0173] This application claims the benefit of Japanese Patent Application No. 2023-146835, filed on September 11, 2023, which is hereby incorporated by reference in its entirety.

Claims

1. A photoelectric conversion device, comprising: a photoelectric conversion element configured to generate an image signal corresponding to an amount of incident light; one or more memories for storing instructions; as well as One or more processors for executing the instructions so that: Perform visual recognition image processing to generate images for visual recognition, performing image recognition image processing to generate an image for image recognition, and Control is performed so that image processing is performed on the image signal generated by the photoelectric conversion element using at least one of the visual recognition image processing and the image recognition image processing.

2. The photoelectric conversion device according to claim 1, wherein The photoelectric conversion element includes a plurality of pixels, each of the plurality of pixels includes a photoelectric conversion unit for emitting a pulse in response to incidence of a photon, a counter for counting the number of pulses, and a memory for storing a count value of the counter, and Here, the photoelectric conversion element generates an image signal based on a difference in count value between a start timing of an accumulation period and an end timing of the accumulation period in each photoelectric conversion unit.

3. The photoelectric conversion device according to claim 1, wherein Control is performed so that the image signal generated in the image recognition accumulation period is output between the end of the image recognition accumulation period and the end of the visual recognition accumulation period, and one full frame period includes the image recognition accumulation period for generating the image recognition image and the visual recognition accumulation period for generating the visual recognition image.

4. The photoelectric conversion device according to claim 1, wherein The one or more processors also execute the instructions to generate images with higher image quality in the visual recognition image processing compared to the image recognition image processing, and to generate images faster in the image recognition image processing compared to the visual recognition image processing.

5. The photoelectric conversion device according to claim 1, wherein The one or more processors further execute the instructions to perform compression for at least reducing the data amount of the image signal generated by the photoelectric conversion element in the image recognition image processing.

6. The photoelectric conversion device according to claim 5, wherein: The one or more processors also execute the instructions to reduce the amount of data by adding or averaging pixel values ​​of multiple pixels in the compression.

7. The photoelectric conversion device according to claim 5, wherein: The one or more processors also execute the instructions to reduce the amount of data by thinning out pixels in the rows or columns of the photoelectric conversion elements in the compression.

8. The photoelectric conversion device according to claim 5, wherein: The one or more processors also execute the instructions to reduce the amount of data by cutting out pixels in a partial area of ​​the photoelectric conversion element in the image recognition image processing.

9. The photoelectric conversion device according to claim 1, wherein The one or more processors also execute the instructions to perform defective pixel correction in the visual recognition image processing.

10. The photoelectric conversion device according to claim 1, wherein The one or more processors also execute the instructions to perform black level correction in the image recognition image processing.

11. The photoelectric conversion device according to claim 1, wherein The one or more processors also execute the instructions to allow the visual recognition image processing and the image recognition image processing to share processing portions.

12. The photoelectric conversion device according to claim 11, wherein The one or more processors also execute the instructions to allow the shared processing portion to use different correction parameters when performing the visual recognition and when performing the image recognition.

13. The photoelectric conversion device according to claim 1, wherein The photoelectric conversion element includes an avalanche photodiode.

14. A movable device comprising: a photoelectric conversion element configured to generate an image signal corresponding to an amount of incident light; one or more memories for storing instructions; as well as One or more processors for executing the instructions so that: Perform visual recognition image processing to generate images for visual recognition, performing image recognition image processing to generate an image for image recognition, and Control is performed so that the image signal generated by the photoelectric conversion element is image processed using at least one of the visual recognition image processing and the image recognition image processing, and the operation of the movable device is controlled.

15. A control method for controlling a photoelectric conversion device, The photoelectric conversion device comprises: a photoelectric conversion element configured to generate an image signal corresponding to an amount of incident light; a visual recognition image processing unit configured to perform visual recognition image processing to generate an image for visual recognition; as well as an image recognition image processing unit configured to perform image recognition image processing to generate an image for image recognition, The control method comprises: Control is performed so that image processing is performed on the image signal generated by the photoelectric conversion element using at least one of the visual recognition image processing unit and the image recognition image processing unit.

16. A non-transitory computer-readable storage medium configured to store a computer program for a photoelectric conversion device, in, The photoelectric conversion device comprises: a photoelectric conversion element configured to generate an image signal corresponding to an amount of incident light; a visual recognition image processing unit configured to perform visual recognition image processing to generate an image for visual recognition; and an image recognition image processing unit configured to perform image recognition image processing to generate an image for image recognition, and The computer program executes the following processing: controlling to perform image processing on the image signal generated by the photoelectric conversion element using at least one of the visual recognition image processing unit and the image recognition image processing unit.

17. A computer program product comprising a computer program, the computer program comprising instructions for a photoelectric conversion device, in, The photoelectric conversion device comprises: a photoelectric conversion element configured to generate an image signal corresponding to an amount of incident light; a visual recognition image processing unit configured to perform visual recognition image processing to generate an image for visual recognition; and an image recognition image processing unit configured to perform image recognition image processing to generate an image for image recognition, and The computer program executes the following processing: controlling to perform image processing on the image signal generated by the photoelectric conversion element using at least one of the visual recognition image processing unit and the image recognition image processing unit.

Images