Image pickup system, mobile device, image pickup method, storage medium, and computer program product

Abstract

The invention provides an image pickup system, a mobile device, an image pickup method, a storage medium and a computer program product. The image pickup system includes: a plurality of pixels each including a sensor unit that emits pulses according to reception of photons and a counter that counts the number of pulses; and one or more processors that execute instructions to generate a first signal based on a difference in count values of the counter between a start time and an end time of a first accumulation period, the image processing unit generates a first signal based on a difference in count values of the counter between a start time and an end time of the first accumulation period, generates a second signal based on a difference in count values of the counter between a start time and an end time of the second accumulation period, performs image processing on the first signal, performs image recognition on the image-processed first signal, and determines an image processing region based on a result of the image recognition. And performing image processing on the second signal based on the image processing area.

Description

Technical Field

[0001] The invention relates to a camera system, a movable device, a camera method and a storage medium. Background Art

[0002] Recently, a photoelectric conversion device has been developed that digitally counts the number of photons incident on an avalanche photodiode (APD) and outputs the count value from a pixel as a photoelectric conversion digital signal. In addition, for example, Japanese Patent No. 7223070 describes a photoelectric conversion device having an APD, which has a configuration that enables multiple video outputs with overlapping storage periods to enable continuous shooting even in low light.

[0003] When a camera outputs an image for the purpose of recognition or viewing, in most cases, the image is output after image processing such as noise reduction, blemish correction, or black level correction is applied to the image. In addition, an image suitable for recognition and viewing can be output by applying image processing. Applying high-performance image processing basically has a small number of defects, thereby making it possible to output images more efficiently.

[0004] In addition, when such a camera is used in a vehicle-mounted camera, it can be used for autonomous driving and collision avoidance. In this case, it is necessary to perform recognition as quickly as possible, and it is expected that the vehicle can be quickly controlled based on the recognition result.

[0005] However, when image processing is applied to the image output from the camera sensor, the processing time increases and a delay may occur when the recognition result is output. In particular, if the processing is complicated, the processing becomes a burden, which leads to an increase in delay time, thereby there is a possibility that the control of the vehicle is delayed. Summary of the invention

[0006] A camera system includes: a plurality of pixels, each of which includes a sensor unit and a counter, the sensor unit being configured to emit pulses according to the reception of photons, the counter being configured to count the number of the pulses; one or more memories storing instructions; and one or more processors executing the instructions to: generate a first signal based on the difference in count values ​​of the counter between the start and end of a first accumulation period, and generate a second signal based on the difference in count values ​​of the counter between the start and end of a second accumulation period, wherein the first accumulation period and the second accumulation period are within a full frame period and the first accumulation period is shorter than the second accumulation period; output the first signal during a period from the end of the first accumulation period to the end of the second accumulation period; perform image processing on the first signal; perform image recognition on the first signal after the image processing; determine an image processing area based on a result of the image recognition; and perform the image processing on the second signal based on the image processing area.

[0007] Furthermore, features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a diagram showing a configuration example of a photoelectric conversion element according to a first embodiment of the present invention.

[0009] Figure 2 2 is a diagram showing a structural example of the sensor plate 11 .

[0010] Figure 3 2 is a diagram showing a structural example of the circuit board 21 .

[0011] Figure 4 It is shown from Figure 2 and Figure 3 1 is a diagram of an equivalent circuit of a pixel 101 and a signal processing circuit 103 corresponding to the pixel 101.

[0012] Figure 5 is a diagram schematically showing the relationship between the operation of the APD 201 and the output signal.

[0013] Figure 6 is a functional block diagram of the imaging apparatus 600 and the movable device 700 according to the first embodiment.

[0014] Figure 7 1 is a diagram for explaining an image capturing method by the camera control unit 605 according to the first embodiment.

[0015] Figure 8 is a diagram showing an example of images of a plurality of divided frames.

[0016] Fig. 9 is a diagram showing the relationship between the memory circuit and the buffer according to the first embodiment.

[0017] Fig.10 is a flowchart showing details of operations when region determination is performed based on a recognition result and image processing is applied to a target region according to the first embodiment.

[0018] Fig.11 is a diagram schematically depicting the result of the identification process according to the first embodiment.

[0019] Fig.12 is a functional block diagram of an imaging apparatus 600 and a movable device 700 according to the second embodiment.

[0020] Fig.13 is a flowchart showing details of an operation example when the image processing area and the image processing intensity are changed based on the recognition result according to the second embodiment. DETAILED DESCRIPTION

[0021] The preferred embodiment of the present invention will be described below with reference to the accompanying drawings. In each figure, the same components or elements are given the same reference numerals, and repeated descriptions are omitted or simplified.

[0022] <First embodiment>

[0023] Figure 1 1 is a diagram showing a configuration example of a photoelectric conversion element according to a first embodiment of the present invention. Hereinafter, the photoelectric conversion element 100 will be described by taking an example of an image pickup apparatus having a so-called stacked structure in which two boards, a sensor board 11 and a circuit board 21, are stacked and electrically connected.

[0024] However, the imaging device may have a so-called non-stacked structure in which the structure included in the sensor board and the structure included in the circuit board are arranged on a common semiconductor layer. The sensor board 11 includes a pixel area 12. The circuit board 21 includes a circuit area 22 for processing a signal detected in the pixel area 12.

[0025] Figure 2 1 is a diagram showing a configuration example of the sensor panel 11. The pixel region 12 in the sensor panel 11 includes a plurality of pixels 101 arranged two-dimensionally in a plurality of rows and columns. Each pixel 101 has a photoelectric conversion unit 102 including an avalanche photodiode (hereinafter referred to as "APD").

[0026] Here, the photoelectric conversion unit 102 functions as a sensor unit that emits pulses at a frequency according to a photon reception frequency. In addition, the number of rows and columns of the pixel array forming the pixel region 12 is not limited to a specific number.

[0027] Figure 3 2 is a diagram showing a configuration example of the circuit board 21. The circuit board 21 includes a circuit board for processing Figure 2 The signal processing circuit 103 of the charge photoelectrically converted by each photoelectric conversion unit 102, the readout circuit 112, the control pulse generating unit 115, the horizontal scanning circuit 111, the vertical signal line 113, the vertical scanning circuit 110 and the output circuit 114.

[0028] The vertical scanning circuit 110 receives the control pulse supplied from the control pulse generating unit 115 and sequentially supplies the control pulse to a plurality of pixels arranged in the row direction. A logic circuit such as a shift register or an address decoder is used for the vertical scanning circuit 110 .

[0029] Each signal processing circuit 103 processes a signal output from the photoelectric conversion unit 102 of each pixel. The signal processing circuit 103 is provided with a counter and a memory, etc., and the memory holds a digital value. The horizontal scanning circuit 111 inputs a control pulse for sequentially selecting each column to each signal processing circuit 103 to read a signal from the memory holding the digital signal of each pixel.

[0030] The vertical signal line 113 receives the output of the signal from the signal processing circuit 103 of the pixel in the row selected by the vertical scanning circuit 110. The signal output to the vertical signal line 113 is output to the outside of the photoelectric conversion element 100 via the readout circuit 112 and the output circuit 114. The readout circuit 112 has a plurality of built-in buffers connected to the vertical signal line 113.

[0031] The plurality of signal processing circuits 103 are as follows Figure 2 and Figure 3 The plan view shown is arranged in a region overlapping with the pixel region 12. Therefore, the vertical scanning circuit 110, the horizontal scanning circuit 111, the readout circuit 112, the output circuit 114, and the control pulse generating unit 115 are arranged in a plan view so that the circuits and the units overlap in the gap between the end of the sensor plate 11 and the end of the pixel region 12.

[0032] In other words, the sensor panel 11 has a pixel area 12 and a non-pixel area arranged around the pixel area 12. In addition, the vertical scanning circuit 110, the horizontal scanning circuit 111, the readout circuit 112, the output circuit 114, and the control pulse generating unit 115 are arranged in a region overlapping with the non-pixel area in a plan view.

[0033] In addition, the configuration of the vertical signal line 113 and the configuration of the readout circuit 112 and the output circuit 114 are not limited to Figure 3 For example, the vertical signal line 113 may be arranged to extend in the row direction, and the readout circuit 112 may be arranged at the end to which the vertical signal line 113 extends. In addition, the signal processing circuit 103 may not necessarily be provided one-to-one for all photoelectric conversion units, and a configuration may be adopted in which a plurality of photoelectric conversion units share one signal processing circuit and perform signal processing sequentially.

[0034] Figure 4 It is shown from Figure 2 and Figure 3 FIG. 1 is a diagram showing an equivalent circuit of a pixel 101 and a signal processing circuit 103 corresponding to the pixel 101. Figure 4 As shown, each pixel 101 includes a photoelectric conversion unit 102 as a sensor portion, a counter circuit 211, a memory circuit 212, and the like.

[0035] The APD 201 included in the photoelectric conversion unit 102 performs photoelectric conversion to generate charge pairs from incident light. One of the two nodes of the APD 201 is connected to a power supply line from which a driving voltage VL (first voltage) is supplied. In addition, the other of the two nodes of the APD 201 is connected to a power supply line from which a driving voltage VH (second voltage) higher than the driving voltage VL is supplied.

[0036] exist Figure 4 In the embodiment, one node of the APD 201 is an anode, and the other node of the APD 201 is a cathode. The anode and cathode of the APD 201 receive a supply of a reverse bias voltage for causing the APD 201 to perform an avalanche multiplication operation. By supplying such a voltage, charges generated from incident light cause avalanche multiplication, and an avalanche current is generated.

[0037] Note that when a reverse bias voltage is supplied, there are a Geiger mode in which the APD is operated with a voltage difference between the anode and the cathode greater than the breakdown voltage, and a linear mode in which the APD is operated with a voltage difference between the anode and the cathode close to or lower than the breakdown voltage. The APD that is operated in the Geiger mode is called a SPAD. For the SPAD, for example, the driving voltage VL (first voltage) is -30V, and the driving voltage VH (second voltage) is 1V.

[0038] The signal processing circuit 103 has a quenching element 202, a waveform shaping unit 210, a counter circuit 211, and a memory circuit 212. The quenching element 202 is connected to a power supply line from which a driving voltage VH is supplied and one node of the anode and the cathode of the APD 201.

[0039] The quenching element 202 serves as a load circuit (quenching circuit) when a signal is multiplied by avalanche multiplication, and has a function (quenching operation) of preventing avalanche multiplication by suppressing the voltage supplied to the APD 201. In addition, the quenching element 202 has a function (recharging operation) of causing a current corresponding to a voltage drop in the quenching operation to flow, thereby returning the voltage supplied to the APD 201 to the driving voltage VH.

[0040] Figure 4 An example is shown in which the signal processing circuit 103 has a waveform shaping unit 210 , a counter circuit 211 , and a memory circuit 212 in addition to the quenching element 202 .

[0041] The waveform shaping unit 210 shapes the voltage change of the cathode of the APD 201 obtained when the photon is detected and outputs a pulse signal. For example, an inverter circuit is used for the waveform shaping unit 210. Figure 4 An example in which one inverter is used as the waveform shaping unit 210 is shown, but a circuit in which a plurality of inverters are connected in series may be used, and other circuits exhibiting a waveform shaping effect may be used.

[0042] The counter circuit 211 counts the number of pulses output from the waveform shaping unit 210 and holds the count value. In addition, when the control pulse RES is supplied through the drive line 213, the signal held in the counter circuit 211 is reset. Here, the counter circuit 211 generates a signal based on the difference in the count value between the start time and the end time of the accumulation period.

[0043] Memory circuit 212 is connected via Figure 4 The driving line 214 ( Figure 3 (not shown) receives Figure 3 The memory circuit 212 controls the supply of the control pulse SEL of the vertical scanning circuit 110, and switches between the electrical connection and disconnection of the counter circuit 211 and the vertical signal line 113. The memory circuit 212 is used as a memory for temporarily storing the count value of the counter circuit 211, and outputs the output signal of the counter circuit 211 from the pixel to the vertical signal line 113.

[0044] A switch such as a transistor may be arranged to switch electrical connection between the quenching element 202 and the APD 201 or between the photoelectric conversion unit 102 and the signal processing circuit 103. Similarly, the supply of the driving voltage VH or the driving voltage VL supplied to the photoelectric conversion unit 102 may be electrically switched by using a switch such as a transistor.

[0045] Figure 5 2 is a diagram schematically showing the relationship between the operation and output signal of the APD 201. The input side of the waveform shaping unit 210 is represented by node A, and the output side thereof is represented by node B. Between time t0 and time t1, a potential difference of VH and VL is applied to the APD 201. When a photon is incident on the APD 201 at time t1, avalanche multiplication occurs in the APD 201, an avalanche multiplication current flows to the quenching element 202, and the voltage of the node A drops.

[0046] When the voltage drop amount further increases and the potential difference applied to the APD 201 decreases, at time t2, avalanche multiplication of the APD 201 stops, and the voltage level of the node A does not drop to a level lower than or equal to a certain value.

[0047] Then, between time t2 and time t3, a current that compensates for the voltage drop from voltage VL flows to node A, and at time t3, node A becomes stable at the original potential level. At this time, the waveform shaping unit 210 shapes the portion of the output waveform at node A that exceeds a certain threshold, and outputs it from node B as a pulse signal.

[0048] Figure 6 1 is a functional block diagram of the imaging apparatus 600 and the movable device 700 according to the first embodiment. In the present embodiment, the imaging system is configured by some functional blocks of the imaging apparatus 600 and the movable device 700. Note that the imaging system is implemented by causing a computer (not shown) included in the imaging apparatus 600 and the movable device 700 to execute a computer program stored in a memory (not shown) serving as a storage medium. Figure 6 Some functional blocks are shown.

[0049] However, some or all of the functional blocks may be implemented as hardware. A dedicated circuit (ASIC) or a processor (reconfigurable processor or DSP) or the like may be used for hardware. Alternatively, Figure 6 The functional blocks shown may not be built into the same housing, or may be configured as individual devices connected to each other via signal paths. Figure 6 The above description is similarly applicable to the following Fig.12 .

[0050] The imaging device 600 has the photoelectric conversion element 100, an imaging optical system 601, an area determination unit 602, an image processing unit 603, a recognition unit 604, a camera control unit 605, a storage unit 606, a communication unit 607, and the like. Figures 1 to 5 As shown, the photoelectric conversion element 100 includes an avalanche diode for photoelectrically converting an optical image.

[0051] The imaging device according to the embodiment is installed in the movable device 700, and the camera unit including the set of the imaging optical system 601 and the photoelectric conversion element 100 is designed to shoot in at least one direction of the movable device, such as the front, rear, and side. Note that a plurality of camera units may be provided in the movable device 700.

[0052] The area determination unit 602 is an area determination unit for determining an area for performing predetermined image processing on an image signal acquired by the photoelectric conversion element 100 and determining a target area for performing predetermined image processing based on a recognition result output from a recognition unit 604 to be described below. In other words, the image processing area is determined based on the result of image recognition. Area information about the area determined by the area determination unit 602 is output to the camera control unit 605.

[0053] The image processing unit 603 performs image processing on the signal output from the pixel. In addition, the image processing unit 603 performs predetermined image processing on the image signal based on the area information output from the camera control unit 605, and generates a final image signal. The predetermined image processing includes, for example, edge enhancement, black level correction, gamma curve adjustment, noise reduction, digital gain adjustment, demosaicing, and data compression.

[0054] Note that in the case where the photoelectric conversion element 100 has an on-chip color filter of R, G, or B, etc., it is desirable that the image processing unit 603 performs processing such as white balance correction or color conversion. Note that the output of the image processing unit 603 is supplied to the recognition unit 604, the electronic control unit (ECU) 701 of the movable device 700, and the camera control unit 605.

[0055] The recognition unit 604 performs image recognition based on the result of the image processing. In other words, by performing image recognition based on the image signal that has been image processed by the image processing unit 603, surrounding people or vehicles are recognized, and a warning or the like is issued as necessary.

[0056] Note that although the movable device 700 is described using a vehicle as an example in this embodiment, the movable device may be any device such as an aircraft, a train, a ship, a drone, an AGV, or a robot as long as it is movable.

[0057] The camera control unit 605 includes a CPU functioning as a computer and a memory storing a computer program, and controls each unit of the imaging apparatus 600 when the CPU causes execution of the computer program stored in the memory.

[0058] Note that the camera control unit 605 functions as a control means, and controls the length of the exposure period of each frame of the photoelectric conversion element 100 or the timing of the control signal CLK or the like via, for example, a control pulse generating unit of the photoelectric conversion element 100 .

[0059] The storage unit 606 includes a recording medium capable of storing and reading image signals, such as a memory card or a hard disk, etc. The communication unit 607 includes a wireless or wired interface that outputs a generated image signal to the outside of the imaging apparatus 600 and receives various signals from the outside.

[0060] The ECU 701 includes a CPU serving as a computer and a memory storing a computer program, and controls each unit of the movable device 700 when the CPU causes the computer program stored in the memory to be executed.

[0061] The output of the ECU 701 is supplied to the vehicle control unit 702 and the display unit 703. The vehicle control unit 702 functions as a movement control means for performing driving, stopping, direction control, and the like of the vehicle as the movable device based on the output of the ECU 701. Note that the vehicle control unit 702 as the movement control means controls the movement of the movable device based on the result of the recognition processing using the signal generated in at least the first accumulation period. Note that the first accumulation period will be described below.

[0062] The display unit 703 serves as a display part, includes a display element such as a liquid crystal device or an organic EL, and is mounted on the movable device 700. The display unit 703 displays various information related to the image acquired by the photoelectric conversion element 100 and the driving state of the vehicle to the driver of the movable device 700 by using, for example, a GUI based on the output of the ECU 701.

[0063] Notice, Figure 6 The image processing unit 603 and the recognition unit 604 etc. in the mobile device 700 may not be installed in the mobile device 700, or may be set in an external terminal etc. which is set separately from the mobile device 700 and is used to remotely control the mobile device 700 or to monitor the travel of the mobile device. The camera device including the camera device 600 of the present embodiment includes a device composed of a plurality of devices connected on such a communication path.

[0064] Figure 71 is a diagram for explaining the imaging method of the camera control unit 605 according to the first embodiment. In the present embodiment, photoelectric conversion is periodically performed at a rate of, for example, 30 full frames / sec. Note that, in the present embodiment, one frame having a length of, for example, 33.3 ms is referred to as a full frame (or main frame), and a frame obtained by dividing a full frame into four is referred to as a frame (or sub frame).

[0065] In other words, if Figure 7 As shown, the full frame 1 is divided into frame 1_1, frame 1_2, frame 1_3 and frame 1_4 with equal time periods (each 8.33 ms) from time T0 to time T4. Figure 7 In and subsequent figures, frame 0_3, frame 0_4, frame 1_1, frame 1_2, frame 1_3, frame 1_4, frame 2_1, frame 2_2, frame 2_3 and frame 2_4 are shown as F0_3, F0_4, F1_1, F1_2, F1_3, F1_4, F2_1, F2_2, F2_3 and F2_4.

[0066] Note that frame 1_1 has an accumulation period from the start time T0 of frame 1 to time T1, and frame 1_2 has an accumulation period from time T0 to time T2. In addition, frame 1_3 has an accumulation period from time T0 to time T3, and frame 1_4 has an accumulation period from time T0 to time T4. Note that in the present embodiment, for example, the accumulation period from time T0 to time T1 is referred to as the first accumulation period, and the accumulation period from time T0 to time T4 is referred to as the second accumulation period.

[0067] Furthermore, at time T0 , the counter circuit 211 is reset, and at times T1 to T4 , respective count values ​​C1_1 , C1_2 , C1_3 , and C1_4 are acquired from the counter circuit 211 .

[0068] Furthermore, the count values ​​C1_1, C1_2, C1_3, and C1_4 are temporarily held in the memory circuit 212. Furthermore, signals corresponding to one row temporarily stored in the memory circuit 212 are sequentially output from the photoelectric conversion elements via the buffer of the readout circuit 112.

[0069] As described above, in the present embodiment, the signal accumulated in the period of frame 1_1 is read out from time T1 to time T2, and is quickly processed by the recognition unit 604. Therefore, image recognition can be performed quickly. Similarly, the signal accumulated in the period of frame 1_2, frame 1_3, and frame 1_4 is sequentially read out at time T2 to T3, time T3 to T4, and time T4 to T1, respectively, so that image recognition can be repeatedly performed.

[0070] Figure 8 is a diagram showing an example of an image of a plurality of segmented frames. Figure 8 As shown, the image of frame 1_1 is dark due to the short accumulation time, and the subject blur of the person jumping in is also small. On the other hand, as the accumulation time becomes longer in the order of frame 1_2, frame 1_3, and frame 1_4, subject blur is likely to occur. Note that blur is less likely to occur on a stopped vehicle or a white line, and the contrast is likely to increase as the accumulation time becomes longer.

[0071] As described above, in the control steps of the camera control unit 605 in the present embodiment, the accumulation operations of the first accumulation period and the second accumulation period are performed within one full frame period. In addition, the first accumulation period is shorter than the second accumulation period, and the signal generated in the first accumulation period is controlled so as to be output during the period from the end of the first accumulation period to the end of the second accumulation period.

[0072] In addition, in the present embodiment, the first accumulation period overlaps with the second accumulation period, and the first accumulation period and the second accumulation period start at the same time. In addition, the end time of the second accumulation period is the disconnection of the frame, and the second accumulation period is an integer multiple of the first accumulation period. In other words, the end time of the second accumulation period coincides with the end time of the full frame period.

[0073] However, the second accumulation period need not be an integer multiple of the first accumulation period.The second accumulation period may be longer than the first accumulation period (the first accumulation period may be shorter than the second accumulation period), and the end of the second accumulation period may be later than the end of the first accumulation period.

[0074] In other words, an image with a shorter accumulation period and an image with a longer accumulation period are created, the timing of the end of the shorter accumulation period is set to be earlier than the timing of the end of the longer accumulation period, and as soon as the shorter accumulation period ends, the image is output to be sent to the recognition unit at a later stage. In addition, the subject is recognized based on the signal generated in at least the first accumulation period. The recognition unit 604 as the recognition means recognizes the subject based on the signal generated in at least the first accumulation period.

[0075] Therefore, in the related art, if one full frame period has not passed, image recognition cannot be performed, but in the present embodiment, image recognition can be performed at the shortest after 1 / 4 full frame period, and for example, when the movable device moves at high speed, obstacles can be quickly recognized. Therefore, the brakes can be applied quickly. Alternatively, obstacles can be avoided earlier.

[0076] Note that since the contrast of an image with a long accumulation period can be improved, it can be used as an image for display. In other words, an image with a short accumulation time is suitable for rapid subject recognition, and an image with a long accumulation time is suitable for a display image. As described above, the display device of this embodiment displays an image using a signal generated at least in the second accumulation period. Note that the recognition unit 604 can also recognize a subject based on a signal generated at least in the second accumulation period.

[0077] In addition, since APD is used in this embodiment, and unlike CMOS sensors, the accumulated charge does not deteriorate due to readout, the accumulation periods can overlap. In addition, since there is no readout noise, the original signal does not deteriorate no matter how many times readout is performed in one accumulation.

[0078] Fig. 9 is a diagram showing the relationship between the memory circuit and the buffer according to the first embodiment. Fig. 9 Show Figure 3 The memory circuits 212 in the signal processing circuit 103 are arranged in N rows and M columns, and the memory circuits are respectively composed of memory 1-1 To Storage N-M In addition, Fig. 9 Buffer 1 to Buffer M represent Figure 3 The readout circuit 112 includes a buffer. Fig. 9 The output circuit 114 corresponds to Figure 3 The output circuit 114 in.

[0079] Fig.10 is a flowchart showing the details of the operation when performing area determination based on the recognition result and applying image processing to the target area according to the first embodiment, and is Figure 6 and Fig.10 A first example of the operation of changing the image processing range based on the recognition result of this embodiment will be described.

[0080] Note that the following operations are performed sequentially by executing a computer program stored in a memory by a CPU serving as a computer included in the camera control unit 605. Fig.10 The operations of each step of the flowchart.

[0081] In step S101, the CPU sets an initial target region to be subjected to image processing using the region determination unit 602. At this time, the initial target region may be a pre-set region or a region used in a previous frame.

[0082] In step S102, the CPU uses the camera sensor to obtain the image signal of frame 1_n. Here, frame 1_n represents Figure 7One of frames (F) 1_1 to 1_4 in .

[0083] In step S103, the CPU applies image processing to the target area in the set image by using the image processing unit 603. In other words, the image processing unit 603 performs image processing on the area set (determined) by the area determination unit 602. Here, step S103 serves as an image processing step of performing image processing on the signal output from the pixel, and performs predetermined image processing on the image processing area.

[0084] Here, the image processing includes, as described above, edge enhancement, black level correction, gamma curve adjustment, noise reduction, digital gain adjustment, demosaicing, and data compression, etc. The image signal subjected to the image processing by the image processing unit 603 is output to the recognition unit 604 .

[0085] In step S104, the CPU applies recognition processing to the processed image by using the recognition unit 604. For the recognition result generated at this time, information related to at least one of the subject position, subject recognition accuracy, and subject type, etc. is output as the recognition result. The output recognition result is output to the area determination unit 602 and the ECU 701.

[0086] Here, step S104 serves as a recognition step of performing image recognition based on the result of image processing, and image recognition is performed based on a signal that has undergone predetermined image processing.

[0087] In step S105, the CPU checks the current frame (n value). For frame 1_1, frame 1_2, and frame 1_3, the process proceeds to step S106. The process of frame 1_4 ends and proceeds to the next frame 2_1. Here, although the maximum value of n is set to 4 in the present embodiment, this is an example, and n may have a value other than 4 (a value that does not satisfy n=4).

[0088] In step S106, the CPU determines whether the subject has been recognized based on the result of the recognition processing. If the subject has been detected, the operation proceeds to step S107. If the subject has not been recognized, the operation proceeds to step S101.

[0089] Since the subject has been recognized in step S107, the CPU updates the region determination unit 602 so that a region including a part or all of the recognized subject is included in the image processing target region. Note that for the target region, a large region on the moving direction side can be determined in consideration of the speed of the recognized subject and the moving direction of the recognized subject.

[0090] In other words, in step S107, the region determination unit 602 determines the region to which the image processing is applied according to the recognition result output from the recognition unit 604. Here, step S107 serves as a region determination step of determining the image processing region based on the result of the image recognition.

[0091] here, Fig.11 11 is a diagram schematically depicting the result of the recognition process according to the first embodiment, and shows an example of an image in which a region has been determined based on the recognition result. Region 1101 in the image is a region in which a vehicle is recognized. Region 1102 is a region in which a person is recognized.

[0092] Region 1103 is a region excluding region 1101 and region 1102 in the field of view. The range to which image processing is applied is a region including a part or all of the recognized subject, such as region 1101 and region 1102. Note that reference numeral 1104 indicates the recognition accuracy of the subject in region 1101 and is Fig.11 On the other hand, reference numeral 1105 indicates the recognition accuracy of the subject in the area 1102 and is displayed as 30%.

[0093] In addition, as for the size of the area, a large area in the moving direction of the subject can be determined in consideration of the recognized subject that will move in the next frame as described above. In the case where the subject is not recognized, a preset initial area can be determined, or the entire angle of view can be determined as the area. In step S107, the area information determined (updated) by the area determination unit 602 is output to the camera control unit 605, and the process proceeds to step S102 by increasing the value n by one.

[0094] In this embodiment, the image processing unit 603 performs predetermined image processing on the image processing area, and the recognition unit 604 performs image recognition based on the signal that has undergone the predetermined image processing as described above. In other words, by changing and processing the range of application of image processing according to the recognition result, the image processing range can be narrowed, and the time required for recognition can be reduced.

[0095] <Second embodiment>

[0096] Fig.12 is a functional block diagram of the imaging device 600 and the movable device 700 according to the second embodiment, and in the second embodiment, the image processing range and the image processing intensity are designed to be changed based on the recognition result and the accumulation period, etc. Note that since Fig.12 in Figure 6 The same reference numerals as in the figure represent the same functional blocks, and their description will be omitted.

[0097] exist Fig.12 In the example, the region information determined by the region determination unit 602 is output to the image processing strength determination unit 608. The image processing strength determination unit 608 determines the image processing strength based on at least the result of the image recognition by the recognition unit.

[0098] In this embodiment, the image processing intensity determination unit 608 determines the image processing intensity when the image processing unit 603 performs image processing based on the recognition result output from the recognition unit 604, the area information output from the area determination unit 602, and the accumulation period output from the camera control unit 605.

[0099] As mentioned above, for example, Fig.11 Reference numeral 1104 represents the recognition accuracy of a vehicle, and here the subject has been recognized as a "vehicle" with an accuracy of "99%". In addition, recognition accuracy 1105 represents the recognition accuracy of a person, and here the subject has been recognized as a "person" with an accuracy of "30%". As described above, in the case where the recognition accuracy differs depending on each subject, the image processing intensity for the subject with a high recognition accuracy is not changed. Alternatively, the image processing intensity is reduced.

[0100] On the other hand, the image processing intensity for the subject with low recognition accuracy is relatively increased. In addition, the image processing intensity is changed according to the accumulation period output from the camera control unit 605. For example, in the first accumulation period and the second accumulation period, since the first accumulation period is shorter, the image is formed to have a lot of noise and is darker.

[0101] For this reason, the noise reduction intensity is set higher for the image signal generated in the first accumulation period than for the image signal acquired in the second accumulation period. Furthermore, the image processing intensity determination unit 608 outputs the area information and the image processing intensity information for each area information to the camera control unit 605.

[0102] The camera control unit 605 receives the region information and the image processing intensity information from the above-mentioned image processing intensity determination unit 608, and outputs the information to the image processing unit 603. In addition, the camera control unit 605 outputs the information related to the accumulation time to the image processing intensity determination unit 608. Then, the image processing unit 603 performs image processing on the region determined by the region determination unit 602, and performs image processing on each region at the intensity determined by the image processing intensity determination unit 608.

[0103] Note that, in the case where the subject is not recognized, as described above, a pre-set area may be determined, or the entire viewing angle may be determined as the area. In addition, the area may be changed for each image processing block (image processing process). For example, the entire image may be processed for black level correction to avoid coloration, and other image processing blocks (image processing processes) may be operated to limit the area.

[0104] Fig.13 605 is a flowchart showing details of an operation example when the image processing area and the image processing intensity are changed based on the recognition result according to the second embodiment. Note that the operation is sequentially performed by the CPU serving as a computer included in the camera control unit 605 executing a computer program stored in a memory. Fig.13 The operations of each step of the flowchart.

[0105] In step S201, the CPU sets an initial target region and an initial image processing strength for image processing. At this time, the initial target region and the initial image processing strength can be a pre-set region and the image processing strength for the region, or a region and the image processing strength for the region used in a previous frame.

[0106] In step S202, the CPU uses the camera sensor to obtain the image signal of frame 1_n. Here, frame 1_n represents Figure 7 One of frames (F) 1_1 to 1_4 in .

[0107] In step S203, the CPU applies image processing to each target area in the image set in step S201 at a set intensity. Here, the image processing includes edge enhancement, black level correction, gamma curve adjustment, noise reduction, digital gain adjustment, demosaicing and data compression.

[0108] In step S204, the CPU applies recognition processing to the processed image. The recognition result generated at this time includes information related to at least one of the position of the object, the accuracy of object recognition, and the type of the object.

[0109] In step S205, the CPU checks the current frame (n value). For frame 1_1, frame 1_2, and frame 1_3, the process proceeds to step S206. The process for frame 1_4 ends and proceeds to the next frame 2_1. Here, the maximum value of n may be a value other than 4 (a value that does not satisfy n=4).

[0110] In step S206, the CPU checks whether a subject has been recognized based on the result of the recognition processing. If a subject has been detected, the operation proceeds to step S207. If a subject has not been recognized, the operation proceeds to step S201.

[0111] Since the subject has been identified in step S207, the CPU updates the area including the subject as the image processing target area. At this time, the target area is an area including a part or all of the identified subject. In addition, for this area, a large area can be set according to the moving direction or moving speed, considering the identified subject that will move in the next frame.

[0112] In step S208, the CPU changes the image processing intensity based on the recognition result and the exposure time information. For example, the recognition accuracy of the recognized subject is output as the recognition result. The operation is performed so that: when the recognition accuracy is low, the image processing intensity is updated in an increasing manner, and when the recognition accuracy is high, the image processing intensity is not changed (or the image processing intensity is reduced). In addition, the image processing intensity can be changed according to the exposure time.

[0113] In other words, in this embodiment, as the n value of frame 1_n is smaller, the exposure time becomes shorter. As the exposure time becomes shorter, the image has a tendency to contain more darkness and noise. Here, in the case of a short exposure time, it is desirable to prevent the recognition accuracy from decreasing by applying image processing at a higher intensity.

[0114] As described above, in the second embodiment, by changing the range of application of image processing according to the recognition result and changing the image processing intensity for each range for processing, the recognition processing range and the image processing range can be reduced, and the delay time can be shortened. In addition, efficient image processing for recognition processing can be performed. In addition, since the accumulated results of a short period of time can be read in the middle of processing according to the second embodiment, image processing and recognition processing can be performed quickly.

[0115] Although the present embodiment has been described in detail above based on the typical embodiment, the present invention is not limited thereto, and various modifications can be made based on the gist of the present invention, and these modifications are not excluded from the scope of the present invention.

[0116] Furthermore, in the above-described embodiment, for example, accumulation is performed in a 1 / 4 full frame period at the shortest, and readout is performed in each 1 / 4 full frame period. Figure 7 The counter circuit is reset midway during the accumulation period of frame 1_1, and the actual accumulation period may be shorter than 1 / 4 of the full frame period.

[0117] Alternatively, you can first Figure 7 The counter circuit is reset at time T1, etc. Thus, the count value read at time T4 can be adjusted.

[0118] Although the present invention has been described with reference to typical embodiments, it should be understood that the present invention is not limited to the disclosed typical embodiments. The scope of the appended claims is consistent with the broadest interpretation to include all such modifications, equivalent structures and functions. In addition, the above-mentioned first embodiment to the second embodiment, etc. can be appropriately combined.

[0119] In addition, as part or all of the control according to the embodiment, a computer program that realizes the functions of the above-described embodiment may be provided to a camera system or the like through a network or a variety of storage media. Then, a computer (or CPU, MPU, etc.) of the camera system or the like may be configured to read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention.

[0120] In addition, the present invention includes content implemented using at least one processor and circuit configured to perform the functions of the above-mentioned embodiments. For example, multiple processors can be used to distribute processing to perform the functions of the above-mentioned embodiments.

[0121] This application claims the benefit of Japanese Patent Application No. 2023-106676, filed on Jun. 29, 2023, which is hereby incorporated by reference in its entirety.

Claims

1. A camera system, comprising: a plurality of pixels, each of which includes a sensor unit and a counter, the sensor unit being configured to emit a pulse according to the reception of a photon, the counter being configured to count the number of the pulses; a generating unit configured to generate a first signal based on a difference in count value of the counter between a start time and an end time of a first accumulation period, and to generate a second signal based on a difference in count value of the counter between a start time and an end time of a second accumulation period, wherein the first accumulation period and the second accumulation period are within one full frame period, and the first accumulation period is shorter than the second accumulation period; an output unit configured to output the first signal during a period from an end of the first accumulation period to an end of the second accumulation period; an image processing unit, configured to perform image processing on the first signal; a recognition unit configured to perform image recognition on the first signal subjected to the image processing; as well as a region determining unit configured to determine an image processing region based on a result of the image recognition, The image processing unit performs the image processing on the second signal based on the image processing area determined by the area determination unit. 2 . The camera system according to claim 1 , further comprising an image processing intensity determination unit configured to determine the image processing intensity based on a result of the image recognition by the recognition unit.

3. The camera system according to claim 1, wherein: The first accumulation period overlaps with the second accumulation period.

4. The camera system according to claim 1, wherein: The first accumulation period and the second accumulation period start simultaneously.

5. The camera system according to claim 1, wherein: An end time of the second accumulation period coincides with an end time of the full frame period. 6 . The camera system according to claim 1 , further comprising a display unit configured to display the second signal as an image.

7. The camera system according to claim 1, wherein: The sensor unit includes an avalanche photodiode.

8. A movable device comprising: The camera system according to any one of claims 1 to 7; as well as A movement control component is configured to control the operation of the movable device based on the result of the recognition processing of the first signal.

9. An imaging method for an imaging system, the imaging system having a plurality of pixels each including a sensor unit and a counter, the sensor unit being configured to emit pulses at a frequency according to a reception frequency of photons, the counter being configured to count the number of the pulses, the imaging method comprising: generating a first signal based on a difference in count value of the counter between a start time and an end time of a first accumulation period, and generating a second signal based on a difference in count value of the counter between a start time and an end time of a second accumulation period, wherein the first accumulation period and the second accumulation period are within one full frame period and the first accumulation period is shorter than the second accumulation period; outputting the first signal during a period from an end of the first accumulation period to an end of the second accumulation period; performing image processing on the first signal; performing image recognition on the first signal that has undergone the image processing; determining an image processing area based on a result of the image recognition; as well as The image processing is performed on the second signal based on the image processing area. 10 . A computer-readable storage medium storing a program for causing a computer to execute each step of the imaging method according to claim 9 . 11 . A computer program product comprising a program for causing a computer to execute each step of the imaging method according to claim 9 .

Images