Image processing apparatus, image processing method, and computer program

Abstract

The present invention provides an image processing device that can reduce the workload of edge detection processing. [Solution] An image processing device for processing output from an image sensor having a counter for counting the output of an avalanche photodiode provided for each pixel, determination means for determining whether the count value of the counter for each pixel has reached a predetermined value within a predetermined determination period from the start of exposure and outputting a determination result, and output means for outputting the determination result and the count value for each pixel, the image processing device having acquisition means for acquiring the count value and the determination result of a plurality of pixels in a predetermined range including a predetermined pixel output by the output means, and edge detection means for detecting edge information by performing a weighting calculation on the count values ​​of the plurality of pixels in the predetermined range according to the determination result.

Description

Technical Field

[0006] , ,

[0005] ,

[0007] ,

[0001] The present invention relates to an image processing apparatus, an image processing method, a computer program, and the like.

Background Art

[0002] There is known an imaging device that performs photoelectric conversion by digitally counting the number of photons arriving at an avalanche photodiode (APD) and outputting the count value from each pixel.

[0003] Patent Document 1 describes an imaging device including an APD, a detection unit that detects an avalanche current, and a switch disposed between the APD and the detection unit. Further, Patent Document 1 describes a reset unit that applies a predetermined potential to an input unit of the switch to reset a node between the switch and the detection unit.

[0004] In Patent Document 1, by resetting the reset unit with a clock pulse at a constant period, it is possible to suppress the power consumption of the imaging device even when photons are incident on the APD at a high frequency. Further, it is possible to obtain accurate signal information in which the linearity between the number of photons incident on the APD and the count value of the photons detected by the imaging device is maintained.

[0005] Also, when the count values corresponding to each exposure time reach a predetermined threshold at a plurality of exposure times shorter than the maximum exposure time, the operating voltage of the APD is changed. Thereby, the counting of the number of photons is paused, and the estimated value of the count is calculated and substituted, so that the power consumption can be reduced.

[0006] Here, the estimated value of the count is a count value that is expected to be obtained when the APD is exposed for the length of the maximum exposure time, based on the count value when the count is paused at an exposure time shorter than the maximum exposure time.

[0007] By using the photoelectric conversion technology disclosed in Patent Document 1, the effects of readout noise can be eliminated and the number of photons can be counted digitally. This allows for the detection of even faint light in dark places and enables the capture of images with a very wide dynamic range.

[0008] On the other hand, as disclosed in Patent Document 2, for example, an edge enhancement function (sometimes called contour enhancement or peaking) that highlights the high-frequency components of an image is known as a technique for visually identifying the in-focus area in an imaging device. [Prior art documents] [Patent Documents]

[0009] [Patent Document 1] Japanese Patent Publication No. 2021-19281 [Patent Document 2] Japanese Patent Publication No. 2006-58683 [Overview of the project] [Problems that the invention aims to solve]

[0010] However, video data that secures a very wide dynamic range using the photoelectric conversion technology disclosed in Patent Document 1 has a deeper bit depth, which makes the edge detection process for conventional edge enhancement processing more computationally intensive.

[0011] Furthermore, because the high-frequency components detected in dark areas have small values, simply superimposing these values ​​onto the video signal makes it difficult to discern the enhancement effect. Therefore, multiplying the detected values ​​by a large gain before superimposing them onto the video signal makes them easier to see. However, if the high-frequency components detected in bright areas are large, applying a large gain similar to that used in dark areas and superimposing them onto the video signal will cause overexposure, making it difficult to discern the enhancement effect.

[0012] One of the objectives of this invention is to provide an image processing apparatus that can reduce the workload of edge detection processing. [Means for solving the problem]

[0013] To solve the above problems, the image processing apparatus of the present invention, as described in one aspect, A counter that counts the output of an avalanche photodiode provided for each pixel, A determination means that determines whether the count value of the counter for each pixel has reached a predetermined value within a predetermined determination period from the start of exposure, and outputs a determination result, An image processing apparatus for processing output from an image sensor, having output means for outputting the determination result and the count value for each pixel, An acquisition means for acquiring the count value and the determination result of a plurality of pixels within a predetermined range including a predetermined pixel output by the output means, An edge detection means that detects edge information by performing a weighting calculation on the count values ​​of a plurality of pixels within the predetermined range according to the determination result, It is characterized by having the following features. [Effects of the Invention]

[0014] According to the present invention, an image processing apparatus that can reduce the workload of edge detection processing can be provided. [Brief explanation of the drawing]

[0015] [Figure 1] This is a functional block diagram showing an example configuration of the imaging device 100 according to Embodiment 1. [Figure 2] This figure shows an example configuration of the photoelectric conversion element 200 according to Embodiment 1. [Figure 3] This figure shows an example configuration of the sensor substrate 21 according to Embodiment 1. [Figure 4] This figure shows an example configuration of the circuit board 23 according to Embodiment 1. [Figure 5] This figure shows an example of an equivalent circuit for pixel 301 and the signal processing circuit 401 corresponding to pixel 301. [Figure 6]This is a diagram schematically showing an example of the relationship between the operation of APD501 according to Embodiment 1 and the output signal. [Figure 7] This is a timing chart for explaining an example of the operation of the signal processing circuit 401 according to Embodiment 1. [Figure 8] This is a diagram showing an example of the relationship between the exposure time for each pixel 301 included in the photoelectric conversion element 200 according to Embodiment 1 and the count value of the counter circuit 511. [Figure 9] This is a diagram showing an example of the exposure time information output according to the exposure time when the count threshold is exceeded. [Figure 10] This is a diagram showing an example of the relationship between the exposure time for each pixel 301 included in the photoelectric conversion element 200 according to Embodiment 1 and the count value of the counter circuit 511. [Figure 11] This is a flowchart showing an example of the operation of the image processing unit 105 in Embodiment 1. [Figure 12] This is a flowchart showing an example of the operation of the image processing unit 105 in Embodiment 2.

Mode for Carrying Out the Invention

[0016] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiments. In each figure, the same members or elements are denoted by the same reference numerals, and duplicate explanations are omitted or simplified.

[0017] <Embodiment 1> FIG. 1 is a functional block diagram showing a configuration example of the imaging device 100 according to Embodiment 1. Note that some of the functional blocks shown in FIG. 1 are realized by causing a CPU or the like as a computer included in the imaging device 100 to execute a computer program stored in a memory as a storage medium.

[0018] However, some or all of these may be implemented in hardware. Hardware options include dedicated circuits (ASICs) and processors (reconfigurable processors, DSPs). Furthermore, each functional block shown in Figure 1 does not necessarily have to be housed in the same enclosure; they may be composed of separate devices connected to each other via signal paths.

[0019] In Figure 1, the imaging device 100 is a device capable of image input, output, and recording. In this embodiment, the imaging device 100 functions as an image processing device that executes an image processing method for processing the output from the image sensor.

[0020] In Figure 1, the CPU 102, ROM 103, RAM 104, image processing unit 105, lens unit 106, imaging unit 107, network module 108, and video terminal unit 109 are connected to the internal bus 101.

[0021] Furthermore, the recording medium interface 110, frame memory 111, operation unit 113, display unit 114, object detection unit 115, and attitude detection unit 116 are connected to the internal bus 101. Each unit connected to the internal bus 101 is configured to exchange data with each other via the internal bus 101.

[0022] The lens unit 106 consists of a lens group including a zoom lens and a focus lens, an aperture mechanism, and a drive motor. The lens unit 106 drives the lens group to change zoom and focus under control from the CPU 102.

[0023] The lens unit 106 also stores lens information such as the specifications of the lens group, the current aperture value, focal length, and the distance to the subject in focus (hereinafter referred to as the focusing distance) in its internal memory, and provides this lens information in response to inquiries from the CPU 102. The lens information is stored in the RAM 104 by the CPU 102 and used for various controls of the imaging device 100.

[0024] The optical image that has passed through the lens unit 106 is received by the imaging unit 107. The imaging unit 107 has R, Gr, Gb, and B color filters in a Bayer array (not shown) and microlenses, and converts the optical signal into an electrical signal using a photoelectric conversion element 200, which will be described later.

[0025] The CPU 102, acting as a computer, controls various parts of the imaging device 100 using the RAM 104 as work memory, according to the computer program stored in the ROM 103, which acts as a storage medium.

[0026] ROM 103 is a non-volatile recording element that stores computer programs and various adjustment parameters for operating the CPU 102. RAM 104 is a volatile memory that uses semiconductor elements, and generally, a slower and smaller capacity version is used compared to frame memory 111.

[0027] The frame memory 111 is an element that can temporarily store video signals and read them out when needed. Since video signals contain a huge amount of data, high bandwidth and large capacity are required.

[0028] For the frame memory 111, for example, DDR4-SDRAM (Dual Data Rate 4-Synchronous Dynamic RAM) can be used. By using the frame memory 111, it becomes possible to perform processes such as combining images that are different in time or extracting only the necessary areas.

[0029] The image processing unit 105 performs various image processing operations on data from the imaging unit 107 or on image data stored in the frame memory 111 or recording medium 112, based on the control of the CPU 102. The image processing unit 105 also functions as a means for generating video signals.

[0030] The image processing performed by the image processing unit 105 includes pixel interpolation, encoding, compression, decoding, scaling (resizing), noise reduction, and color conversion of image data.

[0031] Furthermore, the image processing unit 105 performs processing such as correcting variations in the performance of pixels in the imaging unit 107, correcting defective pixels, correcting white balance, correcting brightness, and correcting distortion and vignetting caused by lens characteristics.

[0032] Furthermore, the image processing unit 105 may be composed of a dedicated circuit block for performing specific image processing. Also, depending on the type of image processing, the CPU 102 may perform the image processing according to the program without using the image processing unit 105.

[0033] Based on the calculation results obtained by the image processing unit 105, the CPU 102 controls the lens unit 106 to optically adjust the image magnification, focal length, and aperture for adjusting the amount of light. It is also possible to perform image stabilization by moving a part of the lens group on a plane perpendicular to the optical axis.

[0034] An operating unit 113 is used as an interface with the outside of the device to receive user input. The operating unit 113 uses mechanical elements such as buttons and switches, and consists of a power switch, a mode selector switch, etc.

[0035] The display unit 114 is a display device that can be viewed by the user, and can display, for example, images processed by the image processing unit 105 or setting menus, allowing the user to check the operating status of the imaging device 100.

[0036] The display unit 114 utilizes small, low-power devices such as LCDs (Liquid Crystal Displays) or organic ELs (Electroluminescence) as display devices. Furthermore, it may also incorporate resistive or capacitive thin-film elements, known as touch panels, and may be used as part of the operation unit 113.

[0037] The CPU 102 generates strings of text to inform the user of the settings status of the imaging device 100, as well as menus for configuring the imaging device 100. These are then superimposed on the image processed by the image processing unit 105 and displayed on the display unit 114. In addition to text information, other imaging assist displays such as histograms, vectorscopes, waveform monitors, zebra stripes, peaking, and false color may also be superimposed.

[0038] The video terminal section 109 may use, for example, SDI (Serial Digital Interface), HDMI (registered trademark) (High Definition Multimedia Interface), DisplayPort (registered trademark), etc.

[0039] By outputting a video signal via the video terminal 109, it is possible to display images in real time on an external monitor (not shown). Furthermore, the CPU 102 can process the image input via the video terminal 109 using the image processing unit 105 and display it on the display unit 114.

[0040] The network module 108 can transmit not only images but also control signals, and is an interface for inputting and outputting video and audio signals. The network module 108 can also communicate with external devices via the internet, etc., and send and receive various data such as files, commands, video signals, and metadata. The transmission method by the network module 108 can be wireless or wired.

[0041] The imaging device 100 not only outputs images to the outside but also has the function of recording them internally. The recording medium 112 can record image data and various setting data, and a large-capacity memory element is used. For example, an HDD (Hard Disk Drive) or SSD (Solid State Drive) is used and is mounted on the recording medium I / F 110.

[0042] When the user presses a designated button on the control unit 113 (hereinafter referred to as the recording button), the CPU 102 begins recording the image data processed by the image processing unit 105 onto the recording medium 112 via the recording medium I / F 110.

[0043] Then, when the user presses the record button again, recording stops. During recording, the imaging device 100 outputs a signal indicating that it is recording via the network module 108 and the video terminal unit 109.

[0044] This makes it possible to inform an external device that recording is in progress, and to record image data output from the imaging device 100 on the external device in conjunction with the recording operation on the imaging device 100.

[0045] The object detection unit 115 is a block for detecting objects using artificial intelligence, such as deep learning using a neural network.

[0046] When performing object detection using deep learning, the CPU 102 transmits the processing program stored in the ROM 103, as well as network structures such as SSD (Single Shot Multibox Detector) and YOLO (You Only Look Once), and weight parameters, to the object detection unit 115.

[0047] The object detection unit 115 performs processing to detect objects from the video signal based on various parameters obtained from the CPU 102, and loads the processing results into the RAM 104.

[0048] The attitude detection unit 116 detects the attitude state of the imaging device 100 using, for example, a gyro sensor or an accelerometer. This makes it possible to detect whether the imaging device 100 is tilted or shaking.

[0049] Figure 2 shows an example of the configuration of a photoelectric conversion element 200 according to Embodiment 1. In the following description, an imaging device in which the photoelectric conversion unit has a so-called stacked structure is described as an example, in which the photoelectric conversion element 200 is composed of two substrates, a sensor substrate 21 and a circuit substrate 23, which are stacked and electrically connected. The circuit substrate 23 includes a circuit region 24 that processes signals detected in the pixel region 22.

[0050] Figure 3 shows an example of the configuration of a sensor substrate 21 according to Embodiment 1. The pixel region 22 of the sensor substrate 21 includes a plurality of pixels 301 arranged in a two-dimensional manner across multiple rows and columns. Each pixel 301 includes a photoelectric conversion unit 302, which includes an avalanche photodiode (hereinafter referred to as APD). The number of rows and columns of the pixel array forming the pixel region 22 is not limited to the example in Figure 3.

[0051] Figure 4 shows an example of the configuration of a circuit board 23 according to Embodiment 1. The circuit board 23 has a signal processing circuit 401 for processing the charge photoelectrically converted by the photoelectric conversion unit 302 in Figure 3, a readout circuit 402, a pulse generation unit 403, a horizontal scanning circuit unit 404, a signal line 405, and a vertical scanning circuit unit 406.

[0052] The signal output from the photoelectric conversion unit 302 of the pixel is processed by the signal processing circuit 401. The signal processing circuit 401 is equipped with a counter and memory, and the memory stores a digital value of the number of photons counted.

[0053] The horizontal scanning circuit 404 inputs pulses to the signal processing circuit 401 to sequentially select each column in order to read a signal from the memory of each pixel in which a digital value is held.

[0054] Signals are output to signal line 405 from the signal processing circuit 401 of the pixels selected by the vertical scanning circuit 406 for the column selected by the horizontal scanning circuit 404. The signals output to signal line 405 are then output to the outside of the photoelectric conversion element 200 via the output circuit 407.

[0055] As shown in Figures 3 and 4, multiple signal processing circuits 401 are arranged in the region that overlaps with the pixel region 22 in a plan view. Then, in a plan view, the vertical scanning circuit section 406, the horizontal scanning circuit section 404, the readout circuit 402, the output circuit 407, and the pulse generation section 403 are arranged so as to overlap between the edge of the sensor substrate 21 and the edge of the pixel region 22.

[0056] These circuit sections and pulse generation sections are non-pixel areas that do not include pixels 301, and the sensor substrate 21 has a pixel area 22 and a non-pixel area arranged around the pixel area 22. The vertical scanning circuit section 406, horizontal scanning circuit section 404, readout circuit 402, output circuit 407, and pulse generation section 403 are arranged in the area that overlaps with the non-pixel area in a plan view.

[0057] The vertical scanning circuit 406 receives pulses supplied from the pulse generation unit 403 and supplies control pulses to each pixel. The vertical scanning circuit 406 is composed of a shift register and an address decoder that connect multiple rows as a single unit, and high-speed reading is achieved by reading multiple rows at once.

[0058] In particular, in imaging devices that digitally count the number of photons arriving at the APD and output the count value as a photoelectrically converted digital signal from the pixel, the operation of the counter circuit that digitally counts the number of photons takes time. Therefore, it is preferable to read out multiple rows simultaneously for high-speed readout.

[0059] In other words, the vertical scanning circuit section 406, which functions as a readout circuit for reading pixel signals from pixels, simultaneously reads out pixel signals from pixels in N rows and pixel signals from pixels in N+1 rows. Furthermore, the pulse generation section 403 sets threshold information, which serves as the judgment criterion, and exposure time information, which indicates the timing for performing the judgment, to the count judgment circuit described later.

[0060] Note that the arrangement of signal line 405, read circuit 402, and output circuit 407 is not limited to Figure 4. For example, signal line 405 may be arranged so as not to extend in the row direction, and read circuit 402 may be placed at the end of the signal line 405.

[0061] Furthermore, the signal processing circuit 401 does not necessarily need to be provided in one unit for each photoelectric conversion unit; a single signal processing unit may be shared by multiple photoelectric conversion units, and sequential signal processing may be performed.

[0062] Figure 5 shows an example of an equivalent circuit of a pixel 301 and a signal processing circuit 401 corresponding to the pixel 301. The APD501 generates charge pairs corresponding to incident light by photoelectric conversion. One of the two nodes of the APD501 is connected to a power line to which the drive voltage VL (first voltage) is supplied.

[0063] Furthermore, the other of the two nodes of the APD501 is connected to a power line supplied with a drive voltage VH (second voltage) higher than voltage VL via a quench element 502. In Figure 5, one node of the APD501 is the anode, and the other node of the APD501 is the cathode.

[0064] A reverse bias voltage is supplied to the anode and cathode of the APD501, causing it to perform avalanche multiplication. This voltage supply causes the charge generated by the incident light to undergo avalanche multiplication, resulting in the generation of an avalanche current.

[0065] Furthermore, the APD501's operating mode is classified into two types depending on the value of the reverse bias voltage used to operate it. The two modes are Geiger mode, in which the Anode and cathode voltage difference is greater than the breakdown voltage, and Linear mode, in which the Anode and cathode voltage difference is near or below the breakdown voltage.

[0066] An APD that operates in Geiger mode is called a SPAD (Single Photon Avalanche Diode). In the case of a SPAD, for example, the voltage VL (first voltage) is -30V and the voltage VH (second voltage) is 1V.

[0067] The quench element 502 is connected to the power line to which the drive voltage VH is supplied and to one of the nodes, either the anode or the cathode, of the APD501. The quench element 502 functions as a load circuit (quench circuit) when the signal is multiplied by avalanche multiplication, suppressing the voltage supplied to the APD501 and thereby suppressing avalanche multiplication (quench operation).

[0068] Furthermore, the quench element 502 has the function of returning the voltage supplied to the APD501 to the drive voltage VH by flowing the current that compensates for the voltage drop caused by the quench operation (recharge operation).

[0069] In this embodiment, the quench element 502 is composed of a MOS transistor, and the on / off state of the quench element 502 is controlled by a control signal CLK connected to the gate of the quench element. The control signal CLK is controlled by the signal generation unit in the pulse generation unit 403. In the following description, the control signal CLK may be abbreviated as control CLK or simply CLK.

[0070] The waveform shaping unit 510 shapes the voltage change at the cathode of the APD501 obtained when the APD501 detects a photon, and outputs a pulse signal. For example, an inverter circuit is used as the waveform shaping unit 510.

[0071] Figure 5 shows an example where one inverter is used as the waveform shaping unit 510, but a circuit with multiple inverters connected in series may also be used, or other circuits that have a waveform shaping effect may be used.

[0072] The counter circuit 511 functions as a counter for counting the output of the avalanche photodiode provided for each pixel, counting the pulse signal output from the waveform shaping unit 510 and storing the count value.

[0073] Furthermore, when the control pulse RES is supplied via the drive line 514, the signal held in the counter circuit 511 is reset. In addition, when the control pulse STOP is supplied via the drive line 517, the counter circuit 511 continues to hold the count value until the control pulse RES is supplied.

[0074] The count determination circuit 512 receives the count value held by the counter circuit 511 via the drive line 516, and a control pulse φt from the pulse generation unit 403 via the drive line 518, which is supplied for example four times at predetermined intervals during one frame period.

[0075] The count determination circuit 512, upon receiving the control pulse φt, compares the count value with a predetermined threshold (default value). If it determines that the count value exceeds the threshold (default value), it supplies a control pulse STOP to the counter circuit 511 via the drive line 517. Furthermore, it outputs the determination result and the count value at the pulse timing to the selection circuit 513.

[0076] Furthermore, the pixel-by-pixel count determination circuit 512 functions as a determination means that determines whether the count value of the counter for each pixel has reached a predetermined value within a predetermined determination period from the start of exposure, and outputs a determination result. The predetermined determination period is, for example, the shortest determination period among a plurality of settable determination periods, i.e., the minimum determination period.

[0077] Furthermore, the count determination circuit 212 outputs exposure time information (the exposure time code when the count threshold is exceeded) and the count value at the timing of the control pulse φt to the selection circuit 213 via the drive line 219.

[0078] For example, if each pixel output is 14 bits, 3 bits are allocated to exposure time information (Tcode), and the remaining 11 bits are allocated to the count value. In this way, the count determination circuit 212 outputs this 14-bit signal as the output of each pixel. The timing chart of the pulse signal will be described later using Figure 7.

[0079] The selection circuit 513 receives a control pulse SEL from the vertical scanning circuit section 406 in Figure 4 via the drive line 515 (not shown in Figure 4) in Figure 5, which switches the electrical connection between the count determination circuit 512 and the signal line 405.

[0080] The selection circuit 513 includes, for example, a buffer circuit for outputting a signal, and outputs the output signal from the pixel count determination circuit 512 to the signal line 405. The selection circuit 513 also functions as an output means that outputs the determination result and count value from the determination means for each pixel.

[0081] Furthermore, switches such as transistors may be placed between the quench element 502 and the APD 501, or between the photoelectric conversion unit 302 and the signal processing circuit 401, to switch the electrical connections. Similarly, the supply of voltage VH or voltage VL to the photoelectric conversion unit 302 may be electrically switched using switches such as transistors.

[0082] Figure 6 schematically shows an example of the relationship between the operation of the APD501 and the output signal according to Embodiment 1. Figure 5 schematically shows the relationship between the control signal CLK of the quench element 502, the voltage at node A, the voltage at node B, and the count value of the counter circuit 511 in the photoelectric conversion element.

[0083] When the control signal CLK is at a high level, the drive voltage VH is less likely to be supplied to the APD501, and when the control signal CLK is at a low level, the drive voltage VH is supplied to the APD501.

[0084] A high level control signal CLK is, for example, 1V, and a low level control signal CLK is, for example, 0V. When the control signal CLK is high level, the quench element 502 is off, and when the control signal CLK is low level, the quench element 502 is on. The resistance value of the quench element 502 when the control signal CLK is high level is higher than the resistance value of the quench element 502 when the control signal CLK is low level.

[0085] When the control signal CLK is at a high level, even if avalanche multiplication occurs in the APD501, recharge operation is less likely to occur, and the voltage supplied to the APD501 becomes below the breakdown voltage of the APD501. Therefore, the avalanche multiplication operation in the APD501 stops.

[0086] At time t1, the control signal CLK changes from a high level to a low level, the quench element 502 turns on, and the recharge operation of the APD501 begins. As a result, the cathode voltage of the APD501 transitions to a high level.

[0087] Then, the difference in voltage applied to the anode and cathode of the APD501 puts the APD501 into an avalanche multiplication state. The cathode voltage is the same as the voltage of node A. Subsequently, when the cathode voltage transitions from a low level to a high level, the voltage of node A becomes greater than or equal to the threshold at time t2.

[0088] Here, the judgment threshold is a voltage value uniquely determined by the electrical characteristics of the waveform shaping unit 510. When the voltage at node A exceeds the judgment threshold, the pulse signal output from node B of the waveform shaping unit 510 is inverted, changing from a high level to a low level.

[0089] Once recharging is complete, a voltage of (drive voltage VH - drive voltage VL) is applied to the APD501. Then, between time t2 and time t3, the control signal CLK becomes high level, and the quench element 502 turns off.

[0090] Next, at time t3, when a photon is incident on APD501, avalanche multiplication occurs in APD501, an avalanche multiplication current flows through the quench element 502, and the cathode voltage drops. In other words, the voltage at node A drops.

[0091] If the voltage at nodeA falls below the threshold during the voltage drop, the voltage at nodeB changes from a low level to a high level. In other words, the portion of the output waveform at nodeA that exceeds the threshold is reshaped by the waveform shaping unit 510 and output as a low-level signal at nodeB.

[0092] Then, the counter circuit 511 counts, and the count value of the counter signal output from the counter circuit 511 increases by 1 LSB.

[0093] Although photons are incident on APD501 between time t3 and time t4, the quench element 502 is in the off state, and the voltage applied to APD501 is not a voltage difference that allows for avalanche multiplication, so the voltage level of node A does not exceed the judgment threshold.

[0094] At time t4, the control signal CLK changes from a high level to a low level, and the quench element 502 turns on. Consequently, a current flows through node A to compensate for the voltage drop from the drive voltage VL, and the voltage at node A returns to its original voltage level.

[0095] At this time, at time t5, the voltage of node A exceeds the threshold, causing the pulse signal of node B to invert and change from a high level to a low level.

[0096] At time t6, the control CLK changes from a low level to a high level, node A settles back to its original voltage level, and the control signal CLK changes from a low level to a high level. In principle, the period during which the control CLK is at a low level only needs to be set to be longer than the period during which node A transitions from a low level to a high level.

[0097] In Figure 6, the period during which the control CLK is at a low level is set to be the same as the period during which nodeA transitions from a low level to a high level. This allows the frequency of the control CLK to be set higher, thereby reducing the effect of the "nonlinear relationship between the number of output signals and the number of input signals," which will be discussed later.

[0098] Subsequently, as explained from time t1 to time t6, the voltages of each node and signal line change in response to the control signal CLK and the incidence of photons. However, when the recharge frequency of the APD is controlled by the control signal CLK, the relationship between the number of output signals and the number of input signals is not linear. The number of input signals refers to the number of photons incident on the APD, and the number of output signals refers to the count value of photons detected by the imaging device.

[0099] In SPADs, when avalanche breakdown occurs, secondary photons are emitted, causing emission crosstalk with adjacent pixels. However, if the effects of emission crosstalk are ignored, the relationship between the number of output signals and the number of input signals can be theoretically derived.

[0100] Specifically, when the number of input signals is Nph, the number of output signals is Nct, the frequency of the control signal CLK (number of control signal CLKs per unit time) is f, and the exposure time is T, it is described by the following equation 1.

number

[0101] Figure 7 is a timing chart illustrating an example of the operation of the signal processing circuit 401 according to Embodiment 1. In this embodiment, the exposure time for each pixel is set by the pulse generation unit 403 to a predetermined exposure time of T / (n to the power of (m-1)).

[0102] T represents the maximum exposure time within one frame. m is any integer such that m≧1, but Figure 7 shows the timing chart when m is set to 1≦m≦4. A control pulse φt is supplied to the drive line 518, which goes Hi only at the moment the exposure time determined by t=T / (n^(m-1)) is reached.

[0103] Furthermore, if the count value of the counter circuit 511 reaches a predetermined count threshold at the moment of exposure time T / (n^(m-1)) for four exposure times when m=1, 2, 3, and 4, the photoelectric conversion element 200 is switched from Geiger mode to linear mode and the APD501 is put into standby mode.

[0104] Once the APD501 is put into a pause state, the pulse signal from the waveform shaping unit 510 is not output, and the count of the counter circuit 511 remains unchanged. The count determination circuit 512 then outputs a 14-bit signal to the selection circuit 513, consisting of, for example, a 3-bit Tcode corresponding to T / (n to the power of (m-1)), which represents the exposure time according to the control pulse φt, and, for example, an 11-bit count value.

[0105] Figure 8 shows an example of the relationship between the exposure time for each pixel 301 included in the photoelectric conversion element 200 according to Embodiment 1 and the count value of the counter circuit 511. In Figure 8, similar to Figure 7, the pulse generation unit 403 sets a predetermined exposure time T / (n to the power of (m-1)) where 1 ≤ m ≤ 4.

[0106] When the upper limit of the count of the counter circuit 511 is set to Cmax, the count threshold is set to Cmax / n for the reasons described later.

[0107] In Figure 8, assume that the count of a certain pixel increases proportionally with time. As described in Figure 7, the count determination circuit 512 determines whether the count value exceeds the count threshold at the moment T / (n to the power of (m-1)), in order of increasing exposure time.

[0108] As shown in Figure 8, when the count value increases, the count threshold is not exceeded at the moments T / n^3 and T / n^2, but it is determined that the count threshold has been exceeded at the moment T / n. Note that n^3 means n to the power of 3, and n^2 means n to the power of 2. At this time, the count is stopped at the moment T / n, and the count value Cout at T / n and the exposure time T / n are output from the photoelectric conversion element 200.

[0109] While the exposure time T / n can be output directly as time information, exposure time information (Tcode) as shown in Figure 9 can also be output.

[0110] Figure 9 shows an example of exposure time information output based on the exposure time when the count threshold is exceeded. The Tcode shown in Figure 9 is one example of the output data format.

[0111] The count estimate (Cest) shown in Figure 8 is calculated as Cest = Cout × n, assuming that the count increases at the same rate during the exposure time from 0 to T / n and the non-exposure time from T / n to T.

[0112] As mentioned earlier, the above count estimation method assumes that the count increases at the same rate during the exposure time from 0 to T / n and the non-exposure time from T / n to T. Therefore, if the upper limit of the count, Cmax, is reached before the exposure time T / n is reached, the rate of increase during the exposure time from 0 to T / n cannot be accurately estimated, resulting in a decrease in the accuracy of the count estimation.

[0113] Therefore, in this embodiment, in order to maintain the accuracy of the count estimation, a threshold determination is performed before the upper count limit (Cmax) is reached. As mentioned above, when the recharge frequency of the APD is controlled by the control signal CLK, the relationship between the number of output signals and the number of input signals is not linear, so in this embodiment, linearity correction is performed before the count estimation.

[0114] Figure 10 shows an example of the relationship between the exposure time for each pixel 301 in the photoelectric conversion element 200 according to Embodiment 1 and the count value of the counter circuit 511. It shows how the count increases when the count value becomes equal to the count threshold at the moment of each exposure time for which threshold determination is performed.

[0115] As mentioned above, in this count estimation method, a threshold determination is made before the upper limit of the count (Cmax) is reached in order to maintain the accuracy of the count estimation. Therefore, if the timing of the threshold determination when the count increases at a rate that reaches Cmax at the moment of the maximum exposure time T is defined as t1, then it is necessary to determine the timing of the threshold determination t2 when the count increases at a rate that reaches Cmax at the moment of t1.

[0116] That is, when the count threshold is Cmax / n, the time it takes to reach Cmax / n when the count increases at a rate that reaches Cmax at the moment of maximum exposure time T is calculated as T × (1 / n), so t1 = T / n. Similarly, t2 = t1 × (1 / n) = T / n^2. In this way, each exposure time for threshold determination is calculated as T / (n to the power of (m-1)).

[0117] As shown in Figure 5, when the APD recharge frequency is controlled by the control signal CLK, the relationship between the number of output signals and the number of input signals is not linear, as shown in Equation 1.

[0118] Therefore, in this embodiment, linearity correction is performed based on the estimated count value calculated by the signal processing circuit 401. Here, linearity correction refers to determining the number of input signals Nph from the number of output signals Nct per exposure time using the following equation 2, where f is the frequency of the control CLK (number of CLKs per unit time) and T is the length of the exposure time.

number

[0119] In this embodiment, the input signal number Nph derived by Equation 2 is the number of photons per exposure time for threshold determination. Therefore, the number of photons in one frame exposure time is calculated as Nph × (n to the power of (m-1)).

[0120] Here, for example in the example in Figure 9, when n=8 and the count value is 11-bit data, the data after linearity correction may be 20 bits or more. Therefore, performing edge detection using the linearity-corrected data would be computationally intensive. In this embodiment, edge detection is performed with minimal processing using the Tcode and count value.

[0121] Figure 11 is a flowchart illustrating an example of the operation of the image processing unit 105 in Embodiment 1. Using Figure 11, the edge detection method performed by the image processing unit 105 in this embodiment will be explained.

[0122] Furthermore, the CPU 102, which acts as the computer within the imaging device, executes the computer program stored in memory, thereby sequentially performing the operations of each step in the flowchart shown in Figure 11.

[0123] In this embodiment, the imaging unit 107 converts incident photons through a Bayer array color filter of R, Gr, Gb, and B (not shown) into photoelectric data, and obtains a Tcode and count value for each of the R, Gr, Gb, and B colors.

[0124] Specifically, each row is configured with pixels of R, Gr, R, Gr, etc., and the adjacent row is configured with pixels of Gb, B, Gb, B, etc. Furthermore, each pixel outputs a 3-bit Tcode and, for example, an 11-bit count value, as shown in Figure 9.

[0125] In step S1101, the image processing unit 105 obtains the horizontal pixel count W and vertical pixel count H of the Gr pixels in the image to be processed and records them in the RAM 104. Hereafter, the position of the Gr pixels will be expressed in orthogonal coordinates (X,Y), with the Gr pixel at the top left of the image being the origin (0,0), the position of the Gr pixel to its right being (0,1), and the position of the Gr pixel below being (1,0).

[0126] In step S1102, the image processing unit 105 initializes the horizontal coordinate X and the vertical coordinate Y to 0 in order to reference the Gr pixels of the captured image.

[0127] In step S1103, the image processing unit 105 obtains the count value and Tcode of each Gr pixel (X,Y) and its surrounding Gr pixels from the imaging unit 107. In this embodiment, the count value and Tcode of a total of 9 pixels are obtained, centered on the Gr pixel (X,Y), including the Gr pixels above, below, to the left and right, and the Gr pixels diagonally to the upper right, upper left, lower right, and lower left.

[0128] Here, step S1103 functions as an acquisition step (acquisition means) that acquires the count values ​​and determination results of multiple pixels within a predetermined range, including a predetermined pixel output by the output means.

[0129] Furthermore, if the central Gr pixel (X,Y) is located at the left, right, top, or bottom edge of the image, some surrounding pixels will have insufficient data, and therefore the image edges will be processed. Image edge processing involves mirroring, which replaces the missing data with data from adjacent Gr pixels. Alternatively, zero-padding, which sets the missing data to 0, may also be used as an edge processing method.

[0130] In step S1104, the image processing unit 105 determines whether all nine Tcodes acquired in step S1103 are the same. If they are the same, the process proceeds to step S1105. On the other hand, if there are pixels with different Tcodes, the process proceeds to step S1106.

[0131] In step S1105, the image processing unit 105 applies edge detection filtering to the counter values ​​of the nine Gr pixels acquired in step S1103 or the edge detection count values ​​calculated in step S1106, which will be described later.

[0132] Here, step S1105 functions as an edge detection step (edge ​​detection means) that detects edge information by performing a weighting calculation on the count values ​​of multiple pixels within a predetermined range according to the determination result.

[0133] In this embodiment, the edge detection filtering process applies, for example, a Laplacian filter. The calculation formula for the Laplacian filter is shown in Equation 3 below. The kernel Kl of the Laplacian filter is shown in Equation 4 below.

number

number

[0134] Here, L(X,Y) is the calculation result of the Laplacian filter, and M(X,Y) is the counter value at the Gr pixel (X,Y). L(X,Y) outputs a large value in the edge regions of the image, and a value of 0 or close to 0 in non-edge regions.

[0135] In this embodiment, a Laplacian filter with 3 pixels vertically and 3 pixels horizontally was used as the edge detection filter, but other methods may be used for edge detection. For example, the number of Gr pixels acquired in step S1103 may be changed to change the number of vertical and horizontal pixels processed by the Laplacian filter, or other edge detection filters such as a Sobel filter may be used.

[0136] In step S1106, the image processing unit 105 calculates an edge detection count value from the count value of each pixel according to the maximum value of the Tcode of the 9 pixels acquired in step S1103.

[0137] Here, we will specifically explain how to calculate the count value for edge detection using the example in Figure 9. For example, when the maximum Tcode value for 9 pixels is "111", the count value of the pixel with a Tcode of "111" is used directly as the count value for edge detection.

[0138] Then, the count value of pixels with Tcode "011" is multiplied by 1 / n, the count value of pixels with Tcode "001" is multiplied by 1 / n^2, and the count value of pixels with Tcode "000" is multiplied by 1 / n^3, and these are used as count values ​​for edge detection.

[0139] Furthermore, for example, if the maximum Tcode value for 9 pixels is "011", the count value of pixels with a Tcode of "011" is used as is for edge detection. Then, the count value of pixels with a Tcode of "001" is multiplied by 1 / n, and the count value of pixels with a Tcode of "000" is multiplied by 1 / n^2 before being used as the count value for edge detection.

[0140] Furthermore, for example, if the maximum Tcode value for 9 pixels is "001", the count value of pixels with a Tcode of "001" is used directly as the count value for edge detection. The count value of pixels with a Tcode of "000" is multiplied by 1 / n and used as the count value for edge detection.

[0141] In other words, the value calculated to decrease the count value of pixels with small Tcode values ​​according to the maximum Tcode value is used for edge detection. Thus, in step S1106, a weighting calculation is performed according to the determination result.

[0142] Conversely, it is also possible to perform edge detection using values ​​calculated to increase the count value of pixels with larger Tcode values ​​according to the minimum Tcode value. However, in that case, the values ​​used for edge detection become large, making the processing heavy.

[0143] Therefore, in this embodiment, the value calculated to decrease the count value of pixels with small Tcode values ​​according to the maximum Tcode value is used for edge detection. In other words, a weighting calculation is performed in which the weight of the count value decreases for pixels with small code values ​​corresponding to the determination results of multiple pixels within a predetermined range.

[0144] Thus, in step S1106, if it is determined in step S1104 that the determination results of multiple pixels within a predetermined range include different determination results, the count values ​​of the multiple pixels within the predetermined range are weighted according to the determination results. Then, using the results of the weighting calculation, edge information is detected in step S1105.

[0145] Furthermore, in step S1105, if the determination results for multiple pixels within a predetermined range are the same in step S1104, edge information is detected using the count values ​​of the multiple pixels within the predetermined range without performing weighting calculations according to the determination results.

[0146] In step S1107, the image processing unit 105 determines whether the horizontal coordinate X is equal to W-1, that is, whether the right edge of the image has been reached. If the right edge has been reached, the process proceeds to step S1108. On the other hand, if the right edge has not been reached, the process proceeds to step S1109.

[0147] In step S1109, the image processing unit 105 assigns the value of the horizontal coordinate X plus 1 to X. This process causes the image processing unit 105 to refer to the Gr pixel one position to the right of the pixel it was currently focusing on. After the processing in step S1109, the process returns to step S1103.

[0148] In step S1108, the image processing unit 105 determines whether the vertical coordinate Y is equal to H-1, that is, whether the bottom edge of the image has been reached. If the bottom edge has been reached, the processing flow shown in Figure 12 is terminated. On the other hand, if the bottom edge has not been reached, the process proceeds to step S1110.

[0149] In step S1110, the image processing unit 105 assigns 0 to the horizontal coordinate X and assigns the value obtained by adding 1 to the vertical coordinate Y to Y. This process causes the image processing unit 105 to refer to the leftmost Gr pixel in the row immediately below the pixel that was previously under consideration. After the processing in step S1110, the process returns to step S1103.

[0150] By performing the above processing, edge detection can be performed similarly in both bright areas with large Tcode values ​​and dark areas with small Tcode values, and it also becomes possible to detect edges in areas with different Tcode values. The image processing unit 105 performs edge detection on the image to be processed and obtains edge information for a number of pixels equal to W in the horizontal direction and H in the vertical direction.

[0151] In this embodiment, a simultaneous color image is generated by demosaicing the signal acquired through a Bayer array color filter. At that time, the image processing unit 105 performs demosaicing on the data after linearity correction.

[0152] In this process, if the number of horizontal pixels in the Gr pixels of the image to be processed by the image processing unit 105 is W and the number of vertical pixels is H, interpolation data is generated for each pixel of the Bayer array by demosaicing, thereby generating a video signal with 2W horizontal pixels and 2H vertical pixels.

[0153] Therefore, edge information obtained in the aforementioned process is superimposed one pixel at a time onto the luminance components of two horizontal pixels and two vertical pixels of the video signal generated by demosaicing. That is, edge enhancement processing is performed by superimposing edge information obtained by stretching the edge information of horizontal pixels W and vertical pixels H by twice in both the horizontal and vertical directions onto the demosaiced video signal.

[0154] In other words, the image processing unit 105, as a means of generating a video signal, generates a video signal using the count value output by the image sensor and the determination result, and superimposes the edge information detected by the edge detection means onto the video signal.

[0155] By performing the above processing, it becomes possible to perform edge enhancement processing in an imaging device that performs photoelectric conversion by digitally counting the number of photons, with minimal edge detection processing and easily discernible enhancement effects.

[0156] Furthermore, as a method for superimposing edge information onto a demosaiced video signal, one could replace the luminance component of the video signal with the edge information, or add the edge information to the luminance component of the video signal. Alternatively, if the luminance of the video signal is too bright and causes overexposure, the edge information could be subtracted.

[0157] Alternatively, edge enhancement processing may be performed by superimposing a value obtained by multiplying the edge information by a predetermined gain onto the luminance component of the video signal. Furthermore, the predetermined gain may be a value that can be set by the user via the operation unit 113. Although the pixels used for edge detection have been described as Gr pixels, Gb pixels may also be used.

[0158] <Embodiment 2> Embodiment 1 describes an example of edge enhancement performed by replacing or adding to the luminance component of the video signal with the detected edge information.

[0159] Embodiment 2 describes an example of edge enhancement processing that makes the enhancement effect easier to discern by changing the color of the edge superimposed according to the Tcode of the central pixel of the pixels used for edge detection.

[0160] Figure 12 is a flowchart illustrating an example of the operation of the image processing unit 105 in Embodiment 2, showing an example of the process of performing edge detection while determining the color of the overlapping edge in Embodiment 2.

[0161] Furthermore, the CPU 102, acting as a computer within the imaging device, executes the computer program stored in memory, thereby sequentially performing the operations of each step in the flowchart of Figure 11. In this embodiment, the same or similar components and steps as in Embodiment 1 are denoted by the same reference numerals, and redundant explanations are omitted.

[0162] In step S1201, the image processing unit 105 determines the color information for the superimposed edges detected in step S1105, according to the Tcode value of the Gr pixel (X,Y) acquired in step S1103.

[0163] Here, we will explain in detail using the example in Figure 9. Specifically, for example, if the Tcode value of a Gr pixel (X,Y) is "000", it is white; if it is "001", it is blue; if it is "011", it is yellow; and if it is "111", it is red. This color information is recorded in RAM 104 along with the position information of the Gr pixel (X,Y).

[0164] In this embodiment, the Tcode takes on four possible values: "000", "001", "011", and "111", and the four colors white, blue, yellow, and red corresponding to these values ​​were described as color information. However, the color information only needs to be different colors depending on the Tcode pattern, and the colors of the color information are not limited to the four colors white, blue, yellow, and red.

[0165] As described above, by performing the processing shown in the flowchart of Figure 12, the image processing unit 105 can detect edges in the image to be processed and obtain edge information for a number of pixels equal to W in the horizontal direction and H in the vertical direction, along with color information when the edges are superimposed.

[0166] Here, if the number of horizontal pixels in the Gr pixels of the image to be processed by the image processing unit 105 is W and the number of vertical pixels is H, interpolation data is generated for each pixel of the Bayer array by demosaicing, thereby generating a video signal with 2W horizontal pixels and 2H vertical pixels.

[0167] Therefore, the edge information obtained in the aforementioned process is superimposed on the horizontal 2 pixels and vertical 2 pixels of the video signal generated by demosaicing, one pixel at a time, according to the color information determined in step S1201.

[0168] Specifically, the edge information of the horizontal pixel count W and vertical pixel count H, along with the color information corresponding to the position of that edge information, are stretched by a factor of two in both the horizontal and vertical directions, and then superimposed onto the demosaiced video signal.

[0169] Furthermore, as a method for superimposing edge information onto a demosaiced video signal, the video signal may be replaced with the color of the edge information, or the color of the edge information may be added to the video signal. Alternatively, the edge information may be multiplied by a predetermined gain and superimposed onto the video signal, and the predetermined gain may be a value that can be set by the user via the operation unit 113.

[0170] In this embodiment, color information is determined according to the determination result of a predetermined pixel, and the edge information detected by the edge detection means is superimposed on the video signal by the video signal generation means according to the above color information.

[0171] By performing the above processing, it becomes possible to perform edge enhancement processing in an imaging device that performs photoelectric conversion by digitally counting the number of photons, with minimal edge detection processing and easily discernible enhancement effects.

[0172] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention.

[0173] Furthermore, the embodiments described above merely illustrate one embodiment of the present invention, and it is possible to combine the embodiments as appropriate. For example, it is possible to operate by partially combining the first to second embodiments. Alternatively, the system may be configured so that the user selects a function from a menu displayed on the imaging device and then executes the control.

[0174] Furthermore, the present invention includes, for example, a system that realizes the functions of the above embodiment using at least one processor such as a CPU, memory, and circuitry (e.g., an ASIC). Alternatively, multiple processors may be used for distributed processing.

[0175] Furthermore, in order to implement some or all of the control in the above embodiment, a computer program that implements the functions of the above embodiment may be supplied to the imaging device, etc., via a network or various storage media.

[0176] The computer (or CPU or MPU, etc.) in the imaging device may read and execute the program. In that case, the program and the storage medium storing the program constitute the present invention. The present invention includes the following combinations.

[0177] (Configuration 1) An image processing apparatus for processing output from an image sensor having a counter for counting the output of an avalanche photodiode provided for each pixel, determination means for determining whether the count value of the counter for each pixel has reached a predetermined value within a predetermined determination period from the start of exposure and outputting a determination result, and output means for outputting the determination result and the count value for each pixel, wherein the image processing apparatus is characterized by having acquisition means for acquiring the count value and the determination result of a plurality of pixels in a predetermined range including a predetermined pixel output by the output means, and edge detection means for detecting edge information by performing a weighting calculation on the count values ​​of the plurality of pixels in the predetermined range according to the determination result.

[0178] (Configuration 2) The image processing apparatus according to Configuration 1, characterized in that when the determination results of a plurality of pixels in a predetermined range are the same, the edge detection means uses the count values ​​of the plurality of pixels in the predetermined range without performing the weighting calculation according to the determination results to detect the edge information.

[0179] (Configuration 3) The image processing apparatus according to Configuration 1 or 2, wherein if the determination results of a plurality of pixels within a predetermined range include different determination results, the edge detection means detects the edge information by performing a weighting calculation on the count values ​​of the plurality of pixels within the predetermined range according to the determination results.

[0180] (Configuration 4) An image processing apparatus according to any one of Configurations 1 to 3, characterized in that the edge detection means detects edge information by performing a weighting calculation in which the weight of the count value decreases for pixels whose code value corresponding to the determination result of a plurality of pixels in a predetermined range is small.

[0181] (Configuration 5) An image processing apparatus according to any one of Configurations 1 to 4, characterized in that it has a video signal generation means that generates a video signal using the count value output by the image sensor and the determination result, and superimposes the edge information detected by the edge detection means onto the video signal.

[0182] (Composition 6) The image processing apparatus according to configuration 5, characterized in that it determines color information according to the determination result of the predetermined pixels, and superimposes the edge information detected by the edge detection means onto the video signal according to the color information using the video signal generation means.

[0183] (Method) An image processing method for processing output from an image sensor having a counter for counting the output of an avalanche photodiode provided for each pixel, determination means for determining whether the count value of the counter for each pixel has reached a predetermined value within a predetermined determination period from the start of exposure and outputting a determination result, and output means for outputting the determination result and the count value for each pixel, the image processing method comprising: an acquisition step for acquiring the count value and the determination result of a plurality of pixels in a predetermined range including a predetermined pixel output by the output means; and an edge detection step for detecting edge information by performing a weighting calculation on the count values ​​of the plurality of pixels in the predetermined range according to the determination result.

[0184] A computer program for controlling each means of the image processing apparatus described in any one of configurations 1 to 6 by a computer. [Explanation of symbols]

[0185] 100: Imaging device 101: Internal bus 102:CPU 103:ROM 104:RAM 105: Image Processing Unit 106: Lens Unit 107: Imaging Unit 108: Network Module 109: Video terminal 110: Recording medium interface 111: Frame memory 112: Recording media 113:Operation unit 114: Display section 115: Object detection unit 116: Attitude detection unit

Claims

1. A counter that counts the output of an avalanche photodiode provided for each pixel, A determination means that determines whether the count value of the counter for each pixel has reached a predetermined value within a predetermined determination period from the start of exposure, and outputs a determination result, An image processing apparatus for processing output from an image sensor, having output means for outputting the determination result and the count value for each pixel, An acquisition means for acquiring the count value and the determination result of a plurality of pixels within a predetermined range including a predetermined pixel output by the output means, An edge detection means that detects edge information by performing a weighting calculation on the count values ​​of a plurality of pixels within the predetermined range according to the determination result, An image processing apparatus characterized by having

2. The image processing apparatus according to claim 1, characterized in that, if the determination results of a plurality of pixels within a predetermined range are the same, the edge detection means detects the edge information by using the count values ​​of the plurality of pixels within the predetermined range without performing the weighting calculation according to the determination results.

3. The image processing apparatus according to claim 1, wherein the edge detection means detects edge information by performing a weighting calculation on the count values ​​of the plurality of pixels in the predetermined range according to the determination results, if the determination results of the plurality of pixels in the predetermined range include different determination results.

4. The image processing apparatus according to claim 1, characterized in that the edge detection means detects the edge information by performing a weighting calculation in which the weight of the count value decreases for pixels with smaller code values ​​corresponding to the determination results of a plurality of pixels within a predetermined range.

5. The image processing apparatus according to claim 1, further comprising a video signal generation means that generates a video signal using the count value output by the image sensor and the determination result, and superimposes the edge information detected by the edge detection means onto the video signal.

6. The image processing apparatus according to claim 5, characterized in that it determines color information according to the determination result of the predetermined pixels, and superimposes the edge information detected by the edge detection means onto the video signal according to the color information using the video signal generation means.

7. A counter that counts the output of an avalanche photodiode provided for each pixel, A determination means that determines whether the count value of the counter for each pixel has reached a predetermined value within a predetermined determination period from the start of exposure, and outputs a determination result, An image processing method for processing output from an image sensor having output means for outputting the determination result and the count value for each pixel, An acquisition step of acquiring the count value and the determination result of a plurality of pixels within a predetermined range including a predetermined pixel output by the output means, An edge detection step in which edge information is detected by performing a weighting calculation on the count values ​​of a plurality of pixels within the predetermined range according to the determination result, An image processing method characterized by having the following features.

8. A computer program for controlling each means of the image processing apparatus described in any one of claims 1 to 6 by a computer.

Images