IMAGING APPARATUS, PROCESSING METHOD AND PROGRAM FOR IMAGING APPARATUS

Abstract

To reduce an influence of a change over time of an imaging unit.SOLUTION: An imaging apparatus has: an imaging unit that includes an avalanche photodiode; an estimated value calculation unit that calculates a change degree estimated value of the imaging unit; and an image processing unit that performs image processing on an output signal from the imaging unit on the basis of an image processing parameter based on the change degree estimated value and a first threshold.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present disclosure relates to an imaging device, a processing method for the imaging device, and a program. [Background technology]

[0002] In recent years, photoelectric conversion devices have become known that digitally count the number of photons arriving at an avalanche photodiode (APD) and output the count value from a pixel as a photoelectrically converted digital signal. Non-Patent Document 1 discloses that a phenomenon called avalanche emission occurs in a photoelectric conversion device having an APD. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] I. Rech et al., “Optical crosstalk in single photon avalanche diode arrays: a new complete model”, OpEx 16(12), 2008 Summary of the Invention [Problem to be solved by the invention]

[0004] In a photoelectric conversion device having an APD, as the time of use increases, the DCR changes, which causes a problem that the output signal changes. DCR is the dark count rate, which is the average count value when there is no incident light.

[0005] The object of the present disclosure is to deterioration The purpose of the present invention is to be able to suppress the influence of the above. [Means for solving the problem]

[0006] The imaging device includes an imaging section including an avalanche photodiode; Based on at least one of information regarding the number of avalanche emissions in the avalanche photodiode, the exposure conditions of the imaging unit, and the temperature of the imaging unit, The imaging section deterioration Degree Show an estimated value calculation unit for calculating an estimated value; deterioration Degree Show The image processing unit performs image processing on the output signal of the imaging unit based on the estimated value and an image processing parameter based on a first threshold value. Effect of the Invention

[0007] According to the present disclosure, deterioration The effects of the above can be suppressed. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a diagram illustrating a configuration example of a photoelectric conversion element. [Diagram 2] FIG. 4 is a diagram illustrating a configuration example of a sensor substrate. [Diagram 3] FIG. 2 is a diagram illustrating an example of a configuration of a circuit board. [Figure 4] 2 is a diagram of an equivalent circuit of a signal processing circuit corresponding to a pixel of a photoelectric conversion element. [Diagram 5] 4 is a timing chart of a photoelectric conversion element. [Figure 6] FIG. 1 is a diagram illustrating an example of the configuration of an imaging system. [Figure 7] FIG. 1 is a diagram illustrating an example of the configuration of a surveillance camera. [Figure 8] 10 is a flowchart showing a processing method of the surveillance camera. [Figure 9] FIG. 1 is a diagram illustrating an example of the configuration of a surveillance camera. [Figure 10] 10 is a flowchart showing a processing method of the surveillance camera. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The present embodiment will be described in detail below. Note that the embodiment described below is an example, and should be appropriately modified or adjusted depending on the configuration of the device to which the present embodiment is applied and various conditions, and is not limited to the following embodiment. Also, in all drawings, parts having the same functions are given the same numbers, and repeated explanations are omitted.

[0010] (First embodiment) <Introduction> FIG. 1 is a diagram showing a configuration example of a photoelectric conversion element 100 according to the first embodiment. In the following, a photoelectric conversion element having a so-called laminated structure in which the photoelectric conversion element 100 is configured by laminating two substrates, a sensor substrate 11 and a circuit substrate 21, and electrically connecting the sensor substrate 11 and the circuit substrate 21, will be described as an example. However, the photoelectric conversion element 100 may have a so-called non-laminated structure in which the configuration included in the sensor substrate 11 and the configuration included in the circuit substrate 21 are arranged on a common semiconductor layer. The sensor substrate 11 has a pixel region 12. The circuit substrate 21 has a circuit region 22 that processes a signal generated in the pixel region 12.

[0011] <Sensor board> Fig. 2 is a diagram showing an example of the configuration of the sensor substrate 11 of Fig. 1. The pixel region 12 of the sensor substrate 11 includes a plurality of pixels 101 arranged two-dimensionally across a plurality of rows and a plurality of columns. Each of the plurality of pixels 101 includes a photoelectric conversion unit 102 including an avalanche photodiode (hereinafter, APD). The photoelectric conversion unit 102 converts light into an electric charge by photoelectric conversion. Note that the number of rows and columns of the pixels 101 is not limited.

[0012] <Circuit board> Fig. 3 is a diagram showing an example of the configuration of the circuit board 21 in Fig. 1. The circuit board 21 has a signal processing circuit 103 that processes charges photoelectrically converted by the photoelectric conversion unit 102 in Fig. 2, a readout circuit 112, a control pulse generation unit 115, a horizontal scanning circuit 111, a signal line 113, a vertical scanning circuit 110, and an output circuit 114.

[0013] The control pulse generating unit 115 supplies control pulses to the vertical scanning circuit 110 , the horizontal scanning circuit 111 , and the readout circuit 112 .

[0014] The vertical scanning circuit 110 receives a control pulse supplied from a control pulse generating unit 115, and supplies a selection pulse for each row of the plurality of signal processing circuits 103. The vertical scanning circuit 110 uses logic circuits such as a shift register and an address decoder.

[0015] The signal processing circuit 103 processes the signal output from the photoelectric conversion unit 102 of the pixel 101 in Fig. 2. The signal processing circuit 103 has a counter, a memory, etc., and digital values ​​are stored in the memory. The signal processing circuit 103 outputs a digital pixel signal based on the signal output from the photoelectric conversion unit 102 to the signal line 113 of each column in row units in response to a selection pulse.

[0016] The readout circuit 112 receives a control pulse supplied from a control pulse generating unit 115 and holds the pixel signals of the signal lines 113 of each column.

[0017] The horizontal scanning circuit 111 receives a control pulse supplied from a control pulse generating unit 115, and outputs a control pulse for sequentially selecting each column to the readout circuit 112. In response to the control pulse, the readout circuit 112 sequentially outputs pixel signals of each column to the output circuit 114. The output circuit 114 sequentially outputs the pixel signals to the outside of the photoelectric conversion element 100.

[0018] <Connection between sensor board and circuit board> 2 and 3, a plurality of signal processing circuits 103 are arranged in each of the regions overlapping the plurality of pixels 101 in a plan view. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged so as to overlap a region between an end of the sensor substrate 11 and an end of the pixel region 12 in a plan view. In other words, the sensor substrate 11 has the pixel region 12 and a non-pixel region arranged around the pixel region 12. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged in a region overlapping the non-pixel region in a plan view.

[0019] The arrangement of the signal lines 113, the readout circuits 112, and the output circuits 114 is not limited to that shown in Fig. 3. For example, the signal lines 113 may be arranged to extend in the row direction, and the readout circuits 112 may be arranged at the ends of the signal lines 113. Also, it is not necessary to provide one signal processing circuit 103 for each photoelectric conversion unit 102. One signal processing circuit 103 may be configured to share a plurality of photoelectric conversion units 102 and perform signal processing sequentially.

[0020] <Pixel equivalent circuit> FIG. 4 is an equivalent circuit diagram showing a configuration example of the photoelectric conversion unit 102 and the signal processing circuit 103 corresponding to the photoelectric conversion unit 102 in FIGS.

[0021] The photoelectric conversion unit 102 has an avalanche photodiode (APD) 201. The APD 201 generates a pair of charges according to incident light by photoelectric conversion. One of the two nodes of the APD 201 is connected to a power supply line to which a driving voltage VL is supplied. The other of the two nodes of the APD 201 is connected, via a quench element 202, to a power supply line to which a driving voltage VH higher than the driving voltage VL is supplied. In FIG. 4, one node of the APD 201 is an anode, and the other node of the APD 201 is a cathode. A reverse bias voltage VR is supplied to the anode and cathode of the APD 201 so that the APD 201 performs an avalanche multiplication operation. Here, the voltage VR is calculated by the following formula (1). VR = VL - VH (1)

[0022] In the APD 201, when such a voltage VR is supplied, the charge generated by the incident light undergoes avalanche multiplication, generating an avalanche current.

[0023] When the voltage VR is supplied, there are Geiger mode and linear mode. The Geiger mode is a mode in which the APD 201 operates with a voltage difference between the anode and cathode that is greater than the breakdown voltage. The linear mode is a mode in which the APD 201 operates with a voltage difference between the anode and cathode that is close to or less than the breakdown voltage. The APD 201 operated in Geiger mode is called a single photon avalanche diode (SPAD). In the case of a SPAD, for example, the drive voltage VL is -30V and the drive voltage VH is 1V.

[0024] The quench element 202 is connected to a power supply line to which a drive voltage VH is supplied and to one of the anode and cathode nodes of the APD 201. The quench element 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication of the APD 201, and has a function of suppressing avalanche multiplication by suppressing the voltage supplied to the APD 201 (quench operation). The quench element 202 also has a function of returning the voltage supplied to the APD 201 to the drive voltage VH by flowing a current equivalent to the voltage drop caused by the quench operation (recharge operation).

[0025] The signal processing circuit 103 has a quench element 202, a waveform shaping unit 210, a counter circuit 211, and a selection circuit 212. In Fig. 4, the signal processing circuit 103 has the quench element 202, the waveform shaping unit 210, the counter circuit 211, and the selection circuit 212, but is not limited to this. The signal processing circuit 103 may have at least one of the waveform shaping unit 210, the counter circuit 211, and the selection circuit 212.

[0026] The waveform shaping unit 210 shapes the waveform of the signal (voltage) of the cathode of the APD 201 obtained at the time of photon detection, and outputs a pulse signal. For example, an inverter circuit is used as the waveform shaping unit 210. Although an example using one inverter is shown in Fig. 4 as the waveform shaping unit 210, a circuit in which a plurality of inverters are connected in series may be used, or another circuit having a waveform shaping effect may be used.

[0027] The counter circuit 211 counts the pulse signal output from the waveform shaping unit 210 and holds the count value. When a control pulse RES is supplied via a drive line 213, the counter circuit 211 resets the signal held in the counter circuit 211.

[0028] A control pulse SEL is supplied to the selection circuit 212 from the vertical scanning circuit 110 in Fig. 3 via a drive line 214, and the selection circuit 212 switches between electrical connection / disconnection between the counter circuit 211 and the signal line 113. The selection circuit 212 includes, for example, a buffer circuit for outputting a signal, and outputs a count value from the counter circuit 211 to the signal line 113. Although an example of switching by the selection circuit 212 is shown in Fig. 4, a switch such as a transistor may be disposed between the quench element 202 and the APD 201, or between the photoelectric conversion unit 102 and the signal processing circuit 103, to switch the electrical connection. Similarly, the supply of the drive voltage VH or the drive voltage VL supplied to the photoelectric conversion unit 102 may be electrically switched using a switch such as a transistor.

[0029] <Circuit drive> 5 is a diagram showing the relationship between the operation of the APD 201 in FIG. 4 and the output signal. The input side of the waveform shaping unit 210 in FIG. 4 is nodeA, and the output side of the waveform shaping unit 210 is nodeB. Between time t0 and time t1, a potential difference of VH-VL is applied to the APD 201. When a photon is incident on the APD 201 at time t1, avalanche multiplication occurs in the APD 201, an avalanche multiplication current flows through the quench element 202, and the voltage of nodeA drops. When the voltage drop amount of nodeA further increases and the potential difference applied to the APD 201 decreases, as at time t2, the avalanche multiplication of the APD 201 stops, and the voltage level of nodeA does not drop by more than a certain value. After that, between time t2 and time t3, a current flows through nodeA to compensate for the voltage drop from drive voltage VL, and at time t3, nodeA is stabilized to its original potential level. At this time, waveform shaping section 210 outputs a signal for nodeB that sets the portion of the waveform value of nodeA lower than the waveform threshold to a high level and sets the portion of the waveform value of nodeA higher than the waveform threshold to a low level.

[0030] Fig. 6 is a diagram showing an example of the configuration of an imaging system 600 according to this embodiment. The imaging system 600 includes a surveillance camera 601, a network 602, a client device 603, an input device 604, and a display device 605. The surveillance camera 601 is an example of an imaging device, includes the photoelectric conversion element 100 of Fig. 1, captures a moving image, and performs image processing on the moving image. The surveillance camera 601 and the client device 603 can communicate with each other via the network 602. The input device 604 and the display device 605 are connected to the client device 603. The client device 603 is an example of an information processing device.

[0031] Fig. 7 is a diagram showing an example of the configuration of the surveillance camera 601 in Fig. 6. The surveillance camera 601 has a photoelectric conversion element 100, an optical system 701, a CPU 702, a ROM 703, a RAM 704, an imaging system control unit 705, and a control unit 706. Furthermore, the surveillance camera 601 has an estimated value calculation unit 707, a camera setting acquisition unit 708, a threshold calculation unit 709, a parameter setting unit 710, an image processing execution unit 711, an encoder unit 712, and a network I / F 713.

[0032] The optical system 701 has a zoom lens, a focus lens, a blur correction lens, an aperture, and a shutter, and collects optical information of a subject. The photoelectric conversion element 100 is a photoelectric conversion element 100 having the APD 201 described in Figs. 1 to 5, is an example of an imaging unit, and obtains color information by combining with a color filter or the like. The photoelectric conversion element 100 changes its DCR and output count value as the usage time increases. The DCR is the dark count rate, which is the average value of the count value of the counter circuit 211 (Fig. 4) detected when there is no incident light.

[0033] A CPU 702 is involved in all the processing of each component, sequentially reading and interpreting programs stored in a ROM (Read Only Memory) 703 and a RAM (Random Access Memory) 704, and executing processing according to the results.

[0034] The imaging system control unit 705 performs control of the optical system 701, such as adjusting the focus, opening the shutter, and adjusting the aperture, in response to instructions from the CPU 702. The control unit 706 performs control in accordance with instructions from the client device 603.

[0035] The estimate calculation unit 707 quantitatively calculates an estimate of the degree of change in aging of the photoelectric conversion element 100 having the APD 201. Here, the estimate of the degree of change is calculated based on information such as the number of avalanche emissions since the start of operation, exposure conditions (aperture, shutter, gain), and temperature.

[0036] The camera setting acquisition unit 708 acquires setting information of various camera functions as camera setting information from the imaging system control unit 705 and the control unit 706. The threshold calculation unit 709 calculates a change threshold for determining the degree of change based on the camera setting information acquired from the camera setting acquisition unit 708. Here, the above-mentioned camera setting information refers to, for example, codec settings for compression processing in the downstream encoder unit 712, and setting information for tone correction, noise reduction processing, defective pixel correction, demosaicing processing, etc. included in the image processing execution unit 711.

[0037] A parameter setting unit 710 compares the change degree estimated value calculated by the estimated value calculation unit 707 with the change threshold value calculated by the threshold calculation unit 709 to calculate and set image processing parameters. Here, the image processing parameters refer to parameters for gradation correction processing, noise reduction processing, defective pixel correction processing, saturation enhancement processing, demosaicing processing, and the like, which are executed by an image processing execution unit 711.

[0038] The image processing execution unit 711 is an image processing unit that executes image processing on the count value output by the photoelectric conversion element 100 based on the image processing parameters set by the parameter setting unit 710, and outputs image data. The image processing includes, for example, tone correction processing, noise reduction processing, defective pixel correction processing, saturation enhancement processing, demosaicing processing, and the like.

[0039] The encoder unit 712 converts the image data output from the image processing execution unit 711 into a file format such as Motion JPEG, H264, H265, etc. The network I / F 713 is an interface used for communication with an external device such as a client device 603 via the network 602.

[0040] The network 602 is a network that connects the surveillance camera 601 and the client device 603. The network is composed of, for example, a plurality of routers, switches, cables, etc. that satisfy a communication standard such as Ethernet (registered trademark). In this embodiment, the network 602 may be any network that can perform communication between the surveillance camera 601 and the client device 603, and the communication standard, scale, and configuration are not important. For example, the network 602 may be composed of the Internet, a wired LAN (Local Area Network), a wireless LAN (Wireless LAN), a WAN (Wide Area Network), etc.

[0041] Fig. 8 is a flowchart showing a processing method of the surveillance camera 601 in Fig. 7. The processing of the flowchart in Fig. 8 may be executed by hardware or software, or may be realized partially by software configuration and the rest by hardware configuration. When executed by software, the processing of the flowchart in Fig. 8 is realized by CPU 702 executing a program stored in ROM 703 or the like, for example. The same applies to the flowchart in Fig. 10 described later.

[0042] In step S801, the estimate calculation unit 707 estimates the degree of change in the aging of the photoelectric conversion element 100 having the APD 201, and calculates an estimate of the degree of change of the photoelectric conversion element 100. For example, the photoelectric conversion element 100 changes its DCR and outputs a count value in response to an increase in the usage time. The estimate calculation unit 707 calculates the degree of change estimate based on the number of avalanche emissions since the start of operation, exposure conditions, temperature, or luminance information of the subject. As an example, the estimate calculation unit 707 calculates the cumulative value of luminance since the start of operation based on the count value output by the photoelectric conversion element 100, and calculates the degree of change estimate based on the cumulative value of luminance. The larger the cumulative value of luminance, the larger the degree of change estimate. The larger the degree of change estimate, the larger the degree of change, and the smaller the degree of change estimate, the smaller the degree of change. The degree of change includes the degree of deterioration.

[0043] Next, in step S802, the camera setting acquisition unit 708 acquires setting information of various camera functions as camera setting information (distribution setting information) from the imaging system control unit 705 and the control unit 706. The camera setting information is setting information of the surveillance camera 601. In this embodiment, an embodiment will be described in which codec setting information of the compression process acquired from the downstream encoder unit 712 is used as an example of the camera setting information. Specifically, the camera setting acquisition unit 708 acquires codec setting information of the encoding method, bit rate, and Qp (Quantization Parameter) value of the encoder unit 712 as the camera setting information.

[0044] The following describes the process and problems associated with the codec setting information acquired in this embodiment. In general, in an imaging system 600 that distributes video via a network 602 as shown in Fig. 6, there are restrictions on the capacity and distribution band of the network server, and therefore video distribution with a reduced bit rate is required. As such, there are techniques called CBR (Contrast bitrate) encoding and VBR (Variable bitrate) encoding as encoding methods for compressing video. CBR encoding is a technique for encoding data so that it always has a constant bit rate, and VBR encoding is a technique for encoding by varying the bit rate value.

[0045] With CBR encoding, image quality changes depending on the bit rate you set. When the bit rate is set to low, frequencies over a wide range, including not only high-frequency components but also low-frequency components, are compressed, resulting in discontinuous changes in brightness, color, and edges between blocks, creating an issue known as block noise.

[0046] It is known that similar issues can occur with VBR encoding depending on the Qp value setting. In general, the smaller the Qp value, the higher the image quality and the larger the data size, while the larger the value, the lower the image quality and the more likely block noise will occur. Images with block noise undergo strong compression processing, including high-frequency noise and pixel noise components.

[0047] The above is the process and issues related to the codec setting information acquired in this embodiment, and in steps from step S803 onwards, a change threshold value is set taking into account the issue of block noise, which will be described later.

[0048] Next, in steps S803 to S809, the threshold calculation unit 709 calculates a change threshold for determining the degree of change based on the above-mentioned camera setting information. First, in step S803, the threshold calculation unit 709 determines whether the encoding method is VBR encoding or CBR encoding based on the encoding method information in the camera setting information. If it is CBR encoding, the process proceeds to step S804, and if it is VBR encoding, the process proceeds to step S807.

[0049] In step S804, the threshold calculation unit 709 determines whether the bit rate of the CBR encoding is equal to or greater than the bit rate threshold based on the bit rate information in the camera setting information. The bit rate threshold is, for example, 8 Mbps. If the bit rate is equal to or greater than the bit rate threshold, the process proceeds to step S805. If the bit rate is not equal to or greater than the bit rate threshold, the process proceeds to step S806.

[0050] In step S805, the threshold calculation unit 709 sets a small first value as the change threshold. The threshold calculation unit 709 is an example of a threshold setting unit. After that, the process proceeds to step S810.

[0051] In step S806, threshold calculation section 709 sets a second value greater than the first value as the change threshold value. Then, the process proceeds to step S810.

[0052] In step S807, the threshold calculation unit 709 determines whether the Qp value of the VBR encoding is equal to or less than the Qp threshold based on the Qp value information in the camera setting information. The Qp threshold is, for example, 10. If the Qp value is equal to or less than the Qp threshold, the process proceeds to step S808, and if the Qp value is not equal to or less than the Qp threshold, the process proceeds to step S809.

[0053] In step S808, threshold calculation section 709 sets a smaller third value as the change threshold value. Then, the process proceeds to step S810.

[0054] In step S809, threshold calculation section 709 sets a fourth value greater than the third value as the change threshold value. Then, the process proceeds to step S810.

[0055] As described above, the change threshold set in steps S803 to S809 is used in comparison with the change degree estimated value in steps S810 to S812. When the change threshold is small, the bit rate is set high (or the Qp value is small), so there is a high possibility that the effects of DCR noise (dark current noise) and defective pixel noise due to aging will become apparent due to the subsequent compression process. On the other hand, when the change threshold is large, the bit rate is set low (or the Qp value is large), so there is a high possibility that the effects of DCR noise and defective pixel noise due to aging will be small due to the subsequent powerful compression process. In consideration of the relationship between the change threshold and image quality as described above, parameter setting unit 710 sets optimal image processing parameters in steps S811 or S812.

[0056] Next, in steps S810 to S812, parameter setting section 710 determines whether or not the change degree estimated value calculated by estimate calculation section 707 is equal to or greater than the change threshold value set by threshold calculation section 709. If the change degree estimated value is equal to or greater than the change threshold value, the process proceeds to step S811, and if the change degree estimated value is not equal to or greater than the change threshold value, the process proceeds to step S812.

[0057] In step S811, the parameter setting unit 710 sets, as the image processing parameters, the first image processing parameters for strongly reducing noise and defective pixel noise. Then, the process proceeds to step S813.

[0058] In step S812, the parameter setting unit 710 sets, as the image processing parameters, second image processing parameters for leaving noise or defective pixel noise as is. The first image processing parameters in step S811 are parameters for performing processing for reducing noise or defective pixel noise more strongly than the second image processing parameters in step S812. Then, the process proceeds to step S813.

[0059] Finally, in step S813, the image processing execution unit 711 performs image processing on the count value (output signal) output by the photoelectric conversion element 100 based on the first image processing parameter or the second image processing parameter set by the parameter setting unit 710.

[0060] As described above, the surveillance camera 601 sets the change threshold value in consideration of the effect of the compression process of the encoder unit 712, and sets optimal image processing parameters according to the change degree estimated value and the change threshold value. Therefore, the surveillance camera 601 can optimally set parameters of the image processing execution unit 711 located at the front stage and the encoder unit 712 located at the rear stage in response to the aging (increase in DCR noise and defective pixel noise) of the photoelectric conversion element 100 having the APD 201. If the compression process of the encoder unit 712 is not taken into consideration, there is a problem that noise and defective pixel noise are unnecessarily over-corrected, or the correction is insufficient, but according to this embodiment, it is possible to solve this problem. The surveillance camera 601 of this embodiment can suppress the influence of aging of the photoelectric conversion element 100 caused by an increase in the usage time.

[0061] Second Embodiment In the second embodiment, a configuration will be described in which image processing parameters are set for each region based on an estimated value of the degree of change for each region.

[0062] Fig. 9 is a diagram showing an example of the configuration of a surveillance camera 601 according to the second embodiment. The surveillance camera 601 in Fig. 9 is provided with a region-specific estimated value calculation unit 901, a region-specific parameter setting unit 902, and a region-specific image processing execution unit 903 instead of the estimated value calculation unit 707, the parameter setting unit 710, and the image processing execution unit 711 of the surveillance camera 601 in Fig. 7. The surveillance camera 601 has the region-specific estimated value calculation unit 901, a camera setting acquisition unit 708, a threshold calculation unit 709, a region-specific parameter setting unit 902, a region-specific image processing execution unit 903, an encoder unit 712, and a network I / F 713.

[0063] FIG. 10 is a flowchart showing a processing method of the surveillance camera 601 according to the second embodiment. First, in step S1001, the area-specific estimated value calculation unit 901 estimates the degree of change in aging change for each area of ​​the photoelectric conversion element 100 having the APD 201, and calculates an estimated value of the degree of change for each area of ​​the photoelectric conversion element 100. For example, the DCR of the photoelectric conversion element 100 changes with an increase in the usage time, and the count value to be output changes. The estimated value calculation unit 707 calculates the estimated value of the degree of change for each area based on the number of avalanche emissions since the start of operation, the exposure conditions, the temperature, or the luminance information of the subject. The unit of the area may be a pixel unit, or a divided area unit prepared for obtaining an evaluation value of AF (Auto Focus), AE (Auto Exposure), or the like.

[0064] Next, in step S1002, the camera setting acquisition unit 708 acquires camera setting information from the imaging system control unit 705 and the control unit 706. The camera setting information is, for example, on / off information of a dark correction function executed by the region-specific image processing execution unit 903. The surveillance camera 601 has a function of improving the visibility of a dark subject by raising the gamma of a low-luminance area. The dark correction by the region-specific image processing execution unit 903 improves the visibility of a subject in a dark area, but there is a problem that noise and defective pixel noise are also noticeable as a commonly known trade-off. In this embodiment, a method of calculating a change threshold for determining an estimated value of a change degree for each region will be described, taking into account the above-mentioned trade-off.

[0065] In steps S1003 to S1005, the threshold calculation unit 709 calculates a change threshold for determining the degree of change for each region based on the above-mentioned camera setting information. First, in step S1003, the threshold calculation unit 709 determines whether the dark compensation function is on or off based on the on / off information of the dark compensation function in the camera setting information. If the dark compensation function is on, the process proceeds to step S1004, and if the dark compensation function is off, the process proceeds to step S1005.

[0066] In step S1004, threshold calculation section 709 sets a smaller fifth value as the change threshold value. Then, the process proceeds to step S1006.

[0067] In step S1005, threshold calculation section 709 sets a sixth value greater than the fifth value as the change threshold value. After that, the process proceeds to step S1006.

[0068] As described above, in steps S1003 to S1005, when the dark compensation function is on, the change threshold is set small to make it easier to correct noise and defective pixel noise that are easily made apparent by dark compensation. On the other hand, when the dark compensation function is off, the change threshold is set large to reduce defective pixel noise and noise more than necessary so as not to lose edge information of the subject.

[0069] Next, in step S1006, region-specific parameter setting unit 902 determines whether or not the change degree estimated value for each region calculated by region-specific estimated value calculation unit 901 is equal to or greater than the change threshold value set by threshold calculation unit 709. For regions where the change degree estimated value is equal to or greater than the change threshold value, the process proceeds to step S1007. For regions where the change degree estimated value is not equal to or greater than the change threshold value, the process proceeds to step S1008.

[0070] In step S1007, the region-specific parameter setting unit 902 sets image processing parameters for each region for strongly reducing noise and defective pixel noise, regarding the image processing parameters for the corresponding region. Then, the process proceeds to step S1009.

[0071] In step S1008, the region-specific parameter setting unit 902 sets image processing parameters for the corresponding region for leaving noise and defective pixel noise as they are, and then the process proceeds to step S1009.

[0072] Finally, in step S1009, the area-specific image processing execution unit 903 executes image processing for each area on the count value output by the photoelectric conversion element 100 based on the image processing parameters for each area set by the parameter setting unit 710.

[0073] As described above, the surveillance camera 601 calculates the change threshold value taking into consideration the influence of noise and defective pixel noise emphasized by the image processing of the regional image processing execution unit 903, and sets optimal image processing parameters for each region according to the change degree estimated value and change threshold value for each region. Therefore, the surveillance camera 601 can optimally set the image processing parameters of the regional image processing execution unit 903 for each region in response to aging (increase in DCR noise and defective pixel noise) occurring in the photoelectric conversion element 100 having the APD 201.

[0074] In the present embodiment, a configuration has been described in which the setting information for the dark correction process is used as the camera setting information acquired by the camera setting acquisition unit 708. However, the camera setting information is not limited to this. For example, the camera setting information may be setting information for a codec, gradation correction, noise reduction process, defective pixel correction, demosaicing process, sharpness process, or saturation enhancement process. The image processing performed by the image processing execution unit 711 includes gradation correction, noise reduction process, defective pixel correction, demosaicing process, sharpness process, saturation enhancement process, dark correction, etc.

[0075] In the above embodiment, a so-called lens-integrated imaging device in which the optical system 701 is integrally formed with the surveillance camera 601 has been described as an example of an imaging device, but the present invention is not limited to this. For example, a so-called lens-interchangeable imaging device in which the surveillance camera 601 and a lens unit including the optical system 701 are separately provided may also be used.

[0076] In the above-described embodiment, the surveillance camera 601 has been described as an example of an imaging device, but the present invention is not limited to this. For example, the present invention may be configured to employ imaging devices other than the surveillance camera 601, such as a digital camera, a digital video camera, a portable device such as a smartphone, or a wearable terminal.

[0077] Furthermore, in the above-described embodiment, the client device 603, which is an information processing device, is, for example, an electronic device such as a personal computer (PC), but is not limited to this. For example, the client device 603 may be configured to employ other electronic devices such as a smartphone or a tablet terminal.

[0078] (Other embodiments) The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) for implementing one or more functions.

[0079] It should be noted that the above-described embodiments are merely illustrative of specific examples of implementing the present disclosure, and the technical scope of the present disclosure should not be interpreted as being limited by these embodiments. In other words, the present disclosure can be implemented in various forms without departing from its technical concept or main features.

[0080] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) an imaging section including an avalanche photodiode; an estimate calculation unit that calculates an estimate of a degree of change of the imaging unit; an image processing unit that performs image processing on an output signal of the imaging unit based on the change degree estimated value and an image processing parameter based on a first threshold value; An imaging device comprising: (Configuration 2) The imaging unit includes: The avalanche photodiode; and a counter that counts the signal from the avalanche photodiode. (Configuration 3) the imaging unit further includes a shaping unit that shapes a waveform of a signal from the avalanche photodiode; 3. The imaging device according to configuration 2, wherein the counter counts the signal shaped by the shaping unit. (Configuration 4) The imaging device according to any one of configurations 1 to 3, wherein the estimated value calculation unit calculates an estimated value of a degree of change in the imaging unit based on the number of avalanche flashes since the start of operation, exposure conditions, temperature, or brightness information of the subject. (Configuration 5) The imaging device according to any one of configurations 1 to 4, wherein the estimated value calculation unit calculates a cumulative value of luminance based on the output signal of the imaging unit, and calculates an estimated value of a degree of change of the imaging unit based on the cumulative value of luminance. (Configuration 6) 6. The imaging device according to any one of configurations 1 to 5, further comprising a threshold setting section that sets the first threshold based on setting information of the imaging device. (Configuration 7) The imaging device according to any one of configurations 1 to 6, wherein the setting information of the imaging device is setting information of a codec, a gradation correction, a noise reduction process, a defective pixel correction, a demosaicing process, a sharpness process, a saturation enhancement process, or a dark area correction. (Configuration 8) the setting information of the imaging device includes a bit rate of an encoder; The imaging device described in configuration 6, characterized in that the threshold setting unit sets a first value as the first threshold when the bit rate is equal to or greater than a second threshold, and sets a second value greater than the first value as the first threshold when the bit rate is not equal to or greater than the second threshold. (Configuration 9) the setting information of the imaging device includes a Qp value of an encoder; The imaging device described in configuration 6, characterized in that the threshold setting unit sets a third value as the first threshold when the Qp value is equal to or less than a third threshold, and sets a fourth value greater than the third value as the first threshold when the Qp value is not equal to or less than the third threshold. (Configuration 10) The setting information of the imaging device includes an encoding method, a bit rate, and a Qp value of an encoder; The threshold setting unit is if the encoding method is VBR encoding and the bit rate is equal to or greater than a second threshold, set the first threshold to a first value; if the encoding method is VBR encoding and the bit rate is not equal to or greater than a second threshold, set the first threshold to a second value that is greater than the first value; if the encoding method is CBR encoding and the Qp value is equal to or smaller than a third threshold, set the first threshold to a third value; The imaging device described in configuration 6, characterized in that if the encoding method is CBR encoding and the Qp value is not less than or equal to a third threshold, a fourth value greater than the third value is set as the first threshold. (Configuration 11) The setting information of the imaging device is on / off information of a dark compensation function, The imaging device described in configuration 6, wherein the threshold setting unit sets a fifth value as the first threshold when the dark compensation function is on, and sets a sixth value larger than the fifth value as the first threshold when the dark compensation function is off. (Configuration 12) a parameter setting unit that sets a first image processing parameter when the change degree estimation value is equal to or greater than the first threshold, and sets a second image processing parameter when the change degree estimation value is not equal to or greater than the first threshold, 12. The imaging device according to any one of configurations 1 to 11, wherein the image processing unit performs image processing based on a first image processing parameter or a second image processing parameter set by the parameter setting unit. (Configuration 13) 13. The imaging device according to configuration 12, wherein the first image processing parameter is a parameter for performing processing to reduce noise or defective pixel noise more strongly than the second image processing parameter. (Configuration 14) The imaging device according to any one of configurations 1 to 13, characterized in that the image processing performed by the image processing unit is gradation correction, noise reduction processing, defective pixel correction, demosaicing processing, sharpness processing, saturation enhancement processing, or dark area correction. (Configuration 15) the estimate calculation unit calculates a change degree estimate value for each region of the imaging unit; The imaging device according to any one of configurations 1 to 14, characterized in that the image processing unit performs image processing for each region on the output signal of the imaging unit based on an estimated value of the degree of change for each region and an image processing parameter for each region that is based on a first threshold value. (Method 1) an estimated value calculation step of calculating an estimated value of a degree of change of an imaging unit including an avalanche photodiode; an image processing step of performing image processing on an output signal of the imaging unit based on the change degree estimated value and an image processing parameter based on a first threshold value; 13. A processing method for an imaging device, comprising: (Program 1) an estimated value calculation step of calculating an estimated value of a degree of change of an imaging unit including an avalanche photodiode; an image processing step of performing image processing on an output signal of the imaging unit based on the change degree estimated value and an image processing parameter based on a first threshold value; A program for causing a computer to execute the following. [Explanation of symbols]

[0081] 100 photoelectric conversion element, 601 surveillance camera, 701 optical system, 702 CPU, 703 ROM, 704 RAM, 705 imaging system control unit, 706 control unit, 707 estimated value calculation unit, 708 camera setting acquisition unit, 709 threshold calculation unit, 710 parameter setting unit, 711 image processing execution unit, 712 encoder unit, 713 network I / F

Claims

1. an imaging section including an avalanche photodiode; an estimate calculation unit that calculates an estimate indicating a degree of deterioration of the imaging unit based on at least one of information regarding the number of avalanche emissions in the avalanche photodiode, an exposure condition of the imaging unit, and a temperature of the imaging unit; an image processing unit that performs image processing on an output signal of the imaging unit based on an image processing parameter that is based on the estimated value indicating the degree of deterioration and a first threshold value; An imaging device comprising:

2. The imaging unit includes: The avalanche photodiode; 2. The image pickup apparatus according to claim 1, further comprising a counter for counting the signal from the avalanche photodiode.

3. the imaging unit further includes a shaping unit that shapes a waveform of a signal from the avalanche photodiode; 3. The imaging device according to claim 2, wherein the counter counts the signal shaped by the shaping unit.

4. The imaging device according to claim 1 , wherein the estimated value calculation unit calculates an estimated value indicating a degree of deterioration of the imaging unit based on the number of avalanche flashes since the start of operation, exposure conditions, temperature, or subject brightness information.

5. The imaging device according to claim 1, characterized in that the estimate value calculation unit calculates a cumulative value of luminance of multiple images obtained based on the output signal of the imaging unit, and calculates an estimate value indicating a degree of deterioration of the imaging unit based on the cumulative value of luminance.

6. 2. The image pickup device according to claim 1, further comprising a threshold setting unit that sets the first threshold based on setting information of the image pickup device.

7. The imaging device according to claim 6 , wherein the setting information of the imaging device is codec setting information.

8. the setting information of the imaging device includes a bit rate of an encoder; 7. The imaging device according to claim 6, wherein the threshold setting unit sets a first value as the first threshold when the bit rate is equal to or greater than a second threshold, and sets a second value greater than the first value as the first threshold when the bit rate is not equal to or greater than the second threshold.

9. the setting information of the imaging device includes a Qp value of an encoder; 7. The imaging device according to claim 6, wherein the threshold setting unit sets a third value as the first threshold when the Qp value is equal to or less than a third threshold, and sets a fourth value greater than the third value as the first threshold when the Qp value is not equal to or less than the third threshold.

10. The setting information of the imaging device includes an encoding method, a bit rate, and a Qp value of an encoder; The threshold setting unit is if the encoding method is VBR encoding and the bit rate is equal to or greater than a second threshold, set the first threshold to a first value; if the encoding method is VBR encoding and the bit rate is not equal to or greater than a second threshold, set the first threshold to a second value greater than the first value; if the encoding method is CBR encoding and the Qp value is equal to or smaller than a third threshold, set the first threshold to a third value; 7. The imaging device according to claim 6, wherein when the encoding method is CBR encoding and the Qp value is not equal to or less than a third threshold, a fourth value greater than the third value is set as the first threshold.

11. The setting information of the imaging device is on / off information of a dark compensation function, 7. The imaging device according to claim 6, wherein the threshold setting unit sets a fifth value as the first threshold when the dark compensation function is on, and sets a sixth value larger than the fifth value as the first threshold when the dark compensation function is off.

12. a parameter setting unit that sets a first image processing parameter when the estimated value indicating the degree of deterioration is equal to or greater than the first threshold, and sets a second image processing parameter when the estimated value indicating the degree of deterioration is not equal to or greater than the first threshold, 2. The imaging apparatus according to claim 1, wherein the image processing unit performs image processing based on a first image processing parameter or a second image processing parameter set by the parameter setting unit.

13. 13. The imaging apparatus according to claim 12, wherein the first image processing parameter is a parameter for performing processing for reducing noise or defective pixel noise more strongly than the second image processing parameter.

14. 2. The imaging apparatus according to claim 1, wherein the image processing performed by the image processing unit is noise reduction processing or defective pixel correction processing.

15. the estimated value calculation unit calculates an estimated value indicating a degree of deterioration for each region of the imaging unit; The imaging device according to claim 1, characterized in that the image processing unit performs image processing for each region on the output signal of the imaging unit based on an estimated value indicating the degree of deterioration for each region and an image processing parameter for each region that is based on a first threshold value.

16. an estimate calculation step of calculating an estimate indicating a degree of deterioration of an imaging section including an avalanche photodiode based on at least one of avalanche light emission counts in the avalanche photodiode, exposure conditions of the imaging section, and temperature; an image processing step of performing image processing on an output signal of the imaging unit based on an image processing parameter based on the estimated value indicating the deterioration degree and a first threshold value; 13. A processing method for an imaging device, comprising:

17. an estimate calculation step of calculating an estimate indicating a degree of deterioration of an imaging section including an avalanche photodiode based on at least one of avalanche light emission counts in the avalanche photodiode, exposure conditions of the imaging section, and temperature; an image processing step of performing image processing on an output signal of the imaging unit based on an image processing parameter based on the estimated value indicating the deterioration degree and a first threshold value; A program for causing a computer to execute the following.

Images