Photoelectric conversion device, information processing device, method for controlling photoelectric conversion device and information processing method

Abstract

To provide a photoelectric conversion device for further reducing a delay time of detecting and correcting a pixel existing outside the range of prescribed reference.SOLUTION: A photoelectric conversion device includes a pixel for generating a signal corresponding to an incident light quantity, a memory for storing the signal, and a processing part for performing signal processing based on the signal held by the memory, the pixel generates the signal during a second storage period including a first storage period and a second storage period including a period after the first storage period, the processing part acquires a first signal corresponding to an incident light quantity in the first storage period from the memory during a period from an end of the first storage period to an end of the second storage period, the processing part generates pixel information showing that a pixel does not satisfy prescribed reference in the case of detecting that the pixel does not satisfy the prescribed reference by using the first signal, the processing part acquires a second signal corresponding to an incident light quantity during the second storage period from the memory after the end of the second storage period, and the processing part corrects the second signal on the basis of the pixel information.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The present invention relates to a photoelectric conversion device, an information processing device, a method for controlling a photoelectric conversion device, and an information processing method. [Background technology]

[0002] Patent Document 1 discloses an imaging device that detects abnormal pixels in each of multiple images captured under different conditions and determines the cause of the abnormality of the detected abnormal pixels based on the difference in output levels of the abnormal pixels. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2008-148129 Summary of the Invention [Problem to be solved by the invention]

[0004] However, there are cases where it is desired to further reduce the processing delay involved in detecting and correcting pixels that fall outside the range of a predetermined standard.

[0005] The present invention aims to provide a photoelectric conversion device, an information processing device, a method for controlling a photoelectric conversion device, and an information processing method in which the delay time in detecting and correcting pixels outside a predetermined standard range is reduced. [Means for solving the problem]

[0006] According to one disclosure of the present specification, there is provided a photoelectric conversion device comprising: a pixel that generates a signal corresponding to an amount of incident light; a memory that stores the signal; and a processing unit that performs signal processing based on the signal stored in the memory, wherein the pixel generates the signal during a second accumulation period that includes a first accumulation period and a period after the first accumulation period; the processing unit acquires from the memory a first signal corresponding to the amount of incident light during the first accumulation period during the period from the end of the first accumulation period to the end of the second accumulation period; when the processing unit detects using the first signal that a pixel does not satisfy a predetermined criterion, it generates pixel information indicating that the pixel does not satisfy the predetermined criterion; the processing unit acquires from the memory a second signal corresponding to the amount of incident light during the second accumulation period after the end of the second accumulation period; and the processing unit corrects the second signal based on the pixel information.

[0007] According to one disclosure of the present specification, there is provided an information processing device comprising: a detection unit that, when detecting that a pixel does not satisfy a predetermined standard using a first signal corresponding to the amount of light incident on the pixel during a first accumulation period, generates pixel information indicating that the pixel does not satisfy the predetermined standard; and a correction unit that corrects, based on the pixel information, a second signal corresponding to the amount of light incident on the pixel during a second accumulation period that includes the first accumulation period and a period after the first accumulation period, wherein the detection unit acquires the first signal during the period from the end of the first accumulation period to the end of the second accumulation period, and the correction unit acquires the second signal after the end of the second accumulation period.

[0008] According to one disclosure of this specification, there is provided a control method for a photoelectric conversion device having a pixel that generates a signal corresponding to the amount of incident light, a memory that stores the signal, and a processing unit that performs signal processing based on the signal stored in the memory, the control method comprising: a step in which the pixel generates the signal during a second accumulation period that includes a first accumulation period and a period after the first accumulation period; a step in which the processing unit acquires from the memory a first signal corresponding to the amount of incident light during the first accumulation period during a period from the end of the first accumulation period to the end of the second accumulation period; a step in which the processing unit generates pixel information indicating that the pixel does not satisfy the predetermined standard when it is detected using the first signal that the pixel does not satisfy the predetermined standard; a step in which the processing unit acquires from the memory a second signal corresponding to the amount of incident light during the second accumulation period after the end of the second accumulation period; and a step in which the processing unit corrects the second signal based on the pixel information.

[0009] According to one disclosure of the present specification, there is provided an information processing method comprising the steps of: generating abnormal pixel information indicating the detection result of an abnormal pixel based on a first signal corresponding to the amount of light incident on a pixel during a first accumulation period; and correcting, based on the abnormal pixel information, a second signal corresponding to the amount of light incident on the pixel during a second accumulation period that includes the first accumulation period and a period after the first accumulation period; wherein the first signal is acquired during the period from the end of the first accumulation period to the end of the second accumulation period, and the second signal is acquired after the end of the second accumulation period. [Effects of the Invention]

[0010] According to the present invention, a photoelectric conversion device, an information processing device, a method for controlling a photoelectric conversion device, and an information processing method are provided in which processing delays caused by detecting and correcting pixels outside a predetermined standard range are further reduced. [Brief explanation of the drawings]

[0011] [Figure 1]1 is a schematic diagram illustrating the overall configuration of a photoelectric conversion device according to a first embodiment. [Figure 2] 2 is a schematic block diagram showing an example of the configuration of a sensor substrate according to the first embodiment. FIG. [Figure 3] 1 is a schematic block diagram showing an example of the configuration of a circuit board according to a first embodiment. [Figure 4] 2 is a schematic block diagram showing an example of the configuration of one pixel of a photoelectric conversion unit and a pixel signal processing unit according to the first embodiment. FIG. [Figure 5] 3A to 3C are diagrams illustrating the operation of the avalanche photodiode according to the first embodiment. [Figure 6] FIG. 2 is a block diagram showing an example of the configuration of a pulse processing unit according to the first embodiment. [Figure 7] 5 is a timing chart illustrating the operation of the pulse processing unit according to the first embodiment. FIG. [Figure 8] 3A and 3B are diagrams illustrating examples of images acquired by the photoelectric conversion device according to the first embodiment. [Figure 9] FIG. 2 is a block diagram showing an example of the configuration of a signal processing unit according to the first embodiment. [Figure 10] 2A and 2B are schematic diagrams illustrating an example of a pixel of interest and peripheral pixels according to the first embodiment. [Figure 11] FIG. 4 is a timing chart illustrating the operation of the signal processing unit according to the first embodiment. [Figure 12] 10A and 10B are schematic diagrams illustrating an example of a pixel of interest and peripheral pixels according to a modified example of the first embodiment. [Figure 13] FIG. 10 is a block diagram showing an example of the configuration of a signal processing unit according to a second embodiment. [Figure 14] FIG. 10 is a timing chart illustrating the operation of the signal processing unit according to the second embodiment. [Figure 15] FIG. 11 is a block diagram showing an example of the configuration of a signal processing unit according to a third embodiment. [Figure 16] FIG. 10 is a timing chart illustrating the operation of a signal processing unit according to the third embodiment. [Figure 17] FIG. 10 is a block diagram showing an example of the configuration of a signal processing unit according to a fourth embodiment. [Figure 18]10 is a flowchart illustrating the operation of the photoelectric conversion device according to the fourth embodiment. [Figure 19] FIG. 11 is a block diagram of a device according to a fifth embodiment. [Figure 20] FIG. 13 is a block diagram of a device according to a sixth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The same or corresponding elements in multiple drawings are designated by common reference numerals, and their description may be omitted or simplified.

[0013] [First embodiment] FIG. 1 is a schematic diagram showing the overall configuration of a photoelectric conversion device according to this embodiment. The photoelectric conversion device 100 includes a sensor substrate 11 and a circuit substrate 21, which are stacked on top of each other. The sensor substrate 11 and the circuit substrate 21 are electrically connected to each other. The sensor substrate 11 includes a pixel region 12 in which a plurality of pixels 101 are arranged in a plurality of rows and a plurality of columns. The circuit substrate 21 includes a first circuit region 22 in which a plurality of pixel signal processing units 103 are arranged in a plurality of rows and a plurality of columns, and a second circuit region 23 arranged on the periphery of the first circuit region 22. The second circuit region 23 may include a circuit for controlling the plurality of pixel signal processing units 103, etc.

[0014] FIG. 2 is a schematic block diagram showing an example of the configuration of the sensor substrate 11 according to this embodiment. A plurality of pixels 101 are arranged in a pixel region 12 to form a plurality of rows and a plurality of columns. Each of the plurality of pixels 101 has a photoelectric conversion unit 102 including an avalanche photodiode (hereinafter referred to as an APD) as a photoelectric conversion element within the substrate. In FIG. 2, some of the m×n pixels 101 arranged in m rows from the first row to the mth row and n columns from the first column to the nth column are shown with reference numerals indicating the row and column numbers. For example, the pixel 101 arranged in the first row and the third column is designated by the reference numeral "P13." The number of rows and columns of the pixel array constituting the pixel region 12 is not particularly limited.

[0015] 3 is a schematic block diagram showing an example of the configuration of a circuit board 21 according to this embodiment. The circuit board 21 has a first circuit area 22, a second circuit area 23, a signal readout circuit 115, and a signal processing unit 117 (processing unit).

[0016] The first circuit area 22 has a plurality of pixel signal processing units 103 arranged in a plurality of rows and a plurality of columns. In Fig. 3, some of the m x n pixel signal processing units 103 arranged in m rows from the first row to the mth row and n columns from the first column to the nth column are shown with reference numerals indicating the row and column numbers. For example, the pixel signal processing unit 103 arranged in the first row and the third column is assigned the reference numeral "S13." The number of rows and columns of the pixel signal processing unit array constituting the first circuit area 22 is not particularly limited.

[0017] A vertical control pulse generating section 110 and a horizontal control pulse generating section 111 are arranged in the second circuit area 23. A vertical control line 112 and a data output line 114 are arranged in each row of the pixel signal processing section array in the first circuit area 22, extending in a first direction (the horizontal direction in FIG. 3). The vertical control line 112 and the data output line 114 are connected to each of the multiple pixel signal processing sections 103 arranged in the first direction. The first direction in which the vertical control line 112 extends may be referred to as the row direction or the horizontal direction.

[0018] A horizontal control line 113 is arranged in each column of the pixel signal processing unit array in the first circuit area 22, extending in the second direction (the vertical direction in FIG. 3). The horizontal control line 113 is connected to each of the multiple pixel signal processing units 103 arranged in the second direction. The second direction in which the horizontal control line 113 extends is sometimes referred to as the column direction or the vertical direction.

[0019] The vertical control line 112 of each row is connected to the vertical control pulse generation unit 110. The vertical control pulse generation unit 110 supplies a control signal for driving the pixel signal processing unit 103 to the pixel signal processing unit 103 via the vertical control line 112. Furthermore, the horizontal control line 113 of each column is connected to the horizontal control pulse generation unit 111. The horizontal control pulse generation unit 111 supplies a control signal for driving the pixel signal processing unit 103 to the pixel signal processing unit 103 via the horizontal control line 113. The vertical control pulse generation unit 110, the horizontal control pulse generation unit 111, and the signal readout circuit 115 are connected via a readout control line 116. The vertical control pulse generation unit 110 supplies a control signal for driving the horizontal control pulse generation unit 111 and the signal readout circuit 115 via the readout control line 116. This allows the vertical control pulse generation unit 110, the horizontal control pulse generation unit 111, and the signal readout circuit 115 to operate in a synchronized state. The vertical control pulse generating section 110 may generate a control signal based on an external trigger (not shown), or may generate a control signal based on an internal signal.

[0020] The data output line 114 of each row is connected to a signal readout circuit 115. The data output line 114 is a signal line for transmitting data held by the pixel signal processing unit 103. The signal readout circuit 115 acquires a plurality of data from the data output line 114 in response to a control signal supplied from the vertical control pulse generation unit 110 via a readout control line 116, and outputs the data to a signal processing unit 117.

[0021] The signal processing unit 117 is an information processing device that processes acquired image data. The signal processing unit 117 reads and executes computer-executable instructions. The signal processing unit 117 may be a computer including one or more processors and one or more memories. The signal processing unit 117 may be configured to include multiple separate computers or multiple separate processors. The signal processing unit 117 may also be configured with one or more processing circuits. The processor or circuit may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA). The processor or circuit may also include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

[0022] In this embodiment, the signal processing unit 117 is disposed within the circuit board 21, but the function of the signal processing unit 117 may be disposed outside the circuit board 21. For example, the function of the signal processing unit 117 may be disposed on an external controller system. In this case, data is output from the signal readout circuit 115 to the controller system via an output interface circuit. That is, in this embodiment, the photoelectric conversion device 100 is an integrated device including the signal processing unit 117, but it may also be configured as a photoelectric conversion system including a photoelectric conversion device and a controller system (information processing device) that are communicatively connected to each other.

[0023] 4 is a schematic block diagram showing an example of the configuration of one pixel of the photoelectric conversion unit 102 and pixel signal processing unit 103 according to this embodiment. Fig. 4 schematically shows a more specific example of the configuration, including the connection relationship between the photoelectric conversion unit 102 arranged on the sensor substrate 11 and the pixel signal processing unit 103 arranged on the circuit board 21.

[0024] The photoelectric conversion unit 102 has an APD 201. The pixel signal processing unit 103 has a pulse generation unit 210 and a pulse processing unit 220. The pulse generation unit 210 has a quench element 211 and a waveform shaping unit 212.

[0025] The APD 201 generates charge pairs according to incident light through photoelectric conversion. A voltage VL (first voltage) is supplied to the anode of the APD 201. A cathode of the APD 201 is connected to a first terminal of the quench element 211 and an input terminal of the waveform shaping unit 212. A voltage VH (second voltage) higher than the voltage VL supplied to the anode is supplied to the cathode of the APD 201. As a result, a reverse bias voltage is supplied to the anode and cathode of the APD 201 such that the APD 201 performs avalanche multiplication. When charges are generated by incident light in the APD 201 to which the reverse bias voltage is supplied, the charges undergo avalanche multiplication, generating an avalanche current.

[0026] The quench element 211 has a function of converting a change in avalanche current occurring in the APD 201 into a voltage signal. Furthermore, the quench element 211 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication. At this time, the quench element 211 suppresses the voltage supplied to the APD 201 to suppress avalanche multiplication (quench operation). The quench element 211 may be, for example, a resistive element.

[0027] The waveform shaping unit 212 shapes the potential change of the cathode of the APD 201 that occurs when a photon is detected, and outputs a pulse signal. The waveform shaping unit 212 uses a circuit having a waveform shaping effect, such as an inverter circuit or a buffer circuit.

[0028] The pulse processing unit 220 receives the photon detection pulses generated by the pulse generating unit 210 and counts the number of photon detection pulses. The configuration and operation of the pulse processing unit 220 will be described later with reference to FIG.

[0029] In this embodiment, a configuration example is shown in which a SPAD (Single Photon Avalanche Diode) using the APD 201 is used as the photoelectric conversion element, but the photoelectric conversion element is not limited to this. Any photoelectric conversion element configured to be able to read information on the amount of incident light obtained by photoelectric conversion during the accumulation period can be applied to the photoelectric conversion device 100 of this embodiment.

[0030] In addition, in this embodiment, all of the multiple pixels 101 in the pixel region 12 are read out, but this is not limited to this. For example, a configuration in which only a portion of the pixel region 12 is read out may be adopted. Alternatively, a configuration in which only a predetermined number of bits of a digital signal acquired in each of the multiple pixels 101 is read out may be adopted. In this embodiment, each of the multiple pixels 101 has a pixel memory and is capable of reading out using a global shutter method, but this is not limited to this. For example, reading out may be performed using a rolling shutter method.

[0031] 5(a), 5(b), and 5(c) are diagrams illustrating the operation of the APD 201 according to this embodiment. FIG. 5(a) is a diagram illustrating the APD 201, the quench element 211, and the waveform shaping unit 212 extracted from FIG. 4. As shown in FIG. 5(a), the connection node between the input terminals of the APD 201, the quench element 211, and the waveform shaping unit 212 is referred to as node A. Also, as shown in FIG. 5(a), the output side of the waveform shaping unit 212 is referred to as node B. In the explanations of FIGS. 5(a), 5(b), and 5(c), it is assumed that the waveform shaping unit 212 is an inverter circuit.

[0032] FIG. 5(b) is a graph showing the time change in the potential of node A in FIG. 5(a). FIG. 5(c) is a graph showing the time change in the potential of node B in FIG. 5(a). From time t0 to time t1, a voltage of VH-VL is applied to the APD 201 in FIG. 5(a). When a photon is incident on the APD 201 at time t1, avalanche multiplication occurs in the APD 201. This causes an avalanche current to flow through the quench element 211, and the potential of node A drops. Thereafter, the amount of potential drop increases further, and the voltage applied to the APD 201 gradually decreases. Then, at time t2, avalanche multiplication in the APD 201 stops. As a result, the voltage level of node A does not drop below a certain value. Then, from time t2 to time t3, a current flows through node A from the node at voltage VH to compensate for the voltage drop, and at time t3, node A settles to its original potential.

[0033] In the above process, the potential of node B becomes high during the period when the potential of node A is lower than a certain threshold. In this way, the waveform of the drop in potential of node A caused by the incidence of a photon is shaped by waveform shaping unit 212 and output as a pulse to node B.

[0034] Fig. 6 is a block diagram showing an example of the configuration of the pulse processing unit 220 according to this embodiment. The pulse processing unit 220 has a pixel counter 221 and a pixel memory 222 (memory). Fig. 6 shows an example in which the pixel counter 221 is capable of outputting an 8-bit digital signal and the pixel memory 222 has an 8-bit storage capacity, but the number of bits is not limited to 8.

[0035] The pixel counter 221 counts the number of photon detection pulses output from the pulse generator 210, starting from the accumulation start time of one frame period. A memory transfer signal is input to the pixel memory 222 via a transfer signal line 118. The pixel memory 222 latches the count value held by the pixel counter 221 when the memory transfer signal is asserted. The memory transfer signal may be output from the vertical control pulse generator 110, the horizontal control pulse generator 111, or a control circuit (not shown). Control of the pixel counter 221 and pixel memory 222, such as starting and stopping operations and resetting held values, is performed for each row by a control signal supplied from the vertical control pulse generator 110 via a vertical control line 112.

[0036] The values ​​held in the pixel memories 222 are output to the data output lines 114 at predetermined timings during accumulation during one frame period and after accumulation is completed. The data output lines 114 are 8-bit parallel buses extending in a first direction and capable of transmitting 8-bit digital signals. A plurality of pixel memories 222 arranged in the horizontal direction are connected to the data output lines 114.

[0037] The vertical control pulse generation unit 110 and the horizontal control pulse generation unit 111 output a selection signal to the pixel memory 222 via a vertical control line 112 and a horizontal control line 113, respectively, to select the pixel signal processing unit 103 that will output the count value. In response to this selection signal, the pixel memory 222 is electrically connected or disconnected from the data output line 114. As a result, the pixel memory 222 of the selected pixel signal processing unit 103 outputs the value it holds to the data output line 114.

[0038] 7 is a timing chart illustrating the operation of the pulse processing unit 220 according to this embodiment, which mainly shows the operation in frame period F1 among three consecutive frame periods F0, F1, and F2.

[0039] In this embodiment, one frame period F1 includes four types of accumulation periods F1_1, F1_2, F1_3, and F1_4. The accumulation period F1_1 is the period from time T0, which is the start time of the frame period F1, to time T1. If the length of the frame period F1 is T, the length of the accumulation period F1_1 is T / 4. The accumulation period F1_2 is the period from time T0 to time T2. The length of the accumulation period F1_2 is 2T / 4. The accumulation period F1_3 is the period from time T0 to time T3. The length of the accumulation period F1_3 is 3T / 4. The accumulation period F1_4 is the period from time T0 to time T4, which is the end time of the frame period F1. The length of the accumulation period F1_4 is T. In other words, the accumulation period F1_4 is the same period as the frame period F1. In this way, the accumulation period F1_4 (second accumulation period) includes the accumulation period F1_1 (first accumulation period) and the period after that (from time T1 to time T4).

[0040] The "count value" in FIG. 7 indicates the count value held in the pixel counter 221. The count value increases over time within the frame period F1 as the pixel counter 221 counts the number of photon detection pulses. The count value is reset when times T0 and T4, at which the frame periods change, have passed. However, the count value is not reset during the frame period F1. That is, the count value is not reset at the end of the accumulation periods F1_1, F1_2, and F1_3 (times T1, T2, and T3). The count values ​​at times T1, T2, T3, T4, and T4 are C1, C2, C3, and C4, respectively. The count value increases monotonically during these periods. Therefore, the count value C2 is equal to or greater than the count value C1, the count value C3 is equal to or greater than the count value C2, and the count value C4 is equal to or greater than the count value C3.

[0041] 7 indicates the count values ​​temporarily held in the pixel memory 222. The signals temporarily stored in the pixel memory 222 are sequentially output to the signal processing unit 117 via the data output line 114 and the signal readout circuit 115. The "output" in FIG. 6 indicates the accumulation period of the signals output from the pixel memory 222.

[0042] In this way, the signals accumulated in the accumulation period F1_1 are read out during the period from time T1 to time T2. The signal processing unit 117 can start processing at this point. Similarly, the signals accumulated in the accumulation period F1_2 are read out during the period from time T2 to time T3, and the signals accumulated in the accumulation period F1_3 are read out during the period from time T3 to time T4. The signals accumulated in the accumulation period F1_4 are read out during the period after time T4. The signal processing unit 117 can acquire signals for one frame period F1 multiple times and perform processing sequentially. Note that the signal processing unit 117 does not need to use all of the signals accumulated in the accumulation period F1_1, the signals accumulated in the accumulation period F1_2, the signals accumulated in the accumulation period F1_3, and the signals accumulated in the accumulation period F1_4. As will be described later, in this embodiment, the signals accumulated in the accumulation period F1_1 and the signals accumulated in the accumulation period F1_4 are used for processing.

[0043] Fig. 8 is a diagram showing an example of an image acquired by the photoelectric conversion device according to this embodiment. Fig. 8 shows images based on signals accumulated in each of the above-mentioned accumulation periods F1_1, F1_2, F1_3, and F1_4. The shading in Fig. 8 indicates pixel values, with areas closer to white having larger pixel values. As shown in Fig. 8, the longer the accumulation period, the larger the pixel values ​​of the subjects (vehicles and trees) that are obtained.

[0044] 9 is a block diagram showing an example of the configuration of the signal processing unit 117 according to this embodiment. The signal processing unit 117 (information processing device) includes buffers 171 and 172, an abnormal pixel detection unit 173, an abnormal pixel information storage unit 174, an abnormal pixel correction unit 175, and a corrected image storage unit 176.

[0045] The buffer 171 holds an image based on signals accumulated during an accumulation period F1_1 among the multiple images output from the pixel signal processing unit 103 of the first circuit area 22 via the signal readout circuit 115. The buffer 172 holds an image based on signals accumulated during an accumulation period F1_4 among the multiple images output from the pixel signal processing unit 103 of the first circuit area 22 via the signal readout circuit 115. In other words, the buffers 171 and 172 are frame buffers having a storage capacity capable of holding one image.

[0046] The abnormal pixel detection unit 173 detects abnormal pixels from an image based on signals accumulated in the buffer 171 during the accumulation period F1_1. This detection of abnormal pixels can be a process of detecting whether the pixel in question does not satisfy a predetermined standard. Here, the predetermined standard is determined taking into account the range within which the pixel in question is treated as acceptable, and is appropriately set taking into account, for example, manufacturing errors of the photoelectric conversion device. The abnormal pixel detection unit 173 then outputs abnormal pixel information indicating the detection result of the abnormal pixel to the abnormal pixel information storage unit 174. The abnormal pixel information can include pixel information indicating that the pixel in question does not satisfy the predetermined standard. The abnormal pixel detection unit 173 is an example of a detection unit in the signal processing unit 117.

[0047] Detection of abnormal pixels in this embodiment will be described. An abnormal pixel is a pixel that outputs a pixel value that is not in accordance with incident light but is unique compared to neighboring pixels. Abnormal pixels can include so-called white defects and black defects. A white defect is an abnormal pixel that has a pixel value that is significantly larger than neighboring pixels, while a black defect is an abnormal pixel that has a pixel value that is significantly smaller than neighboring pixels. As such, the pixel value of an abnormal pixel is often significantly different from the pixel values ​​of its surrounding pixels. Therefore, by comparing the pixel of interest with the pixel values ​​of its surroundings (e.g., eight neighboring pixels), it is possible to determine whether the pixel of interest is an abnormal pixel based on whether the pixel value of the pixel of interest is significantly different.

[0048] FIG. 10 is a schematic diagram showing an example of a pixel of interest and peripheral pixels according to this embodiment. FIG. 10 illustrates three rows and three columns of a plurality of pixels 101, along with their pixel values. FIG. 10 shows a pixel of interest 101a located at the center and eight peripheral pixels 101b located around the pixel of interest 101a. The numbers written in the circles representing each pixel indicate the pixel value. As shown in FIG. 10, the pixel value of the pixel of interest 101a is significantly larger than the pixel value of the peripheral pixels 101b. Therefore, an abnormal pixel can be detected by, for example, calculating the difference between the pixel value of the pixel of interest 101a and the average pixel value of the peripheral pixels 101b, and determining that the pixel of interest 101a is an abnormal pixel if the difference is equal to or greater than a predetermined threshold.

[0049] Note that, in this embodiment, the image of the accumulation period F1_1 used for detecting abnormal pixels is an image with smaller pixel values ​​than the image of the accumulation period F1_4 for image generation, as shown in FIG. 8 . Therefore, a value smaller than the difference between the pixel value of the target pixel 101a assumed in the image of the accumulation period F1_4 for image generation and the average pixel value of the peripheral pixels 101b may be used as the threshold for determination in the abnormal pixel detection unit 173. The length of the accumulation period F1_1 is ¼ of the length of the accumulation period F1_4. Therefore, it is desirable to set the threshold to a relatively small value, approximately ¼ of the difference between the pixel value of the target pixel 101a assumed in the image of the accumulation period F1_4 for image generation and the average pixel value of the peripheral pixels 101b.

[0050] The abnormal pixel information output from the abnormal pixel detection unit 173 is stored in the abnormal pixel information storage unit 174. The abnormal pixel information may include information indicating the coordinates of the abnormal pixels among the multiple pixels 101. The abnormal pixel information may be map data in the form of a matrix having the number of rows and columns of the pixels 101. Each element of this matrix is ​​a 1-bit value indicating whether the pixel is normal or abnormal. Alternatively, the abnormal pixel information may be table data indicating the XY coordinate information (row number and column number) of the abnormal pixel.

[0051] The abnormal pixel correction unit 175 corrects abnormal pixels in the image for the accumulation period F1_4 stored in the buffer 172 based on the abnormal pixel information stored in the abnormal pixel information storage unit 174. The method of correcting the abnormal pixels may involve, for example, identifying the coordinates of the abnormal pixel from the abnormal pixel information and replacing the value of the abnormal pixel at those coordinates with an interpolated value calculated from the values ​​of multiple surrounding pixels. The image in which the abnormal pixels have been corrected by the abnormal pixel correction unit 175 is output to the corrected image storage unit 176. The corrected image storage unit 176 temporarily stores the corrected pixels. The corrected pixels may be output from the corrected image storage unit 176 to a downstream circuit. The abnormal pixel correction unit 175 is an example of a correction unit in the signal processing unit 117.

[0052] Fig. 11 is a timing diagram illustrating the operation of the signal processing unit 117 according to this embodiment. Fig. 11 shows the timing of signal input to the signal processing unit 117, signal processing in the signal processing unit 117, etc. The notations for frame period, "pixel memory," and "output" are generally the same as in Fig. 7, and therefore will not be described here.

[0053] "Detection processing" in FIG. 11 indicates the timing at which the abnormal pixel detection unit 173 performs the abnormal pixel detection processing. "Information" in FIG. 11 indicates the information stored in the abnormal pixel information storage unit 174 and the timing at which it is stored. "Correction processing" in FIG. 11 indicates the timing at which the abnormal pixel correction unit 175 performs the abnormal pixel correction processing. Note that the configuration of the signal processing unit 117 in FIG. 9 does not include a buffer for storing images from accumulation periods F1_2 and F1_3, and processing using these images is not performed. Therefore, the "Pixel memory" and "Output" columns in FIG. 11 do not include operation timings for accumulation periods F1_2 and F1_3 and count values ​​C2 and C3.

[0054] After the output of the image (first signal) of the accumulation period F1_1 starts at time T1, the abnormal pixel detection unit 173 starts the abnormal pixel detection process ("detection 1" in FIG. 11). After a predetermined time has elapsed since the start of the abnormal pixel detection process, the abnormal pixel information holding unit 174 updates the abnormal pixel information ("abnormal pixel information 1" in FIG. 11). After the output of the image (second signal) of the accumulation period F1_4 starts at time T4, the abnormal pixel correction unit 175 starts the abnormal pixel correction process ("correction 1" in FIG. 11).

[0055] As described above, because the period during which the abnormal pixel detection process is performed overlaps with the accumulation period F1_4, the abnormal pixel detection process and accumulation are performed in parallel. Therefore, the processing time for the abnormal pixel detection process does not affect the length of the frame period F1, and the abnormal pixel detection process can be performed without affecting the overall processing time. Furthermore, because the abnormal pixel detection process can be completed early, the abnormal pixel correction process can be started immediately after the end of the accumulation period F1_4. Therefore, this embodiment provides a photoelectric conversion device, an information processing device, a photoelectric conversion device control method, and an information processing method that further reduce processing delays caused by detecting and correcting pixels outside the predetermined standard range.

[0056] Furthermore, since the accumulation period F1_1 for the abnormal pixel detection process is shorter than the accumulation period F1_4 for image generation, signal saturation is less likely to occur, thereby reducing the possibility of missing an abnormal pixel due to signal saturation.

[0057] In this embodiment, the upper limit of the time that can be allocated to the abnormal pixel detection process is 3T / 4, which is relatively long. In Fig. 11, the abnormal pixel detection process is performed in a time of about T / 4, but it is also possible to improve the accuracy of abnormal pixel detection by performing additional processes to make the total processing time longer than T / 4.

[0058] An example of additional processing will be described. When the temperature of the sensor substrate 11 is high, the number of abnormal pixels increases, and the abnormal pixels may be adjacent to each other. FIG. 12 is a schematic diagram showing an example of a pixel of interest and peripheral pixels according to a modified example of this embodiment. FIG. 12 shows an example of pixel values ​​when the temperature of the sensor substrate 11 is high, and peripheral pixel 101c, which is one of eight peripheral pixels arranged around pixel of interest 101a, is an abnormal pixel. In such a case, the difference between the pixel value of pixel of interest 101a and the average value of the pixel values ​​of peripheral pixels 101b and 101c becomes small, which may reduce the accuracy of determining abnormal pixels.

[0059] In such cases, the accuracy of determining whether an abnormal pixel is an isolated point can be improved by adding a process for determining whether the abnormal pixel is an isolated point. This process will be described below. Adjacent pixels are determined to be connected when their pixel values ​​are close (the difference between the pixel values ​​is within a predetermined range). The connected pixels are then treated as a single pixel group, and the number of pixels (connection count) in each pixel group is counted. In the example of FIG. 12, there are seven pixels with pixel values ​​between 20 and 40 (connection count of 7), while there are two pixels with pixel values ​​between 80 and 100 (connection count of 2). After the connection process, pixel groups with a connection count of N (N is a positive integer) or less are determined to be isolated points. For example, when N=2, the two pixels with pixel values ​​between 80 and 100 are determined to be isolated points and both are abnormal pixels. If the temperature of the sensor substrate 11 is higher and the density of abnormal pixels is higher, the connection count threshold N may need to be set to 3 or more. In this way, when the connection number threshold N is large, the range of pixels whose pixel values ​​are referenced during processing becomes wider, and the processing load, such as memory access and the amount of calculation, increases. However, even when the processing load increases and processing takes a long time, in this embodiment, the abnormal pixel detection process can be started and completed early, thereby reducing processing delays.

[0060] [Second embodiment] The first embodiment describes a method in which the signal processing unit 117 acquires one image during accumulation in the frame period F1 and performs abnormal pixel detection processing. In contrast, the present embodiment describes a method in which the signal processing unit 117 acquires multiple images during accumulation in the frame period F1 and performs abnormal pixel detection processing multiple times. In the present embodiment, descriptions of elements common to the first embodiment may be omitted or simplified.

[0061] Fig. 13 is a block diagram showing an example of the configuration of the signal processing unit 117 according to this embodiment. The signal processing unit 117 has four buffers 171a, 171b, 171c, and 172. That is, in this embodiment, the buffer 171 in Fig. 9 is replaced with three buffers 171a, 171b, and 171c. The other configuration in Fig. 13 is the same as that in Fig. 9.

[0062] The buffer 171a holds an image based on signals accumulated during the accumulation period F1_1 among a plurality of images output from the pixel signal processing unit 103 of the first circuit area 22 via the signal readout circuit 115. Similarly, the buffers 171b, 171c, and 172 hold images based on signals accumulated during the accumulation periods F1_2, F1_3, and F1_4, respectively.

[0063] The abnormal pixel detection unit 173 acquires multiple images based on signals accumulated during accumulation periods F1_1, F1_2, and F1_3 from the buffers 171a, 171b, and 171c. The abnormal pixel detection unit 173 detects abnormal pixels from each of these images. The lengths of the accumulation periods F1_1, F1_2, and F1_3 are 1 / 4, 2 / 4, and 3 / 4 of the length of the accumulation period F1_4, respectively. Therefore, the threshold value used for detecting abnormal pixels may be different for each of these images. These threshold values ​​may be set to, for example, 1 / 4, 2 / 4, and 3 / 4, respectively, of the difference between the pixel value of the target pixel 101a and the average pixel value of the surrounding pixels 101b, which are assumed in the image generated during the accumulation period F1_4.

[0064] The abnormal pixel detection unit 173 integrates the abnormal pixel detection results from each of the multiple images to generate abnormal pixel information. The abnormal pixel information output from the abnormal pixel detection unit 173 is stored in the abnormal pixel information storage unit 174. The abnormal pixel information may be map data in the form of a matrix having the number of rows and columns of the pixels 101. Each element of this matrix is ​​a 1-bit value indicating whether the pixel is normal or abnormal. Alternatively, the abnormal pixel information may be table data indicating the XY coordinate information (row number and column number) of the abnormal pixel.

[0065] We will now explain examples of methods for integrating the detection results of abnormal pixels from multiple images. As a first example, there is a method for determining abnormal pixels based on the logical product of the detection results. In this method, if an abnormal pixel is detected at the same coordinate in all of the multiple images, the pixel at that coordinate is determined to be an abnormal pixel.

[0066] When the abnormal pixel information is map data with a 1-bit value, the logical product can be obtained as follows. First, if an abnormal pixel is detected in the image of the accumulation period F1_1, a value of "1" indicating an abnormality is written to the corresponding coordinate in the map data stored in the abnormal pixel information storage unit 174. Furthermore, a value of "0" indicating normality is written to other coordinates. If an abnormal pixel is detected in the images of the accumulation periods F1_2 and F1_3, if a "1" has been written to the corresponding coordinate in the map data, it is overwritten with a "1"; otherwise, a "0" is written. By writing values ​​to the map data in this manner, map data is generated in which only the coordinates where three detection results indicating an abnormality are obtained are set to "1." By obtaining the logical product of multiple detection results in this way and determining whether an abnormal pixel is present, the possibility of erroneous detection of an abnormal pixel can be reduced.

[0067] A second example of a method for integrating abnormal pixel detection results is a method for determining abnormal pixels based on the logical sum of the detection results. In this method, if an abnormal pixel is detected in at least one of multiple images, the pixel at that coordinate is determined to be an abnormal pixel. If the abnormal pixel information is map data, when an abnormal pixel is detected in the images during accumulation periods F1_1, F1_2, and F1_3, a "1" is written to that coordinate regardless of the original value. By writing values ​​to the map data in this manner, map data is generated in which the coordinates where an abnormal detection result indicating an abnormality was obtained at least once out of three times are set to "1." By determining abnormal pixels by obtaining the logical sum of multiple detection results in this way, the possibility of missing an abnormal pixel can be reduced. It is also possible to skip detection processing for coordinates where an abnormal pixel has once been detected, which can reduce the computational load and processing time.

[0068] A third example of a method for integrating the detection results of abnormal pixels is a method for determining abnormal pixels based on the majority rule of the detection results. In this method, if an abnormal pixel is detected in at least two (i.e., the majority) of multiple images, the pixel at that coordinate is determined to be an abnormal pixel. This allows for a good balance between the possibility of falsely detecting an abnormal pixel and the possibility of missing an abnormal pixel.

[0069] Fig. 14 is a timing diagram illustrating the operation of the signal processing unit 117 according to this embodiment. Fig. 14 shows the timing of signal input to the signal processing unit 117, signal processing in the signal processing unit 117, etc. The notation of each item in Fig. 14 is generally the same as in Fig. 11, and therefore description thereof will be omitted.

[0070] At time T1, after the output of an image for the accumulation period F1_1 is started, the abnormal pixel detection unit 173 starts the abnormal pixel detection process ("Detection 1" in FIG. 14). After a predetermined time has elapsed since the start of the abnormal pixel detection process, the abnormal pixel information storage unit 174 updates the abnormal pixel information ("Information 1" in FIG. 14). At time T2, after the output of an image for the accumulation period F1_2 is started, the abnormal pixel detection unit 173 starts the abnormal pixel detection process ("Detection 2" in FIG. 14). After a predetermined time has elapsed since the start of the abnormal pixel detection process, the abnormal pixel information storage unit 174 updates the abnormal pixel information ("Information 2" in FIG. 14). At time T3, after the output of an image for the accumulation period F1_3 is started, the abnormal pixel detection unit 173 starts the abnormal pixel detection process ("Detection 3" in FIG. 14). After a predetermined time has elapsed since the start of the abnormal pixel detection process, the abnormal pixel information storage unit 174 updates the abnormal pixel information ("Information 3" in FIG. 14). After the output of the image for the accumulation period F1_4 starts at time T4, the abnormal pixel corrector 175 starts the abnormal pixel correction process ("correction" in FIG. 14).

[0071] As described above, the abnormal pixel detection process is performed multiple times at least partially in parallel with the accumulation operation during the accumulation period F1_4. Therefore, even if the abnormal pixel detection process is performed multiple times, the processing time does not significantly affect the frame period F1, and the abnormal pixel detection process can be performed without significantly affecting the overall processing time. Furthermore, by integrating the detection results from multiple times, the accuracy of abnormal pixel detection can be improved.

[0072] Therefore, according to this embodiment, a photoelectric conversion device, an information processing device, a method for controlling a photoelectric conversion device, and an information processing method are provided that can obtain the same effects as those of Embodiment 1. Furthermore, according to this embodiment, the accuracy of detecting abnormal pixels can be improved by integrating multiple detection results.

[0073] [Third embodiment] In the first and second embodiments, the signal processing unit 117 has a buffer for storing images in the middle of accumulation, and an abnormal pixel information storage unit 174 for storing abnormal pixel information. In contrast, in this embodiment, an example of a configuration will be described in which abnormal pixels are detected and corrected in real time without using these storage units. In this embodiment, descriptions of elements common to the first and second embodiments may be omitted or simplified.

[0074] FIG. 15 is a block diagram showing an example of the configuration of the signal processing unit 117 according to this embodiment. Compared to the configuration of the signal processing unit 117 in FIG. 9, FIG. 15 does not include buffers 171 and 172 and an abnormal pixel information storage unit 174. That is, in this embodiment, a frame buffer with a storage capacity sufficient to store one frame image is not provided. Data input to the signal processing unit 117 via the signal readout circuit 115 is input to the abnormal pixel detection unit 173 and the abnormal pixel correction unit 175 via multiple data paths 177 and 178 in raster scan order. The abnormal pixel detection unit 173 and the abnormal pixel correction unit 175 each include a hardware calculator that processes the input image at a predetermined clock cycle and a delay buffer that stores data for several lines of one image. The hardware calculator and delay buffer sequentially process and output images while absorbing differences in processing time. In this way, the signal processing unit 117 is configured to generate abnormal pixel information by sequentially processing multiple signals that make up one frame image without simultaneously temporarily storing them when acquiring signals from the signal readout circuit 115.

[0075] Data based on signals accumulated during accumulation period F1_1 among the multiple images output from the pixel signal processing unit 103 of the first circuit area 22 via the signal readout circuit 115 passes through a data path 177 and is input to the abnormal pixel detection unit 173. Data based on signals accumulated during accumulation period F1_4 among the multiple images output from the pixel signal processing unit 103 of the first circuit area 22 via the signal readout circuit 115 passes through a data path 178 and is input to the abnormal pixel correction unit 175. The abnormal pixel detection unit 173 detects abnormal pixels based on the input image of accumulation period F1_1 and outputs abnormal pixel information to the abnormal pixel correction unit 175. The abnormal pixel correction unit 175 corrects abnormal pixels included in the input image of accumulation period F1_4 in real time based on the abnormal pixel information, and outputs the corrected image to the corrected image holding unit 176.

[0076] Fig. 16 is a timing diagram illustrating the operation of the signal processing unit 117 according to this embodiment. Fig. 16 shows the timing of signal input to the signal processing unit 117, signal processing in the signal processing unit 117, etc. The notation of each item in Fig. 16 is generally the same as in Fig. 11, and therefore description thereof will be omitted.

[0077] At time T1, the image of the accumulation period F1_1 is stored in the pixel memory 222. In this embodiment, since the signal processing unit 117 does not have a frame buffer capable of storing one frame image, the image of the accumulation period F1_1 is not output and remains stored in the pixel memory 222 between time T1 and time T3. At time T3, after output of the image of the accumulation period F1_1 starts, the abnormal pixel detection unit 173 starts the abnormal pixel detection process ("Detection 1" in FIG. 16). After a predetermined time has elapsed after the start of the abnormal pixel detection process, the abnormal pixel detection unit 173 updates the abnormal pixel information to be output to the abnormal pixel correction unit 175 ("Information 1" in FIG. 16). After a predetermined time has elapsed, the abnormal pixel information 1 is updated. At time T4, after output of the image of the accumulation period F1_4 starts, the abnormal pixel correction unit 175 starts the abnormal pixel correction process ("Correction" in FIG. 16) using the abnormal pixel information output from the abnormal pixel detection unit 173.

[0078] According to this embodiment, a photoelectric conversion device, an information processing device, a photoelectric conversion device control method, and an information processing method are provided that can obtain the same effects as those of Embodiment 1. Furthermore, according to this embodiment, a frame buffer that can hold frame images is not provided, and the abnormal pixel detection process and the abnormal pixel correction process are performed in real time, so the storage capacity of the memory used in the signal processing unit 117 can be reduced.

[0079] [Fourth embodiment] In the first to third embodiments, an abnormal pixel is detected and corrected each time an image is input to the signal processing unit 117. In contrast, in this embodiment, an example of a configuration is described in which an abnormal pixel is detected when a predetermined change in situation occurs and abnormal pixel information is updated. In this embodiment, the description of elements common to any of the first to third embodiments may be omitted or simplified.

[0080] Fig. 17 is a block diagram showing an example of the configuration of the signal processing unit 117 according to this embodiment. The signal processing unit 117 in Fig. 17 further includes a temperature detection unit 181, a time detection unit 182, and a control unit 183 in addition to the configuration of the signal processing unit 117 in Fig. 9. The signal processing unit 117 in Fig. 17 also includes a first information storage unit 184 and a second information storage unit 185. The first information storage unit 184 and the second information storage unit 185 correspond to the abnormal pixel information storage unit 174 in Fig. 9.

[0081] In this embodiment, abnormal pixel information is stored in both nonvolatile memory and volatile memory. The first information storage unit 184 includes a nonvolatile memory such as a flash memory. The first information storage unit 184 pre-stores abnormal pixel information detected during inspection before shipping from the factory or abnormal pixel information acquired during past operations. The second information storage unit 185 includes a volatile memory such as an SRAM or a DRAM. The second information storage unit 185 temporarily stores abnormal pixel information acquired through the abnormal pixel detection process. The abnormal pixel information stored in the second information storage unit 185 is updated as appropriate when the abnormal pixel detection process is performed. Furthermore, the abnormal pixel information stored in the second information storage unit 185 is lost when the photoelectric conversion device 100 is powered off.

[0082] The temperature detection unit 181 acquires temperature information from a temperature sensor disposed on the sensor substrate 11 and outputs the information to the control unit 183. This enables the control unit 183 to detect temperature changes in the pixels 101.

[0083] The time detection unit 182 acquires time information corresponding to the accumulated operating time and outputs it to the control unit 183. This time information may be acquired, for example, based on the number of avalanche multiplications detected by a pulse detection sensor disposed in the first circuit area 22, or may be acquired based on the operating time of the photoelectric conversion device 100. This allows the control unit 183 to detect the amount of change in the accumulated operating time.

[0084] The control unit 183 controls the abnormal pixel detection unit 173, the first information storage unit 184, and the second information storage unit 185 based on the temperature information acquired by the temperature detection unit 181 and the time information acquired by the time detection unit 182.

[0085] The abnormal pixel correction unit 175 acquires the abnormal pixel information stored in the second information storage unit 185. Based on the abnormal pixel information, the abnormal pixel correction unit 175 corrects the abnormal pixels of the image output from the buffer 172 and outputs the corrected image to the corrected image storage unit 176.

[0086] Fig. 18 is a flowchart illustrating the operation of the photoelectric conversion device 100 according to this embodiment. Fig. 18 shows the processing procedure from when the photoelectric conversion device 100 is started up until when the operation is terminated.

[0087] In step S01 immediately after starting up the photoelectric conversion device 100, the control unit 183 controls the loading of abnormal pixel information from the first information storage unit 184 including a nonvolatile memory to the second information storage unit 185 including a volatile memory. This enables the abnormal pixel correction unit 175 to acquire the abnormal pixel information.

[0088] In step S02, the control unit 183 acquires time information from the time detection unit 182 and determines whether the amount of change in the cumulative operating time of the first circuit area 22 exceeds a predetermined threshold. If the amount of change in the cumulative operating time exceeds the threshold (YES in step S02), the process proceeds to step S04. If the amount of change in the cumulative operating time does not exceed the threshold (NO in step S02), the process proceeds to step S03.

[0089] In step S04, the control unit 183 outputs a control signal to instruct the abnormal pixel detection unit 173 to perform abnormal pixel detection processing. The abnormal pixel detection unit 173 performs the same abnormal pixel detection processing as described in the first embodiment, and outputs abnormal pixel information to the second information storage unit 185.

[0090] In step S05, the second information storage unit 185 updates the abnormal pixel information stored in the volatile memory. Then, in step S06, the first information storage unit 184 acquires the abnormal pixel information stored in the volatile memory of the second information storage unit 185 and updates the abnormal pixel information stored in the nonvolatile memory. Then, the process proceeds to step S09.

[0091] In step S03, control unit 183 acquires temperature information from temperature detection unit 181 and determines whether the amount of change in temperature exceeds a predetermined threshold. If the amount of change in temperature exceeds the threshold (YES in step S03), the process proceeds to step S07. If the amount of change in temperature does not exceed the threshold (NO in step S03), the process proceeds to step S09.

[0092] In step S07, the control unit 183 outputs a control signal to instruct the abnormal pixel detection unit 173 to perform abnormal pixel detection processing. The abnormal pixel detection unit 173 performs the same abnormal pixel detection processing as described in the first embodiment, and outputs abnormal pixel information to the second information storage unit 185.

[0093] In step S08, the second information storage unit 185 updates the abnormal pixel information stored in the volatile memory, after which the process proceeds to step S09.

[0094] The subsequent operations from step S09 to step S12 are generally similar to the abnormal pixel correction process described in the first embodiment. In step S09, the abnormal pixel correction unit 175 acquires abnormal pixel information stored in the volatile memory of the second information storage unit 185. In step S10, the abnormal pixel correction unit 175 acquires an image from the buffer 172. In step S11, the abnormal pixel correction unit 175 corrects abnormal pixels in the acquired image based on the abnormal pixel information and outputs the image to the corrected image storage unit 176. In step S12, the corrected image storage unit 176 outputs the corrected image to a downstream circuit.

[0095] In step S13, the control unit 183 determines whether it is time to end the imaging operation in the photoelectric conversion device 100. If it is time to end the imaging operation (YES in step S13), the processing in Fig. 18 ends. If it is not time to end the imaging operation (NO in step S13), the processing proceeds to step S02, and the same abnormal pixel detection processing and abnormal pixel correction processing are repeated.

[0096] As described above, in this embodiment, abnormal pixels are detected and updated in accordance with the amount of change in temperature or the amount of change in cumulative operating time. The effects of performing such an operation will be described.

[0097] In general, the number of abnormal pixels depends on the temperature of the sensor substrate 11 on which the pixels 101 are arranged. For example, as the temperature of the sensor substrate 11 rises, the number of abnormal pixels tends to increase compared to when the temperature is low. Therefore, if abnormal pixel information detected when the temperature is low is used to correct abnormal pixels at high temperatures, correction may be missed due to the increase in abnormal pixels caused by the rise in temperature. Therefore, in this embodiment, when the temperature detection unit 181 detects a temperature change exceeding a predetermined threshold, the abnormal pixel detection unit 173 detects abnormal pixels and updates the abnormal pixel information in the second information storage unit 185.

[0098] Furthermore, as the cumulative operating time of the photoelectric conversion device 100 increases, the number of abnormal pixels may increase due to aging or other factors. Therefore, if old abnormal pixel information is used to correct abnormal pixels in the photoelectric conversion device 100, correction may be missed due to an increase in abnormal pixels caused by aging or other factors. Therefore, in this embodiment, when the time detection unit 182 detects that a predetermined cumulative operating time has elapsed, the abnormal pixel detection unit 173 detects abnormal pixels and updates the abnormal pixel information in the first information storage unit 184 and the second information storage unit 185.

[0099] As a result, when it is effective to update the abnormal pixel information due to temperature changes or the passage of time, the abnormal pixel information is updated, and when it is not effective, the abnormal pixel detection process is omitted, thereby reducing the system load and power consumption required for abnormal pixel detection.

[0100] Note that the temperature of the sensor substrate 11 changes in a relatively short cycle depending on the operating time of the photoelectric conversion device 100 and the surrounding environment in which it is used, and therefore the abnormal pixel information may also be updated in a relatively short cycle. Therefore, it is desirable that the update of the abnormal pixel information due to temperature changes be applied only to the volatile memory of the second information holding unit 185. On the other hand, changes in the abnormal pixels due to aging and the like occur in a relatively long cycle and are irreversible. Therefore, it is desirable that the update of the abnormal pixel information due to the passage of time be applied also to the nonvolatile memory of the first information holding unit 184. This allows the same abnormal pixel information to be reused even after the photoelectric conversion device 100 is restarted.

[0101] According to this embodiment, a photoelectric conversion device, an information processing device, a method for controlling a photoelectric conversion device, and an information processing method are provided that can obtain the same effects as those of Embodiment 1. Furthermore, according to this embodiment, the system load and power consumption required for detecting abnormal pixels can be reduced.

[0102] [Fifth embodiment] The photoelectric conversion device 100 in the above-described embodiment can be applied to various devices. Examples of such devices include digital still cameras, digital camcorders, camera heads, copiers, fax machines, mobile phones, vehicle-mounted cameras, observation satellites, and surveillance cameras. Fig. 19 shows a block diagram of a digital still camera as an example of such a device. Fig. 19 shows an example in which the above-described photoelectric conversion device 100 is applied to a digital still camera.

[0103] The device 70 shown in FIG. 19 includes a barrier 706, a lens 702, an aperture 704, and an image capture device 700 (an example of the photoelectric conversion device 100). The device 70 also includes a signal processing unit (processing device) 708, a timing generating unit 720, an overall control / calculation unit 718 (control device), a memory unit 710 (storage device), a recording medium control I / F unit 716, a recording medium 714, and an external I / F unit 712. The information processing device 30 of the above-described embodiment may be included in the image capture device 700 or the signal processing unit 708. At least one of the barrier 706, the lens 702, and the aperture 704 is an optical device corresponding to the device. The barrier 706 protects the lens 702, and the lens 702 forms an optical image of a subject on the image capture device 700. The aperture 704 varies the amount of light passing through the lens 702. The image capture device 700 converts the optical image formed by the lens 702 into image data (image signals). The signal processing unit 708 performs various corrections, data compression, etc. on the imaging data output from the imaging device 700. The timing generation unit 720 outputs various timing signals to the imaging device 700 and the signal processing unit 708. The overall control / calculation unit 718 controls the entire digital still camera, and the memory unit 710 temporarily stores image data. The recording medium control I / F unit 716 is an interface for recording or reading image data to or from the recording medium 714, which is a removable recording medium such as a semiconductor memory for recording or reading imaging data. The external I / F unit 712 is an interface for communicating with an external computer, etc. Timing signals, etc., may be input from outside the device. The device 70 may further include a display device (monitor, electronic viewfinder, etc.) that displays information obtained by the photoelectric conversion device. The device includes at least a photoelectric conversion device. The device 70 also includes at least one of an optical device, a control device, a processing device, a display device, a storage device, and a mechanical device that operates based on information obtained by the photoelectric conversion device. The mechanical device is a movable part (for example, a robot arm) that operates in response to a signal from the photoelectric conversion device.

[0104] Each pixel may include a plurality of photoelectric conversion units (a first photoelectric conversion unit and a second photoelectric conversion unit). The signal processing unit 708 may be configured to process a pixel signal based on the charge generated in the first photoelectric conversion unit and a pixel signal based on the charge generated in the second photoelectric conversion unit, and acquire information about the distance from the image capturing device 700 to the subject.

[0105] [Sixth embodiment] 20(a) and 20(b) are block diagrams of devices related to an in-vehicle camera according to this embodiment. FIGS. 20(a) and 20(b) show an example in which the photoelectric conversion device 100 described above is applied to a moving body such as a vehicle. The device 80 includes an imaging device 800 (an example of the photoelectric conversion device 100 or a photoelectric conversion system) and a signal processing device (processing device) that processes signals from the imaging device 800. The device 80 includes an image processing unit 801 that performs image processing on multiple pieces of image data acquired by the imaging device 800, and a parallax calculation unit 802 that calculates parallax (phase difference of parallax images) from the multiple pieces of image data acquired by the device 80. The information processing device 30 according to the above embodiment may be included in the imaging device 800 or the image processing unit 801. The device 80 also includes a distance measurement unit 803 that calculates the distance to an object based on the calculated parallax, and a collision determination unit 804 that determines whether or not there is a possibility of a collision based on the calculated distance. Here, the parallax calculation unit 802 and the distance measurement unit 803 are an example of a distance information acquisition unit that acquires distance information to an object. That is, the distance information is information related to the parallax, the defocus amount, the distance to the object, etc. The collision determination unit 804 may determine the possibility of a collision using any of this distance information. The distance information acquisition unit may be realized by dedicated hardware or a software module. Furthermore, it may be realized by an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or a combination thereof.

[0106] The device 80 is connected to a vehicle information acquisition device 810 and can acquire vehicle information such as vehicle speed, yaw rate, and steering angle. The device 80 is also connected to a control ECU 820, which is a control device that outputs a control signal to generate a braking force for the vehicle based on the determination result of the collision determination unit 804. The device 80 is also connected to an alarm device 830 that issues an alarm to the driver based on the determination result of the collision determination unit 804. For example, if the determination result of the collision determination unit 804 indicates a high collision possibility, the control ECU 820 performs vehicle control to avoid the collision and mitigate damage by applying the brakes, releasing the accelerator, suppressing engine output, etc. The alarm device 830 warns the user by sounding an alarm, displaying alarm information on the screen of a car navigation system, etc., or vibrating the seat belt or steering wheel. The device 80 functions as a control means for controlling the operation of controlling the vehicle as described above.

[0107] In this embodiment, the device 80 captures images of the surroundings of the vehicle, for example, the front or rear. FIG. 20(b) shows the device when capturing an image of the area in front of the vehicle (image capturing range 850). A vehicle information acquisition device 810, which serves as an image capturing control means, sends an instruction to the device 80 or the image capturing device 800 to perform an image capturing operation. This configuration can further improve the accuracy of distance measurement.

[0108] Although the above describes an example of control to prevent collision with other vehicles, the present invention can also be applied to control of automatic driving by following other vehicles, control of automatic driving to prevent deviation from a lane, etc. Furthermore, the present invention is not limited to vehicles such as automobiles, but can be applied to moving objects (moving devices) such as ships, aircraft, artificial satellites, industrial robots, and consumer robots. In addition, the present invention can be applied to a wide range of devices that use object recognition or biometric recognition, such as intelligent transport systems (ITS) and surveillance systems, without being limited to moving objects.

[0109] [Modified embodiment] The present invention is not limited to the above-described embodiments and can be modified in various ways. For example, an example in which part of the configuration of one embodiment is added to another embodiment, or an example in which part of the configuration of one embodiment is replaced with part of the configuration of another embodiment, is also an embodiment of the present invention.

[0110] The disclosure of this specification includes the complement of the concepts described in this specification. In other words, if this specification states, for example, that "A is B" (A=B), then this specification is deemed to disclose or suggest that "A is not B" even if the statement that "A is not B" (A≠B) is omitted. This is because when "A is B," it is assumed that the case where "A is not B" is taken into consideration.

[0111] The disclosure of this specification includes the following configurations or methods. (Configuration 1) a pixel that generates a signal according to the amount of incident light; a memory for storing the signal; a processing unit that performs signal processing based on the signal stored in the memory; and the pixel generates the signal during a second accumulation period that includes a first accumulation period and a period after the first accumulation period; the processing unit acquires, from the memory, a first signal corresponding to the amount of incident light during the first accumulation period, during a period from the end of the first accumulation period to the end of the second accumulation period; When the processing unit detects, using the first signal, that a pixel does not satisfy a predetermined criterion, it generates pixel information indicating that the pixel does not satisfy the predetermined criterion; the processing unit acquires, after the second accumulation period has ended, from the memory a second signal corresponding to the amount of incident light during the second accumulation period; The processing unit corrects the second signal based on the pixel information. A photoelectric conversion device characterized by: (Configuration 2) A period during which the processing unit detects whether the pixel does not satisfy a predetermined criterion and the second accumulation period at least partially overlap. 2. The photoelectric conversion device according to configuration 1, (Configuration 3) the pixel includes an avalanche photodiode; The memory stores a count value based on photons incident on the avalanche photodiode as the signal. 3. The photoelectric conversion device according to configuration 1 or 2. (Configuration 4) The count value is not reset during the period from the start of the second accumulation period to the end of the second accumulation period. 4. The photoelectric conversion device according to configuration 3. (Configuration 5) A plurality of the pixels are included, The pixel information includes information indicating coordinates of a pixel that is detected as not satisfying the predetermined criterion among the plurality of pixels. 5. The photoelectric conversion device according to any one of configurations 1 to 4. (Configuration 6) The processing unit corrects the second signal corresponding to the pixel at the coordinate. 6. The photoelectric conversion device according to configuration 5. (Configuration 7) the second accumulation period includes a plurality of the first accumulation periods, the processing unit acquires a plurality of the first signals during a period from the start of the second accumulation period to the end of the second accumulation period; The processing unit generates the pixel information based on a plurality of the first signals. 7. The photoelectric conversion device according to any one of configurations 1 to 6. (Configuration 8) The processing unit generates a detection result of a 1-bit value based on each of the plurality of first signals, and generates the pixel information based on a logical product of the plurality of detection results. 8. The photoelectric conversion device according to configuration 7. (Configuration 9) The processing unit generates a detection result of a 1-bit value based on each of the plurality of first signals, and generates the pixel information based on a logical sum of the plurality of detection results. 8. The photoelectric conversion device according to configuration 7. (Configuration 10) The processing unit generates a detection result of a 1-bit value based on each of the plurality of first signals, and generates the pixel information based on a majority logic of the plurality of detection results. 8. The photoelectric conversion device according to configuration 7. (Configuration 11) The processing unit generates the pixel information further based on a plurality of threshold values ​​different from each other and set corresponding to the plurality of first signals, respectively. 11. The photoelectric conversion device according to any one of configurations 7 to 10. (Configuration 12) A plurality of the pixels are included, The processing unit generates the pixel information by sequentially processing the first signals output from the pixels without simultaneously temporarily storing them. 12. The photoelectric conversion device according to any one of configurations 1 to 11. (Configuration 13) a storage unit for storing the pixel information; a temperature detection unit that detects the temperature of the pixel; and The holding unit updates the pixel information based on the amount of change in the temperature. 13. The photoelectric conversion device according to any one of configurations 1 to 12. (Configuration 14) a storage unit for storing the pixel information; a time detection unit that detects an accumulated operating time of the photoelectric conversion device; and The holding unit updates the pixel information based on the amount of change in the cumulative operating time. 14. The photoelectric conversion device according to any one of configurations 1 to 13. (Configuration 15) the storage unit includes a nonvolatile memory; The storage unit updates the pixel information stored in the nonvolatile memory. 15. The photoelectric conversion device according to configuration 14. (Configuration 16) The photoelectric conversion device according to any one of configurations 1 to 15, an optical device corresponding to the photoelectric conversion device; a control device that controls the photoelectric conversion device; a processing device that processes a signal output from the photoelectric conversion device; a display device that displays information obtained by the photoelectric conversion device; a storage device that stores information obtained by the photoelectric conversion device; and and a mechanical device that operates based on information obtained by the photoelectric conversion device. (Configuration 17) 17. The device according to claim 16, wherein the processing device acquires distance information from the photoelectric conversion device to the subject. (Configuration 18) a detection unit that, when detecting that a pixel does not satisfy a predetermined standard using a first signal corresponding to the amount of light incident on the pixel during a first accumulation period, generates pixel information indicating that the pixel does not satisfy the predetermined standard; a correction unit that corrects a second signal corresponding to the amount of light incident on the pixel during a second accumulation period that includes the first accumulation period and a period after the first accumulation period, based on the pixel information; and the detection unit acquires the first signal during a period from the end of the first accumulation period to the end of the second accumulation period; The correction unit acquires the second signal after the second accumulation period ends. 1. An information processing device comprising: (Method 19) a pixel that generates a signal according to the amount of incident light; a memory for storing the signal; a processing unit that performs signal processing based on the signal stored in the memory; A method for controlling a photoelectric conversion device having generating the signal during a second integration period, the second integration period including a first integration period and a period after the first integration period; the processing unit acquiring, from the memory, a first signal corresponding to the amount of incident light during the first accumulation period, during a period from the end of the first accumulation period to the end of the second accumulation period; When detecting that a pixel does not satisfy a predetermined criterion using the first signal, the processing unit generates pixel information indicating that the pixel does not satisfy the predetermined criterion; the processing unit acquiring, after the second accumulation period has ended, from the memory a second signal corresponding to the amount of incident light during the second accumulation period; the processing unit correcting the second signal based on the pixel information; A method for controlling a photoelectric conversion device, comprising: (Method 20) generating abnormal pixel information indicating a result of detection of an abnormal pixel based on a first signal corresponding to the amount of light incident on the pixel during a first accumulation period; correcting a second signal corresponding to the amount of light incident on the pixel during a second accumulation period that includes the first accumulation period and a period after the first accumulation period, based on the abnormal pixel information; and the first signal is acquired during a period from the end of the first accumulation period to the end of the second accumulation period; The second signal is acquired after the second accumulation period ends. An information processing method comprising:

[0112] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0113] It should be noted that the above-described embodiments are merely examples of specific embodiments for carrying out the present invention, and the technical scope of the present invention should not be construed as being limited by these embodiments. In other words, the present invention can be carried out in various forms without departing from its technical concept or main features. [Explanation of symbols]

[0114] 100 Photoelectric conversion device 101 pixels 117 Signal Processing Unit 222 pixel memory

Claims

1. a pixel that generates a signal according to the amount of incident light; a memory for storing the signal; a processing unit that performs signal processing based on the signal stored in the memory; and the pixel generates the signal during a second accumulation period that includes a first accumulation period and a period after the first accumulation period; the processing unit acquires, from the memory, a first signal corresponding to an amount of incident light during the first accumulation period, during a period from the end of the first accumulation period to the end of the second accumulation period; When the processing unit detects, using the first signal, that a pixel does not satisfy a predetermined criterion, it generates pixel information indicating that the pixel does not satisfy the predetermined criterion; the processing unit acquires, after the second accumulation period has ended, from the memory a second signal corresponding to the amount of incident light during the second accumulation period; The processing unit corrects the second signal based on the pixel information. A photoelectric conversion device characterized by:

2. A period during which the processing unit detects whether the pixel does not satisfy a predetermined criterion and the second accumulation period at least partially overlap.

2. The photoelectric conversion device according to claim 1.

3. the pixel includes an avalanche photodiode; The memory stores a count value based on photons incident on the avalanche photodiode as the signal.

2. The photoelectric conversion device according to claim 1.

4. The count value is not reset during the period from the start of the second accumulation period to the end of the second accumulation period.

4. The photoelectric conversion device according to claim 3.

5. A plurality of the pixels are included, The pixel information includes information indicating coordinates of a pixel that is detected as not satisfying the predetermined criterion among the plurality of pixels.

2. The photoelectric conversion device according to claim 1.

6. The processing unit corrects the second signal corresponding to the pixel at the coordinate.

6. The photoelectric conversion device according to claim 5.

7. the second accumulation period includes a plurality of the first accumulation periods, the processing unit acquires a plurality of the first signals during a period from the start of the second accumulation period to the end of the second accumulation period; The processing unit generates the pixel information based on a plurality of the first signals.

2. The photoelectric conversion device according to claim 1.

8. The processing unit generates a detection result of a 1-bit value based on each of the plurality of first signals, and generates the pixel information based on a logical product of the plurality of detection results.

8. The photoelectric conversion device according to claim 7.

9. The processing unit generates a detection result of a 1-bit value based on each of the plurality of first signals, and generates the pixel information based on a logical sum of the plurality of detection results.

8. The photoelectric conversion device according to claim 7.

10. The processing unit generates a detection result of a 1-bit value based on each of the plurality of first signals, and generates the pixel information based on a majority logic of the plurality of detection results.

8. The photoelectric conversion device according to claim 7.

11. The processing unit generates the pixel information further based on a plurality of threshold values ​​different from each other and set corresponding to the plurality of first signals, respectively.

8. The photoelectric conversion device according to claim 7.

12. A plurality of the pixels are included, The processing unit generates the pixel information by sequentially processing the first signals output from the pixels without temporarily storing them simultaneously.

2. The photoelectric conversion device according to claim 1.

13. a storage unit for storing the pixel information; a temperature detection unit that detects the temperature of the pixel; and The holding unit updates the pixel information based on the amount of change in the temperature.

2. The photoelectric conversion device according to claim 1.

14. a storage unit for storing the pixel information; a time detection unit that detects an accumulated operating time of the photoelectric conversion device; and The holding unit updates the pixel information based on the amount of change in the cumulative operating time.

2. The photoelectric conversion device according to claim 1.

15. the storage unit includes a nonvolatile memory; The storage unit updates the pixel information stored in the nonvolatile memory.

15. The photoelectric conversion device according to claim 14.

16. The photoelectric conversion device according to any one of claims 1 to 15, an optical device corresponding to the photoelectric conversion device; a control device that controls the photoelectric conversion device; a processing device that processes a signal output from the photoelectric conversion device; a display device that displays information obtained by the photoelectric conversion device; a storage device that stores information obtained by the photoelectric conversion device; and and a mechanical device that operates based on information obtained by the photoelectric conversion device.

17. 17. The device according to claim 16, wherein the processing device acquires distance information from the photoelectric conversion device to the subject.

18. a detection unit that, when detecting that a pixel does not satisfy a predetermined standard using a first signal corresponding to the amount of light incident on the pixel during a first accumulation period, generates pixel information indicating that the pixel does not satisfy the predetermined standard; a correction unit that corrects a second signal corresponding to an amount of light incident on the pixel during a second accumulation period that includes the first accumulation period and a period after the first accumulation period, based on the pixel information; and the detection unit acquires the first signal during a period from the end of the first accumulation period to the end of the second accumulation period; The correction unit acquires the second signal after the second accumulation period ends.

1. An information processing device comprising:

19. a pixel that generates a signal according to the amount of incident light; a memory for storing the signal; a processing unit that performs signal processing based on the signal stored in the memory; A method for controlling a photoelectric conversion device having generating the signal during a second integration period, the second integration period including a first integration period and a period after the first integration period; the processing unit acquiring, from the memory, a first signal corresponding to the amount of incident light during the first accumulation period, during a period from the end of the first accumulation period to the end of the second accumulation period; When detecting that a pixel does not satisfy a predetermined criterion using the first signal, the processing unit generates pixel information indicating that the pixel does not satisfy the predetermined criterion; the processing unit acquiring, after the second accumulation period has ended, from the memory a second signal corresponding to the amount of incident light during the second accumulation period; the processing unit correcting the second signal based on the pixel information; A method for controlling a photoelectric conversion device, comprising:

20. generating abnormal pixel information indicating a detection result of the abnormal pixel based on a first signal corresponding to the amount of light incident on the pixel during a first accumulation period; correcting a second signal corresponding to the amount of light incident on the pixel during a second accumulation period that includes the first accumulation period and a period after the first accumulation period, based on the abnormal pixel information; and the first signal is acquired during a period from the end of the first accumulation period to the end of the second accumulation period; The second signal is acquired after the second integration period ends.

1. An information processing method comprising:

Images