Information processing device and information processing method

Abstract

To provide an information processing device with improved correction accuracy.SOLUTION: An information processing device comprises: a first memory unit which stores first array data based on pixel values output from each of a plurality of pixels; a second memory unit which stores second array data including a plurality of coefficients used for correcting the pixel values; a first calculation unit which corrects a pixel value of a first target pixel on the basis of the pixel value of a first target pixel among the first array data, the pixel value of a second target pixel adjacent to the first target pixel among the first array data, and the second array data; a third memory unit which stores third array data based on the pixel value output from the first calculation unit; a second calculation unit which detects a third target pixel from the third array data; and a third calculation unit which corrects the pixel value of the third target pixel on the basis of the third array data.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to an information processing device and an information processing method. [Background technology]

[0002] Patent Document 1 discloses a method for correcting defective pixels, which identifies defective pixels and replaces the output value of the defective pixels with the average value of the output values ​​of surrounding pixels. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Application Publication No. 2007 / 0030365 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the correction method described in Patent Document 1, the accuracy of correction is sometimes insufficient.

[0005] An object of the present invention is to provide an information processing apparatus and an information processing method with improved correction accuracy. [Means for solving the problem]

[0006] According to one disclosure of the present specification, there is provided an information processing device comprising: a first memory unit that stores first array data based on pixel values ​​output from each of a plurality of pixels; a second memory unit that stores second array data including a plurality of coefficients used to correct the pixel values; a first calculation unit that corrects the pixel value of a first target pixel based on the pixel value of the first target pixel in the first array data, the pixel value of a second target pixel adjacent to the first target pixel in the first array data, and the second array data; a third memory unit that stores third array data based on the pixel values ​​output from the first calculation unit; a second calculation unit that detects a third target pixel from the third array data; and a third calculation unit that corrects the pixel value of the third target pixel based on the third array data.

[0007] According to one disclosure of the present specification, there is provided an information processing device comprising: a first calculation unit that corrects the pixel value of a first target pixel based on a pixel value of the first target pixel in first array data based on pixel values ​​output from each of a plurality of pixels, a pixel value of a second target pixel adjacent to the first target pixel in the first array data, and second array data including a plurality of coefficients; and a third calculation unit that corrects a third target pixel detected from third array data based on the pixel values ​​output from the first calculation unit.

[0008] According to one disclosure of the present specification, there is provided an information processing method comprising the steps of: acquiring first array data based on pixel values ​​output from each of a plurality of pixels; acquiring second array data including a plurality of coefficients used to correct the pixel values; correcting the pixel value of the first target pixel based on the pixel value of the first target pixel in the first array data, the pixel value of a second target pixel adjacent to the first target pixel in the first array data, and the second array data; acquiring third array data based on the corrected pixel value of the first target pixel; detecting a third target pixel from the third array data; and correcting the pixel value of the third target pixel based on the third array data. [Effects of the Invention]

[0009] According to the present invention, an information processing apparatus and an information processing method with improved correction accuracy are provided. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a schematic diagram illustrating the overall configuration of a photoelectric conversion device according to a first embodiment. [Figure 2] 2 is a schematic block diagram showing an example of the configuration of a sensor substrate of the photoelectric conversion device according to the first embodiment. FIG. [Figure 3] 2 is a schematic block diagram showing an example of the configuration of a circuit board of the photoelectric conversion device according to the first embodiment. FIG. [Figure 4] 1 is a schematic block diagram showing an example of the configuration of one pixel of a photoelectric conversion unit and a pixel signal processing unit of the photoelectric conversion device according to the first embodiment. [Figure 5] 3A to 3C are diagrams illustrating the operation of the avalanche photodiode of the photoelectric conversion device according to the first embodiment. [Figure 6] FIG. 2 is a schematic explanatory diagram of a correction process performed in the information processing device according to the first embodiment. [Figure 7] 1 is a block diagram of a photoelectric conversion system according to a first embodiment. [Figure 8] 3 is a flowchart of processing performed in the photoelectric conversion system according to the first embodiment. [Figure 9] FIG. 10 is a schematic explanatory diagram of a correction process performed in an information processing device according to a second embodiment. [Figure 10] FIG. 10 is a block diagram of a photoelectric conversion system according to a second embodiment. [Figure 11] 10 is a flowchart of a process performed in a photoelectric conversion system according to a second embodiment. [Figure 12] FIG. 10 is a block diagram of a device according to a third embodiment. [Figure 13] FIG. 10 is a block diagram of a device according to a fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The same or corresponding elements in multiple drawings are designated by common reference numerals, and their description may be omitted or simplified.

[0012] [First embodiment] The information processing device according to this embodiment is a device that processes image data acquired by a photoelectric conversion device. Prior to describing the information processing device, the configuration and operation of the photoelectric conversion device will be described using FIGS. 1 to 5(c). In this description, it is assumed that the information processing device is provided outside the photoelectric conversion device, and that the photoelectric conversion device and the information processing device constitute a photoelectric conversion system. However, the information processing device may also be disposed within the photoelectric conversion device, for example.

[0013] FIG. 1 is a schematic diagram showing the overall configuration of a photoelectric conversion device 100 according to this embodiment. The photoelectric conversion device 100 includes a sensor substrate 11 and a circuit substrate 21, which are stacked on top of each other. The sensor substrate 11 and the circuit substrate 21 are electrically connected to each other. The sensor substrate 11 includes a pixel region 12 in which a plurality of pixels 101 are arranged in a plurality of rows and a plurality of columns. The circuit substrate 21 includes a first circuit region 22 in which a plurality of pixel signal processing units 103 are arranged in a plurality of rows and a plurality of columns, and a second circuit region 23 arranged on the periphery of the first circuit region 22. The second circuit region 23 may include a circuit for controlling the plurality of pixel signal processing units 103, etc.

[0014] The sensor substrate 11 has a first semiconductor layer having a photoelectric conversion unit 102 (described later) and a first wiring structure. The circuit board 21 has a second semiconductor layer having circuits such as a pixel signal processing unit 103 (described later) and a second wiring structure. The photoelectric conversion device 100 is configured by laminating the second semiconductor layer, the second wiring structure, the first wiring structure, and the first semiconductor layer in this order. The photoelectric conversion device 100 is a back-illuminated photoelectric conversion device in which light is incident from a first surface of the first semiconductor layer and the circuit board 21 is disposed on the second surface side of the first semiconductor layer.

[0015] In the following description, the sensor substrate 11 and the circuit board 21 are described as being diced chips, but the sensor substrate 11 and the circuit board 21 are not limited to being chips. For example, the sensor substrate 11 and the circuit board 21 may be wafers. Furthermore, if the sensor substrate 11 and the circuit board 21 are diced chips, the photoelectric conversion device 100 may be manufactured by stacking them in the wafer state and then dicing them, or by stacking them after dicing.

[0016] FIG. 2 is a schematic block diagram showing an example of the configuration of the sensor substrate 11 of the photoelectric conversion device 100 according to this embodiment. A plurality of pixels 101 are arranged in a pixel region 12, forming a plurality of rows and a plurality of columns. Each of the plurality of pixels 101 has a photoelectric conversion unit 102 including an avalanche photodiode (hereinafter referred to as an APD) as a photoelectric conversion element within the substrate. The pixel 101 is typically a circuit that outputs a signal for forming an image; however, when the photoelectric conversion device 100 is a sensor used for time of flight (TOF), forming an image is not essential. For example, the pixel 101 may be a circuit that measures the time and amount of light that arrives.

[0017] 3 is a schematic block diagram showing an example of the configuration of a circuit board 21 of a photoelectric conversion device according to this embodiment. The circuit board 21 has a first circuit area 22 in which a plurality of pixel signal processing units 103 are arranged in a plurality of rows and a plurality of columns.

[0018] Also arranged on the circuit board 21 are a vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, pixel output signal lines 113, an output circuit 114, and a control signal generation unit 115. The multiple photoelectric conversion units 102 shown in Fig. 2 and the multiple pixel signal processing units 103 shown in Fig. 3 are electrically connected to each other via connection wiring provided for each pixel 101.

[0019] The control signal generation unit 115 is a control circuit that generates control signals for driving the vertical scanning circuit 110, the horizontal scanning circuit 111, and the readout circuit 112, and supplies these signals to each of these units. In this way, the control signal generation unit 115 controls the drive timing of each unit, etc.

[0020] The vertical scanning circuit 110 supplies a control signal to each of the plurality of pixel signal processing units 103 based on the control signal supplied from the control signal generation unit 115. The vertical scanning circuit 110 supplies a control signal to each pixel signal processing unit 103 for each row via a drive line provided for each row in the first circuit area 22. Note that, as will be described later, there may be multiple drive lines for each row. The vertical scanning circuit 110 may include logic circuits such as a shift register and an address decoder. In this way, the vertical scanning circuit 110 selects a row for outputting a signal from the pixel signal processing unit 103.

[0021] The signal output from the photoelectric conversion unit 102 of the pixel 101 is processed by the pixel signal processing unit 103. The pixel signal processing unit 103 has circuits such as a counter and a memory. The memory of the pixel signal processing unit 103 holds the digital signal.

[0022] The horizontal scanning circuit 111 supplies a control signal to the readout circuit 112 based on a control signal supplied from the control signal generation unit 115. The pixel signal processing units 103 are connected to the readout circuit 112 via pixel output signal lines 113 provided for each column of the first circuit area 22. The pixel output signal line 113 for one column is shared by multiple pixel signal processing units 103 for the corresponding column. The pixel output signal line 113 includes multiple wirings and has at least the function of outputting a digital signal from each pixel signal processing unit 103 to the readout circuit 112 and the function of supplying the pixel signal processing unit 103 with a control signal for selecting a column from which to output a signal. The readout circuit 112 outputs a signal to a storage unit or signal processing unit external to the photoelectric conversion device 100 via the output circuit 114 based on the control signal supplied from the control signal generation unit 115.

[0023] The photoelectric conversion units 102 in the pixel region 12 may be arranged one-dimensionally. Furthermore, the function of the pixel signal processing unit 103 does not necessarily have to be provided for each pixel 101. For example, one pixel signal processing unit 103 may be shared by multiple pixels 101. In this case, the pixel signal processing unit 103 provides the signal processing function to each pixel 101 by sequentially processing the signals output from each photoelectric conversion unit 102.

[0024] 2 and 3, a first circuit region 22 in which a plurality of pixel signal processing units 103 are arranged is arranged in a region overlapping the pixel region 12 in a planar view. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control signal generation unit 115 are arranged so as to overlap between an end of the sensor substrate 11 and an end of the pixel region 12 in a planar view. In other words, the sensor substrate 11 has the pixel region 12 and a non-pixel region arranged around the pixel region 12. A second circuit region 23 in which the vertical scanning circuit 110, the horizontal scanning circuit 111, the readout circuit 112, the output circuit 114, and the control signal generation unit 115 are arranged is arranged in a region of the circuit substrate 21 overlapping the non-pixel region in a planar view.

[0025] Fig. 4 is a schematic block diagram showing an example of the configuration of one pixel of the photoelectric conversion unit and pixel signal processing unit of the photoelectric conversion device according to this embodiment. Fig. 4 schematically shows a more specific example of the configuration, including the connection relationship between the photoelectric conversion unit 102 arranged on the sensor substrate 11 and the pixel signal processing unit 103 arranged on the circuit board 21. In Fig. 4, the drive lines between the vertical scanning circuit 110 and the pixel signal processing unit 103 in Fig. 3 are shown as drive lines 213 and 214.

[0026] The photoelectric conversion unit 102 has an APD 201. The pixel signal processing unit 103 has a quenching element 202, a waveform shaping unit 210, a counter circuit 211, and a selection circuit 212. It is sufficient that the pixel signal processing unit 103 has at least one of the waveform shaping unit 210, the counter circuit 211, and the selection circuit 212.

[0027] The APD 201 generates charge pairs according to incident light through photoelectric conversion. A voltage VL (first voltage) is supplied to the anode of the APD 201. A cathode of the APD 201 is connected to a first terminal of the quench element 202 and an input terminal of the waveform shaping unit 210. A voltage VH (second voltage) higher than the voltage VL supplied to the anode is supplied to the cathode of the APD 201. As a result, a reverse bias voltage is supplied to the anode and cathode of the APD 201 such that the APD 201 performs avalanche multiplication. When charges are generated by incident light in the APD 201 to which the reverse bias voltage is supplied, the charges undergo avalanche multiplication, generating an avalanche current.

[0028] The APD 201 can operate in either Geiger mode or linear mode when a reverse bias voltage is supplied to it. The Geiger mode is a mode in which the APD operates with a potential difference between the anode and cathode greater than the breakdown voltage, while the linear mode is a mode in which the APD operates with a potential difference between the anode and cathode close to or less than the breakdown voltage.

[0029] An APD operated in Geiger mode is called a SPAD (Single Photon Avalanche Diode). The APD 201 may be operated in either linear mode or Geiger mode. A SPAD is preferable because the potential difference is larger than in a linear mode APD, making the avalanche multiplication effect more pronounced.

[0030] The quench element 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication. The quench element 202 also suppresses the voltage supplied to the APD 201 to suppress avalanche multiplication (quench operation). The quench element 202 also returns the voltage supplied to the APD 201 to voltage VH by passing a current corresponding to the voltage drop caused by the quench operation (recharge operation). The quench element 202 may be, for example, a resistor or a transistor.

[0031] The waveform shaping unit 210 shapes the potential change of the cathode of the APD 201 obtained when a photon is detected, and outputs a pulse signal. For example, an inverter circuit is used as the waveform shaping unit 210. While Fig. 4 shows an example in which one inverter is used as the waveform shaping unit 210, it may be a circuit in which multiple inverters are connected in series, or may be any other circuit that has a waveform shaping effect.

[0032] The counter circuit 211 counts the pulse signals output from the waveform shaping unit 210 and holds a digital signal indicating the count value. When a control signal is supplied from the vertical scanning circuit 110 via a drive line 213, the counter circuit 211 resets the signal it holds.

[0033] A control signal is supplied to the selection circuit 212 from the vertical scanning circuit 110 shown in Fig. 3 via a drive line 214 shown in Fig. 4. In response to this control signal, the selection circuit 212 switches between electrical connection and disconnection between the counter circuit 211 and the pixel output signal line 113. The selection circuit 212 includes, for example, a buffer circuit for outputting a signal corresponding to the value held in the counter circuit 211.

[0034] 4, the selection circuit 212 switches between electrical connection and disconnection between the counter circuit 211 and the pixel output signal line 113, but the method for controlling the signal output to the pixel output signal line 113 is not limited to this. For example, a switch such as a transistor may be disposed at a node between the quench element 202 and the APD 201, or between the photoelectric conversion unit 102 and the pixel signal processing unit 103, and the like, to switch between electrical connection and disconnection, thereby controlling the signal output to the pixel output signal line 113. Alternatively, the signal output to the pixel output signal line 113 may be controlled by changing the value of the voltage VH or voltage VL supplied to the photoelectric conversion unit 102 using a switch such as a transistor.

[0035] FIG. 4 shows an example configuration using a counter circuit 211. However, instead of the counter circuit 211, a time-to-digital converter (hereinafter, referred to as TDC) and a memory may be used to acquire the timing for detecting a pulse. In this case, the generation timing of the pulse signal output from the waveform shaping unit 210 is converted into a digital signal by the TDC. In this case, a control signal (reference signal) may be supplied to the TDC from the vertical scanning circuit 110 in FIG. 3 via a drive line. The TDC acquires a signal indicating the relative time of the input timing of the pulse based on the control signal as a digital signal. Note that the arrangement of the pixel output signal lines 113, the readout circuits 112, and the output circuits 114 is not limited to the example shown in FIG. 3. For example, the pixel output signal lines 113 may extend in the row direction, and the readout circuits 112 may be connected to the pixel output signal lines 113 extending in the row direction.

[0036] 5(a), 5(b), and 5(c) are diagrams illustrating the operation of the APD 201 according to this embodiment. FIG. 5(a) is a diagram illustrating the APD 201, the quench element 202, and the waveform shaping unit 210 extracted from FIG. 4. As shown in FIG. 5(a), the connection node between the APD 201, the quench element 202, and the input terminals of the waveform shaping unit 210 is referred to as node A. Also, as shown in FIG. 5(a), the output side of the waveform shaping unit 210 is referred to as node B.

[0037] FIG. 5(b) is a graph showing the time change in the potential of node A in FIG. 5(a). FIG. 5(c) is a graph showing the time change in the potential of node B in FIG. 5(a). From time t0 to time t1, a voltage of VH-VL is applied to the APD 201 in FIG. 5(a). When a photon is incident on the APD 201 at time t1, avalanche multiplication occurs in the APD 201. This causes an avalanche current to flow through the quench element 202, and the potential of node A drops. Thereafter, the amount of potential drop increases further, and the voltage applied to the APD 201 gradually decreases. Then, at time t2, avalanche multiplication in the APD 201 stops. As a result, the voltage level of node A does not drop below a certain value. Then, from time t2 to time t3, a current flows to node A from the node at voltage VH to compensate for the voltage drop, and at time t3, node A settles to its original potential.

[0038] In the above process, the potential of node B becomes high during the period when the potential of node A is lower than a certain threshold. In this way, the waveform of the drop in the potential of node A caused by the incidence of a photon is shaped by waveform shaping section 210 and output as a pulse to node B.

[0039] Next, an information processing device according to this embodiment will be described with reference to Figs. 6 to 8. The information processing device according to this embodiment is a device that corrects array data based on pixel values ​​output from a plurality of pixels 101 in the photoelectric conversion device described with reference to Figs. 1 to 5(c). First, an outline of the processing performed by the information processing device according to this embodiment will be described with reference to Fig. 6. Fig. 6 is a schematic explanatory diagram of the correction processing performed in the information processing device according to this embodiment.

[0040] The abnormal pixels to be corrected in this embodiment will be described below. An abnormal pixel refers to a pixel that has a unique pixel value compared to its surrounding pixels. In the following, we will mainly describe abnormal pixels that have a significantly larger pixel value compared to its surrounding pixels. However, pixels that have a significantly smaller pixel value compared to its surrounding pixels can also be called abnormal pixels.

[0041] One method for correcting abnormal pixels involves identifying a pixel whose pixel value is significantly different from that of neighboring pixels as an abnormal pixel and replacing the pixel value of the abnormal pixel with the average or median of the pixel values ​​of multiple pixels surrounding the abnormal pixel. For example, in a photoelectric conversion device using an APD, crosstalk can occur due to the avalanche light emission phenomenon that occurs when charges generated by avalanche multiplication recombine. When this crosstalk occurs, the abnormal pixel can affect the characteristics of multiple pixels, resulting in an image that appears to contain multiple abnormal pixels. Such abnormal pixels are called clustered abnormal pixels. Note that devices that may experience clustered abnormal pixels due to crosstalk are not limited to photoelectric conversion devices using APDs. Clustered abnormal pixels can also occur in CMOS sensors.

[0042] FIG. 6(a) schematically illustrates the distribution of pixel values ​​surrounding a cluster of uncorrected abnormal pixels contained in image data based on the output signal of the photoelectric conversion device 100. The circles illustrated in a matrix in FIG. 6(a) represent pixels. The hatching around each pixel in FIG. 6(a) schematically illustrates the pixel value. That is, the whiter the hatching, the larger the pixel value, and the blacker the hatching, the smaller the pixel value. In FIG. 6(a), the white pixel in the center is an abnormal pixel with a large pixel value. Also, in FIG. 6(a), due to crosstalk, the abnormal pixel affects two pixels located above, below, left, and right of the central abnormal pixel, as well as four pixels diagonally opposite the central defective pixel. In this manner, an abnormal pixel may affect surrounding pixels due to crosstalk.

[0043] Figure 6(b) shows a schematic diagram of the process for correcting the effects of crosstalk on such clustered abnormal pixels. In this process, two-dimensional array data including the clustered abnormal pixels is multiplied by two-dimensional correction array data that has been acquired in advance. This generates image data in which the effects of crosstalk have been corrected, as shown in Figure 6(c). In this image data after crosstalk correction, the abnormal pixel in the center remains, but the influence of the abnormal pixel on the surrounding pixels has been reduced.

[0044] Here, the correction array data represents the influence of surrounding pixels on one pixel as a matrix of crosstalk probability (crosstalk matrix). Furthermore, "multiplication" may include a calculation process in which elements of the image data array are multiplied by elements of the correction array data. For example, "multiplication" may include a calculation in which each element of the crosstalk matrix is ​​multiplied by a constant using a numerical value based on the pixel value of the abnormal pixel, or may include a convolution calculation using a fast Fourier transform or the like. Figure 6(b) shows an example of a convolution calculation and an example of a matrix used in the convolution calculation.

[0045] After that, the image data after the crosstalk correction is corrected by replacing the pixel value of the abnormal pixel with the average or median value of the pixel values ​​of multiple surrounding pixels, as shown in Figure 6(d). Through the above procedure, the cluster abnormal pixel correction is completed.

[0046] Next, the correction process of Fig. 6 will be described in more detail with reference to Fig. 7 and Fig. 8. Fig. 7 is a block diagram of a photoelectric conversion system according to this embodiment. The photoelectric conversion system includes an information processing device 30, an image acquisition unit 31, and a readout unit 32.

[0047] The image acquisition unit 31 and the readout unit 32 correspond to the above-mentioned photoelectric conversion device. The image acquisition unit 31 is, for example, the plurality of pixels 101 shown in FIG. 2. The readout unit 32 is, for example, the pixel signal processing unit 103, vertical scanning circuit 110, horizontal scanning circuit 111, readout circuit 112, pixel output signal line 113, output circuit 114, and control signal generation unit 115 shown in FIGS. 3 and 4. The image data generated in the image acquisition unit 31 and the readout unit 32 is input to the information processing device 30.

[0048] The information processing device 30 processes image data acquired by the image acquisition unit 31 and the readout unit 32. The information processing device 30 reads and executes computer-executable instructions. The information processing device 30 may be a computer including one or more processors and one or more memories. The information processing device 30 may be configured to include multiple separate computers or multiple separate processors. The information processing device 30 may also be configured with one or more processing circuits. The processor or circuit may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA). The processor or circuit may also include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

[0049] The information processing device 30 has a first memory unit 301, a second memory unit 302, a crosstalk correction unit 303, a third memory unit 304, an abnormal pixel detection unit 305, and an abnormal pixel correction unit 306. The first memory unit 301, the second memory unit 302, and the third memory unit 304 may be configured with a memory such as a static RAM (SRAM). The functions of the crosstalk correction unit 303, the abnormal pixel detection unit 305, and the abnormal pixel correction unit 306 may be realized, for example, by a processor reading and executing computer-executable instructions from the memory.

[0050] 8 is a flowchart of the processing performed in the photoelectric conversion system according to this embodiment. The processing of this embodiment will be described in more detail with mutual reference to FIGS.

[0051] In step S11, the image acquisition unit 31 outputs a signal based on the incident light incident on the plurality of pixels 101, and the readout unit 32 acquires image data based on the signal from the image acquisition unit 31. The first storage unit 301 stores this image data (first array data). The image data stored in the first storage unit 301 is array data including pixel values ​​of the plurality of pixels 101.

[0052] In step S12, the crosstalk correction unit 303 selects correction array data (second array data) to be used in the crosstalk correction process. The second storage unit 302 stores one or more types of correction array data in advance. If the second storage unit 302 stores one type of correction array data, the crosstalk correction unit 303 selects that one type of correction array data. If the second storage unit 302 stores multiple types of correction array data, the crosstalk correction unit 303 selectively acquires one of the multiple types of correction array data based on setting values, shooting environment, etc., and performs processing.

[0053] The correction array data is crosstalk matrix data with N rows and M columns. One of the number of rows N and the number of columns M is an integer greater than or equal to 2, and the other is an integer greater than or equal to 1. In other words, the correction array data may be one-dimensional or two-dimensional as long as it contains two or more elements.

[0054] It is preferable that the number of rows N and the number of columns M are both integers of 3 or greater. In other words, it is preferable that the correction array data has a size of 3 rows and 3 columns or greater. In this case, the pixel data to be corrected also has a size of 3 rows and 3 columns or greater. This allows pixels adjacent to one pixel in eight directions to be taken into account during correction, making it possible to perform crosstalk correction of pixels around an abnormal pixel with high accuracy. It is also preferable that the number of rows N and the number of columns M are both odd numbers. Since the crosstalk probability is symmetric in the row and column directions around the abnormal pixel that caused the crosstalk, correction can be performed efficiently by using odd numbers of rows N and columns M.

[0055] In step S13, the image data stored in the first storage unit 301 and the correction array data stored in the second storage unit 302 are input to the crosstalk correction unit 303 (first calculation unit). The crosstalk correction unit 303 performs crosstalk correction processing to correct the image data based on the correction array data. The crosstalk-corrected image data (third array data) obtained by this processing is stored in the third storage unit 304. As described above, this crosstalk correction processing can be a process of convolving the matrix of the image data with the crosstalk matrix, which is the correction array data. However, the crosstalk correction processing is not limited to convolution processing as long as it can correct crosstalk by using one pixel (first target pixel), its adjacent pixel (second target pixel), and second array data including multiple coefficients. The processing of step S13 corrects clustered abnormal pixels in the image data to resemble isolated abnormal pixels, as shown in FIG. 6(c).

[0056] In step S14, the abnormal pixel detection unit 305 (second calculation unit) detects abnormal pixels (third target pixels) from the crosstalk-corrected image data stored in the third storage unit 304, and outputs the detection result to the abnormal pixel correction unit 306. The abnormal pixel detection process may be a process of detecting abnormal pixels based on pixel values ​​of the crosstalk-corrected image data. Alternatively, the abnormal pixel detection process may be a process of detecting abnormal pixels based on position information of abnormal pixels acquired in advance, or a process of treating both known abnormal pixels and abnormal pixels detected from the crosstalk-corrected image data as abnormal pixels.

[0057] In step S15, the abnormal pixel correction unit 306 (third calculation unit) performs an abnormal pixel correction process on the crosstalk-corrected image data. The abnormal pixel correction unit 306 performs a correction to replace the pixel value of a pixel detected as an abnormal pixel by the abnormal pixel detection unit 305 with the average or median value of multiple pixel values ​​of surrounding pixels. This allows the abnormal pixel to be corrected as shown in FIG. 6(d).

[0058] As described above, the information processing device 30 according to this embodiment performs crosstalk correction on image data to correct pixels surrounding an abnormal pixel, and then corrects the abnormal pixel itself. That is, a wide-area correction is performed to correct clusters of abnormal pixels that occur over a wide area, followed by a narrow-area correction to correct isolated abnormal pixels. The effect of performing the two-stage correction in this order will be described below.

[0059] Crosstalk can occur due to abnormal pixels, which can result in clusters of abnormal pixels. When correcting image data containing such clusters of abnormal pixels, it is desirable to take the influence of the abnormal pixels into account when calculating the crosstalk correction process. For example, when crosstalk correction is performed after correcting the abnormal pixels, some of the abnormal pixels in the clusters of abnormal pixels have disappeared by the time the crosstalk correction process is performed. In such cases, the missing abnormal pixels are not taken into account during the crosstalk correction process, which can lead to a problem in that sufficient correction accuracy cannot be ensured.

[0060] In contrast, in this embodiment, crosstalk correction processing is performed before the correction of abnormal pixels, so the effects of crosstalk caused by the abnormal pixels are appropriately corrected. Therefore, this embodiment provides an information processing device 30 and an information processing method with improved correction accuracy. The processing order of performing crosstalk correction processing before the correction of abnormal pixels in this embodiment can be more generally rephrased as performing correction processing in the reverse order of the noise generation. This allows appropriate correction to be performed even when noise from multiple sources overlaps, such as when one noise generates another noise.

[0061] [Second embodiment] In this embodiment, a modification of the correction process of the first embodiment will be described with reference to Figures 9 to 11. In this embodiment, the description of elements common to the first embodiment may be omitted or simplified in some cases.

[0062] First, an outline of the processing performed by the information processing device of this embodiment will be described with reference to Fig. 9. Fig. 9 is a schematic explanatory diagram of the correction processing performed by the information processing device according to this embodiment.

[0063] Similar to FIG. 6(a), FIG. 9(a) schematically illustrates the distribution of pixel values ​​around a cluster of abnormal pixels included in image data based on the output signal of the photoelectric conversion device 100. In this embodiment, as shown in FIG. 9(b), nonlinearity correction processing is performed on such image data. Furthermore, as shown in FIG. 9(c), in parallel with the nonlinearity correction processing, multiplication (convolution) of correction array data is performed on the image data before correction. This multiplication generates subtraction image data for crosstalk correction processing, as shown in FIG. 9(d). Then, crosstalk correction processing is performed as shown in FIG. 9(e) by subtracting the subtraction image data of FIG. 9(d) from the image data after nonlinearity correction processing of FIG. 9(b).

[0064] Then, the image data after the crosstalk correction is corrected by replacing the pixel value of the abnormal pixel with the average or median value of the pixel values ​​of the surrounding pixels, as shown in Figure 9(f). Through the above procedure, the cluster abnormal pixel correction is completed.

[0065] The effects of performing nonlinearity correction processing and generating subtraction image data from image data before nonlinearity correction processing are described below. One example of the configuration of a photoelectric conversion device 100 that acquires image data is a circuit configuration using an APD 201 with an active recharge type drive circuit, typically a clock recharge type. In a photoelectric conversion device 100 using an active recharge type drive circuit, a recharge operation that resets the APD 201 to a state where avalanche multiplication is possible is performed at a predetermined interval. In a photoelectric conversion device 100 using such an active recharge type drive circuit, a pile-up phenomenon may occur. When a pile-up phenomenon occurs, some incident photons may not be properly counted, resulting in a nonlinear response characteristic of the count value relative to the number of incident photons. Furthermore, depending on the configuration or drive method of the photoelectric conversion device 100, the response characteristic may become nonlinear due to the influence of dead time during which photons are not counted. As such, the response characteristic of the photoelectric conversion device 100 may be nonlinear. Therefore, it is desirable to perform nonlinearity correction processing to bring this response characteristic closer to linearity.

[0066] The nonlinear response function indicating the above-mentioned nonlinear response can be expressed by the following equation (1).

number

[0067] Here, we discuss the relationship between the count value measured when crosstalk is present and the number of incident photons. If the count value is Nct and the number of incident photons is Nph, then Nct = f(Nph). The amount of light emitted due to crosstalk is proportional to the count value Nct, not the number of incident photons Nph. Meanwhile, the total count value Nct' of photons incident on a pixel affected by crosstalk, Nc, is expressed as Nct' = f(Nc + Ns), where Nc is the number of spurious signal photons due to crosstalk, and Ns is the number of signal photons, Ns, and Nc + Ns is the sum of these. Therefore, when estimating the distribution of spurious signal photons generated by crosstalk, estimating it based on count values ​​after nonlinearity correction may result in a large error compared to the actual distribution of spurious signal photons.

[0068] Considering the above, it is more appropriate to estimate the distribution of false signal photons generated by crosstalk based on the count value before nonlinearity correction. Therefore, in this embodiment, subtraction image data for correction (FIG. 9(d)) is generated based on pixel data before nonlinearity correction (FIG. 9(a)). By subtracting the subtraction image data for correction obtained in this manner from image data after nonlinearity correction, the influence of deterioration in correction accuracy of clustered abnormal pixels caused by nonlinearity is reduced even in scenes where pixels with high pixel values ​​exist and nonlinearity is affecting. Therefore, accurate correction can be performed even when clustered abnormal pixels are close to each other or when the amount of signal light corresponding to the subject is high.

[0069] Next, the correction process of Fig. 9 will be described in more detail with reference to Fig. 10 and Fig. 11. Fig. 10 is a block diagram of a photoelectric conversion system according to this embodiment. In addition to the configuration shown in Fig. 7, the information processing device 30 of this embodiment further includes a nonlinearity correction unit 307, a black level correction unit 308, and a noise reduction processing unit 309. The functions of the nonlinearity correction unit 307, the black level correction unit 308, and the noise reduction processing unit 309 can be realized, for example, by a processor reading and executing computer-executable instructions from a memory.

[0070] 11 is a flowchart of the processing performed in the photoelectric conversion system according to this embodiment. The processing of this embodiment will be described in more detail with mutual reference to FIGS.

[0071] In step S11, the image data (first array data) is stored in the first storage unit 301, similarly to FIG.

[0072] In step S16, the nonlinearity correction unit 307 (sixth calculation unit) performs nonlinearity correction processing on the image data stored in the first storage unit. As described above, the response characteristics of the photoelectric conversion device 100 may be nonlinear. In this case, performing correction based on a linear response during crosstalk correction processing may result in overcorrection. Therefore, in this embodiment, the nonlinearity correction unit 307, which is located upstream of the crosstalk correction unit 303, performs nonlinearity correction processing on the image data before crosstalk correction processing. This makes it possible to reduce the impact of overcorrection on signal quality. Note that the nonlinearity correction processing is performed, for example, by referring to a lookup table that indicates the correspondence between input pixel values ​​and output pixel values.

[0073] In step S12, the crosstalk correction unit 303 selects correction sequence data (second sequence data) to be used in the crosstalk correction process.

[0074] In step S13, the image data before the nonlinearity correction processing stored in the first storage unit 301, the correction array data stored in the second storage unit 302, and the image data after the nonlinearity correction processing are input to the crosstalk correction unit 303. The crosstalk correction unit 303 convolves the image data before the nonlinearity correction processing with the correction array data to generate subtraction image data for the crosstalk correction processing. The crosstalk correction unit 303 generates the image data after the crosstalk correction processing by subtracting the subtraction image data from the image data after the nonlinearity correction processing.

[0075] In step S17, the black level correction unit 308 (fourth calculation unit) performs black level correction processing to correct the black level by subtracting a predetermined correction value corresponding to the black level from the pixel values ​​of the image data after crosstalk correction. The image after black level correction is stored in the third storage unit 304. The correction value used in the black level correction processing may be generated, for example, based on pixel values ​​output from light-shielded pixels included in an optical black (OB) region of the pixel region 12, whose light incident surface side is shielded by a light-shielding film. Alternatively, the value used in the black level correction processing may be generated based on a frequency distribution of pixel values ​​output from a plurality of light-shielded pixels. Alternatively, the black level correction unit 308 may acquire the correction value by referring to a lookup table in which temperature information is associated with correction values ​​based on temperature information output from a temperature sensor mounted in the photoelectric conversion device 100.

[0076] It should be noted that instead of the process of step S17, a process of adding a correction value for black level correction to the subtraction image data may be added to the process of step S13. Black level correction can also be performed in this configuration.

[0077] The black level correction process in step S17 may be performed before the crosstalk correction process in step S13. However, the pixel values ​​of pixels that cause crosstalk include the influence of the black level, and crosstalk to surrounding pixels is influenced by the black level. Therefore, it is desirable to perform the black level correction process after the crosstalk correction process. This allows the crosstalk correction process to be performed on image data that includes the contribution of the black level, so the influence of crosstalk is appropriately corrected while taking the contribution of the black level into consideration.

[0078] In step S14, the abnormal pixel detection unit 305 performs an abnormal pixel detection process on the black level corrected image data stored in the third storage unit 304. Then, in step S15, the abnormal pixel correction unit 306 performs an abnormal pixel correction process on the black level corrected image data. The contents of these processes are the same as those in the first embodiment.

[0079] In the abnormal pixel detection process of step S14, if the image data contains the influence of the black level, the accuracy of detecting abnormal pixels with small pixel values ​​will decrease. Therefore, it is desirable to perform the black level correction process of step S17 before the abnormal pixel detection process of step S14. This allows for accurate detection and correction of abnormal pixels with small pixel values.

[0080] In step S18, the noise reduction processing unit 309 (fifth calculation unit) performs noise reduction processing on the image data after the abnormal pixel correction processing to reduce noise other than that of the cluster abnormal pixels. Examples of noise to be processed in the noise reduction processing include thermal noise and random noise due to statistical fluctuations. This reduces noise caused by these factors. The noise reduction processing may be, for example, a smoothing processing using a spatial filter. By performing the smoothing processing, thermal noise, random noise, etc. can be reduced. Alternatively, the noise reduction processing may be processing using a noise reduction processing model generated by machine learning such as deep learning.

[0081] Note that if the noise reduction process of step S18 is performed before the abnormal pixel detection process of step S14, the accuracy of detecting abnormal pixels will decrease. Therefore, it is desirable to perform the noise reduction process of step S18 after the abnormal pixel detection process of step S14. This allows abnormal pixels to be detected and corrected with high accuracy. In this embodiment, the noise reduction process of step S18 is performed after the abnormal pixel correction process of step S15, but this is not limited to this. The noise reduction process of step S18 may be performed after the abnormal pixel detection process of step S14, and then the abnormal pixel correction process of step S15 may be performed.

[0082] According to this embodiment, similarly to the first embodiment, an information processing device 30 and an information processing method with improved correction accuracy are provided. Furthermore, according to this embodiment, by performing nonlinearity correction processing before crosstalk correction processing, it is possible to suppress overcorrection during crosstalk correction processing. Furthermore, according to this embodiment, by performing black level correction processing before abnormal pixel detection processing, it is possible to improve the accuracy of detecting abnormal pixels. Furthermore, according to this embodiment, by performing noise reduction processing, it is possible to improve signal quality.

[0083] [Third embodiment] The information processing device 30 and photoelectric conversion system in the above-described embodiment can be applied to various devices. Examples of such devices include digital still cameras, digital camcorders, camera heads, copiers, fax machines, mobile phones, vehicle-mounted cameras, observation satellites, and surveillance cameras. Fig. 12 shows a block diagram of a digital still camera as an example of such a device. Fig. 12 shows an example in which the photoelectric conversion system shown in Fig. 7 is applied to a digital still camera.

[0084] The device 70 shown in FIG. 12 includes a barrier 706, a lens 702, an aperture 704, and an imaging device 700 (an example of the photoelectric conversion device 100 or a photoelectric conversion system). The device 70 further includes a signal processing unit (processing device) 708, a timing generating unit 720, an overall control / calculation unit 718 (control device), a memory unit 710 (storage device), a recording medium control I / F unit 716, a recording medium 714, and an external I / F unit 712. The information processing device 30 of the above-described embodiment may be included in the imaging device 700 or the signal processing unit 708. At least one of the barrier 706, the lens 702, and the aperture 704 is an optical device corresponding to the device. The barrier 706 protects the lens 702, and the lens 702 forms an optical image of a subject on the imaging device 700. The aperture 704 varies the amount of light passing through the lens 702. The imaging device 700 converts the optical image formed by the lens 702 into image data (image signals). The signal processing unit 708 performs various corrections, data compression, etc. on the imaging data output from the imaging device 700. The timing generation unit 720 outputs various timing signals to the imaging device 700 and the signal processing unit 708. The overall control / calculation unit 718 controls the entire digital still camera, and the memory unit 710 temporarily stores image data. The recording medium control I / F unit 716 is an interface for recording or reading image data to or from the recording medium 714, which is a removable recording medium such as a semiconductor memory for recording or reading imaging data. The external I / F unit 712 is an interface for communicating with an external computer, etc. Timing signals, etc., may be input from outside the device. The device 70 may further include a display device (monitor, electronic viewfinder, etc.) that displays information obtained by the photoelectric conversion device. The device includes at least a photoelectric conversion device. The device 70 also includes at least one of an optical device, a control device, a processing device, a display device, a storage device, and a mechanical device that operates based on information obtained by the photoelectric conversion device. The mechanical device is a movable part (for example, a robot arm) that operates in response to a signal from the photoelectric conversion device.

[0085] Each pixel may include a plurality of photoelectric conversion units (a first photoelectric conversion unit and a second photoelectric conversion unit). The signal processing unit 708 may be configured to process a pixel signal based on the charge generated in the first photoelectric conversion unit and a pixel signal based on the charge generated in the second photoelectric conversion unit, and acquire information about the distance from the image capturing device 700 to the subject.

[0086] [Fourth embodiment] 13(a) and 13(b) are block diagrams of devices related to an in-vehicle camera according to this embodiment. FIGS. 13(a) and 13(b) show an example in which the photoelectric conversion system shown in FIG. 7 is applied to a moving body such as a vehicle. The device 80 includes an imaging device 800 (an example of the photoelectric conversion device 100 or the photoelectric conversion system) and a signal processing device (processing device) that processes signals from the imaging device 800. The device 80 includes an image processing unit 801 that performs image processing on multiple pieces of image data acquired by the imaging device 800, and a parallax calculation unit 802 that calculates parallax (phase difference of parallax images) from the multiple pieces of image data acquired by the device 80. The information processing device 30 according to the above-described embodiment may be included in the imaging device 800 or the image processing unit 801. The device 80 also includes a distance measurement unit 803 that calculates the distance to an object based on the calculated parallax, and a collision determination unit 804 that determines whether or not there is a possibility of a collision based on the calculated distance. Here, the parallax calculation unit 802 and the distance measurement unit 803 are an example of a distance information acquisition unit that acquires distance information to an object. That is, the distance information is information related to the parallax, the defocus amount, the distance to the object, etc. The collision determination unit 804 may determine the possibility of a collision using any of this distance information. The distance information acquisition unit may be realized by dedicated hardware or a software module. Furthermore, it may be realized by an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or a combination thereof.

[0087] The device 80 is connected to a vehicle information acquisition device 810 and can acquire vehicle information such as vehicle speed, yaw rate, and steering angle. The device 80 is also connected to a control ECU 820, which is a control device that outputs a control signal to generate a braking force for the vehicle based on the determination result of the collision determination unit 804. The device 80 is also connected to an alarm device 830 that issues an alarm to the driver based on the determination result of the collision determination unit 804. For example, if the determination result of the collision determination unit 804 indicates a high collision possibility, the control ECU 820 performs vehicle control to avoid the collision and mitigate damage by applying the brakes, releasing the accelerator, suppressing engine output, etc. The alarm device 830 warns the user by sounding an alarm, displaying alarm information on the screen of a car navigation system, etc., or vibrating the seat belt or steering wheel. The device 80 functions as a control means for controlling the operation of controlling the vehicle as described above.

[0088] In this embodiment, the device 80 captures images of the surroundings of the vehicle, for example, the front or rear. Fig. 13(b) shows the device when capturing an image of the area in front of the vehicle (image capturing range 850). A vehicle information acquisition device 810, which serves as an image capturing control means, sends an instruction to the device 80 or the image capturing device 800 to perform an image capturing operation. This configuration can further improve the accuracy of distance measurement.

[0089] Although the above describes an example of control to prevent collision with other vehicles, the present invention can also be applied to control of automatic driving by following other vehicles, control of automatic driving to prevent deviation from a lane, etc. Furthermore, the present invention is not limited to vehicles such as automobiles, but can be applied to moving objects (moving devices) such as ships, aircraft, artificial satellites, industrial robots, and consumer robots. In addition, the present invention can be applied to a wide range of devices that use object recognition or biometric recognition, such as intelligent transport systems (ITS) and surveillance systems, without being limited to moving objects.

[0090] [Modified embodiment] The present invention is not limited to the above-described embodiments and can be modified in various ways. For example, an example in which part of the configuration of one embodiment is added to another embodiment, or an example in which part of the configuration of one embodiment is replaced with part of the configuration of another embodiment, is also an embodiment of the present invention.

[0091] The disclosure of this specification includes the complement of the concepts described in this specification. In other words, if this specification states, for example, that "A is B" (A=B), then this specification is deemed to disclose or suggest that "A is not B" even if the statement that "A is not B" (A≠B) is omitted. This is because when "A is B," it is assumed that the case where "A is not B" is taken into consideration.

[0092] The disclosure of this specification includes the following configurations or methods. (Configuration 1) a first storage unit that stores first array data based on pixel values ​​output from each of the plurality of pixels; a second storage unit that stores second array data including a plurality of coefficients used to correct the pixel values; a first calculation unit that corrects the pixel value of a first target pixel based on a pixel value of the first target pixel in the first array data, a pixel value of a second target pixel adjacent to the first target pixel in the first array data, and the second array data; a third storage unit that stores third array data based on the pixel values ​​output from the first calculation unit; a second calculation unit that detects a third target pixel from the third array data; a third calculation unit that corrects the pixel value of the third target pixel based on the third array data; An information processing device comprising: (Configuration 2) The correction in the first calculation unit includes multiplication of pixel values ​​of the first array data by coefficients of the second array data. 2. The information processing device according to configuration 1, (Configuration 3) The correction in the first calculation unit includes a convolution calculation of the first array data and the second array data. 3. The information processing device according to configuration 1 or 2. (Configuration 4) The number of rows and the number of columns of the second array data are both 3 or more. 4. The information processing device according to any one of configurations 1 to 3. (Configuration 5) The number of rows and the number of columns of the second array data are both odd numbers. 5. The information processing device according to configuration 4. (Configuration 6) The correction in the third calculation unit includes a process of replacing the pixel value of the third target pixel with another value. 6. The information processing device according to any one of configurations 1 to 5. (Configuration 7) The correction in the third calculation unit includes a process of replacing the pixel value of the third target pixel with an average value or a median value of pixel values ​​of a plurality of pixels surrounding the third target pixel. 7. The information processing device according to configuration 6. (Configuration 8) a fourth calculation unit that subtracts a black level from the pixel value output from the first calculation unit, The third array data is generated based on pixel values ​​after the black level is subtracted. 8. The information processing device according to any one of configurations 1 to 7. (Configuration 9) The black level is calculated based on pixel values ​​output from light-shielded pixels. 9. The information processing device according to configuration 8. (Configuration 10) Further, a fifth calculation unit is provided that performs noise reduction processing on the output of the second calculation unit or the third calculation unit. 10. The information processing device according to any one of configurations 1 to 9. (Configuration 11) The fifth calculation unit performs the noise reduction process by smoothing the image using a spatial filter. 11. The information processing device according to configuration 10. (Configuration 12) Further, a sixth calculation unit corrects the nonlinearity of the first array data before the correction in the first calculation unit. 12. The information processing device according to any one of configurations 1 to 11. (Configuration 13) The correction in the first calculation unit is performed based on the first array data before the nonlinearity correction, the first array data after the nonlinearity correction, and the second array data. 13. The information processing device according to configuration 12. (Configuration 14) the second storage unit includes a plurality of types of the second array data; One of the plurality of types of second sequence data is selectively input to the first calculation unit. 14. The information processing device according to any one of configurations 1 to 13. (Configuration 15) a first calculation unit that corrects the pixel value of a first target pixel based on a pixel value of the first target pixel in first array data based on pixel values ​​output from each of a plurality of pixels, a pixel value of a second target pixel adjacent to the first target pixel in the first array data, and second array data that includes a plurality of coefficients; a third calculation unit that corrects a third target pixel detected from third array data based on the pixel value output from the first calculation unit; An information processing device comprising: (Configuration 16) the plurality of pixels; The information processing device according to any one of configurations 1 to 15, A photoelectric conversion device comprising: (Configuration 17) the photoelectric conversion device according to Configuration 16; an optical device corresponding to the photoelectric conversion device; a control device that controls the photoelectric conversion device; a processing device that processes a signal output from the photoelectric conversion device; a display device that displays information obtained by the photoelectric conversion device; a storage device that stores information obtained by the photoelectric conversion device; and and a mechanical device that operates based on information obtained by the photoelectric conversion device. (Configuration 18) 18. The device according to claim 17, wherein the processing device acquires distance information from the photoelectric conversion device to the subject. (Method 19) acquiring first array data based on pixel values ​​output from each of the plurality of pixels; obtaining second array data including a plurality of coefficients used to correct the pixel values; correcting the pixel value of the first target pixel based on a pixel value of the first target pixel in the first array data, a pixel value of a second target pixel adjacent to the first target pixel in the first array data, and the second array data; obtaining third array data based on the corrected pixel values ​​of the first target pixels; detecting a third target pixel from the third array data; correcting the pixel value of the third target pixel based on the third array data; An information processing method comprising: (Configuration 20) A program for causing a computer to execute the information processing method described in Method 19.

[0093] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0094] It should be noted that the above-described embodiments are merely examples of specific embodiments for carrying out the present invention, and the technical scope of the present invention should not be construed as being limited by these embodiments. In other words, the present invention can be carried out in various forms without departing from its technical concept or main features. [Explanation of symbols]

[0095] 30 Information processing equipment 301 1st memory section 302 2nd memory section 303 Crosstalk correction section 304 Third memory section 305 Abnormal pixel detection unit 306 Abnormal pixel correction unit

Claims

1. a first storage unit that stores first array data based on pixel values ​​output from each of the plurality of pixels; a second storage unit configured to store second array data including a plurality of coefficients used for correcting the pixel values; a first calculation unit that corrects the pixel value of a first target pixel based on a pixel value of the first target pixel in the first array data, a pixel value of a second target pixel adjacent to the first target pixel in the first array data, and the second array data; a third storage unit that stores third array data based on the pixel values ​​output from the first calculation unit; a second calculation unit that detects a third target pixel from the third array data; a third calculation unit that corrects the pixel value of the third target pixel based on the third array data; An information processing device comprising:

2. The correction in the first calculation unit includes multiplication of pixel values ​​of the first array data by coefficients of the second array data.

2. The information processing apparatus according to claim 1, wherein:

3. The correction in the first calculation unit includes a convolution calculation of the first array data and the second array data.

2. The information processing apparatus according to claim 1, wherein:

4. The number of rows and the number of columns of the second array data are both 3 or more.

2. The information processing apparatus according to claim 1, wherein:

5. The number of rows and the number of columns of the second array data are both odd numbers.

5. The information processing apparatus according to claim 4,

6. The correction in the third calculation unit includes a process of replacing the pixel value of the third target pixel with another value.

2. The information processing apparatus according to claim 1, wherein:

7. The correction in the third calculation unit includes a process of replacing the pixel value of the third target pixel with an average value or a median value of pixel values ​​of a plurality of pixels surrounding the third target pixel.

7. The information processing apparatus according to claim 6,

8. a fourth calculation unit that subtracts a black level from the pixel value output from the first calculation unit, The third array data is generated based on pixel values ​​after the black level is subtracted.

2. The information processing apparatus according to claim 1, wherein:

9. The black level is calculated based on pixel values ​​output from light-shielded pixels.

9. The information processing apparatus according to claim 8,

10. Further, a fifth calculation unit is provided to perform noise reduction processing on the output of the second calculation unit or the third calculation unit.

2. The information processing apparatus according to claim 1, wherein:

11. The fifth calculation unit performs the noise reduction process by smoothing processing using a spatial filter.

11. The information processing apparatus according to claim 10,

12. The method further includes a sixth calculation unit that corrects the nonlinearity of the first array data before the correction in the first calculation unit.

2. The information processing apparatus according to claim 1, wherein:

13. The correction in the first calculation unit is performed based on the first array data before the nonlinearity correction, the first array data after the nonlinearity correction, and the second array data.

13. The information processing apparatus according to claim 12.

14. the second storage unit includes a plurality of types of the second array data; One of the plurality of types of second array data is selectively input to the first calculation unit.

2. The information processing apparatus according to claim 1, wherein:

15. a first calculation unit that corrects the pixel value of a first target pixel based on a pixel value of the first target pixel in first array data based on pixel values ​​output from each of a plurality of pixels, a pixel value of a second target pixel adjacent to the first target pixel in the first array data, and second array data that includes a plurality of coefficients; a third calculation unit that corrects a third target pixel detected from third array data based on the pixel value output from the first calculation unit; An information processing device comprising:

16. the plurality of pixels; An information processing device according to any one of claims 1 to 15; A photoelectric conversion device comprising:

17. The photoelectric conversion device according to claim 16 ; an optical device corresponding to the photoelectric conversion device; a control device that controls the photoelectric conversion device; a processing device that processes a signal output from the photoelectric conversion device; a display device that displays information obtained by the photoelectric conversion device; a storage device that stores information obtained by the photoelectric conversion device; and and a mechanical device that operates based on information obtained by the photoelectric conversion device.

18. 18. The device according to claim 17, wherein the processing device acquires distance information from the photoelectric conversion device to a subject.

19. acquiring first array data based on pixel values ​​output from each of the plurality of pixels; obtaining second array data including a plurality of coefficients used to correct the pixel values; correcting the pixel value of the first target pixel based on a pixel value of the first target pixel in the first array data, a pixel value of a second target pixel adjacent to the first target pixel in the first array data, and the second array data; obtaining third array data based on the corrected pixel values ​​of the first target pixels; detecting a third target pixel from the third array data; correcting the pixel value of the third target pixel based on the third array data; An information processing method comprising:

20. A program for causing a computer to execute the information processing method according to claim 19.

Images