Information processing apparatus and information processing method

Abstract

To provide an information processing apparatus that can more suitably correct an abnormal pixel.SOLUTION: An information processing apparatus has: a first connection unit that, when the difference between pixel values of adjacent two pixels, of a plurality of pixels arranged to form a plurality of rows and a plurality of columns, is within a first range, determines that the two pixels belong to the same first connected pixel group; a pixel value determination unit that, when the order of a pixel value of a target pixel is higher than an upper limit of a second range or lower than a lower limit of the second range, in a neighboring pixel group including the target pixel and a neighboring pixel arranged in the neighborhood of the target pixel, determines that a first condition is satisfied; a first connection number determination unit that, when the number of pixels belonging to the first connected pixel group including the target pixel is equal to or less than a first threshold, determines that a second condition is satisfied; and a correction unit that corrects the pixel value of the target pixel when at least the first condition and the second condition are satisfied, and does not correct the pixel value of the target pixel when the first condition is not satisfied or when the second condition is not satisfied.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing device and an information processing method. [Background technology]

[0002] Some imaging devices, such as digital cameras, have an array of multiple imaging elements that convert incident light into electrical signals through photoelectric conversion. In imaging devices that include multiple imaging elements, the output characteristics of signals from some of the imaging elements may differ from those of the other imaging elements, and some of the imaging elements may output abnormally high or low signals.

[0003] If such an abnormal signal output from an image sensor is used without correction, the image quality may be degraded by the abnormal pixels. For this reason, Cited Documents 1 to 3 propose methods for detecting and correcting abnormal pixels. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2005-236749 A [Patent Document 2] JP 2004-015191 A [Patent Document 3] JP 2005-223796 A Summary of the Invention [Problem to be solved by the invention]

[0005] The occurrence rate and distribution of abnormal pixels are diverse, and a method for dynamically correcting abnormal pixels that takes into account various occurrence modes is required.

[0006] An object of the present invention is to provide an information processing apparatus and an information processing method that can more appropriately correct abnormal pixels. [Means for solving the problem]

[0007] According to one disclosure of the present specification, there is provided an information processing device comprising: a first connection unit that determines that two adjacent pixels in a plurality of pixels arranged in a plurality of rows and a plurality of columns belong to the same first connected pixel group if the difference in pixel values ​​between the two adjacent pixels is within a first range; a pixel value determination unit that determines that a first condition is satisfied if the ranking of the pixel value of the target pixel in a neighborhood pixel group including the target pixel and a neighborhood pixel arranged near the target pixel is higher than an upper limit of a second range or lower than a lower limit of the second range; a first connection number determination unit that determines that a second condition is satisfied if the number of pixels belonging to the first connected pixel group including the target pixel is equal to or less than a first threshold; and a correction unit that corrects the pixel value of the target pixel if at least the first condition and the second condition are satisfied, and does not correct the pixel value of the target pixel if the first condition is not satisfied or the second condition is not satisfied.

[0008] According to one disclosure of the present specification, there is provided an information processing method comprising the steps of: determining that two adjacent pixels in a plurality of pixels arranged in a plurality of rows and a plurality of columns belong to the same first connected pixel group if the difference in pixel values ​​between the two adjacent pixels is within a first range; determining that a first condition is satisfied if the ranking of the pixel value of the target pixel in a neighboring pixel group including the target pixel and neighboring pixels arranged near the target pixel is higher than the upper limit of a second range or lower than the lower limit of the second range; determining that a second condition is satisfied if the number of pixels belonging to the first connected pixel group including the target pixel is equal to or less than a first threshold; and correcting the pixel value of the target pixel if at least the first condition and the second condition are satisfied, wherein the pixel value of the target pixel is not corrected if the first condition is not satisfied or the second condition is not satisfied. Effect of the Invention

[0009] According to the present invention, an information processing device and an information processing method capable of correcting abnormal pixels more suitably are provided. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing a hardware configuration of an information processing device according to a first embodiment. [Diagram 2] 3 is a functional block diagram relating to a correction processing function of the information processing device according to the first embodiment. FIG. [Diagram 3] FIG. 2 is a diagram illustrating a pixel arrangement according to the first embodiment. [Figure 4] 5 is a flowchart showing a correction process executed by the information processing device according to the first embodiment. [Diagram 5] 5A to 5C are diagrams illustrating examples of pixel values ​​and pixel states in the correction process according to the first embodiment. [Figure 6] 5A to 5C are diagrams illustrating examples of pixel values, pixel states, and data holding methods in the correction process according to the first embodiment. [Figure 7] 11 is a graph showing an example of threshold values ​​in the correction process according to the first embodiment. [Figure 8] 5A to 5C are diagrams illustrating examples of pixel values ​​and pixel states in the correction process according to the first embodiment. [Figure 9] FIG. 11 is a functional block diagram relating to a correction processing function of an information processing device according to a second embodiment. [Figure 10] 10 is a flowchart showing a correction process executed by an information processing device according to a second embodiment. [Figure 11] 13A to 13C are diagrams illustrating examples of pixel values ​​in the correction process according to the second embodiment. [Figure 12] 13 is a graph showing an example of threshold values ​​in the correction process according to the second embodiment. [Figure 13] FIG. 11 is a diagram illustrating a pixel arrangement according to a modified example of the second embodiment. [Figure 14] FIG. 11 is a functional block diagram relating to a correction processing function of an information processing device according to a third embodiment. [Figure 15] 13 is a flowchart showing a correction process executed by an information processing device according to the third embodiment. [Figure 16]13A to 13C are diagrams illustrating examples of pixel values ​​in the correction process according to the third embodiment. [Figure 17] FIG. 13 is a functional block diagram relating to a correction processing function of an information processing device according to a fourth embodiment. [Figure 18] 13 is a flowchart showing a correction process executed by an information processing device according to a fourth embodiment. [Figure 19] 13A to 13C are diagrams illustrating examples of pixel values ​​in the correction process according to the fourth embodiment. [Figure 20] FIG. 11 is a schematic diagram showing the overall configuration of a photoelectric conversion device according to a fifth embodiment. [Figure 21] FIG. 13 is a schematic block diagram showing an example of the configuration of a sensor substrate according to a fifth embodiment. [Figure 22] FIG. 13 is a schematic block diagram showing an example of the configuration of a circuit board according to a fifth embodiment. [Diagram 23] FIG. 13 is a schematic block diagram showing an example of the configuration of one pixel of a photoelectric conversion unit and a pixel signal processing unit according to the fifth embodiment. [Figure 24] 13A to 13C are diagrams illustrating the operation of the avalanche photodiode according to the fifth embodiment. [Diagram 25] FIG. 13 is a block diagram of an apparatus according to a sixth embodiment. [Figure 26] FIG. 13 is a block diagram of an apparatus according to a seventh embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The same elements or corresponding elements in multiple drawings are denoted by the same reference numerals, and the description thereof may be omitted or simplified.

[0012] [First embodiment] Fig. 1 is a block diagram showing a hardware configuration of an information processing device 1 according to this embodiment. The information processing device 1 performs information processing such as correction on pixel data acquired by a photoelectric conversion device. Fig. 1 shows an example in which information processing in the information processing device 1 is performed by a general computer, but the information processing function of the information processing device 1 may be realized by other devices. For example, the information processing device 1 may be an image processing device specialized for image processing functions, or may be an image processing unit incorporated in a photoelectric conversion device.

[0013] The information processing device 1 has a data input unit 121, a data storage unit 122, a display unit 123, and an input unit 124. The information processing device 1 also has a CPU (Central Processing Unit) 125, a RAM (Ramdom Access Memory) 126, and a ROM (Read Only Memory) 127. The information processing device 1 also has a communication unit 128 and an information processing unit 129. These units are connected to each other via a bus. FIG. 1 shows an example of the configuration of the information processing device 1, and some of the units shown in FIG. 1 may be arranged in a device external to the information processing device 1, and devices other than the units shown in FIG. 1 may be further arranged in the information processing device 1.

[0014] The data input unit 121 includes a photoelectric conversion device such as an image sensor. The photoelectric conversion device includes a plurality of pixel circuits arranged to form a plurality of rows and a plurality of columns. Each of the plurality of pixel circuits photoelectrically converts incident light to generate an electric signal. The electric signal generated by each of the plurality of pixel circuits is converted into a digital signal (pixel value). In this way, the data input unit 121 has a function of generating pixel values ​​of each of the pixels arranged to form a plurality of rows and a plurality of columns as image data and inputting the pixel values ​​to the information processing device 1. Note that, when the photoelectric conversion device is arranged outside the information processing device, the data input unit 121 may be an interface that acquires image data from the photoelectric conversion device.

[0015] The data storage unit 122 is a recording medium that holds data used in information processing, such as image data and parameters. The recording medium may be, for example, a computer-readable non-volatile recording medium such as a hard disk, a solid state drive (SSD), or a flexible disk. The recording medium may be an optical disk such as a compact disc (CD)-ROM, a recordable CD-R (CD-R), a digital versatile disc (DVD), or a Blu-ray (registered trademark). The recording medium may be a semiconductor memory such as a memory card, a compact flash (CF) card, a smart media, an SD card, a memory stick, an xD picture card, or a universal serial bus (USB) memory. The data storage unit 122 may store data other than the program and image data. Alternatively, a part of the storage capacity of the RAM 126 may be used as the data storage unit 122. Alternatively, an external recording device that is communicatively connected to the information processing device 1 by the communication unit 128 may be used as the data storage unit 122.

[0016] The display unit 123 is a device that displays an image before image processing, an image after image processing, or an image for operation such as a graphical user interface. The display unit 123 may be a CRT (Cathode-Ray Tube) display, a liquid crystal display, an organic EL (Electro-Luminescence) display, or the like. Alternatively, the display unit 123 may be an external display that is provided outside the information processing device 1 and is communicatively connected via a cable or the like.

[0017] The input unit 124 is a device for a user to input instructions or data. The input unit 124 includes a keyboard, a pointing device, and the like. Examples of pointing devices include a mouse, a trackball, a trackpad, and a tablet. Alternatively, when the information processing device 1 of the present embodiment is applied to a device such as a digital camera or a printer, the input unit 124 may be a button, a dial, and the like. The input unit 124 may also be a software keyboard displayed on a screen by software. In this case, the input unit 124 may be configured so that a user operates a button, a dial, or a pointing device to input characters into the software keyboard.

[0018] Note that, like a touch screen device, the same device may have the functions of both the display unit 123 and the input unit 124. In that case, information input to the information processing device 1 by a user operating an operation screen displayed on the touch screen device is treated as input information from the input unit 124.

[0019] The input unit 124 may also be configured to receive instructions from a user through gesture recognition processing. In this case, the input unit 124 includes an input device that inputs an image captured using visible light or infrared light, and a recognition device that recognizes a user's gesture from the image and converts it into a command. The data input unit 121 may also function as the input device. The recognition device may be added as a dedicated gesture recognition circuit, or may be realized by the CPU 125 executing a gesture recognition program.

[0020] The input unit 124 may also be configured to receive user instructions through voice recognition processing. In this case, the input unit 124 includes a microphone device and a recognition device that recognizes the user's speech from voice data acquired by the microphone device and converts it into a command. The recognition device may be added as a dedicated voice recognition circuit, or may be realized by the CPU 125 executing a voice recognition program.

[0021] The gesture recognition process and the voice recognition process described above may be performed by a device external to the information processing device 1. In this case, the information processing device 1 is connected for communication with the external device or a server on a network via the communication unit 128, and transmits image data or voice data to the external device or the server. The external device or the server is configured to receive the image data or voice data according to a predetermined communication procedure, perform a recognition process, and transmit data indicating the recognition result to the information processing device 1.

[0022] The CPU 125 is a processor that controls each part of the information processing device 1 and processes information. The RAM 126 and the ROM 127 provide the CPU 125 with programs, data, a working area, and the like required for control and information processing. When a program is stored in the data storage unit 122 or the ROM 127, the program is once loaded into the RAM 126 and then executed by the CPU 125. The information processing device 1 may also be configured to receive a program from the outside via the communication unit 128. In that case, the program is once stored in the data storage unit 122 and then loaded into the RAM 126, or is directly loaded from the communication unit 128 into the RAM 126 and then executed by the CPU 125.

[0023] 1 illustrates only one block representing the CPU 125, the number of CPUs 125 is not limited to one. That is, the information processing device 1 may include a plurality of CPUs 125.

[0024] The communication unit 128 is an interface for performing communication between devices. The communication unit 128 may be based on a wired communication method such as a wired network, RS-232C, USB, IEEE1284, IEEE1394, or telephone line. Alternatively, the communication unit 128 may be based on a wireless communication method such as infrared (IrDA), IEEE802.11a, IEEE802.11b, IEEE802.11g, IEEE802.11n, IEEE802.11ac, or IEEE802.11ax. Alternatively, the communication unit 128 may be based on another wireless communication method such as Bluetooth (registered trademark), UWB (Ultra Wide Band), a wireless telephone line, or NFC (Near Field Communication). Alternatively, the communication unit 128 may be based on an inter-chip communication method such as I2C (Inter-Integrated Circuit) or SPI (Serial Peripheral Interface).

[0025] Furthermore, the communication unit 128 may support more than one communication method, not just one. For example, the communication unit 128 may be configured to support two or more of the various communication methods described above. Alternatively, the information processing device 1 may have a plurality of communication units 128 that support different communication methods. Even in this case, the plurality of communication units 128 will be collectively referred to as the communication unit 128 in the present embodiment.

[0026] The information processing unit 129 is a signal processing circuit including a DSP (Digital Signal Processor), a logic circuit, etc. Alternatively, the information processing unit 129 may be a GPU (Graphics Processing Unit). The information processing unit 129 performs arithmetic processing on image data input from the data input unit 121 or image data held in the RAM 126, the data storage unit 122, etc. The processing result in the information processing unit 129 may be output to the RAM 126, the data storage unit 122, the display unit 123, etc., or may be output to an external device of the information processing device 1 via the communication unit 128. Note that when the load of the arithmetic processing is small (when a high arithmetic speed is not required or when the amount of data to be calculated is small), the CPU 125 may also serve as the information processing unit 129.

[0027] Although not shown in Fig. 1, a register circuit may be added as necessary. The register circuit holds operation parameters of the CPU 125 or the information processing unit 129. The values ​​held in the register circuit may be set by the CPU 125 or the information processing unit 129, or may be set by an external device via the communication unit 128.

[0028] In the case where the information processing device 1 is a camera device, the display unit 123 may have a function of displaying a preview image of a subject and a function of displaying a captured image. However, these images may be displayed on another device (e.g., a smartphone) connected via the communication unit 128. In that case, the display unit 123 may be omitted. Similarly, the other device connected via the communication unit 128 may receive an instruction from a user, and the information processing device 1 may receive a command corresponding to the instruction via the communication unit 128, so that the information processing device 1 performs an operation corresponding to the command. In this case, the process of identifying an operation corresponding to the command from the command may be performed by the CPU 125 or the information processing unit 129.

[0029] Alternatively, when processing and control by software are not required, the CPU 125, the ROM 127, etc. may be omitted. An example of such a case is when a logic circuit for realizing the necessary processing and control is arranged in the information processing unit 129.

[0030] Alternatively, the information processing device 1 may be a stacked sensor in which a substrate on which a photoelectric conversion element is arranged and a substrate on which a signal processing circuit is arranged are stacked. In this case, a logic circuit, a memory, a CPU 125, and the like can be arranged inside the stacked sensor. In such a configuration, the data input unit 121 may be configured to include a photoelectric conversion element and its peripheral circuits. In addition, the data storage unit 122, the CPU 125, the RAM 126, the ROM 127, the communication unit 128, and the information processing unit 129 may be arranged inside the stacked sensor. In this case, the display unit 123 and the input unit 124 may be omitted.

[0031] 2 is a functional block diagram relating to a correction processing function of the information processing device 1 according to this embodiment. The information processing device 1 has a pixel data holding unit 131, a correction data generating unit 132, a first pixel value determining unit 133, a first pixel state holding unit 134, a first connecting unit 135, a correction unit 136, and a first connecting number determining unit 137.

[0032] The functions of the pixel data holding unit 131 and the first pixel state holding unit 134 are realized, for example, by the data storage unit 122 in Fig. 1. The functions of the correction data generation unit 132, the first pixel value determination unit 133, the first connection unit 135, the correction unit 136, and the first connection number determination unit 137 are realized, for example, by the information processing unit 129 in Fig. 1. The functions of the correction data generation unit 132, the first pixel value determination unit 133, the first connection unit 135, the correction unit 136, and the first connection number determination unit 137 may be realized by the CPU 125 executing a correction processing program. This correction processing program may be stored in advance in the data storage unit 122 or the ROM 127, or may be obtained from another device via the communication unit 128. The operation of each of these units will be described later.

[0033] 3 to 8(b), the correction process executed by the information processing device 1 according to this embodiment will be described. First, prior to describing a specific processing procedure, the configuration of pixel data acquired by the photoelectric conversion device and held in the pixel data holding unit 131 will be described.

[0034] Fig. 3 is a diagram showing a pixel arrangement according to this embodiment. Fig. 3 shows a plurality of pixels P11 to P56 constituting image data held in the pixel data holding unit 131, and the pixel values ​​of each pixel. The plurality of pixels P11 to P56 are arranged to form a plurality of rows and a plurality of columns. For the sake of simplicity, only five rows and six columns of pixels are shown in Fig. 3, but in reality, more pixels may be arranged.

[0035] In the symbols attached to the pixels, the two numbers after "P" indicate the row number and column number, respectively. In Fig. 3, the values ​​written in the circles representing pixels P11 to P56 indicate the pixel values. For example, "20" is written in the circle of pixel P11, which indicates that the pixel value of pixel P11 is 20.

[0036] In the description of this embodiment, for the sake of simplicity, the pixel array is described assuming that the image to be processed is a grayscale image, but the image to be processed may be a color image. For example, if the pixel array is an array of red, green, and blue color channels, the correction process of this embodiment can be similarly applied by performing independent processing on each color channel. Also, if the pixel array is a Bayer array, the correction process of this embodiment can be similarly applied by performing independent processing on each color channel.

[0037] The abnormal pixels to be corrected in this embodiment will be described. An abnormal pixel means a pixel that has a unique pixel value compared to the pixels nearby the pixel. The abnormal pixel may include so-called white defects and black defects. A white defect is an abnormal pixel that has a pixel value that is significantly larger than the pixels nearby, and a black defect is an abnormal pixel that has a pixel value that is significantly smaller than the pixels nearby.

[0038] However, a pixel having a unique pixel value is not necessarily an independent single pixel. In other words, two or more abnormal pixels may be connected, so that the connected abnormal pixels must be taken into consideration when correcting the abnormal pixels. Here, two abnormal pixels being connected means that the two abnormal pixels are in a continuous positional relationship in either the vertical, horizontal, or diagonal direction in FIG. 3.

[0039] The following formula (1) indicates the probability that at least one of eight pixels in the vicinity of a certain abnormal pixel is also an abnormal pixel.

number

[0040] In this case, the expected number of connected abnormal pixels is roughly expressed by the following formula (2).

number

[0041] For example, in an image with 4096 pixel columns and 2048 pixel rows, even if the ratio P is a very small value of 0.1%, the expected number of anomalous pixels calculated by formula (2) is about 67. If we assume that all of these anomalous pixels are pairs of two anomalous pixels and that no three or more anomalous pixels are connected, there will be about 33 pairs of connected anomalous pixels in the image. This corresponds to a state in which, when an image is divided into 32 8x4 images, each divided image will have about one pair of consecutive anomalous pixels. In other words, the probability of occurrence of an event in which anomalous pixels are adjacent to each other cannot be ignored.

[0042] Furthermore, we will discuss the probability that there are two or more abnormal pixels in the eight pixels surrounding an abnormal pixel, i.e., the probability that three or more abnormal pixels are connected. Equation (1) shows the probability that there is one or more abnormal pixels in the eight pixels surrounding an abnormal pixel. Therefore, by subtracting the probability that there is only one abnormal pixel in the eight pixels surrounding an abnormal pixel from equation (1), we can calculate the probability that there are two or more abnormal pixels in the eight pixels surrounding an abnormal pixel.

[0043] The probability that there is only one abnormal pixel among the eight pixels neighboring an abnormal pixel is expressed by the following equation (3).

number

[0044] Therefore, the expected number of occurrences of three or more connected abnormal pixels is roughly expressed by the following formula (4).

number

[0045] As an example, in an image having 4096 pixel columns and 2048 pixel rows, the expected value of the number of abnormal pixels in which three or more pixels are connected is calculated. When P=0.1%, the expected value of the number of abnormal pixels in which three or more pixels are connected is 0.234. When P=0.5%, the expected value of the number of abnormal pixels in which three or more pixels are connected is 28.8. When P=1.0%, the expected value of the number of abnormal pixels in which three or more pixels are connected is 226.

[0046] As described above, even if the ratio P is a small value of 1% or less, the possibility of the presence of three or more connected abnormal pixels cannot be ignored. Furthermore, depending on the values ​​of P, H, and W, it may be necessary to consider four or more connected abnormal pixels. As described above, taking into account the case where multiple abnormal pixels are connected, the information processing device 1 of this embodiment performs correction processing in response to N or less connected abnormal pixels (N is a positive integer).

[0047] 4 is a flowchart showing the correction process executed by the information processing device 1 according to this embodiment. The correction process method according to this embodiment will be described with reference to the flowchart of FIG.

[0048] The process of FIG. 4 is a process of detecting abnormal pixels from an image captured by a photoelectric conversion device and correcting the detected abnormal pixels. The process of FIG. 4 is started, for example, after the photoelectric conversion device captures an image and the image is stored in the pixel data storage unit 131. Note that in the process of this embodiment, position information of abnormal pixels acquired in advance (so-called flaw map) and an image captured in a light-shielded state (so-called dark image) are not used. Also, it is assumed that the pixel data storage unit 131 has a storage capacity sufficient to simultaneously store pixel values ​​for each of a plurality of pixels in an entire image for one frame. Similarly, it is assumed that the first pixel state storage unit 134 has a storage capacity sufficient to simultaneously store pixel states in an entire image for one frame.

[0049] In step S11, the first connecting unit 135 acquires pixel values ​​of each of a plurality of pixels constituting an image from the pixel data storage unit 131. Then, the first connecting unit 135 determines that two adjacent pixels belong to the same first connected pixel group when the pixel values ​​of the two adjacent pixels are close to each other (the difference between the pixel values ​​of the two adjacent pixels is within a first range) (first connecting process). The first pixel state storage unit 134 stores information indicating the determination result of the first connecting process. In this process, the pixel values ​​of a pixel region of three rows and three columns including the pixel to be processed (pixel of interest) and its eight neighboring pixels are referenced for each pixel, and the pixel values ​​are compared and determined. The determination result is stored in a data region of the first pixel state storage unit 134 corresponding to the referenced pixel region of three rows and three columns.

[0050] Hereinafter, the process in which the first connecting unit 135 determines that two pixels belong to the same first connected pixel group and the first pixel state holding unit 134 holds the pixel states of these pixels may be expressed as "connecting". Among the multiple pixels after the connection process is completed, those with a connection count of N (N is a positive integer) or less are called isolated points. Here, the connection count is the number of pixels that can be reached by tracing connected pixels from a certain pixel, that is, the number of pixels that belong to the same first connected pixel group. For example, when a certain pixel is not connected to any pixel, the connection count is 1. The value of N indicates the upper limit number of abnormal pixel connections to be considered, and can be set appropriately depending on the required accuracy of abnormal pixel detection, image quality, etc.

[0051] First, with reference to Figures 5(a) to 7(d), we will explain an example of the first connecting process when the value of the threshold N for the number of connections in determining an isolated point is 1, i.e., when a pixel that is not connected to other pixels is considered to be an isolated point.

[0052] FIG. 5(a) is a diagram showing an example of pixel values ​​in the correction process according to this embodiment. The notation in FIG. 5(a) is the same as in FIG. 3. FIG. 5(b) is a diagram showing an example of pixel states in the correction process according to this embodiment. FIG. 5(b) shows pixel state data stored in the first pixel state storage unit 134 corresponding to each pixel. The connecting line N1 shows the relationship between two adjacent pixels P11 and P12, and indicates that the pixel value "20" of pixel P11 and the pixel value "22" of pixel P12 are determined to be close values ​​and are connected.

[0053] In FIG. 5(b), the numerical values ​​written in the eight outermost positions in the circle corresponding to each pixel are linked data indicating whether the pixel value of the pixel in question and the pixel value of the adjacent pixel are close to each other. In the linked data, "0" indicates that the pixel value of the pixel in question and the pixel value of the adjacent pixel in that direction are not close to each other and are not linked. Also, "1" indicates that the pixel value of the pixel in question and the pixel value of the adjacent pixel in that direction are close to each other and are linked. Also, for the edge pixels, there are less than eight adjacent pixels. Thus, when there are no adjacent pixels, the value of the linked data is 0.

[0054] FIG. 6(a) is a diagram showing an example of pixel values ​​in the vicinity of pixel P11. FIG. 6(b) is a diagram showing an example of pixel states in the vicinity of pixel P11. As shown in FIG. 6(b), the right side, lower side, and lower right side of the connected data of pixel P11 are 1, and pixel P11 is connected to adjacent pixels by connecting lines N1. Specifically, pixel P11 has a pixel value close to that of pixels P12, P21, and P22, so pixel P11 is connected to pixels P12, P21, and P22. Pixel P11 is a pixel located at an end, and there are no adjacent pixels in any other direction, so the connected data of pixel P11 is 0 except for the right side, lower side, and lower right side.

[0055] Figures 6(c) and 6(d) are diagrams showing a typical method of storing concatenated data. The eight values ​​included in the concatenated data can be stored as 8-bit data by, for example, lining up in a clockwise direction based on a value in a certain direction as shown in Figure 6(c). Figure 6(d) shows the 8-bit data "00011100" ("1C" in hexadecimal) obtained in this way.

[0056] 5(a) and 5(b), the eight values ​​included in the connected data of the hatched pixels P15, P43, and P45 are all 0. In this way, the pixels P15, P43, and P45, whose eight values ​​are all 0, are not connected to other pixels. Since the number of connections of such pixels is 1, that is, N or less, the pixels P15, P43, and P45 are isolated points.

[0057] 7(a) to 7(d) are graphs showing examples of thresholds used to determine whether or not the pixel values ​​of two adjacent pixels are close to each other in the first connecting portion 135. Note that in comparisons of magnitude relationships using thresholds described below, "greater than or equal to" can be interpreted interchangeably, and "less than or equal to" can also be interpreted interchangeably.

[0058] 7(a) to 7(d), the horizontal axis indicates the pixel value of the pixel of interest, and the vertical axis indicates the pixel values ​​of the neighboring pixels adjacent to the pixel of interest. In FIG. 7(a) to 7(d), a line L1 has a slope of 1 and an intercept of 0 (i.e., y=x), and indicates the lower limit threshold of the pixel values ​​of the neighboring pixels to be connected.

[0059] In FIG. 7(a), the line L2 indicates the upper threshold of the pixel value of the neighboring pixel to be connected. The slope of the line L2 is greater than 1, and the intercept of the line L2 is greater than 0. In addition, on the line L2, when x=x1, y=y2. When the pixel value of the pixel of interest is x1, and the pixel value of the neighboring pixel is greater than or equal to y1 (=x1) and less than or equal to y2, the pixel of interest and the neighboring pixel are connected. That is, the first connecting unit 135 connects two pixels when the pixel values ​​of the pixel of interest and the neighboring pixel are in an area between the lines L1 and L2 (the difference between the pixel values ​​of the pixel of interest and the neighboring pixel is within a first range determined by the upper threshold and the lower threshold). At this time, the first pixel state holding unit 134 holds a pixel state representing the connection state by holding "1" as the value of the corresponding bit of the connection data of the pixel of interest and the neighboring pixel. That is, a "1" is held in each of the bit indicating the direction of the neighboring pixel of the linked data corresponding to the pixel of interest and the bit indicating the direction of the pixel of interest of the linked data corresponding to the neighboring pixel.

[0060] The function for determining the threshold value is not limited to that shown in FIG. 7(a). For example, as shown in FIG. 7(b), the upper limit threshold value may be given by a broken line L3 composed of a plurality of line segments. On the straight line L3, when x = x1, y = y3. When the pixel value of the target pixel is x1, the target pixel and the neighboring pixel are connected when the pixel value of the neighboring pixel is greater than or equal to y1 (= x1) and less than or equal to y3.

[0061] Also, for example, as shown in FIG. 7(c), the upper limit threshold value may be given by a curve L4. On the straight line L4, when x = x1, y = y4. When the pixel value of the target pixel is x1, the target pixel and the neighboring pixel are connected when the pixel value of the neighboring pixel is greater than or equal to y1 (= x1) and less than or equal to y4.

[0062] Also, for example, as shown in FIG. 7(d), a lower limit threshold value may be given by a straight line L5 different from the straight line L1. The slope of the straight line L5 is less than 1, and the intercept of the straight line L5 is less than 0. Also, on the straight line L5, when x = x1, y = y5. When the pixel value of the target pixel is x1, the target pixel and the neighboring pixel are connected when the pixel value of the neighboring pixel is greater than or equal to y5 (<x1) and less than or equal to y2.

[0063] In the example of FIG. 7(d), one or both of the straight line L2 and the straight line L5 may be replaced by a broken line or may be replaced by a curve. Also, the straight line L2 and the straight line L5 may be replaced by different types of lines.

[0064] A method of acquiring a threshold (upper or lower threshold) will be described. When the graph shape of the threshold is simple, the first connection unit 135 may calculate the threshold by a function that inputs the pixel value of the pixel of interest and outputs the value of the threshold. In this case, it is desirable to use the coefficient of a term included in the function as a parameter so that the function can be adjusted. In addition, it is not essential that the mathematical formula of this function is the same over the entire graph, and for example, this function may be a piecewise linear function. In addition, the first connection unit 135 may include a lookup table in which the pixel value of the pixel of interest and the threshold are associated with each other. In this case, the first connection unit 135 acquires the threshold by referring to the pixel value of the pixel of interest as an index. Here, the lookup table may hold values ​​for all indexes, or may hold values ​​only for indexes of some representative points. When the lookup table holds values ​​only for indexes of representative points, the first connection unit 135 may calculate thresholds for indexes other than the representative points by interpolation.

[0065] Referring again to FIG. 4, the processing after the first connection processing in step S11 will be described. In step S12 after the first connection processing in step S11 is completed, the first pixel value determination unit 133 determines whether or not the pixel value of the pixel of interest among the multiple pixels is unique (determination of the first condition). If it is determined that the pixel value of the pixel of interest is unique (YES in step S12), the processing proceeds to step S13. If it is determined that the pixel value of the pixel of interest is not unique (NO in step S12), the processing proceeds to step S15. In this case, the pixel of interest is not a correction target.

[0066] The determination in step S12 will be described in more detail. Note that this description is also based on the assumption that the value of N is 1, that is, that a pixel that is not connected to other pixels is regarded as an isolated point.

[0067] The first pixel value determination unit 133 compares pixel values ​​of a pixel region of three rows and three columns (neighborhood pixel group) including the pixel of interest and its eight neighboring pixels. The eight neighboring pixels are arranged to surround the pixel of interest. If the pixel value of the pixel of interest is the maximum among the pixel values ​​of this pixel region, the first pixel value determination unit 133 determines that the pixel value of the pixel of interest is a convex singular value. If the pixel value of the pixel of interest is the minimum among the pixel values ​​of this pixel region, the first pixel value determination unit 133 determines that the pixel value of the pixel of interest is a concave singular value. If the pixel value of the pixel of interest is neither the maximum nor the minimum among the pixel values ​​of this pixel region, the first pixel value determination unit 133 determines that the pixel value of the pixel of interest is not singular. The first pixel state storage unit 134 stores information indicating these determination results in a data area corresponding to the pixel of interest.

[0068] In FIG. 5(b), the 2-bit value written in the center of the circle corresponding to each pixel is unevenness data indicating whether the pixel value of the pixel is maximum or minimum in a pixel region of 3 rows and 3 columns including the pixel (pixel of interest) and 8 neighboring pixels. When the left bit of the unevenness data is 1, the pixel value of the pixel is maximum in the pixel region of 3 rows and 3 columns. When the right bit of the unevenness data is 1, the pixel value of the pixel is minimum in the pixel region of 3 rows and 3 columns. For example, the unevenness data value of pixel P11 is "01", indicating that the pixel value of pixel P11 is minimum (i.e., a concave singular value) in the pixel region of 3 rows and 3 columns. Also, the unevenness data value of pixel P45 is "10", indicating that the pixel value of pixel P45 is maximum (i.e., a convex singular value) in the pixel region of 3 rows and 3 columns.

[0069] The first connection number determination unit 137 determines whether the pixel of interest is a convex or concave isolated point based on the 8-bit connected data and the 2-bit uneven data (i.e., a total of 10-bit pixel state data) stored in the first pixel state storage unit 134 (determination of the second condition). That is, in step S13 when the pixel value of the pixel of interest is singular, the first connection number determination unit 137 determines whether the pixel of interest is an isolated point based on whether the number of connections calculated from the connected data is equal to or less than a threshold N (first threshold). If it is determined that the pixel value of the pixel of interest is an isolated point (YES in step S13), the process proceeds to step S14. In this case, the pixel of interest is a correction target. If it is determined that the pixel value of the pixel of interest is not an isolated point (NO in step S13), the process proceeds to step S15. In this case, the pixel of interest is not a correction target.

[0070] For example, when pixel P45, which has a convex singular value with a concave / convex data value of "10", is referred to, all values ​​of the 8-bit concatenated data are 0. Therefore, the number of connections of pixel P45 is 1, and pixel P45 is determined to be a convex isolated point in the processing of step S13. Therefore, pixel P45 is a correction target.

[0071] In addition, when pixel P15, which has a concave singular value with a concave data value of "01", is referred to, the values ​​of the 8-bit concave data are all 0. Therefore, the number of connections of pixel P15 is 1, and pixel P15 is determined to be a concave isolated point in the processing of step S13. Therefore, pixel P15 is also a correction target.

[0072] Note that pixel P43 is an isolated point because all values ​​of the 8-bit concatenated data are 0, but it is not a singular value because the unevenness data value is "00." Since it is determined in the process of step S12 that the pixel value is not singular and the process proceeds to step S15, pixel P43 is not a correction target.

[0073] Although not shown in Fig. 5(b), in a pixel region of three rows and three columns including the pixel of interest and its eight neighboring pixels, if all nine pixels have the same pixel value, the pixel value of the pixel of interest is both maximum and minimum, and so the unevenness data value may be "11". In this case, since the pixel values ​​of the pixel of interest and the eight neighboring pixels are the same, the number of connections of the pixel of interest is nine or more, so the pixel of interest is not an isolated point and is not subject to correction.

[0074] In step S14, the correction unit 136 corrects the pixel value of the pixel of interest by replacing the pixel value of the pixel of interest with the correction value generated by the correction data generation unit 132 and outputting the replaced pixel value. In step S15, the correction unit 136 does not correct the correction value of the pixel of interest. This process may involve outputting the pixel value held in the pixel data holding unit 131 as is.

[0075] The correction process in the correction data generating unit 132 is not particularly limited, but may be, for example, a process in which nine pixel values ​​in a pixel region of three rows and three columns including one pixel of interest and eight neighboring pixels are referenced, and the median of these nine pixel values ​​is taken as the correction value. Also, instead of the median, the average or weighted average of the pixel region may be taken as the correction value. Also, the median, average or weighted average may be calculated using pixel values ​​of the nine pixel values ​​of the pixel region excluding singular values.

[0076] The processes from step S12 to step S15 are performed for each of the multiple pixels, and may be performed in parallel for each of the multiple pixels, or may be performed sequentially for each of the multiple pixels.

[0077] In addition, in the process of step S12, nine pixels in a pixel area of ​​three rows and three columns including the pixel of interest and its neighboring eight pixels are referenced, but the range of the pixel area to be referenced is not limited to this. For example, 25 pixels in a pixel area of ​​five rows and five columns including the pixel of interest and its neighboring 24 pixels may be referenced.

[0078] In addition, in the judgment of whether a singular value is a convex or concave singular value in step S12, the judgment criterion is whether it is the maximum or minimum value in the pixel region, but the judgment criterion of the singular value is not limited to this. In step S12, if the rank of the pixel value of the pixel of interest in the pixel region is higher than the upper limit of a predetermined range (second range), it may be judged to be a convex type, and if the rank of the pixel value of the pixel of interest in the pixel region is lower than the lower limit of the predetermined range, it may be judged to be a concave type. For example, if this predetermined range is from 3rd to 7th, if the rank of the pixel value of the pixel of interest in the pixel region is 1st (maximum) or 2nd, it is judged to be a convex type, and if the rank of the pixel value of the pixel of interest in the pixel region is 9th (minimum) or 8th, it is judged to be a concave type. If the upper limit of this predetermined range is set to 2nd and the lower limit is set to the rank of the number of pixels in the pixel region minus 1, the same process as when the judgment criterion is whether it is the maximum or minimum value in the pixel region is performed. In other words, if the pixel value of the pixel of interest is ranked first (maximum) in the pixel region, it is determined to be a convex shape, and if the pixel value of the pixel of interest is ranked last (minimum) in the pixel region, it is determined to be a concave shape.

[0079] In the above example, for the sake of simplicity, a case has been described in which the value of the threshold N for the number of connections in determining an isolated point is 1, i.e., a pixel that is not connected to other pixels is determined to be an isolated point, but the value of the threshold N may be 2 or more. By setting the value of the threshold N to 2 or more, a determination can be made that takes connected abnormal pixels into consideration. Below, the correction process of this embodiment will be described in a more generalized manner, taking into consideration a case in which the value of the threshold N is set to a value other than 1. In this case, a pixel included in the first group of connected pixels, whose number of connections is N or less, is an isolated point.

[0080] First, with reference to Figures 8(a) and 8(b), the isolated point determination process in step S13 in Figure 4 will be described. In this description, it is assumed that the value of the isolated point threshold N is 3.

[0081] Fig. 8(a) is a diagram showing an example of pixel values ​​in the correction process according to the present embodiment. Fig. 8(b) is a diagram showing an example of pixel states in the correction process according to the present embodiment. In Fig. 8(a) and Fig. 8(b), the notation method of these figures is the same as Fig. 5(a) and Fig. 5(b). In addition, the pixel values, linked data, and unevenness data shown in Fig. 8(a) and Fig. 8(b) are also the same as Fig. 5(a) and Fig. 5(b).

[0082] 8(a) and 8(b), the hatched pixels P15, P43, and P45 are not connected to other pixels and have a connectivity count of 1, i.e., the connectivity count is equal to or less than the threshold (N=3), so the pixels P15, P43, and P45 are isolated points. This is similar to what is shown in FIGS. 5(a) and 5(b).

[0083] 8(b), the first connected pixel group G1 indicated by the dashed line includes pixels P42, P52, and P53. With reference to the connected data of pixels P42, P52, and P53, the pixels P42, P52, and P53 are connected to one another. Therefore, the connected number of pixels P42, P52, and P53 in the first connected pixel group G1 is 3. In other words, since the connected number is equal to or less than the threshold value (N=3), the pixels P42, P52, and P53 are also isolated points. This connected number can be determined by tracing the connected relationships with reference to the connected data of each pixel.

[0084] Next, the process of determining whether or not to correct in steps S12 and S13 in FIG. 4 when the threshold N of the number of connections is 2 or more will be described.

[0085] The determination of whether a pixel of interest is a convex isolated point or a concave isolated point is the same as when the threshold N is 1. However, when the connection number threshold N is 2 or more, there may be a plurality of pixels in the first connected pixel group, in which case all pixels in the first connected pixel group may not be determined to be the same type of isolated point. For example, in the example of FIG. 8(b), of pixels P42, P52, and P53, the pixel value of pixel P52 is convex. However, the pixel values ​​of pixels P42 and P53 are not convex because pixel P52 is located nearby.

[0086] In this way, when the number of connections is two or more, if the first connected pixel group determined to be an isolated point contains at least one pixel determined to be convex, all pixels of the first connected pixel group are considered to be convex isolated points. Also, if the first connected pixel group determined to be an isolated point contains at least one pixel determined to be concave, all pixels of the first connected pixel group are considered to be concave isolated points. That is, in either case, all pixels of the first connected pixel group are subject to correction (step S14 in FIG. 4). Also, if the first connected pixel group determined to be an isolated point does not contain any pixel determined to be convex or concave, all pixels of the first connected pixel group are not subject to correction (step S15 in FIG. 4).

[0087] There may be cases where the first connected pixel group determined to be an isolated point includes both pixels determined to be convex and pixels determined to be concave. It is desirable to define an exception process that can handle such cases. As a specific example of the exception process, all pixels of the first connected pixel group may be treated as convex isolated points, or all pixels of the first connected pixel group may be treated as concave isolated points. As another specific example of the exception process, all pixels of the first connected pixel group may be treated as neither convex nor concave isolated points, and may be excluded from correction.

[0088] In the above example, it is assumed that the same correction process is performed on the pixels in the first connected pixel group that are determined to be isolated points, but this is not limited to the above. For example, only the pixels determined to be convex or concave among the first connected pixel group that are determined to be isolated points may be subject to correction.

[0089] The threshold N of the number of connections may be set to different values ​​for convex isolated points and concave isolated points. Also, the value of the threshold N of the number of connections may be changed for each imaging frame period of the photoelectric conversion device.

[0090] The data reference range in the process of determining the number of connections in step S13 will now be described in more detail. The first connection number determination unit 137 determines the number of connections by referring to a data area of ​​pixels in a range of (2N-1)×(2N-1) with the pixel of interest at the center, among the data stored in the first pixel state storage unit 134.

[0091] The reason for setting the reference range of the data area as described above will be explained. If it can be determined that the number of pixels that can be reached by tracing the connected pixels starting from the target pixel is N or less, including the target pixel, then the number of connections of the first connected pixel group including the target pixel is N or less. Also, if N pixels are connected in a straight line in the same direction from the target pixel and an adjacent pixel, then the number of connections is N+1 or more. Therefore, in order to determine whether the number of connections is (N+1) or more, it is sufficient to refer to the pixel states in a range of (N-1) pixels each up, down, left and right from the target pixel as the center. In other words, it is possible to determine whether the number of connections is N or less by referring to the data area of ​​pixels in a range of (2N-1) x (2N-1) centered on the target pixel.

[0092] As an example, a case where the threshold N of the number of connections is 2 will be described. In this case, the first connection number determination unit 137 determines the number of connections by referring to a data area of ​​pixels in a 3×3 range centered on the pixel of interest, that is, the pixel of interest and its surrounding 8 pixels. If the pixel of interest and its surrounding pixels are not connected, the number of connections of the first connected pixel group including the pixel of interest is 1. If the pixel of interest and two or more surrounding pixels are connected, the number of connections of the first connected pixel group including the pixel of interest is greater than 2. If the pixel of interest and one of its surrounding pixels are connected, and the connected pixel is connected only to the pixel of interest, the number of connections of the first connected pixel group including the pixel of interest is 2. If the pixel of interest and one of its surrounding pixels are connected, and the connected pixel is connected to pixels other than the pixel of interest, the number of connections of the first connected pixel group including the pixel of interest is greater than 2. By calculating the number of connections in this manner, it is possible to determine whether the number of connections is 2 or less.

[0093] When the connection number is determined, the first connection process by first connection unit 135 and the determination of convex or concave singular values ​​by first pixel value determination unit 133 have been completed for at least the pixels corresponding to the above-mentioned data area. First connection unit 135 and first pixel value determination unit 133 refer to pixel values ​​of pixels in a 3×3 range centered on the pixel of interest. Therefore, the range of pixel values ​​related to the determination of the connection number by first connection number determination unit 137 is a (2N+1)×(2N+1) range.

[0094] In this embodiment, for the sake of simplicity, it is assumed that the pixel data storage unit 131 and the first pixel state storage unit 134 store data for one frame. However, as described above, the first connection number determination unit 137 refers to a data area of ​​pixels in a range of (2N-1)×(2N-1) centered on the pixel of interest, so data from the (N-1)th row before the pixel of interest to the (N-1)th row after the pixel of interest is sufficient. However, in the process of the first connection unit 135, data from the (N-1)th row after the pixel of interest is compared with data from the Nth row after the pixel of interest, so data from the Nth row after the pixel of interest is also necessary. Therefore, the first pixel state storage unit 134 can be configured by a band memory having a storage capacity capable of storing data from the (N-1)th row before the pixel of interest to the Nth row after the pixel of interest.

[0095] Furthermore, first connecting unit 135, first pixel value determination unit 133, first connection number determination unit 137, and correction data generation unit 132 refer to a data area of ​​pixels in a 3×3 range centered on the pixel of interest from pixel data holding unit 131. Here, the pixel of interest for first connecting unit 135 and first pixel value determination unit 133 is (N-1) rows behind the pixel of interest in the processing of correction data generation unit 132. Therefore, pixel data holding unit 131 can also be configured with a band memory having a storage capacity capable of holding data of about (N+1) rows.

[0096] Furthermore, an algorithm for determining the number of connections, such as Union-Find, may be used in determining the connection state in first connection number determination unit 137. Even in this case, it is sufficient to refer to the data area of ​​pixels in the range of (2N-1) x (2N-1) centered on the pixel of interest in order to determine whether the number of connections is N or less, and therefore the band memory described above can be applied.

[0097] As described above, according to this embodiment, in the correction process for abnormal pixels, connected abnormal pixels equal to or smaller than the threshold value N can be detected and corrected. The value of N can be set appropriately depending on the occurrence status of the abnormal pixels, etc. Therefore, according to this embodiment, an information processing device and an information processing method capable of more suitably correcting abnormal pixels are provided.

[0098] In addition, in the process of this embodiment, position information of the abnormal pixels acquired in advance and images captured in a light-shielded state are not required. Therefore, the abnormal pixels can be corrected more easily. In addition, since the memory capacity for storing these data is not required, the memory capacity of the information processing device can be reduced.

[0099] [Second embodiment] In this embodiment, a modification of the correction process of the first embodiment will be described. In this embodiment, the description of elements common to the first embodiment may be omitted or simplified.

[0100] A subject may contain a pattern such as a line, and such a pattern may cause pixel values ​​to become peculiar. This peculiar image value is caused by the subject's pattern and should not be corrected. A group of pixels having such peculiar image values ​​is called a texture. In a texture, pixels with large fluctuations in pixel values ​​may appear at a relatively high density.

[0101] In this embodiment, a configuration example will be shown in which it is determined whether or not the pixel of interest is a pixel that constitutes a texture during the correction process of the first embodiment, thereby preventing erroneous correction of a part of the texture.

[0102] 9 is a functional block diagram relating to the correction processing function of the information processing device 1 according to this embodiment. In addition to the same configuration as in the first embodiment, the information processing device 1 further includes a second pixel value determination unit 138, a determination result storage unit 139, a second linkage unit 140, a second pixel state storage unit 141, and a second linkage number determination unit 142.

[0103] The functions of the determination result holding unit 139 and the second pixel state holding unit 141 are realized, for example, by the data storage unit 122 in Fig. 1. The functions of the second pixel value determination unit 138, the second connecting unit 140 and the second connecting number determination unit 142 are realized, for example, by the information processing unit 129 in Fig. 1. The functions of the second pixel value determination unit 138, the second connecting unit 140 and the second connecting number determination unit 142 may be realized by the CPU 125 executing a correction processing program. This correction processing program may be stored in advance in the data storage unit 122 or ROM 127, or may be obtained from another device via the communication unit 128. The operation of each of these units will be described later.

[0104] Fig. 10 is a flowchart showing the correction process executed by the information processing device 1 according to this embodiment. The correction process method of this embodiment will be described with reference to the flowchart of Fig. 10. In the flowchart of Fig. 10, steps S11, S12, S13, S14, and S15 are the same as those in Fig. 4, and therefore description thereof will be omitted or simplified.

[0105] In step S21, the second pixel value determination unit 138, the determination result storage unit 139, the second connection unit 140, and the second pixel state storage unit 141 perform a connection process of pixels having high pixel values ​​(second connection process). The operation of each unit in the second connection process will be described with further reference to Fig. 11. Fig. 11 is a diagram showing a schematic example of pixel values ​​in the correction process according to this embodiment.

[0106] The second pixel value determination unit 138 acquires the pixel values ​​of each of the pixels constituting the image from the pixel data storage unit 131. The second pixel value determination unit 138 then determines whether or not the pixel value of the pixel of interest is high based on a representative value determined by the pixel values ​​of a pixel region of three rows and three columns including the pixel of interest and its eight neighboring pixels. The representative value is, for example, the median value of the pixel values ​​of a pixel region of three rows and three columns including the pixel of interest and its eight neighboring pixels, and in the following description, the representative value is assumed to be the median value.

[0107] First, focus on pixel P32 in FIG. 11. The pixel value of pixel P33 is "500". The median (representative value) of the pixel values ​​of the pixel region of three rows and three columns including pixel P32 and its eight neighboring pixels is "250". Therefore, the pixel value of pixel P32 is determined to be high. Next, focus on pixel P33 in FIG. 11. The pixel value of pixel P33 is "1000". The median (representative value) of the pixel values ​​of the pixel region of three rows and three columns including pixel P33 and its eight neighboring pixels is "500". Therefore, the pixel value of pixel P33 is determined to be high. If a similar determination is made for other pixels, in the example of FIG. 11, the four hatched pixels P32, P33, P34, and P35 are determined to have high pixel values.

[0108] The determination result storage unit 139 stores, for each pixel, information indicating the determination result by the second pixel value determination unit 138. This information may be, for example, one-bit data per pixel, in which a pixel having a high pixel value is set to "1" and other pixels are set to "0."

[0109] The second connecting unit 140 refers to the information stored in the determination result storage unit 139, and when two adjacent pixels (first pixel and second pixel) both have high pixel values, it determines that the two pixels belong to the same second connected pixel group (second connecting process). The second pixel state storage unit 141 stores information indicating the determination result of the second connecting process in a format similar to that of the connected data described in the first embodiment. The process in which the second connecting unit 140 determines that two pixels belong to the same second connected pixel group and the second pixel state storage unit 141 stores the pixel states of these pixels may also be expressed as "connecting".

[0110] 11, pixels P32 and P33 are connected by a connecting line N2, pixels P33 and P34 are connected by a connecting line N3, and pixels P34 and P35 are connected by a connecting line N4. Therefore, the number of connections among pixels P32, P33, P34, and P35 in second connected pixel group G2 is 4.

[0111] Referring again to FIG. 10, the second connection process in step S21 and subsequent processes will be described. In this embodiment, if it is determined in step S13 that the pixel value of the pixel of interest is an isolated point (YES in step S13), the process proceeds to step S22. In step S22, the second connection number determination unit 142 determines whether the pixel of interest is a texture or not based on whether the connection number calculated from the connection data stored in the second pixel state storage unit 141 is greater than a threshold M (second threshold) (determination of the third condition). The threshold M is a positive integer, but it is preferable that M is 2 or more from the viewpoint of detecting that high pixel values ​​are connected. If the connection number is greater than the threshold M (YES in step S22), the pixel of interest is determined to be a texture, and the process proceeds to step S15. In this case, the pixel of interest is not a correction target. If the connection number is equal to or less than the threshold M (NO in step S22), the pixel of interest is determined to be not a texture, and the process proceeds to step S14. In this case, the pixel of interest is a correction target. In step S14 or step S15, the correction unit 136 performs the same pixel value output process as in the first embodiment.

[0112] For the same reason as described in the first embodiment, the data reference range in the process of determining the number of connections in step S22 is a range of (2-1) x (2-1). That is, the second connection number determination unit 142 determines the number of connections by referring to a data area of ​​pixels in a range of (2-1) x (2-1) centered on the pixel of interest, among the data held in the second pixel state holding unit 141.

[0113] 11, the number of connections of pixels P32, P33, P34, and P35 in second connected pixel group G2 is 4, so pixels P32, P33, P34, and P35 are texture. Therefore, correction processing is not performed on pixels P32, P33, P34, and P35.

[0114] Figures 12(a) to 12(d) are graphs showing examples of thresholds used in second pixel value determination unit 138 to determine whether the pixel value of a pixel of interest is high or not, using a representative value as a reference. In Figures 12(a) to 12(d), the horizontal axis shows the representative value calculated from the pixel values ​​of a pixel region of three rows and three columns including the pixel of interest and its eight neighboring pixels, and the vertical axis shows the pixel value of the pixel of interest. In Figures 12(a) to 12(d), line L6 has a slope of 1 and an intercept of 0 (i.e., y=x).

[0115] In Fig. 12(a), line L7 indicates the lower limit threshold at which the pixel value of the pixel of interest is determined to be high. The slope of line L7 is greater than 1, and the intercept of line L2 is greater than 0. Furthermore, on line L7, when x = x2, y = y7. When the representative value is x2, if the pixel value of the pixel of interest is equal to or greater than y7, the pixel value of the pixel of interest is determined to be high.

[0116] The function for determining the threshold is not limited to that shown in Fig. 12(a). For example, the lower threshold may be given by a broken line L8 composed of multiple line segments as shown in Fig. 12(b). On the line L8, when x=x2, y=y8. When the representative value is x2, if the pixel value of the pixel of interest is equal to or greater than y8, the pixel value of the pixel of interest is determined to be high.

[0117] Also, for example, the lower limit threshold may be given by a curve L9 as shown in Fig. 12(c). On the line L9, when x = x2, y = y9. When the representative value is x2, if the pixel value of the pixel of interest is equal to or greater than y9, the pixel value of the pixel of interest is determined to be high.

[0118] In the above example, the second pixel value determination unit 138 determines that the pixel value of the pixel of interest is high based on the representative value, but may determine that the pixel value of the pixel of interest is low based on the representative value. In this case, the pixel of interest may be determined to be texture when the pixel value of the pixel of interest is low. When such a determination condition is applied, for example, the upper threshold may be given by a straight line L10 as shown in FIG. 12(d). The slope of the straight line L10 is smaller than 1, and the intercept of the straight line L10 is smaller than 0. Also, on the straight line L10, when x=x2, y=y10. When the representative value is x2, when the pixel value of the pixel of interest is equal to or less than y10, the pixel value of the pixel of interest is determined to be low.

[0119] It is also possible to perform a determination whether the pixel value of the pixel of interest is high or low. In this example, when the representative value is x2, the pixel value of the pixel of interest is determined to be high when the pixel value of the pixel of interest is y7 or more, and is determined to be low when the pixel value of the pixel of interest is y10 or less. In other words, the determination of the second pixel value determination unit 138 in this embodiment can be said to determine whether the difference between the pixel value of the pixel of interest and the representative value is outside a predetermined range (outside the third range or the fourth range).

[0120] 12(d), one or both of the straight lines L7 and L10 may be replaced with a broken line or a curved line. Also, the straight lines L7 and L10 may be replaced with a different type of line.

[0121] The above-mentioned lower limit threshold and upper limit threshold can be acquired by the same method as the method of acquiring the threshold used in the processing of the first connecting unit 135 described in the first embodiment. That is, the above-mentioned lower limit threshold and upper limit threshold may be calculated by a function or acquired by a lookup table. In addition, the above-mentioned lower limit threshold and upper limit threshold may be the same value as the threshold used in the processing of the first connecting unit 135 described in the first embodiment, or may be a different value.

[0122] 12(d), when both a determination is made when the pixel value of the pixel of interest is high and a determination is made when the pixel value of the pixel of interest is low, the threshold M of the number of connections may be different between the textures with high pixel values ​​and the textures with low pixel values. Also, the value of the threshold M of the number of connections may be changed for each imaging frame period of the photoelectric conversion device.

[0123] In this embodiment, for the sake of simplicity, the pixel data storage unit 131 and the second pixel state storage unit 141 store data for one frame. However, as described above, the second connection number determination unit 142 refers to a data area of ​​pixels in a range of (2M-1)×(2M-1) centered on the pixel of interest, so data from the (M-1)th row before the pixel of interest to the (M-1)th row after the pixel of interest is sufficient. However, in the process of the second connection unit 140, data from the (M-1)th row after the pixel of interest is compared with data from the (M-1)th row after the pixel of interest, so data from the Mth row after the pixel of interest is also necessary. Therefore, the second pixel state storage unit 141 can be configured by a band memory having a storage capacity capable of storing data from the (M-1)th row before the pixel of interest to the Mth row after the pixel of interest.

[0124] Further, the second pixel value determination unit 138 refers to a pixel data area in a 3×3 range centered on the pixel of interest from the pixel data storage unit 131. The second pixel state storage unit 141 stores the determination results of the correction data generation unit 132 and the second connection number determination unit 142 for the pixel of interest up to M rows later. Therefore, the pixel data storage unit 131 needs to store information up to (M+1) rows later. Furthermore, the correction data generation unit 132 also refers to a pixel data area in a 3×3 range centered on the pixel of interest. For the above reasons, the pixel data storage unit 131 can be configured with a band memory having a storage capacity capable of storing data of about (M+2) rows. However, this is not limited to the case where N>M.

[0125] Furthermore, in determining the connection state in second connection number determination unit 142, an algorithm for determining the number of connections, such as Union-Find, may be used. Even in this case, it is sufficient to refer to the data area of ​​pixels in the range of (2M-1) x (2M-1) centered on the pixel of interest in order to determine whether the number of connections is M or less, and therefore the band memory described above can be applied.

[0126] As described above, according to this embodiment, an information processing device and an information processing method are provided that can obtain the same effects as those of the first embodiment. Furthermore, in this embodiment, the influence of erroneous correction of texture on image quality can be reduced by determining the texture and excluding it from the correction target.

[0127] In the above description, it is assumed that the image to be processed is a grayscale image, but the same processing can be applied to an image including data of multiple channels, such as a color image. In this case, the same processing can be performed by applying the processing of this embodiment to a channel image that collects data of the same channel (data of the same color).

[0128] An example of application to a Bayer image, which is an example of an image containing data of multiple channels, will be described below. Here, a Bayer image refers to an image before demosaic processing, captured by an imaging device in which color filters in a Bayer array are arranged. A Bayer image has a repeating array with a basic unit of 2 rows and 2 columns. Red pixels (R) and green pixels (Gr) are arranged in the first row of the 2 rows and 2 columns, and green pixels (Gb) and blue pixels (B) are arranged in the second row.

[0129] In a Bayer image, one pixel holds data of only one channel. For example, if there is a red line-shaped texture on the image, a line of high pixel values ​​may appear only in the red (R) channel. On the other hand, if there is a white line-shaped texture on the image, a line of high pixel values ​​may appear in all of the red (R), green (Gr, Gb) and blue (B) channels. In consideration of these, a modified example will be described in which the processing content is expanded so that the second connecting unit 140 performs two types of connecting processing, that is, between different channels and between the same channels.

[0130] Fig. 13 is a diagram showing a schematic arrangement of pixels according to a modified example of this embodiment. In Fig. 13, pixel P33 is a pixel of interest. In Fig. 13, pixels shown in solid line circles (pixels P11, P13, P31, etc.) are pixels of the same channel as the pixel of interest P33, and pixels shown in dashed line circles (pixels P12, P22, P32, etc.) are pixels of a different channel from the pixel of interest P33. Pixels P22, P23, P24, P32, P34, P42, P43, P44 are pixels of a different channel from the pixel of interest P33 and are adjacent to the pixel of interest P33. Pixels P11, P13, P15, P31, P35, P51, P53, P55 are pixels of the same channel as the pixel of interest P33 and are adjacent to the pixel of interest P33 in a channel image that collects pixels of the same channel. The second connecting unit 140 regards the above-mentioned pixels P22, P23, P24, P32, P34, P42, P43, P44 and pixels P11, P13, P15, P31, P35, P51, P53, P55 as candidates for connection destinations from the target pixel P33. This allows connection processing to be performed taking into account both a single color texture and a white texture. In this modified example, since the number of connection destination pixels is 16, the connection data is 16-bit data.

[0131] [Third embodiment] In this embodiment, a modification of the correction process of the second embodiment will be described. In this embodiment, the description of elements common to the first and second embodiments may be omitted or simplified.

[0132] In the second embodiment, an example was shown in which pixels determined to be textures were excluded from correction. However, there are cases in which the textures contain abnormal pixels that have pixel values ​​that are extremely different from the surrounding pixels. In this embodiment, a configuration example is shown in which abnormal pixels in the textures are detected and corrected.

[0133] 14 is a functional block diagram relating to the correction processing function of the information processing device 1 according to this embodiment. The information processing device 1 further includes a protrusion determination unit 143 in addition to the same configuration as in the second embodiment.

[0134] The function of the protrusion determination unit 143 is realized, for example, by the information processing unit 129 in Fig. 1. The function of the protrusion determination unit 143 may be realized by the CPU 125 executing a correction processing program. This correction processing program may be stored in advance in the data storage unit 122 or the ROM 127, or may be obtained from another device via the communication unit 128. The operation of each of these units will be described later.

[0135] Fig. 15 is a flowchart showing the correction process executed by the information processing device 1 according to this embodiment. The correction process method of this embodiment will be described with reference to the flowchart of Fig. 15. In the flowchart of Fig. 15, steps S11, S12, S13, S14, S15, S21, and S22 are the same as those in Fig. 10, and therefore description thereof will be omitted or simplified.

[0136] In this embodiment, if the number of connections is greater than the threshold value M in step S22 (YES in step S22), the process proceeds to step S31. In step S31, the protrusion determination unit 143 refers to the pixel values ​​of a pixel area of ​​three rows and three columns including the pixel of interest and its neighboring eight pixels to determine whether the pixel value of the pixel of interest protrudes (determination of the fourth condition). If it is determined that the pixel value of the pixel of interest protrudes (YES in step S31), the process proceeds to step S14. In this case, the pixel of interest is a correction target. If it is not determined that the pixel value of the pixel of interest protrudes (NO in step S31), the process proceeds to step S15. In this case, the pixel of interest is not a correction target.

[0137] Here, a pixel value that is convex refers to a situation where the pixel value of a pixel of interest is extremely convex or extremely convex with respect to the surrounding pixels. "Extremely convex" refers to a situation where the pixel value is equal to or greater than a threshold value (third threshold value). "Extremely concave" refers to a situation where the pixel value is equal to or less than a threshold value (fourth threshold value). More specific examples of "extremely convex" and "extremely concave" will be described. If the pixel value of the pixel of interest is the maximum among the pixel of interest and its eight neighboring pixels, and the pixel value of the pixel of interest is Q times (Q is a number equal to or greater than 1) (third threshold value or greater) the maximum pixel value of the eight pixels excluding the pixel of interest, the pixel value of the pixel of interest is deemed to be extremely convex. If the pixel value of the pixel of interest is the minimum among the pixel of interest and its eight neighboring pixels, and the pixel value of the pixel of interest is (1 / Q) times or less (fourth threshold value or less) the minimum pixel value of the eight pixels excluding the pixel of interest, the pixel value of the pixel of interest is deemed to be extremely convex. This method of setting the thresholds is just one example, and for example, thresholds as shown in FIG. 12(a) to FIG. 12(d) may be set.

[0138] Hereinafter, a specific example of the process of the protrusion determination unit 143 will be described assuming that M is 3 and Q is 3. FIG. 16 is a diagram showing an example of pixel values ​​in the correction process according to this embodiment. FIG. 16 shows an example in which pixel P44 has an extremely large pixel value "8000". Pixel P44 is set as a pixel of interest, and pixel values ​​of a pixel area of ​​three rows and three columns including eight pixels in the vicinity of pixel P44 are referenced. In this case, the pixel value "8000" of pixel P44 is three times or more larger than the second largest pixel value "1000" of pixel P33 or pixel P34, and therefore is an extremely convex pixel value. In addition, when the determination is performed using the method described in the second embodiment, the number of connections of pixels P32, P33, P34, P35, and P44 is 5, which is larger than the threshold value M, and therefore these are textures. In this embodiment, in such a case, pixels P32, P33, P34, and P35 are determined to be outside the scope of correction, but pixel P44 is determined to be a correction target because it is an extremely convex pixel and protrudes.

[0139] As described above, according to this embodiment, an information processing device and an information processing method are provided that can obtain the same effects as those of the second embodiment. Furthermore, in this embodiment, abnormal pixels in a texture can be detected and corrected.

[0140] [Fourth embodiment] In this embodiment, another example of the texture determination process in the second embodiment will be described. In this embodiment, the description of elements common to the first or second embodiment may be omitted or simplified.

[0141] 17 is a functional block diagram relating to a correction processing function of the information processing device 1 according to this embodiment. The information processing device 1 further includes a density determination unit 144 instead of the second connecting unit 140, the second pixel state storage unit 141, and the second connecting number determination unit 142 in FIG.

[0142] The function of density determination unit 144 is realized, for example, by information processing unit 129 in Fig. 1. The function of density determination unit 144 may be realized by CPU 125 executing a correction processing program. This correction processing program may be stored in advance in data storage unit 122 or ROM 127, or may be obtained from another device via communication unit 128. The operation of each of these units will be described later.

[0143] Fig. 18 is a flowchart showing the correction process executed by the information processing device 1 according to this embodiment. The correction process method of this embodiment will be described with reference to the flowchart of Fig. 18. In the flowchart of Fig. 18, steps S11, S12, S13, S14, and S15 are the same as those in Fig. 10, and therefore description thereof will be omitted or simplified.

[0144] In step S41, the second pixel value determination unit 138 and the determination result storage unit 139 perform a determination process for a pixel having a high pixel value (pixel value determination process). This process is generally similar to the process of the second pixel value determination unit 138 and the determination result storage unit 139 described in the second embodiment. That is, the second pixel value determination unit 138 determines whether or not the pixel value of the pixel of interest is high based on a representative value determined by pixel values ​​of a pixel area of ​​three rows and three columns including the pixel of interest and eight pixels in the vicinity of the pixel of interest. Then, the determination result storage unit 139 stores information indicating the determination result by the second pixel value determination unit 138 for each pixel. Note that the pixel value determination process may be a process for determining a pixel having a low pixel value, as described in the second embodiment. That is, the determination by the second pixel value determination unit 138 in this embodiment can be rephrased as a process for determining that the difference between the pixel value of the pixel of interest and the representative value is outside a predetermined range (outside a fifth range).

[0145] In this embodiment, if it is determined in step S13 that the pixel value of the pixel of interest is an isolated point (YES in step S13), the process proceeds to step S42. In step S42, the density determination unit 144 acquires the determination result stored in the determination result storage unit 139, and calculates the number of pixels determined to have high pixel values ​​in a pixel area of ​​5 rows and 5 columns including the pixel of interest and 24 pixels in the vicinity of the pixel of interest. In other words, the density determination unit 144 calculates the density of pixels determined to have high pixel values ​​in a predetermined range including the pixel of interest. The density determination unit 144 determines whether the pixel of interest is a texture or not based on whether the number of pixels determined to have high pixel values ​​is greater than a threshold R (fifth threshold) (determination of the fifth condition). The threshold R is a positive integer, but it is preferable that R is 2 or more from the viewpoint of detecting that pixels with high pixel values ​​exist at a high density. If the number of pixels determined to have high pixel values ​​is greater than the threshold R (YES in step S42), the pixel of interest is determined to be a texture, and the process proceeds to step S15. In this case, the pixel of interest is not a correction target. If the number of pixels determined to have high pixel values ​​is equal to or less than the threshold value R (NO in step S42), the pixel of interest is determined not to be texture, and the process proceeds to step S14. In step S14 or step S15, the correction unit 136 performs pixel value output processing similar to that in the first embodiment.

[0146] FIG. 19 is a diagram showing an example of pixel values ​​in the correction process according to the present embodiment. A specific example of the process according to the present embodiment will be described assuming that the threshold value R in the above process is 3. In the example of FIG. 19, as in FIG. 11, four hatched pixels P32, P33, P34, and P35 are determined to have high pixel values. There are four pixels determined to have high pixel values ​​in a pixel region R1 of five rows and five columns including the pixel P33 as the pixel of interest and 24 pixels in the vicinity of the pixel P33. Since this number is greater than the threshold value R, the density determination unit 144 determines that the pixel P33 is texture, and excludes the pixel P33 from the correction target.

[0147] In this embodiment, for the sake of simplicity, it is assumed that the pixel data storage unit 131 and the determination result storage unit 139 store one frame's worth of data. However, as described above, the density determination unit 144 references the data area of ​​the pixel region R1 in the vicinity of the pixel of interest, and therefore data for the number of rows in the pixel region R1 is sufficient. Therefore, the determination result storage unit 139 can also be configured with a band memory having a storage capacity capable of storing data for the number of rows in the pixel region R1 referenced in the processing by the density determination unit 144.

[0148] Furthermore, correction data generation unit 132 refers to a pixel data area of ​​a 3×3 range centered on the pixel of interest. Then, second pixel value determination unit 138 refers to a pixel data area of ​​a 3×3 range centered on the pixel of the last row in pixel area R1 near the pixel of interest. Therefore, pixel data storage unit 131 can also be configured with a band memory having a storage capacity capable of storing data from one row before the pixel of interest to the row next to the last row in pixel area R1 near the pixel of interest.

[0149] As described above, according to the present embodiment, an information processing device and an information processing method are provided that can obtain the same effects as those of the first embodiment. Also, in the present embodiment, similarly to the second embodiment, the influence of erroneous correction of texture on image quality can be reduced by determining texture and excluding it from correction targets.

[0150] The method of detecting and correcting abnormal pixels in a texture described in the third embodiment can also be applied to this embodiment.

[0151] [Fifth embodiment] In this embodiment, a specific configuration example of a photoelectric conversion device including an avalanche photodiode that can be applied to the data input unit 121 of the information processing device 1 according to the first to fourth embodiments will be described. The configuration example of this embodiment is just an example, and the photoelectric conversion device that can be applied to the data input unit 121 is not limited to this.

[0152] FIG. 20 is a schematic diagram showing the overall configuration of a photoelectric conversion device 100 according to this embodiment. The photoelectric conversion device 100 has a sensor substrate 11 (first substrate) and a circuit substrate 21 (second substrate) that are stacked on each other. The sensor substrate 11 and the circuit substrate 21 are electrically connected to each other. The sensor substrate 11 has a pixel region 12 in which a plurality of pixel circuits 101 are arranged to form a plurality of rows and a plurality of columns. The circuit substrate 21 has a first circuit region 22 in which a plurality of pixel signal processing sections 103 are arranged to form a plurality of rows and a plurality of columns, and a second circuit region 23 arranged on the periphery of the first circuit region 22. The second circuit region 23 may include a circuit for controlling the plurality of pixel signal processing sections 103. The sensor substrate 11 has a light incident surface that receives incident light and a connection surface that faces the light incident surface. The sensor substrate 11 is connected to the circuit substrate 21 on the connection surface side. That is, the photoelectric conversion device 100 is a so-called back-illuminated type.

[0153] In this specification, "planar view" refers to a view from a direction perpendicular to the surface opposite to the light incident surface. Also, a cross section refers to a surface in a direction perpendicular to the surface opposite to the light incident surface of the sensor substrate 11. Note that the light incident surface may be rough when viewed microscopically, and in such a case, the planar view is defined based on the light incident surface when viewed macroscopically.

[0154] In the following description, the sensor substrate 11 and the circuit substrate 21 are described as being diced chips, but the sensor substrate 11 and the circuit substrate 21 are not limited to being chips. For example, the sensor substrate 11 and the circuit substrate 21 may be wafers. In addition, when the sensor substrate 11 and the circuit substrate 21 are diced chips, the photoelectric conversion device 100 may be manufactured by stacking them in a wafer state and then dicing them, or may be manufactured by stacking them after dicing.

[0155] 21 is a schematic block diagram showing an example of the arrangement of the sensor substrate 11. A plurality of pixel circuits 101 are arranged in a plurality of rows and a plurality of columns in the pixel region 12. Each of the plurality of pixel circuits 101 has a photoelectric conversion unit 102 including an avalanche photodiode (hereinafter referred to as APD) as a photoelectric conversion element within the substrate.

[0156] The conductivity type of the charge pair generated in the APD and used as the signal charge is called the first conductivity type. The first conductivity type refers to a conductivity type in which the charge of the same polarity as the signal charge is the majority carrier. The conductivity type opposite to the first conductivity type, that is, the conductivity type in which the charge of the opposite polarity to the signal charge is the majority carrier, is called the second conductivity type. In the APD described below, the anode of the APD is at a fixed potential, and a signal is taken out from the cathode of the APD. Therefore, the semiconductor region of the first conductivity type is an N-type semiconductor region, and the semiconductor region of the second conductivity type is a P-type semiconductor region. The cathode of the APD may be at a fixed potential, and the signal may be taken out from the anode of the APD. In this case, the semiconductor region of the first conductivity type is a P-type semiconductor region, and the semiconductor region of the second conductivity type is an N-type semiconductor region. In the following, a case in which one node of the APD is at a fixed potential will be described, but the potentials of both nodes may be fluctuating.

[0157] 22 is a schematic block diagram showing a configuration example of the circuit board 21. The circuit board 21 has a first circuit area 22 in which a plurality of pixel signal processing units 103 are arranged to form a plurality of rows and a plurality of columns.

[0158] Further, a vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, pixel output signal lines 113, an output circuit 114, and a control signal generating unit 115 are arranged on the circuit board 21. The multiple photoelectric conversion units 102 shown in Fig. 21 and the multiple pixel signal processing units 103 shown in Fig. 22 are electrically connected to each other via connection wiring provided for each pixel circuit 101.

[0159] The control signal generating unit 115 is a control circuit that generates control signals for driving the vertical scanning circuit 110, the horizontal scanning circuit 111, and the readout circuit 112, and supplies these circuits with these signals. In this way, the control signal generating unit 115 controls the drive timing of each circuit.

[0160] The vertical scanning circuit 110 supplies a control signal to each of the pixel signal processing units 103 based on the control signal supplied from the control signal generation unit 115. The vertical scanning circuit 110 supplies a control signal to each pixel signal processing unit 103 for each row via a drive line provided for each row of the first circuit area 22. As will be described later, there may be multiple drive lines for each row. The vertical scanning circuit 110 may include logic circuits such as a shift register and an address decoder. In this way, the vertical scanning circuit 110 selects a row for outputting a signal from the pixel signal processing unit 103.

[0161] A signal output from the photoelectric conversion unit 102 of the pixel circuit 101 is processed by a pixel signal processing unit 103. The pixel signal processing unit 103 counts the number of pulses output from the APD included in the photoelectric conversion unit 102 to obtain and hold a digital signal having multiple bits.

[0162] The pixel signal processing unit 103 does not necessarily have to be provided for each pixel circuit 101. For example, one pixel signal processing unit 103 may be shared by a plurality of pixel circuits 101. In this case, the pixel signal processing unit 103 provides a signal processing function to each pixel circuit 101 by sequentially processing the signals output from each photoelectric conversion unit 102.

[0163] The horizontal scanning circuit 111 supplies a control signal to the readout circuit 112 based on the control signal supplied from the control signal generation unit 115. The pixel signal processing unit 103 is connected to the readout circuit 112 via a pixel output signal line 113 provided for each column of the first circuit region 22. The pixel output signal line 113 of one column is shared by a plurality of pixel signal processing units 103 of the corresponding column. The pixel output signal line 113 includes a plurality of wirings, and has at least a function of outputting a digital signal from each pixel signal processing unit 103 to the readout circuit 112 and a function of supplying a control signal for selecting a column for outputting a signal to the pixel signal processing unit 103. The readout circuit 112 outputs a signal to a storage unit or a signal processing unit outside the photoelectric conversion device 100 via the output circuit 114 based on the control signal supplied from the control signal generation unit 115.

[0164] The photoelectric conversion units 102 in the pixel region 12 may be arranged one-dimensionally. Moreover, the function of the pixel signal processing unit 103 does not necessarily have to be provided for each pixel circuit 101. For example, one pixel signal processing unit 103 may be shared by a plurality of pixel circuits 101. In this case, the pixel signal processing unit 103 provides a signal processing function to each pixel circuit 101 by sequentially processing signals output from each photoelectric conversion unit 102.

[0165] 21 and 22, a first circuit region 22 in which a plurality of pixel signal processing units 103 are arranged is arranged in a region overlapping the pixel region 12 in a plan view. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control signal generating unit 115 are arranged so as to overlap between an end of the sensor substrate 11 and an end of the pixel region 12 in a plan view. In other words, the sensor substrate 11 has the pixel region 12 and a non-pixel region arranged around the pixel region 12. In a region of the circuit substrate 21 overlapping the non-pixel region in a plan view, a second circuit region 23 in which the vertical scanning circuit 110, the horizontal scanning circuit 111, the readout circuit 112, the output circuit 114, and the control signal generating unit 115 are arranged is arranged.

[0166] The arrangement of the pixel output signal lines 113, the readout circuits 112, and the output circuits 114 are not limited to those shown in Fig. 22. For example, the pixel output signal lines 113 may be arranged to extend in the row direction and shared by a plurality of pixel signal processing units 103 in the corresponding row. The readout circuits 112 may be arranged so that the pixel output signal lines 113 in each row are connected to each other.

[0167] Fig. 23 is a schematic block diagram showing a configuration example of one pixel of the photoelectric conversion unit 102 and pixel signal processing unit 103 according to this embodiment. Fig. 23 shows a more specific configuration example including a connection relationship between the photoelectric conversion unit 102 arranged on the sensor substrate 11 and the pixel signal processing unit 103 arranged on the circuit substrate 21. Note that in Fig. 23, the drive lines between the vertical scanning circuit 110 and the pixel signal processing unit 103 in Fig. 22 are shown as drive lines 213 and 214.

[0168] The photoelectric conversion unit 102 has an APD 201. The pixel signal processing unit 103 has a quenching element 202, a waveform shaping unit 210, a counter circuit 211, and a selection circuit 212. It is sufficient that the pixel signal processing unit 103 has at least one of the waveform shaping unit 210, the counter circuit 211, and the selection circuit 212.

[0169] The APD201 generates charge pairs according to incident light by photoelectric conversion. A voltage VL (first voltage) is supplied to the anode of the APD201. A cathode of the APD201 is connected to a first terminal of the quench element 202 and an input terminal of the waveform shaping unit 210. A voltage VH (second voltage) higher than the voltage VL supplied to the anode is supplied to the cathode of the APD201. As a result, a reverse bias voltage is supplied to the anode and cathode of the APD201 such that the APD201 performs avalanche multiplication. When charges are generated by incident light in the APD201 to which the reverse bias voltage is supplied, the charges cause avalanche multiplication, generating an avalanche current.

[0170] In addition, there are two operation modes when a reverse bias voltage is supplied to the APD 201: Geiger mode and linear mode. The Geiger mode is a mode in which the APD 201 operates with a potential difference between the anode and cathode that is greater than the breakdown voltage, and the linear mode is a mode in which the APD 201 operates with a potential difference between the anode and cathode that is close to or less than the breakdown voltage.

[0171] An APD operated in Geiger mode is called a SPAD (Single Photon Avalanche Diode). In this case, for example, the voltage VL (first voltage) is −30 V, and the voltage VH (second voltage) is 1 V. The APD 201 may be operated in either linear mode or Geiger mode. In the case of a SPAD, the potential difference is larger than that of an APD in linear mode, and the effect of avalanche multiplication is more pronounced, so a SPAD is preferable.

[0172] The quench element 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication. The quench element 202 suppresses the voltage supplied to the APD 201 to suppress avalanche multiplication (quench operation). The quench element 202 also returns the voltage supplied to the APD 201 to the voltage VH by passing a current corresponding to the voltage drop caused by the quench operation (recharge operation). The quench element 202 can be, for example, a resistive element.

[0173] The waveform shaping unit 210 shapes the potential change of the cathode of the APD 201 obtained when a photon is detected, and outputs a pulse signal. For example, an inverter circuit is used as the waveform shaping unit 210. Although an example using one inverter as the waveform shaping unit 210 is shown in Fig. 23, the waveform shaping unit 210 may be a circuit in which a plurality of inverters are connected in series, or may be another circuit having a waveform shaping effect.

[0174] The counter circuit 211 counts the pulse signals output from the waveform shaping unit 210 and holds a digital signal indicating the count value. When a control signal is supplied from the vertical scanning circuit 110 via the drive line 213, the counter circuit 211 resets the signal it holds.

[0175] A control signal is supplied to the selection circuit 212 from the vertical scanning circuit 110 shown in Fig. 22 via a drive line 214 shown in Fig. 23. In response to this control signal, the selection circuit 212 switches between electrical connection and non-connection between the counter circuit 211 and the pixel output signal line 113. The selection circuit 212 includes, for example, a buffer circuit for outputting a signal corresponding to a value held in the counter circuit 211.

[0176] 23, the selection circuit 212 switches between electrical connection and disconnection between the counter circuit 211 and the pixel output signal line 113, but the method of controlling the signal output to the pixel output signal line 113 is not limited to this. For example, a switch such as a transistor may be disposed at a node between the quench element 202 and the APD 201, between the photoelectric conversion unit 102 and the pixel signal processing unit 103, or the like, and the signal output to the pixel output signal line 113 may be controlled by switching between electrical connection and disconnection. Alternatively, the signal output to the pixel output signal line 113 may be controlled by changing the value of the voltage VH or voltage VL supplied to the photoelectric conversion unit 102 using a switch such as a transistor.

[0177] 23 shows a configuration example using a counter circuit 211. However, instead of the counter circuit 211, a time to digital converter (hereinafter, TDC) and a memory may be used to acquire the timing for detecting a pulse. At this time, the generation timing of the pulse signal output from the waveform shaping unit 210 is converted into a digital signal by the TDC. In this case, a control signal (reference signal) may be supplied to the TDC from the vertical scanning circuit 110 in FIG. 22 via a driving line. The TDC acquires a signal indicating the relative time of the input timing of the pulse based on the control signal as a digital signal.

[0178] Fig. 24(a), Fig. 24(b) and Fig. 24(c) are diagrams for explaining the operation of the APD 201 according to this embodiment. Fig. 24(a) is a diagram showing the APD 201, the quench element 202 and the waveform shaping unit 210 extracted from Fig. 23. As shown in Fig. 24(a), the connection node of the APD 201, the quench element 202 and the input terminals of the waveform shaping unit 210 is referred to as nodeA. Also, as shown in Fig. 24(a), the output side of the waveform shaping unit 210 is referred to as nodeB.

[0179] FIG. 24(b) is a graph showing the time change of the potential of nodeA in FIG. 24(a). FIG. 24(c) is a graph showing the time change of the potential of nodeB in FIG. 24(a). In the period from time t0 to time t1, a voltage of VH-VL is applied to the APD 201 in FIG. 24(a). When a photon is incident on the APD 201 at time t1, avalanche multiplication occurs in the APD 201. As a result, an avalanche current flows through the quench element 202, and the potential of nodeA drops. Thereafter, the amount of potential drop becomes larger, and the voltage applied to the APD 201 gradually decreases. Then, at time t2, the avalanche multiplication in the APD 201 stops. As a result, the voltage level of nodeA does not drop below a certain value. Thereafter, during the period from time t2 to time t3, a current that compensates for the voltage drop from the node of voltage VH flows to nodeA, and at time t3, nodeA settles to its original potential.

[0180] In the above process, the potential of node B becomes high during the period when the potential of node A is lower than a certain threshold. In this way, the waveform of the drop in the potential of node A caused by the incidence of a photon is shaped by the waveform shaping unit 210 and output as a pulse to node B.

[0181] According to this embodiment, a photoelectric conversion device using an avalanche photodiode is provided, which can be applied to the data input unit 121 of the information processing device 1 of the first to fourth embodiments. A photoelectric conversion device using an avalanche photodiode may have more abnormal pixels than a photoelectric conversion device using a general photodiode without avalanche multiplication. The information processing device 1 of the first to fourth embodiments can appropriately correct abnormal pixels, and is therefore effective for correcting the output signal of a photoelectric conversion device using an avalanche photodiode.

[0182] [Sixth embodiment] The information processing device 1 in the above-described embodiment can be applied to various devices. Examples of the devices include digital still cameras, digital camcorders, camera heads, copiers, fax machines, mobile phones, vehicle-mounted cameras, observation satellites, and surveillance cameras. Fig. 25 shows a block diagram of a digital still camera as an example of a device. Fig. 25 shows an example in which the information processing device 1 shown in Fig. 1 is applied to a digital still camera.

[0183] The device 70 shown in FIG. 25 includes a barrier 706, a lens 702, an aperture 704, and an imaging device 700 (an example of a photoelectric conversion device). The device 70 further includes a signal processing unit (processing device) 708, a timing generating unit 720, an overall control / calculation unit 718 (control device), a memory unit 710 (storage device), a recording medium control I / F unit 716, a recording medium 714, and an external I / F unit 712. At least one of the barrier 706, the lens 702, and the aperture 704 is an optical device corresponding to the device. The barrier 706 protects the lens 702, and the lens 702 forms an optical image of a subject on the imaging device 700. The aperture 704 makes the amount of light passing through the lens 702 variable. The imaging device 700 converts the optical image formed by the lens 702 into image data (image signal). The signal processing unit 708 performs various corrections, data compression, etc. on the imaging data output from the imaging device 700. The timing generating unit 720 outputs various timing signals to the imaging device 700 and the signal processing unit 708. The overall control and calculation unit 718 controls the entire digital still camera, and the memory unit 710 temporarily stores image data. The recording medium control I / F unit 716 is an interface for recording or reading image data to or from the recording medium 714, which is a removable recording medium such as a semiconductor memory for recording or reading imaging data. The external I / F unit 712 is an interface for communicating with an external computer or the like. Timing signals and the like may be input from outside the device. The device 70 may further include a display device (monitor, electronic viewfinder, etc.) that displays information obtained by the photoelectric conversion device. The device includes at least a photoelectric conversion device. The device 70 further includes at least one of an optical device, a control device, a processing device, a display device, a storage device, and a mechanical device that operates based on information obtained by the photoelectric conversion device. The mechanical device is a movable part (for example, a robot arm) that operates by receiving a signal from the photoelectric conversion device.

[0184] Each pixel may include a plurality of photoelectric conversion units (a first photoelectric conversion unit and a second photoelectric conversion unit). The signal processing unit 708 may be configured to process a pixel signal based on the charge generated in the first photoelectric conversion unit and a pixel signal based on the charge generated in the second photoelectric conversion unit, and to acquire distance information from the imaging device 700 to the subject.

[0185] [Seventh embodiment] FIG. 26(a) and FIG. 26(b) are block diagrams of devices related to an on-board camera in this embodiment. FIG. 26 is an example in which the information processing device 1 shown in FIG. 1 is applied to a moving body such as a vehicle. The device 80 has an imaging device 800 (an example of a photoelectric conversion device) and a signal processing device (processing device) that processes a signal from the imaging device 800. The device 80 has an image processing unit 801 that performs image processing on a plurality of image data acquired by the imaging device 800, and a parallax calculation unit 802 that calculates parallax (phase difference of parallax images) from a plurality of image data acquired by the device 80. The device 80 also has a distance measurement unit 803 that calculates a distance to an object based on the calculated parallax, and a collision determination unit 804 that determines whether or not there is a possibility of collision based on the calculated distance. Here, the parallax calculation unit 802 and the distance measurement unit 803 are examples of distance information acquisition means that acquire distance information to an object. That is, the distance information is information on parallax, defocus amount, distance to an object, and the like. The collision determination unit 804 may determine the possibility of a collision using any of these pieces of distance information. The distance information acquisition means may be realized by dedicated hardware, or may be realized by a software module. In addition, it may be realized by an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or a combination of these.

[0186] The device 80 is connected to a vehicle information acquisition device 810, and can acquire vehicle information such as vehicle speed, yaw rate, and steering angle. In addition, the device 80 is connected to a control ECU 820, which is a control device that outputs a control signal to generate a braking force for the vehicle based on the judgment result of the collision judgment unit 804. In addition, the device 80 is also connected to an alarm device 830 that issues an alarm to the driver based on the judgment result of the collision judgment unit 804. For example, when the judgment result of the collision judgment unit 804 indicates that there is a high possibility of a collision, the control ECU 820 performs vehicle control to avoid a collision and reduce damage by applying the brakes, releasing the accelerator, suppressing engine output, etc. The alarm device 830 warns the user by sounding an alarm such as a sound, displaying alarm information on the screen of a car navigation system, etc., and applying vibrations to a seat belt or steering wheel. The device 80 functions as a control means that controls the operation of controlling the vehicle as described above.

[0187] In this embodiment, the surroundings of the vehicle, for example, the front or rear, are captured by the device 80. Fig. 26(b) shows the device when capturing an image of the area in front of the vehicle (imaging range 850). A vehicle information acquisition device 810, which serves as an imaging control means, sends an instruction to the device 80 or the imaging device 800 to perform an imaging operation. This configuration can further improve the accuracy of distance measurement.

[0188] Although the above describes an example of control to prevent collision with other vehicles, the present invention can also be applied to control of automatic driving by following other vehicles, control of automatic driving to prevent deviation from lanes, etc. Furthermore, the device is not limited to vehicles such as automobiles, but can be applied to moving bodies (moving devices) such as ships, aircraft, artificial satellites, industrial robots, and consumer robots. In addition, the present invention can be applied to devices that use object recognition or biometric recognition, such as intelligent transport systems (ITS) and surveillance systems, in addition to moving bodies.

[0189] [Modified embodiment] The present invention is not limited to the above-described embodiments and can be modified in various ways. For example, an example in which a part of the configuration of any of the embodiments is added to another embodiment, or an example in which a part of the configuration of any of the embodiments is replaced with a part of the configuration of another embodiment, is also an embodiment of the present invention.

[0190] The disclosure of this specification includes the complement of the concepts described in this specification. In other words, if the specification states, for example, that "A is B" (A=B), the specification is deemed to disclose or suggest that "A is not B" even if the statement that "A is not B" (A≠B) is omitted. This is because when it states that "A is B," it is assumed that the case that "A is not B" is taken into consideration.

[0191] The disclosure of this specification includes the following configurations or methods. (Configuration 1) a first connection unit that determines that two adjacent pixels in a plurality of pixels arranged in a plurality of rows and a plurality of columns belong to the same first connected pixel group when a difference between the pixel values ​​of the two adjacent pixels is within a first range; a pixel value determination unit that determines that the first condition is satisfied when a ranking of the pixel value of the pixel of interest in a neighborhood pixel group including the pixel of interest and neighborhood pixels arranged in the neighborhood of the pixel of interest is higher than an upper limit of a second range or lower than a lower limit of the second range; a first link number determination unit that determines that a second condition is satisfied when the number of pixels belonging to the first link pixel group including the target pixel is equal to or smaller than a first threshold; a correction unit that corrects a pixel value of the pixel of interest when at least the first condition and the second condition are satisfied, and does not correct the pixel value of the pixel of interest when the first condition is not satisfied or when the second condition is not satisfied; 13. An information processing device comprising: (Configuration 2) The upper limit of the second range is second place, and the lower limit of the second range is the rank obtained by subtracting 1 from the number of pixels included in the group of neighboring pixels. 2. The information processing device according to configuration 1. (Configuration 3) The first threshold is equal to or greater than 2. 3. The information processing device according to configuration 1 or 2. (Configuration 4) The neighboring pixels are arranged in the same row as the pixel of interest or in an adjacent row, and in the same column as the pixel of interest or in an adjacent column. 4. The information processing device according to any one of configurations 1 to 3. (Configuration 5) The neighboring pixel group includes the pixel of interest and a plurality of the neighboring pixels arranged to surround the pixel of interest. 5. The information processing device according to configuration 4. (Configuration 6) a second connection unit that determines that, for two adjacent first and second pixels, the first pixel and the second pixel belong to the same second connected pixel group when a difference between a pixel value of the first pixel and a representative value of a neighborhood pixel group including a neighborhood pixel arranged near the first pixel is outside a third range and a difference between a pixel value of the second pixel and a representative value of a neighborhood pixel group including a neighborhood pixel arranged near the second pixel is outside a fourth range; a second link number determination unit that determines that a third condition is satisfied when the number of pixels belonging to the second linked pixel group including the target pixel is greater than a second threshold; Further comprising: the correction unit corrects a pixel value of the pixel of interest when both the first condition and the second condition are satisfied and the third condition is not satisfied; The correction unit does not correct the pixel value of the pixel of interest when the first condition is not satisfied, when the second condition is not satisfied, or when the third condition is satisfied. 2. The information processing device according to configuration 1. (Configuration 7) The second threshold is equal to or greater than 2. 7. The information processing device according to configuration 6. (Configuration 8) The representative value is a median of pixel values ​​of a plurality of pixels included in the group of neighboring pixels. 8. The information processing device according to configuration 6 or 7. (Configuration 9) a second connection unit that determines that, for two adjacent first and second pixels, the first pixel and the second pixel belong to the same second connected pixel group when a difference between a pixel value of the first pixel and a representative value of a neighborhood pixel group including a neighborhood pixel arranged near the first pixel is outside a third range and a difference between a pixel value of the second pixel and a representative value of a neighborhood pixel group including a neighborhood pixel arranged near the second pixel is outside a fourth range; a second link number determination unit that determines that a third condition is satisfied when the number of pixels belonging to the second linked pixel group including the target pixel is greater than a second threshold; a protrusion determination unit that determines that a fourth condition is satisfied when the pixel value of the pixel of interest is equal to or greater than a third threshold value or equal to or less than a fourth threshold value; and the correction unit corrects a pixel value of the pixel of interest when both the first condition and the second condition are satisfied and the third condition is not satisfied; the correction unit corrects the pixel value of the pixel of interest even when all of the first condition, the second condition, the third condition, and the fourth condition are satisfied; The correction unit does not correct the pixel value of the pixel of interest when the first condition is not satisfied, when the second condition is not satisfied, or when the fourth condition is not satisfied. 2. The information processing device according to configuration 1. (Configuration 10) a density determination unit that determines that a fifth condition is satisfied when the number of pixels in a predetermined range including the pixel of interest, in which a difference between a pixel value of a certain pixel and a representative value of a group of neighboring pixels including neighboring pixels arranged near the certain pixel, is outside a fifth range, is greater than a fifth threshold value; the correction unit corrects a pixel value of the pixel of interest when both the first condition and the second condition are satisfied and the fifth condition is not satisfied; The correction unit does not correct the pixel value of the pixel of interest when the first condition is not satisfied, when the second condition is not satisfied, or when the fifth condition is satisfied. 2. The information processing device according to configuration 1. (Configuration 11) The fifth threshold is equal to or greater than 2. 11. The information processing device according to configuration 10. (Configuration 12) The pixel value is generated based on a signal obtained by a photoelectric conversion element photoelectrically converting incident light. 12. The information processing device according to any one of configurations 1 to 11. (Configuration 13) The photoelectric conversion element includes an avalanche photodiode. 13. The information processing device according to configuration 12. (Configuration 14) A plurality of photoelectric conversion elements arranged in a plurality of rows and a plurality of columns; The information processing device according to any one of configurations 1 to 13, to which pixel values ​​based on signals output from the plurality of photoelectric conversion elements are input; A photoelectric conversion device comprising: (Configuration 15) The photoelectric conversion device according to configuration 14, an optical device corresponding to the photoelectric conversion device; A control device for controlling the photoelectric conversion device; a processing device that processes a signal output from the photoelectric conversion device; a display device that displays information obtained by the photoelectric conversion device; A storage device that stores information obtained by the photoelectric conversion device; and and a mechanical device that operates based on information obtained by the photoelectric conversion device. (Configuration 16) 16. The device according to configuration 15, wherein the processing device acquires distance information from the photoelectric conversion device to a subject. (Method 17) determining that two adjacent pixels in a plurality of pixels arranged in a plurality of rows and a plurality of columns belong to the same first connected pixel group when a difference between pixel values ​​of the two adjacent pixels is within a first range; determining that the first condition is satisfied when a rank of the pixel value of the pixel of interest in a neighborhood pixel group including the pixel of interest and neighborhood pixels arranged in the neighborhood of the pixel of interest is higher than an upper limit of a second range or lower than a lower limit of the second range; determining that a second condition is satisfied if the number of pixels belonging to the first connected pixel group including the pixel of interest is equal to or less than a first threshold; correcting a pixel value of the pixel of interest when at least the first condition and the second condition are satisfied; having When the first condition is not satisfied or when the second condition is not satisfied, the pixel value of the pixel of interest is not corrected. 23. An information processing method comprising: (Configuration 18) A program for causing a computer to execute the information processing method according to method 17. (Configuration 19) A recording medium storing a program for causing a computer to execute the information processing method according to method 17.

[0192] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions.

[0193] It should be noted that the above-mentioned embodiments are merely examples of the implementation of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these embodiments. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features. [Explanation of symbols]

[0194] 1. Information processing device 133 First pixel value determination unit 135 1st connection part 136 Correction section 137 First link number determination section

Claims

1. a first connection unit that determines that two adjacent pixels in a plurality of pixels arranged in a plurality of rows and a plurality of columns belong to the same first connected pixel group when a difference between the pixel values ​​of the two adjacent pixels is within a first range; a pixel value determination unit that determines that a first condition is satisfied when a ranking of a pixel value of the pixel of interest in a neighborhood pixel group including the pixel of interest and neighborhood pixels arranged in the neighborhood of the pixel of interest is higher than an upper limit of a second range or lower than a lower limit of the second range; a first link number determination unit that determines that a second condition is satisfied when a number of pixels belonging to the first link pixel group including the target pixel is equal to or smaller than a first threshold; a correction unit that corrects a pixel value of the pixel of interest when at least the first condition and the second condition are satisfied, and does not correct the pixel value of the pixel of interest when the first condition is not satisfied or the second condition is not satisfied; 13. An information processing device comprising:

2. The upper limit of the second range is the second rank, and the lower limit of the second range is the rank obtained by subtracting 1 from the number of pixels included in the group of neighboring pixels.

2. The information processing apparatus according to claim 1,

3. The first threshold is equal to or greater than 2.

2. The information processing apparatus according to claim 1,

4. The neighboring pixels are arranged in the same row as the pixel of interest or in an adjacent row, and in the same column as the pixel of interest or in an adjacent column.

2. The information processing apparatus according to claim 1,

5. The neighboring pixel group includes the pixel of interest and a plurality of the neighboring pixels arranged to surround the pixel of interest.

5. The information processing apparatus according to claim 4.

6. a second connection unit that determines that, for two adjacent first and second pixels, the first pixel and the second pixel belong to the same second connected pixel group when a difference between a pixel value of the first pixel and a representative value of a neighborhood pixel group including a neighborhood pixel arranged near the first pixel is outside a third range and a difference between a pixel value of the second pixel and a representative value of a neighborhood pixel group including a neighborhood pixel arranged near the second pixel is outside a fourth range; a second link number determination unit that determines that a third condition is satisfied when the number of pixels belonging to the second link pixel group including the target pixel is greater than a second threshold value; and the correction unit corrects a pixel value of the pixel of interest when both the first condition and the second condition are satisfied and the third condition is not satisfied; The correction unit does not correct the pixel value of the pixel of interest when the first condition is not satisfied, when the second condition is not satisfied, or when the third condition is satisfied.

2. The information processing apparatus according to claim 1,

7. The second threshold is equal to or greater than 2.

7. The information processing apparatus according to claim 6,

8. The representative value is a median of pixel values ​​of a plurality of pixels included in the group of neighboring pixels.

7. The information processing apparatus according to claim 6,

9. a second connection unit that determines that, for two adjacent first and second pixels, the first pixel and the second pixel belong to the same second connected pixel group when a difference between a pixel value of the first pixel and a representative value of a neighborhood pixel group including a neighborhood pixel arranged near the first pixel is outside a third range and a difference between a pixel value of the second pixel and a representative value of a neighborhood pixel group including a neighborhood pixel arranged near the second pixel is outside a fourth range; a second link number determination unit that determines that a third condition is satisfied when the number of pixels belonging to the second link pixel group including the target pixel is greater than a second threshold value; a protrusion determination unit that determines that a fourth condition is satisfied when a pixel value of the pixel of interest is equal to or greater than a third threshold value or equal to or less than a fourth threshold value; Further comprising: the correction unit corrects a pixel value of the pixel of interest when both the first condition and the second condition are satisfied and the third condition is not satisfied; the correction unit corrects the pixel value of the pixel of interest even when all of the first condition, the second condition, the third condition, and the fourth condition are satisfied; The correction unit does not correct the pixel value of the pixel of interest when the first condition is not satisfied, when the second condition is not satisfied, or when the fourth condition is not satisfied.

2. The information processing apparatus according to claim 1,

10. a density determination unit that determines that a fifth condition is satisfied when a number of pixels in a predetermined range including the target pixel, the number of pixels in which a difference between a pixel value of a certain pixel and a representative value of a neighborhood pixel group including neighboring pixels arranged near the certain pixel is outside a fifth range, is greater than a fifth threshold value; the correction unit corrects a pixel value of the pixel of interest when both the first condition and the second condition are satisfied and the fifth condition is not satisfied; The correction unit does not correct the pixel value of the pixel of interest when the first condition is not satisfied, when the second condition is not satisfied, or when the fifth condition is satisfied.

2. The information processing apparatus according to claim 1,

11. The fifth threshold is equal to or greater than 2.

11. The information processing apparatus according to claim 10,

12. The pixel value is generated based on a signal obtained by a photoelectric conversion element photoelectrically converting incident light.

2. The information processing apparatus according to claim 1,

13. The photoelectric conversion element includes an avalanche photodiode.

13. The information processing apparatus according to claim 12.

14. A plurality of photoelectric conversion elements arranged in a plurality of rows and a plurality of columns; The information processing device according to claim 1 , wherein pixel values ​​based on signals output from the plurality of photoelectric conversion elements are input; A photoelectric conversion device comprising:

15. The photoelectric conversion device according to claim 14 ; an optical device corresponding to the photoelectric conversion device; A control device for controlling the photoelectric conversion device; a processing device that processes a signal output from the photoelectric conversion device; a display device that displays information obtained by the photoelectric conversion device; A storage device that stores information obtained by the photoelectric conversion device; and and a mechanical device that operates based on information obtained by the photoelectric conversion device.

16. 16. The device according to claim 15, wherein the processing device acquires distance information from the photoelectric conversion device to a subject.

17. determining that two adjacent pixels in a plurality of pixels arranged in a plurality of rows and a plurality of columns belong to the same first connected pixel group when a difference between pixel values ​​of the two adjacent pixels is within a first range; determining that the first condition is satisfied when a ranking of the pixel value of the pixel of interest in a neighborhood pixel group including the pixel of interest and neighborhood pixels arranged in the neighborhood of the pixel of interest is higher than an upper limit of a second range or lower than a lower limit of the second range; determining that a second condition is satisfied when a number of pixels belonging to the first connected pixel group including the pixel of interest is equal to or smaller than a first threshold; correcting a pixel value of the pixel of interest when at least the first condition and the second condition are satisfied; having When the first condition is not satisfied or when the second condition is not satisfied, the pixel value of the pixel of interest is not corrected.

23. An information processing method comprising:

18. A program for causing a computer to execute the information processing method according to claim 17.

19. A recording medium storing a program for causing a computer to execute the information processing method according to claim 17.

Images