Information processing apparatus and information processing method

Abstract

To provide an information processing apparatus that can more suitably correct an abnormal pixel.SOLUTION: An information processing apparatus has: a first connection unit that, when the difference between pixel values of adjacent two pixels is within a first range, determines that the two pixels belong to the same first connection pixel group; a pixel value determination unit that, when the order of the pixel value of a target pixel is outside a second range in a neighborhood pixel group including the target pixel and a neighborhood pixel arranged in the neighborhood of the target pixel, determines that a first condition is satisfied; a first connection number determination unit that, when the number of pixels belonging to the first connection pixel group including the target pixel is equal to or less than a first threshold, determines that a second condition is satisfied; a correction unit that, in a first case where at least the first condition and the second condition are satisfied, performs correction of the pixel value of the target pixel, and in a case where the first condition is not satisfied or in a case where the second condition is satisfied, does not perform, for the pixel value of the target pixel, the correction to be performed in the first case; and a first threshold determination unit that determines the first threshold on the basis of temperature information and a first representative value.SELECTED DRAWING: Figure 20

Description

Technical Field

[0001] The present invention relates to an information processing apparatus and an information processing method.

Background Art

[0002] Some imaging devices such as digital cameras have a plurality of imaging elements that convert incident light into an electrical signal by photoelectric conversion. In an imaging device including a plurality of imaging elements, the output characteristics of signals from some of the plurality of imaging elements may be different from those of other imaging elements, and some of the plurality of imaging elements may output an abnormally high signal or a low signal.

[0003] If such abnormal signals output from the imaging element are used as they are without correction, the image quality may deteriorate due to abnormal pixels. Therefore, in Cited Documents 1 to 3, methods for detecting and correcting abnormal pixels have been proposed.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0005] The occurrence ratio and distribution of abnormal pixels are diverse, and a method for dynamically correcting abnormal pixels in consideration of various occurrence modes is required.

[0006] An object of the present invention is to provide an information processing apparatus and an information processing method that can more suitably correct abnormal pixels.

Means for Solving the Problems

[0007] According to one disclosure of this specification, a first connection unit that determines that two adjacent pixels among a plurality of pixels arranged in a plurality of rows and a plurality of columns belong to the same first connected pixel group when the difference in pixel values between the two pixels is within a first range; a pixel value determination unit that determines that a first condition is satisfied when the rank of the pixel value of the target pixel is higher than the upper limit of a second range or lower than the lower limit of the second range among a group of neighboring pixels including the target pixel and neighboring pixels arranged in the vicinity of the target pixel; a first connection number determination unit that determines that a second condition is satisfied when the number of pixels belonging to the first connected pixel group including the target pixel is equal to or less than a first threshold; a correction unit that corrects the pixel value of the target pixel in a first case where at least the first condition and the second condition are satisfied, and does not perform the correction performed in the first case on the pixel value of the target pixel when the first condition is not satisfied or the second condition is not satisfied; an information acquisition unit that acquires temperature information; a representative value calculation unit that calculates a first representative value of the group of neighboring pixels; and a first threshold determination unit that determines the first threshold based on the temperature information and the first representative value. An information processing apparatus characterized by having these components is provided.

[0008] According to one disclosure of the present specification, a first connection unit that determines that two adjacent pixels among a plurality of pixels arranged in a plurality of rows and a plurality of columns belong to the same first connected pixel group when the difference in pixel values of the two pixels is within a first range; a pixel value determination unit that determines that a first condition is satisfied when the rank of the pixel value of the target pixel is higher than the upper limit of a second range or lower than the lower limit of the second range among a group of neighboring pixels including the target pixel and neighboring pixels arranged in the vicinity of the target pixel; a first connection number determination unit that determines that a second condition is satisfied when the number of pixels belonging to the first connected pixel group including the target pixel is less than or equal to a first threshold; a correction unit that corrects the pixel value of the target pixel in a first case where at least the first condition and the second condition are satisfied, and does not perform the correction performed in the first case on the pixel value of the target pixel when the first condition is not satisfied or the second condition is not satisfied; a representative value calculation unit that calculates a first representative value of the group of neighboring pixels; an information acquisition unit that acquires a look-up table showing the relationship between the first representative value and the first threshold generated based on temperature information; and a first threshold determination unit that determines the first threshold based on the first representative value and the look-up table, an information processing apparatus is provided.

[0009] According to one disclosure of this specification, a step of determining that two adjacent pixels in a plurality of pixels arranged in a plurality of rows and a plurality of columns belong to the same first connected pixel group when the difference between the pixel values of the two pixels is within a first range; a step of determining that a first condition is satisfied when the rank of the pixel value of the target pixel is higher than the upper limit of a second range or lower than the lower limit of the second range among a group of neighboring pixels including the target pixel and neighboring pixels arranged in the vicinity of the target pixel; a step of acquiring temperature information; a step of calculating a first representative value of the group of neighboring pixels; a step of determining a first threshold value based on the temperature information and the first representative value; a step of determining that a second condition is satisfied when the number of pixels belonging to the first connected pixel group including the target pixel is equal to or less than the first threshold value; and a step of correcting the pixel value of the target pixel in a first case where at least the first condition and the second condition are satisfied, and an information processing method is provided, characterized in that when the first condition is not satisfied or the second condition is not satisfied, the correction performed in the first case is not performed on the pixel value of the target pixel.

[0010] According to one disclosure of this specification, when the difference between the pixel values of two adjacent pixels among a plurality of pixels arranged in a plurality of rows and a plurality of columns is within a first range, determining that the two pixels belong to the same first connected pixel group; determining that a first condition is satisfied when, among a neighborhood pixel group including a target pixel and neighborhood pixels arranged in the neighborhood of the target pixel, the rank of the pixel value of the target pixel is higher than the upper limit of a second range or lower than the lower limit of the second range; calculating a first representative value of the neighborhood pixel group; obtaining a look-up table showing the relationship between the first representative value and a first threshold value, which is generated based on temperature information; determining the first threshold value based on the first representative value and the look-up table; determining that a second condition is satisfied when the number of pixels belonging to the first connected pixel group including the target pixel is less than or equal to the first threshold value; and correcting the pixel value of the target pixel in a first case where at least the first condition and the second condition are satisfied. When the first condition is not satisfied or the second condition is not satisfied, an information processing method is provided, which is characterized by not performing the correction performed in the first case on the pixel value of the target pixel.

Effect of the Invention

[0011] According to the present invention, there are provided an information processing apparatus and an information processing method capable of more suitably correcting abnormal pixels.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Figure 38

Embodiments for Carrying Out the Invention

[0013] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The same elements or corresponding elements are given common reference numerals throughout the plurality of drawings, and the description thereof may be omitted or simplified.

[0014] [First Embodiment] FIG. 1 is a block diagram showing the hardware configuration of an information processing apparatus 1 according to the present embodiment. The information processing apparatus 1 performs information processing such as correction on pixel data acquired by a photoelectric conversion device. In FIG. 1, an example is shown in which the information processing in the information processing apparatus 1 is performed by a general computer, but the information processing function of the information processing apparatus 1 may be realized by another device. For example, the information processing apparatus 1 may be an image processing apparatus specialized for the function of image processing, or may be an image processing unit incorporated in a photoelectric conversion device.

[0015] The information processing apparatus 1 has a data input unit 121, a data storage unit 122, a display unit 123, and an input unit 124. Further, the information processing apparatus 1 has a CPU (Central Processing Unit) 125, a RAM (Random Access Memory) 126, and a ROM (Read Only Memory) 127. Further, the information processing apparatus 1 has a communication unit 128 and an information processing unit 129. These units are interconnected via a bus. FIG. 1 shows an example of the configuration of the information processing apparatus 1, and a part of the units shown in FIG. 1 may be arranged in a device external to the information processing apparatus 1, or a device other than the units shown in FIG. 1 may be further arranged in the information processing apparatus 1.

[0016] The data input unit 121 includes a photoelectric conversion device such as an image sensor. The photoelectric conversion device includes a plurality of pixel circuits arranged in a plurality of rows and a plurality of columns. Each of the plurality of pixel circuits photoelectrically converts incident light to generate an electrical signal. The electrical signal generated by each of the plurality of pixel circuits is converted into a digital signal (pixel value). In this way, the data input unit 121 has a function of generating the pixel value of each pixel arranged in a plurality of rows and a plurality of columns as image data and inputting it to the information processing device 1. When the photoelectric conversion device is arranged outside the information processing device, the data input unit 121 may be an interface for acquiring image data from the photoelectric conversion device.

[0017] The data storage unit 122 is a recording medium that holds data used for information processing such as image data and parameters. The recording medium can be, for example, a computer-readable non-volatile recording medium such as a hard disk, an SSD (Solid State Drive), or a flexible disk. The recording medium may be an optical disk such as a CD (Compact Disc)-ROM, a CD-R (Recordable), a DVD (Digital Versatile Disc), or a Blu-ray (registered trademark). The recording medium may be a semiconductor memory such as a memory card, a CF (Compact Flash) card, a smart media, an SD card, a memory stick, an xD picture card, or a USB (Universal Serial Bus) memory. Programs and data other than image data may be stored in the data storage unit 122. Alternatively, a part of the storage capacity of the RAM 126 may be used as the data storage unit 122. Or, alternatively, an external recording device communicatively connected to the information processing device 1 by the communication unit 128 may be used as the data storage unit 122.

[0018] The display unit 123 is a device that displays an image before image processing, an image after image processing, or an operation image such as a graphical user interface. The display unit 123 can be a CRT (Cathode-Ray Tube) display, a liquid crystal display, an organic EL (Electro-Luminescence) display, or the like. Alternatively, the display unit 123 may be an external display provided outside the information processing apparatus 1 and communicatively connected by a cable or the like.

[0019] The input unit 124 is a device for a user to input instructions or data. The input unit 124 includes a keyboard, a pointing device, and the like. Examples of the pointing device include a mouse, a trackball, a track pad, a tablet, and the like. Alternatively, when the information processing apparatus 1 of the present embodiment is applied to devices such as a digital camera or a printer, the input unit 124 may be a button, a dial, or the like. Further, the input unit 124 may be a software keyboard displayed on the screen by software. In this case, the input unit 124 may be configured such that the user operates a button, a dial, or a pointing device to input characters to the software keyboard.

[0020] Note that the same device may also function as both the display unit 123 and the input unit 124, such as a touch screen device. In this case, information input to the information processing apparatus 1 by the user operating the operation screen displayed on the touch screen device is treated as input information from the input unit 124.

[0021] Further, the input unit 124 may be configured to receive an instruction from the user by gesture recognition processing. In this case, the input unit 124 includes an input device that inputs an image captured by visible light or infrared rays, and a recognition device that recognizes the user's gesture from the image and converts it into a command. The data input unit 121 may also function as the input device. The recognition device may be added as a dedicated gesture recognition circuit, or may be realized by the CPU 125 executing a gesture recognition program.

[0022] Further, the input unit 124 may be configured to receive a user's instruction through voice recognition processing. In this case, the input unit 124 includes a microphone device and a recognition device that recognizes the user's speech from the voice data acquired by the microphone device and converts it into a command. The recognition device may be added as a dedicated voice recognition circuit, or may be realized by the CPU 125 executing a voice recognition program.

[0023] Note that the above-described gesture recognition processing and voice recognition processing may be performed by a device external to the information processing device 1. In that case, the information processing device 1 communicates and connects with an external device or a server on the network via the communication unit 128, and transmits image data or voice data to the external device or the server. The external device or the server is configured to receive the image data or the voice data according to a predetermined communication procedure, perform recognition processing, and transmit data indicating the recognition result to the information processing device 1.

[0024] The CPU 125 is a processor that controls each part of the information processing device 1 and performs information processing. The RAM 126 and the ROM 127 provide the CPU 125 with programs, data, work areas, etc. necessary for control and information processing. Also, when the program is stored in the data storage unit 122 or the ROM 127, the program is once read into the RAM 126 and then executed by the CPU 125. Further, the information processing device 1 may be configured to receive a program from the outside via the communication unit 128. In that case, the program is either stored in the data storage unit 122 once and then read into the RAM 126, or directly read from the communication unit 128 into the RAM 126 and then executed by the CPU 125.

[0025] Note that in FIG. 1, only one block indicating the CPU 125 is shown, but the number of CPUs 125 is not limited to one. That is, the information processing device 1 may have a plurality of CPUs 125.

[0026] The communication unit 128 is an interface for performing communication between devices. The communication unit 128 can be based on a wired communication method such as, for example, a wired network, RS-232C, USB, IEEE1284, IEEE1394, a telephone line, etc. Alternatively, the communication unit 128 can be based on a wireless communication method such as infrared (IrDA), IEEE802.11a, IEEE802.11b, IEEE802.11g, IEEE802.11n, IEEE802.11ac, IEEE802.11ax, etc. Alternatively, the communication unit 128 can be based on another wireless communication method such as Bluetooth (registered trademark), UWB (Ultra Wide Band), a wireless telephone line, NFC (Near Field Communication), etc. Alternatively, the communication unit 128 may be based on an inter-chip communication method such as I2C (Inter-Integrated Circuit) or SPI (Serial Peripheral Interface).

[0027] Also, the communication methods supported by the communication unit 128 are not limited to one, and there may be a plurality. For example, the communication unit 128 may be configured to support two or more of the various communication methods described above. Alternatively, the information processing apparatus 1 may have a plurality of communication units 128 that support different communication methods. Even in that case, in the present embodiment, the plurality of communication units 128 are collectively referred to as the communication unit 128 for explanation.

[0028] The information processing unit 129 is a signal processing circuit including a DSP (Digital Signal Processor), a logic circuit, etc. Alternatively, the information processing unit 129 may be a GPU (Graphics Processing Unit). The information processing unit 129 performs arithmetic processing on the image data input from the data input unit 121 or the image data held in the RAM 126, the data storage unit 122, etc. The processing result in the information processing unit 129 may be output to the RAM 126, the data storage unit 122, the display unit 123, etc., or may be output to a device external to the information processing apparatus 1 via the communication unit 128. Note that when the load of the arithmetic processing is small (when a high arithmetic speed is not required, or when the amount of data to be processed is small), the CPU 125 may also serve as the information processing unit 129.

[0029] Although not shown in FIG. 1, a register circuit may be added as necessary. The register circuit holds the operation parameters of the CPU 125 or the information processing unit 129. The value held in the register circuit may be set by the CPU 125 or the information processing unit 129, or may be set by an external device via the communication unit 128.

[0030] When the information processing apparatus 1 is a camera apparatus, the display unit 123 may have a function of displaying a preview image of the subject and a function of displaying the captured image. However, these images may also be displayed on another device (for example, a smartphone) connected via the communication unit 128. In that case, the display unit 123 may be omitted. Similarly, when another device connected via the communication unit 128 receives an instruction from the user and the information processing apparatus 1 receives a command corresponding to the instruction via the communication unit 128, the information processing apparatus 1 may perform an operation corresponding to the command. In this case, the CPU 125 or the information processing unit 129 may perform the process of specifying the operation corresponding to the command.

[0031] Alternatively, when software-based processing and control are unnecessary, the CPU 125, ROM 127, etc. may be omitted. As an example of such a case, there is a case where a logic circuit for realizing necessary processing and control is arranged in the information processing unit 129.

[0032] Alternatively, the information processing apparatus 1 may be a stacked sensor in which a substrate on which a photoelectric conversion element is arranged and a substrate on which a signal processing circuit is arranged are stacked. In this case, a logic circuit, a memory, the CPU 125, etc. can be arranged inside the stacked sensor. In such a configuration, the data input unit 121 may be configured to include a photoelectric conversion element and its peripheral circuit. Further, the data storage unit 122, the CPU 125, the RAM 126, the ROM 127, the communication unit 128, and the information processing unit 129 may be arranged inside the stacked sensor. In this case, the display unit 123 and the input unit 124 may be omitted.

[0033] FIG. 2 is a functional block diagram regarding the correction processing function of the information processing apparatus 1 according to the present embodiment. The information processing apparatus 1 includes a pixel data holding unit 131, a correction data generation unit 132, a first pixel value determination unit 133, a first pixel state holding unit 134, a first connection unit 135, a correction unit 136, and a first connection number determination unit 137.

[0034] The functions of the pixel data holding unit 131 and the first pixel state holding unit 134 are realized, for example, by the data storage unit 122 in FIG. 1. The functions of the correction data generation unit 132, the first pixel value determination unit 133, the first connection unit 135, the correction unit 136, and the first connection number determination unit 137 are realized, for example, by the information processing unit 129 in FIG. 1. The functions of the correction data generation unit 132, the first pixel value determination unit 133, the first connection unit 135, the correction unit 136, and the first connection number determination unit 137 may be realized by the CPU 125 executing a correction processing program. This correction processing program may be held in advance in the data storage unit 122 or the ROM 127, or may be acquired from another device via the communication unit 128. The operations of these respective units will be described later.

[0035] Hereinafter, the correction process executed by the information processing apparatus 1 according to the present embodiment will be described with reference to FIGS. 3 to 8(b). First, prior to the description of the specific processing procedure, the configuration of the pixel data acquired by the photoelectric conversion device and held in the pixel data holding unit 131 will be described.

[0036] FIG. 3 is a diagram schematically showing the arrangement of pixels according to the present embodiment. FIG. 3 schematically shows a plurality of pixels P11 to P56 constituting the image data held in the pixel data holding unit 131 and the pixel values of each pixel. The plurality of pixels P11 to P56 are arranged so as to form a plurality of rows and a plurality of columns. For simplicity of explanation, only 5 rows and 6 columns of pixels are shown in FIG. 3, but actually more pixels may be arranged.

[0037] In the reference numerals attached to the pixels, the two numerical values after "P" indicate the row number and the column number, respectively. In FIG. 3, the values described in the circles indicating the pixels P11 to P56 indicate the pixel values. For example, "20" is written in the circle of the pixel P11, indicating that the pixel value of the pixel P11 is 20.

[0038] In the description of the present embodiment, for simplicity, the pixel array will be described assuming that the image to be processed is a grayscale image, but the image to be processed may be a color image. For example, when the pixel array is an array for each of the red, green, and blue color channels, the correction process of the present embodiment can be similarly applied by performing independent processing for each color channel. Also, even when the pixel array is a Bayer array, the correction process of the present embodiment can be similarly applied by performing independent processing for each color channel.

[0039] The abnormal pixels to be corrected in the present embodiment will be described. An abnormal pixel means a pixel having a specific pixel value compared with the pixels in the vicinity of the pixel. Abnormal pixels can include so-called white scratches and black scratches. A white scratch is an abnormal pixel having a significantly larger pixel value compared with the neighboring pixels, and a black scratch is an abnormal pixel having a significantly smaller pixel value compared with the neighboring pixels.

[0040] However, pixels having unique pixel values are not always independent by only one. That is, since two or more abnormal pixels may be connected, it is required to consider the connected abnormal pixels in the correction of abnormal pixels. Here, that two abnormal pixels are connected means that the two abnormal pixels are in a continuous positional relationship in any of the vertical direction, horizontal direction, and diagonal direction in FIG. 3.

[0041] The following formula (1) shows the probability that at least one of the eight pixels in the neighborhood of a certain abnormal pixel is an abnormal pixel.

Equation

[0042] At this time, the expected value of the number of generated connected abnormal pixels is generally expressed by the following formula (2).

Equation

[0043] For example, in an image where the number of columns of pixels is 4096 and the number of rows of pixels is 2048, even if the ratio P is a very small value of 0.1%, the expected value of the number of abnormal pixels calculated by formula (2) is about 67. Assuming that all these abnormal pixels are pairs of two abnormal pixels and no three or more abnormal pixels are connected, there will be about 33 pairs of connected abnormal pixels in the image. This corresponds to a state where there is about 1 pair of continuous abnormal pixels in each divided image when the image is divided into 32 pieces of 8×4. That is, the occurrence probability of the event that abnormal pixels are adjacent to each other cannot be ignored.

[0044] Furthermore, the probability that there are two or more abnormal pixels among the eight pixels adjacent to a certain abnormal pixel, that is, the probability that three or more abnormal pixels are connected, will be described. Equation (1) shows the probability that there is one or more abnormal pixels among the eight pixels adjacent to a certain abnormal pixel. Therefore, by subtracting the probability that there is only one abnormal pixel among the eight pixels adjacent to a certain abnormal pixel from Equation (1), the probability that there are two or more abnormal pixels among the eight pixels adjacent to a certain abnormal pixel can be calculated.

[0045] The probability that there is only one abnormal pixel among the eight pixels adjacent to a certain abnormal pixel is represented by the following Equation (3).

Equation

[0046] Therefore, the expected value of the number of generated abnormal pixels connected by three or more is generally represented by the following Equation (4).

Equation

[0047] As an example, in an image where the number of pixel columns is 4096 and the number of pixel rows is 2048, the expected value of the number of abnormal pixels connected by three or more is calculated. When P = 0.1%, the expected value of the number of abnormal pixels connected by three or more is 0.234. When P = 0.5%, the expected value of the number of abnormal pixels connected by three or more is 28.8. When P = 1.0%, the expected value of the number of abnormal pixels connected by three or more is 226.

[0048] As described above, even if the ratio P is a small value of 1% or less, the possibility that there are abnormal pixels connected by three or more cannot be ignored. Furthermore, depending on the values of P, H, and W, it may be necessary to consider abnormal pixels connected by four or more. As described above, considering the case where a plurality of abnormal pixels are connected, the information processing apparatus 1 of the present embodiment performs correction processing corresponding to abnormal pixels connected by N (N is a positive integer) or less.

[0049] FIG. 4 is a flowchart showing correction processing executed by the information processing apparatus 1 according to the present embodiment. A correction processing method of the present embodiment will be described along the flowchart of FIG. 4.

[0050] The processing in FIG. 4 is processing for detecting abnormal pixels from an image captured by a photoelectric conversion device and correcting the detected abnormal pixels. The processing in FIG. 4 is started, for example, after the photoelectric conversion device performs imaging and an image is held in the pixel data holding unit 131. In the processing of the present embodiment, position information of abnormal pixels (so-called scratch map) acquired in advance and an image captured in a light-shielded state (so-called dark image) are not used. Also, it is assumed that the pixel data holding unit 131 has a storage capacity capable of simultaneously holding pixel values for each of a plurality of pixels in the entire image for one frame. Similarly, it is assumed that the first pixel state holding unit 134 has a storage capacity capable of simultaneously holding the pixel states of the entire image for one frame.

[0051] In step S11, the first connection unit 135 acquires the pixel value of each of the plurality of pixels constituting the image from the pixel data holding unit 131. Then, when the pixel values of two adjacent pixels are close values (the difference between the pixel values of two adjacent pixels is within the first range), the first connection unit 135 determines that the two pixels belong to the same first connected pixel group (first connection process). The first pixel state holding unit 134 holds information indicating the determination result of the first connection process. In this process, for each pixel, the pixel values in a pixel region of three rows and three columns including the pixel to be processed (target pixel) and eight adjacent pixels thereof are referred to, and comparison and determination of the pixel values are performed. The determination result is held in a data region corresponding to the three-row and three-column pixel region referred to by the first pixel state holding unit 134.

[0052] Hereinafter, the process in which the first connection unit 135 determines that two pixels belong to the same first connected pixel group and the first pixel state holding unit 134 holds the pixel states of these pixels may be expressed as "connecting". Among a plurality of pixels after the connection process is completed, those with a connection number of N (N is a positive integer) or less are called isolated points. Here, the connection number is the number of pixels reachable by tracing the pixels connected to a certain pixel, that is, the number of pixels belonging to the same first connected pixel group. For example, when a certain pixel is not connected to any other pixel, the connection number is 1. The value of N indicates the upper limit number considering the connection of abnormal pixels and can be appropriately set according to the required accuracy of abnormal pixel detection, the quality of the image, etc.

[0053] First, an example of the first connection process when the value of the connection number threshold N in the determination of isolated points is 1, that is, when a pixel not connected to other pixels is regarded as an isolated point, will be described with reference to FIGS. 5(a) to 7(d).

[0054] FIG. 5(a) is a diagram schematically showing an example of pixel values in the correction process according to the present embodiment. The notation in FIG. 5(a) is the same as that in FIG. 3. FIG. 5(b) is a diagram schematically showing an example of pixel states in the correction process according to the present embodiment. FIG. 5(b) schematically shows the data of the pixel states held in the first pixel state holding unit 134 corresponding to each pixel. The connection line N1 shows the relationship between two adjacent pixels P11 and P12, indicating that the pixel value "20" of pixel P11 and the pixel value "22" of pixel P12 are determined to be close values and are connected.

[0055] In FIG. 5(b), the numerical values described at eight locations outside the circle corresponding to each pixel are connection data indicating whether the pixel value of the pixel and the pixel values of adjacent pixels are close. In the connection data, "0" indicates that the pixel value of the pixel and the pixel value of the adjacent pixel in that direction are not close and are not connected. Also, "1" indicates that the pixel value of the pixel and the pixel value of the adjacent pixel in that direction are close and are connected. For the pixels at the ends, the number of adjacent pixels is less than eight. Thus, when there are no adjacent pixels, the value of the connection data is 0.

[0056] FIG. 6(a) is a diagram schematically showing an example of pixel values in the vicinity of pixel P11. FIG. 6(b) is a diagram schematically showing an example of pixel states in the vicinity of pixel P11. As shown in FIG. 6(b), among the connection data of pixel P11, the right, bottom, and bottom-right are 1, and pixel P11 is connected to adjacent pixels by connection line N1 respectively. Specifically, since pixel P11 has pixel values close to any of pixels P12, P21, and P22, pixel P11 is connected to pixels P12, P21, and P22. Pixel P11 is a pixel arranged at the end, and there are no adjacent neighboring pixels in other directions. Therefore, except for the right, bottom, and bottom-right of the connection data of pixel P11, the others are 0.

[0057] FIGS. 6(c) and 6(d) are diagrams schematically showing a method of holding connection data. The eight values included in the connection data can be held as 8-bit data, for example, by arranging them in a row in clockwise order based on the value in a certain direction as shown in FIG. 6(c). FIG. 6(d) shows the 8-bit data "00011100" (hexadecimal "1C") obtained in this way.

[0058] In FIGS. 5(a) and 5(b), all eight values included in the connection data of the hatched pixels P15, P43, and P45 are 0. Thus, the pixels P15, P43, and P45 for which all eight values are 0 are not connected to other pixels. Since the connection number of such pixels is 1, i.e., N or less, the pixels P15, P43, and P45 are isolated points.

[0059] FIGS. 7(a) to 7(d) are graphs showing examples of thresholds used to determine whether the pixel values of two adjacent pixels in the first connection portion 135 are close. In the comparison of magnitude relationships using the thresholds described below, "greater than or equal to" and "greater than" are interchangeable, and "less than or equal to" and "less than" are also interchangeable.

[0060] In FIGS. 7(a) to 7(d), the horizontal axis represents the pixel value of the pixel of interest, and the vertical axis represents the pixel value of the neighboring pixel adjacent to the pixel of interest. In FIGS. 7(a) to 7(d), the straight line L1 is a straight line with a slope of 1 and an intercept of 0 (i.e., y = x), and represents the lower threshold value of the pixel value of the connected neighboring pixel.

[0061] In FIG. 7(a), the straight line L2 represents the upper threshold value of the pixel value of the connected neighboring pixel. The slope of the straight line L2 is greater than 1, and the intercept of the straight line L2 is greater than 0. Also, on the straight line L2, when x = x1, y = y2. When the pixel value of the pixel of interest is x1, the pixel of interest and the neighboring pixel are connected when the pixel value of the neighboring pixel is greater than or equal to y1 (= x1) and less than or equal to y2. That is, the first connection portion 135 connects two pixels when the relationship that the pixel values of the pixel of interest and the neighboring pixel are in the region sandwiched between the straight line L1 and the straight line L2 (the difference between the pixel values of the pixel of interest and the neighboring pixel is within the first range determined by the upper threshold value and the lower threshold value) is satisfied. At this time, the first pixel state holding unit 134 holds the pixel state indicating the connection state by holding "1" as the value of the corresponding bit of the connection data between the pixel of interest and the neighboring pixel. That is, "1" is held in each of the bit indicating the direction of the neighboring pixel of the connection data corresponding to the pixel of interest and the bit indicating the direction of the pixel of interest of the connection data corresponding to the neighboring pixel.

[0062] The function for determining the threshold value is not limited to that shown in FIG. 7(a). For example, as shown in FIG. 7(b), the upper limit threshold value may be given by a broken line L3 composed of a plurality of line segments. On the straight line L3, when x = x1, y = y3. When the pixel value of the target pixel is x1, the target pixel and the neighboring pixel are connected when the pixel value of the neighboring pixel is y1 (= x1) or more and y3 or less.

[0063] Also, for example, as shown in FIG. 7(c), the upper limit threshold value may be given by a curve L4. On the straight line L4, when x = x1, y = y4. When the pixel value of the target pixel is x1, the target pixel and the neighboring pixel are connected when the pixel value of the neighboring pixel is y1 (= x1) or more and y4 or less.

[0064] Also, for example, as shown in FIG. 7(d), a lower limit threshold value may be given by a straight line L5 different from the straight line L1. The slope of the straight line L5 is smaller than 1, and the intercept of the straight line L5 is smaller than 0. Also, on the straight line L5, when x = x1, y = y5. When the pixel value of the target pixel is x1, the target pixel and the neighboring pixel are connected when the pixel value of the neighboring pixel is y5 (<x1) or more and y2 or less.

[0065] In the example of FIG. 7(d), one or both of the straight line L2 and the straight line L5 may be replaced with a broken line or may be replaced with a curve. Also, the straight line L2 and the straight line L5 may be replaced with different types of lines.

[0066] A method for obtaining a threshold value (upper threshold value or lower threshold value) will be described. When the graph shape of the threshold value is simple, the first connection unit 135 may calculate the threshold value by a function that takes the pixel value of the pixel of interest as an input and outputs the value of the threshold value. In this case, it is desirable to use the coefficients of the terms included in the function as parameters so that the function can be adjusted. Also, it is not essential that the mathematical formula of this function be the same throughout the entire graph. For example, this function may be a piecewise linear function. Further, the first connection unit 135 may be provided with a look-up table in which the pixel value of the pixel of interest and the threshold value are associated. In this case, the first connection unit 135 refers to the pixel value of the pixel of interest as an index and obtains the threshold value. Here, the look-up table may hold values for all indexes, or may hold values only for the indexes of some representative points. When the look-up table holds values only for the indexes of the representative points, the first connection unit 135 may calculate the threshold values for indexes other than the representative points by interpolation.

[0067] Referring again to FIG. 4, the processing after the first connection processing in step S11 will be described. In step S12 after the completion of the first connection processing in step S11, the first pixel value determination unit 133 determines whether the pixel value of the pixel of interest among the plurality of pixels is abnormal (determination of the first condition). If it is determined that the pixel value of the pixel of interest is abnormal (YES in step S12), the processing proceeds to step S13. If it is determined that the pixel value of the pixel of interest is not abnormal (NO in step S12), the processing proceeds to step S15. In this case, the pixel of interest is not a correction target.

[0068] The determination in step S12 will be described in more detail. In this description as well, it is assumed that the value of N described above is 1, that is, a pixel not connected to other pixels is regarded as an isolated point.

[0069] The first pixel value determination unit 133 compares the pixel values of a pixel region (neighboring pixel group) of three rows and three columns including the target pixel and eight neighboring pixels in the vicinity thereof. The eight neighboring pixels are arranged so as to surround the target pixel. When the pixel value of the target pixel is the maximum among the pixel values of this pixel region, the first pixel value determination unit 133 determines that the pixel value of the target pixel is a convex singular value. When the pixel value of the target pixel is the minimum among the pixel values of this pixel region, the first pixel value determination unit 133 determines that the pixel value of the target pixel is a concave singular value. When the pixel value of the target pixel is neither the maximum nor the minimum among the pixel values of this pixel region, the first pixel value determination unit 133 determines that the pixel value of the target pixel is not singular. The first pixel state holding unit 134 holds information indicating these determination results in a data region corresponding to the target pixel.

[0070] In FIG. 5(b), the 2-bit value described at the center of the circle corresponding to each pixel is concavity / convexity data indicating whether the pixel value of the pixel (target pixel) is the maximum or the minimum in a pixel region of three rows and three columns including the pixel and eight neighboring pixels in the vicinity thereof. When the left bit of the concavity / convexity data is 1, the pixel value of the pixel is the maximum in the pixel region of three rows and three columns. When the right bit of the concavity / convexity data is 1, the pixel value of the pixel is the minimum in the pixel region of three rows and three columns. For example, in pixel P11, the value of the concavity / convexity data is "01", indicating that the pixel value of pixel P11 is the minimum (i.e., a concave singular value) in the pixel region of three rows and three columns. Also, in pixel P45, the value of the concavity / convexity data is "10", indicating that the pixel value of pixel P45 is the maximum (i.e., a convex singular value) in the pixel region of three rows and three columns.

[0071] The first connectivity number determination unit 137 determines whether the pixel of interest is a convex or concave isolated point based on the 8-bit connectivity data and 2-bit concavity / convexity data (i.e., a total of 10-bit pixel state data) held in the first pixel state holding unit 134 (determination of the second condition). That is, in step S13 when the pixel value of the pixel of interest is abnormal, the first connectivity number determination unit 137 determines whether the pixel of interest is an isolated point based on whether the connectivity number calculated from the connectivity data is less than or equal to a threshold value N (the first threshold value). When it is determined that the pixel value of the pixel of interest is an isolated point (YES in step S13), the process proceeds to step S14. In this case, the pixel of interest is a correction target. When it is determined that the pixel value of the pixel of interest is not an isolated point (NO in step S13), the process proceeds to step S15. In this case, the pixel of interest is not a correction target.

[0072] For example, referring to pixel P45 that has a convex abnormal value with the value of the concavity / convexity data being "10", the values of the 8-bit connectivity data are all 0. Therefore, the connectivity number of pixel P45 is 1, and in the process of step S13, pixel P45 is determined to be a convex isolated point. Therefore, pixel P45 is a correction target.

[0073] Also, referring to pixel P15 that has a concave abnormal value with the value of the concavity / convexity data being "01", the values of the 8-bit connectivity data are all 0. Therefore, the connectivity number of pixel P15 is 1, and in the process of step S13, pixel P15 is determined to be a concave isolated point. Therefore, pixel P15 is also a correction target.

[0074] Note that although pixel P43 is an isolated point because the values of the 8-bit connectivity data are all 0, it is not an abnormal value because the value of the concavity / convexity data is "00". Since it is determined in the process of step S12 that the pixel value is not abnormal and the process proceeds to step S15, pixel P43 is not a correction target.

[0075] Although not shown in FIG. 5(b), in a pixel region of three rows and three columns including the target pixel and the eight neighboring pixels, if all the pixel values of the nine pixels are the same value, the pixel value of the target pixel may be the maximum and the minimum, so that the value of the concavity / convexity data can be "11". In this case, since the pixel values of the target pixel and the eight neighboring pixels are the same, the connectivity number of the target pixel is 9 or more, so the target pixel is not an isolated point and is not a correction target.

[0076] In step S14, the correction unit 136 corrects the pixel value of the target pixel by replacing the pixel value of the target pixel with the correction value generated by the correction data generation unit 132 and outputting it. In step S15, the correction unit 136 does not correct the correction value of the target pixel. This process may be to output the pixel value held in the pixel data holding unit 131 as it is.

[0077] The correction process in the correction data generation unit 132 is not particularly limited. For example, it may refer to the nine pixel values in a pixel region of three rows and three columns including one target pixel and the eight neighboring pixels, and use the median value of these nine pixel values as the correction value. Also, instead of the median value, the average value or the weighted average value of the pixel region may be used as the correction value. Further, among the nine pixel values of the pixel region, the median value, the average value, or the weighted average value may be calculated using the pixel values excluding the singular values.

[0078] Note that the processes from step S12 to step S15 are performed for each of the plurality of pixels. The processes may be performed in parallel for each of the plurality of pixels, or may be performed sequentially for each of the plurality of pixels.

[0079] Also, in the process of step S12, although the nine pixels in a pixel region of three rows and three columns including the target pixel and the eight neighboring pixels are referred to, the range of the pixel region to be referred to is not limited to this. For example, the 25 pixels in a pixel region of five rows and five columns including the target pixel and the 24 neighboring pixels may be referred to.

[0080] In addition, in the determination of the convex or concave singular value in step S12, the criterion is whether it is the maximum or minimum value in the pixel region. However, the criterion for determining the singular value is not limited to this. In step S12, when the rank of the pixel value of the pixel of interest in the pixel region is higher than the upper limit of a predetermined range (second range), it may be determined as convex. When the rank of the pixel value of the pixel of interest in the pixel region is lower than the lower limit of the predetermined range, it may be determined as concave. For example, when this predetermined range is from the 3rd to the 7th rank, when the rank of the pixel value of the pixel of interest in the pixel region is the 1st (maximum) or 2nd rank, it is determined as convex. When the rank of the pixel value of the pixel of interest in the pixel region is the 9th (minimum) or 8th rank, it is determined as concave. If the upper limit of this predetermined range is set to the 2nd rank and the lower limit is set to the rank of the number obtained by subtracting 1 from the number of pixels in the pixel region, the same processing as when the criterion is whether it is the maximum or minimum value in the pixel region is performed. That is, when the rank of the pixel value of the pixel of interest in the pixel region is the 1st (maximum) rank, it is determined as convex. When the rank of the pixel value of the pixel of interest in the pixel region is the lowest (minimum) rank, it is determined as concave.

[0081] In the above example, for the sake of simplicity of explanation, the case where the value of the threshold N for the number of connections in the determination of isolated points is 1, that is, the case where a pixel not connected to other pixels is regarded as an isolated point, is described. However, the value of the threshold N may be 2 or more. By setting the value of the threshold N to 2 or more, it is possible to perform a determination considering connected abnormal pixels. Hereinafter, considering the case where the value of the threshold N is set to a value other than 1 in the correction process of this embodiment, the correction process will be described more generally. In this case, the pixels included in the first connected pixel group with the number of connections being N or less are isolated points.

[0082] First, referring to FIGS. 8(a) and 8(b), the determination process of isolated points in step S13 of FIG. 4 will be described. In this description, it is assumed that the value of the threshold N for isolated points is 3.

[0083] FIG. 8(a) is a diagram schematically showing an example of pixel values in the correction process according to the present embodiment. FIG. 8(b) is a diagram schematically showing an example of pixel states in the correction process according to the present embodiment. In FIGS. 8(a) and 8(b), the notation method of these figures is the same as that in FIGS. 5(a) and 5(b). Also, the pixel values, connection data, and concavo-convex data shown in FIGS. 8(a) and 8(b) are the same as those in FIGS. 5(a) and 5(b).

[0084] In FIGS. 8(a) and 8(b), the hatched pixels P15, P43, and P45 are not connected to other pixels, and the connection number is 1, that is, since the connection number is equal to or less than the threshold value (N = 3), the pixels P15, P43, and P45 are isolated points. This is the same as that shown in FIGS. 5(a) and 5(b).

[0085] Also, in FIG. 8(b), the first connected pixel group G1 indicated by the broken line includes pixels P42, P52, and P53. Referring to the connection data of the pixels P42, P52, and P53, the pixels P42, P52, and P53 are connected to each other. Therefore, the connection number of the pixels P42, P52, and P53 in the first connected pixel group G1 is 3. That is, since the connection number is equal to or less than the threshold value (N = 3), the pixels P42, P52, and P53 are also isolated points. This connection number can be determined by tracing the connection relationship with reference to the connection data of each pixel.

[0086] Next, when the threshold value N of the connection number is 2 or more, the determination process of the correction target in steps S12 and S13 of FIG. 4 will be described.

[0087] The determination of the convex isolated point or the concave isolated point for the target pixel is the same as when the threshold value N is 1. However, when the threshold value N of the connection number is 2 or more, the number of pixels in the first connected pixel group may be plural, and in that case, it may not be determined that all the pixels in the first connected pixel group are the same type of isolated point. For example, in the example of FIG. 8(b), among the pixels P42, P52, and P53, the pixel value of the pixel P52 is convex. However, the pixel values of the pixels P42 and P53 are not convex because the pixel P52 is in the vicinity.

[0088] When the linking number is 2 or more in this way, among the first connected pixel groups determined to be isolated points, if at least one pixel determined to be convex is included, then all the pixels of the first connected pixel group shall be considered convex isolated points. Also, among the first connected pixel groups determined to be isolated points, if at least one pixel determined to be concave is included, then all the pixels of the first connected pixel group shall be considered concave isolated points. That is, in any case, all the pixels of the first connected pixel group are to be the correction targets (step S14 in FIG. 4). Also, when neither a pixel determined to be convex nor a pixel determined to be concave is included among the first connected pixel groups determined to be isolated points, all the pixels of the first connected pixel group are outside the correction targets (step S15 in FIG. 4).

[0089] Note that there may be a case where the first connected pixel group determined to be an isolated point includes both a pixel determined to be convex and a pixel determined to be concave. It is desirable that exception handling corresponding to such a case be defined. As a specific example of the exception handling, all the pixels of the first connected pixel group may be regarded as convex isolated points, or all the pixels of the first connected pixel group may be regarded as concave isolated points. Also, as another specific example of the exception handling, all the pixels of the first connected pixel group may be neither convex isolated points nor concave isolated points and may be outside the correction targets.

[0090] Note that in the above example, it is assumed that the same correction process is performed on the pixels within the first connected pixel group determined to be an isolated point, but this is not limited thereto. For example, only the pixels determined to be convex or concave among the first connected pixel groups determined to be isolated points may be the correction targets.

[0091] The threshold value N of the linking number may be set to different values for convex isolated points and concave isolated points. Also, the value of the threshold value N of the linking number may be changed for each imaging frame period of the photoelectric conversion device.

[0092] More specifically, the data reference range in the process of determining the linking number in step S13 will be described. The first linking number determination unit 137 refers to the data area of the pixels in the range of (2N - 1) × (2N - 1) centered on the target pixel among the data held in the first pixel state holding unit 134, and determines the linking number.

[0093] The reason for setting the reference range of the data area as described above will be explained. If it can be determined that the number of pixels reachable by tracing the pixels connected to the target pixel starting from the target pixel is N or less including the target pixel, it can be said that the linking number of the first connected pixel group including the target pixel is N or less. Also, if N pixels are linearly connected in the same direction from the pixel adjacent to the target pixel, it can be said that the linking number is N + 1 or more. Therefore, in order to determine whether the linking number is (N + 1) or more, it is sufficient to refer to the pixel states in the range of (N - 1) pixels each in the up, down, left, and right directions from the target pixel centered on the target pixel. In other words, it is possible to determine whether the linking number is N or less by referring to the data area of the pixels in the range of (2N - 1) × (2N - 1) centered on the target pixel.

[0094] As an example, the case where the linking number threshold N is 2 will be described. In this case, the first linking number determination unit 137 refers to the pixels in the range of 3 × 3 centered on the target pixel, that is, the data area of the target pixel and its surrounding 8 pixels, and determines the linking number. If the target pixel and its surrounding pixels are not connected, the linking number of the first connected pixel group including the target pixel is 1. If the target pixel and two or more of its surrounding pixels are connected, the linking number of the first connected pixel group including the target pixel is greater than 2. If the target pixel is connected to one of its surrounding pixels and the connected pixel is only connected to the target pixel, the linking number of the first connected pixel group including the target pixel is 2. If the target pixel is connected to one of its surrounding pixels and the connected pixel is also connected to pixels other than the target pixel, the linking number of the first connected pixel group including the target pixel is greater than 2. In this way, by calculating the linking number, it is possible to determine whether the linking number is 2 or less.

[0095] Note that when determining the linking number, for at least the pixels corresponding to the above data area, the first linking process by the first linking unit 135 and the determination of the singular value of the convex or concave type by the first pixel value determination unit 133 are completed. The first linking unit 135 and the first pixel value determination unit 133 refer to the pixel values of the pixels in the 3×3 range centered on the target pixel. Therefore, the range of pixel values related to the determination of the linking number by the first linking number determination unit 137 is in the range of (2N + 1)×(2N + 1).

[0096] In the present embodiment, for simplicity of explanation, it is assumed that the pixel data holding unit 131 and the first pixel state holding unit 134 hold data for one frame. However, as described above, since the first linking number determination unit 137 refers to the data area of the pixels in the range of (2N - 1)×(2N - 1) centered on the target pixel, it is sufficient to have the data from (N - 1) rows before the target pixel to (N - 1) rows after the target pixel. However, in the process of the first linking unit 135, since the data (N - 1) rows after the target pixel and the data N rows after the target pixel are compared, the data N rows after the target pixel is also required. Therefore, the first pixel state holding unit 134 can also be configured by a band memory having a storage capacity capable of holding the data from (N - 1) rows before the target pixel to N rows after the target pixel.

[0097] Also, the first linking unit 135, the first pixel value determination unit 133, the first linking number determination unit 137, and the correction data generation unit 132 refer to the data area of the pixels in the 3×3 range centered on the target pixel from the pixel data holding unit 131. Here, for the target pixel in the process of the correction data generation unit 132, the target pixels of the first linking unit 135 and the first pixel value determination unit 133 are (N - 1) rows after. Therefore, the pixel data holding unit 131 can also be configured by a band memory having a storage capacity capable of holding data of about (N + 1) rows.

[0098] In addition, in the determination of the connection state in the first connection number determination unit 137, an algorithm for determining the connection number such as Union-Find may be used. Even in this case, in order to determine whether the connection number is N or less, it is sufficient to refer to the data area of the pixels in the range of (2N - 1) × (2N - 1) centered on the target pixel. Therefore, the above-described band memory can be applied.

[0099] As described above, according to the present embodiment, in the correction process of abnormal pixels, it is possible to detect and correct connected abnormal pixels with a connection number of N or less. The value of N can be appropriately set according to the occurrence status of abnormal pixels and the like. Therefore, according to the present embodiment, an information processing apparatus and an information processing method capable of more suitably correcting abnormal pixels are provided.

[0100] In addition, in the process of the present embodiment, the position information of the abnormal pixels acquired in advance and the image captured in the light-shielded state are unnecessary. Therefore, abnormal pixels can be corrected more simply. In addition, since the storage capacity for holding these data is also unnecessary, the storage capacity of the information processing apparatus can be reduced.

[0101] [Second Embodiment] In the present embodiment, a modification example of the correction process of the first embodiment will be described. In the present embodiment, the description of the elements common to the first embodiment may be omitted or simplified.

[0102] The subject may include a pattern such as a line, and in such a case, the pixel value may become a specific value due to such a pattern. This specific image value is caused by the pattern of the subject and should not be corrected originally. A pixel group having such a specific image value is called a texture. In a texture, pixels with large fluctuations in pixel values may appear at a relatively high density.

[0103] In the present embodiment, a configuration example for preventing partial miscorrection of the texture is shown by determining whether the target pixel is a pixel constituting the texture in the process of the correction process of the first embodiment.

[0104] FIG. 9 is a functional block diagram of the correction processing function of the information processing apparatus 1 according to the present embodiment. The information processing apparatus 1 further includes a second pixel value determination unit 138, a determination result holding unit 139, a second connection unit 140, a second pixel state holding unit 141, and a second connection number determination unit 142 in addition to the configuration similar to that of the first embodiment.

[0105] The functions of the determination result holding unit 139 and the second pixel state holding unit 141 are realized, for example, by the data storage unit 122 in FIG. 1. The functions of the second pixel value determination unit 138, the second connection unit 140, and the second connection number determination unit 142 are realized, for example, by the information processing unit 129 in FIG. 1. The functions of the second pixel value determination unit 138, the second connection unit 140, and the second connection number determination unit 142 may be realized by the CPU 125 executing a correction processing program. This correction processing program may be held in advance in the data storage unit 122 or the ROM 127, or may be acquired from another device via the communication unit 128. The operations of these respective units will be described later.

[0106] FIG. 10 is a flowchart showing the correction processing executed by the information processing apparatus 1 according to the present embodiment. The correction processing method of the present embodiment will be described along the flowchart of FIG. 10. In the flowchart of FIG. 10, steps S11, S12, S13, S14, and S15 are the same as those in FIG. 4, and thus the description thereof will be omitted or simplified.

[0107] In step S21, the second pixel value determination unit 138, the determination result holding unit 139, the second connection unit 140, and the second pixel state holding unit 141 perform a connection process on pixels having a high pixel value (second connection process). The operations of the respective units will be described with further reference to FIG. 11. FIG. 11 is a diagram schematically showing an example of pixel values in the correction processing according to the present embodiment.

[0108] The second pixel value determination unit 138 acquires the pixel value of each of the plurality of pixels constituting the image from the pixel data holding unit 131. Then, the second pixel value determination unit 138 determines whether the pixel value of the pixel of interest is high, based on representative values (second representative value, third representative value) determined by the pixel values of the pixel region of three rows and three columns including the pixel of interest and its eight neighboring pixels. Note that the representative value is, for example, the median value of the pixel values of the pixel region of three rows and three columns including the pixel of interest and its eight neighboring pixels, and in the following description, the representative value is assumed to be the median value.

[0109] First, focus on pixel P32 in FIG. 11. The pixel value of pixel P33 is "500". The median value (representative value) of the pixel values of the pixel region of three rows and three columns including pixel P32 and its eight neighboring pixels is "250". Therefore, it is determined that the pixel value of pixel P32 is high. Next, focus on pixel P33 in FIG. 11. The pixel value of pixel P33 is "1000". The median value (representative value) of the pixel values of the pixel region of three rows and three columns including pixel P33 and its eight neighboring pixels is "500". Therefore, it is determined that the pixel value of pixel P33 is high. When the same determination is made for other pixels, in the example of FIG. 11, the four hatched pixels P32, P33, P34, and P35 are determined to be pixels having high pixel values.

[0110] The determination result holding unit 139 holds information indicating the determination result by the second pixel value determination unit 138 for each pixel. This information can be, for example, 1-bit data per pixel such that a pixel having a high pixel value is "1" and a pixel not having a high pixel value is "0".

[0111] The second connection unit 140 refers to the information held by the determination result holding unit 139, and determines that two adjacent pixels (the first pixel and the second pixel) belong to the same second connected pixel group when both have high pixel values (second connection process). The second pixel state holding unit 141 holds information indicating the determination result of the second connection process in the same format as the connection data described in the first embodiment. The process in which the second connection unit 140 determines that two pixels belong to the same second connected pixel group and the second pixel state holding unit 141 holds the pixel states of these pixels may also be expressed as "connect".

[0112] In the example of FIG. 11, the pixel P32 and the pixel P33 are connected by the connection line N2, the pixel P33 and the pixel P34 are connected by the connection line N3, and the pixel P34 and the pixel P35 are connected by the connection line N4. Therefore, the connection number of the pixels P32, P33, P34, and P35 in the second connected pixel group G2 is 4.

[0113] Referring to FIG. 10 again, the processes after the second connection process in step S21 will be described. In the present embodiment, when it is determined in step S13 that the pixel value of the target pixel is an isolated point (YES in step S13), the process proceeds to step S22. In step S22, the second connection number determination unit 142 determines whether the target pixel is a texture based on whether the connection number calculated from the connection data held by the second pixel state holding unit 141 is greater than a threshold value M (second threshold value) (determination of the third condition). The threshold value M is a positive integer, but it is desirable that M be 2 or more from the viewpoint of detecting that high pixel values are connected. When the connection number is greater than the threshold value M (YES in step S22), the target pixel is determined to be a texture, and the process proceeds to step S15. In this case, the target pixel is not a correction target. When the connection number is less than or equal to the threshold value M (NO in step S22), the target pixel is determined not to be a texture, and the process proceeds to step S14. In this case, the target pixel is a correction target. In step S14 or step S15, the correction unit 136 performs the same pixel value output process as in the first embodiment.

[0114] For the same reason as described in the first embodiment, the data reference range in the process of determining the linking number in step S22 is in the range of (2M - 1) × (2M - 1). That is, the second linking number determination unit 142 refers to the data area of the pixels in the range of (2M - 1) × (2M - 1) centered on the target pixel among the data held in the second pixel state holding unit 141, and determines the linking number.

[0115] For example, assuming that the value of the threshold M is 3, in the example of FIG. 11, since the linking numbers of the pixels P32, P33, P34, and P35 in the second linked pixel group G2 are 4, the pixels P32, P33, P34, and P35 are textures. Therefore, the correction process for the pixels P32, P33, P34, and P35 is not performed.

[0116] FIGS. 12(a) to 12(d) are graphs showing examples of thresholds used for determining whether the pixel value of the target pixel is high based on the representative value in the second pixel value determination unit 138. In FIGS. 12(a) to 12(d), the horizontal axis represents the representative value calculated from the pixel values of the pixels in a 3-row and 3-column pixel region including the target pixel and its neighboring 8 pixels, and the vertical axis represents the pixel value of the target pixel. In FIGS. 12(a) to 12(d), the straight line L6 is a straight line with a slope of 1 and an intercept of 0 (i.e., y = x).

[0117] In FIG. 12(a), the straight line L7 indicates the lower threshold for determining that the pixel value of the target pixel is high. The slope of the straight line L7 is greater than 1, and the intercept of the straight line L2 is greater than 0. Also, on the straight line L7, when x = x2, y = y7. When the representative value is x2, if the pixel value of the target pixel is y7 or more, it is determined that the pixel value of the target pixel is high.

[0118] The function for determining the threshold is not limited to that shown in FIG. 12(a). For example, as shown in FIG. 12(b), the lower threshold may be given by a broken line L8 composed of a plurality of line segments. On the straight line L8, when x = x2, y = y8. When the representative value is x2, if the pixel value of the target pixel is y8 or more, it is determined that the pixel value of the target pixel is high.

[0119] Also, for example, a lower threshold value may be given by a curve L9 as shown in FIG. 12(c). On the straight line L9, when x = x2, y = y9. When the representative value is x2, if the pixel value of the target pixel is y9 or more, it is determined that the pixel value of the target pixel is high.

[0120] In the above example, the second pixel value determination unit 138 determines that the pixel value of the target pixel is high based on the representative value. However, it may be determined that the pixel value of the target pixel is low based on the representative value. In this case, when the pixel value of the target pixel is low, it can be determined that the target pixel is a texture. When such a determination condition is applied, for example, an upper threshold value may be given by a straight line L10 as shown in FIG. 12(d). The slope of the straight line L10 is smaller than 1, and the intercept of the straight line L10 is smaller than 0. Also, on the straight line L10, when x = x2, y = y10. When the representative value is x2, if the pixel value of the target pixel is y10 or less, it is determined that the pixel value of the target pixel is low.

[0121] Determination may be made for both the case where the pixel value of the target pixel is high and the case where it is low. In this example, when the representative value is x2, if the pixel value of the target pixel is y7 or more, it is determined that the pixel value of the target pixel is high, and if the pixel value of the target pixel is y10 or less, it is determined that the pixel value of the target pixel is low. That is, it can be rephrased that the determination by the second pixel value determination unit 138 in the present embodiment is to determine that the difference between the pixel value of the target pixel and the representative value is outside a predetermined range (outside the third range or outside the fourth range).

[0122] In the example of FIG. 12(d), one or both of the straight line L7 and the straight line L10 may be replaced with a broken line or may be replaced with a curve. Also, the straight line L7 and the straight line L10 may be replaced with different types of lines.

[0123] The above lower threshold value and upper threshold value can be obtained by the same method as the method for obtaining the threshold value used in the processing of the first connection portion 135 described in the first embodiment. That is, the above lower threshold value and upper threshold value may be calculated by a function or may be obtained by a look-up table. Further, the above lower threshold value and upper threshold value may be the same value as the threshold value used in the processing of the first connection portion 135 described in the first embodiment, or may be different values.

[0124] In the example of FIG. 12(d), when both the determination when the pixel value of the pixel of interest is high and the determination when the pixel value of the pixel of interest is low are performed, the threshold value M of the number of connections may be a different value between the texture with a high pixel value and the texture with a low pixel value. Further, the value of the threshold value M of the number of connections may be changed for each imaging frame period of the photoelectric conversion device.

[0125] In the present embodiment, for simplicity of explanation, it is assumed that the pixel data holding unit 131 and the second pixel state holding unit 141 hold data for one frame. However, as described above, since the second number-of-connections determination unit 142 refers to the data area of the pixels in the range of (2M - 1)×(2M - 1) centered on the pixel of interest, it is sufficient if there is data from (M - 1) rows before the pixel of interest to (M - 1) rows after the pixel of interest. However, in the processing of the second connection portion 140, since the data (M - 1) rows after the pixel of interest and the data M rows after the pixel of interest are compared, the data M rows after the pixel of interest is also necessary. Therefore, the second pixel state holding unit 141 can also be configured by a band memory having a storage capacity capable of holding data from (M - 1) rows before the pixel of interest to M rows after the pixel of interest.

[0126] Also, the second pixel value determination unit 138 refers to the data areas of the pixels in a 3×3 range centered on the target pixel from the pixel data holding unit 131. Then, the second pixel state holding unit 141 holds the determination results up to M rows after for the target pixel of the correction data generation unit 132 and the second connectivity number determination unit 142. Therefore, the pixel data holding unit 131 needs to hold information up to (M + 1) rows after. Furthermore, the correction data generation unit 132 also refers to the data areas of the pixels in a 3×3 range centered on the target pixel. As described above, the pixel data holding unit 131 can also be configured by a band memory having a storage capacity capable of holding data of about (M + 2) rows. However, this is not the case when N > M.

[0127] Also, in the determination of the connection state in the second connectivity number determination unit 142, an algorithm for determining the connectivity number such as Union-Find may be used. Even in this case, in order to determine whether the connectivity number is M or less, it is sufficient to refer to the data areas of the pixels in a (2M - 1)×(2M - 1) range centered on the target pixel, so the above-described band memory can be applied.

[0128] As described above, according to the present embodiment, an information processing apparatus and an information processing method that obtain the same effects as those of the first embodiment are provided. Also, in the present embodiment, by determining the texture and excluding it from the correction target, it is possible to reduce the influence on the image quality due to incorrect correction of the texture.

[0129] In the above description, it is assumed that the image to be processed is a grayscale image, but the same processing is applicable to an image including data of a plurality of channels such as a color image. In this case, by applying the processing of the present embodiment to a channel image in which data of the same channel (data of the same color) is collected, the same processing can be performed.

[0130] An application example to a Bayer image, which is an example of an image including data of multiple channels, will be described below. Here, the Bayer image refers to an image before demosaicing processed by an imaging device provided with a color filter of a Bayer array. The Bayer image forms a repeating array with a basic unit of two rows and two columns. In the first row of the two rows and two columns, a red pixel (R) and a green pixel (Gr) are arranged, and in the second row, a green pixel (Gb) and a blue pixel (B) are arranged.

[0131] In the Bayer image, only data of one channel is held in one pixel. For example, when there is a red line-shaped texture on the image, high pixel values of the red (R) channel may appear only. On the other hand, when there is a white line-shaped texture on the image, high pixel values in the form of a line may appear in all channels of red (R), green (Gr, Gb), and blue (B). Considering these, a modified example in which the processing content is extended so that the connection process in the second connection unit 140 is performed for two types, between different channels and between the same channels, will be described.

[0132] FIG. 13 is a diagram schematically showing an arrangement of pixels according to a modification of the present embodiment. In FIG. 13, pixel P33 is a pixel of interest. In FIG. 13, the pixels (such as pixels P11, P13, P31, etc.) indicated by the solid circles are pixels of the same channel as the pixel of interest P33, and the pixels (such as pixels P12, P22, P32, etc.) indicated by the broken circles are pixels of a channel different from the pixel of interest. Pixels P22, P23, P24, P32, P34, P42, P43, P44 are pixels of a channel different from the pixel of interest P33 and are adjacent to the pixel of interest P33. Pixels P11, P13, P15, P31, P35, P51, P53, P55 are pixels of the same channel as the pixel of interest P33 and are adjacent to the pixel of interest P33 in a channel image obtained by gathering pixels of the same channel. The second connection unit 140 uses the above-described pixels P22, P23, P24, P32, P34, P42, P43, P44 and pixels P11, P13, P15, P31, P35, P51, P53, P55 as candidates for connection destinations from the pixel of interest P33. Thereby, connection processing can be performed in consideration of both a texture of one color and a white texture. In this modification, since the number of pixels of the connection destination is 16, the connection data is 16-bit data.

[0133] [Third Embodiment] In the present embodiment, a modification of the correction process of the second embodiment will be described. In the present embodiment, descriptions of elements common to the first embodiment or the second embodiment may be omitted or simplified.

[0134] In the second embodiment, an example in which pixels determined as textures are excluded from correction targets was shown. However, there may be cases where abnormal pixels having pixel values extremely different from surrounding pixels are included in the texture. In the present embodiment, a configuration example for detecting and correcting abnormal pixels in the texture will be shown.

[0135] FIG. 14 is a functional block diagram regarding the correction function of the information processing apparatus 1 according to the present embodiment. The information processing apparatus 1 further includes a protrusion determination unit 143 in addition to the same configuration as that of the second embodiment.

[0136] The function of the protrusion determination unit 143 is realized, for example, by the information processing unit 129 in FIG. 1. The function of the protrusion determination unit 143 may be realized by the CPU 125 executing a correction processing program. This correction processing program may be held in advance in the data storage unit 122 or the ROM 127, or may be acquired from another device via the communication unit 128. The operations of these respective units will be described later.

[0137] FIG. 15 is a flowchart showing the correction processing executed by the information processing apparatus 1 according to the present embodiment. The correction processing method of the present embodiment will be described along the flowchart of FIG. 15. In the flowchart of FIG. 15, steps S11, S12, S13, S14, S15, S21, and S22 are the same as those in FIG. 10, and thus the description thereof will be omitted or simplified.

[0138] In the present embodiment, when the linking number is greater than the threshold value M in step S22 (YES in step S22), the process proceeds to step S31. In step S31, the protrusion determination unit 143 refers to the pixel values of the pixel region of three rows and three columns including the target pixel and its eight neighboring pixels, and determines whether the pixel value of the target pixel protrudes (determination of the fourth condition). When it is determined that the pixel value of the target pixel protrudes (YES in step S31), the process proceeds to step S14. In this case, the target pixel is a correction target. When it is not determined that the pixel value of the target pixel protrudes (NO in step S31), the process proceeds to step S15. In this case, the target pixel is not a correction target.

[0139] Here, when the pixel value is prominent, it refers to a situation where the pixel value of the target pixel is extremely convex or extremely concave with respect to the surrounding pixels. "Extremely convex" refers to a situation where, for example, the pixel value is equal to or greater than a threshold value (the third threshold value). Also, "extremely concave" refers to a situation where, for example, the pixel value is equal to or less than a threshold value (the fourth threshold value). More specific examples of "extremely convex" and "extremely concave" will be described. When the pixel value of the target pixel is the largest among the target pixel and its eight neighboring pixels, and the pixel value of the target pixel is Q times (Q is a number greater than or equal to 1) or more (equal to or greater than the third threshold value) with respect to the largest pixel value among the eight pixels excluding the target pixel, the pixel value of the target pixel is considered to be extremely convex. Also, when the pixel value of the target pixel is the smallest among the target pixel and its eight neighboring pixels, and the pixel value of the target pixel is (1 / Q) times or less (equal to or less than the fourth threshold value) with respect to the smallest pixel value among the eight pixels excluding the target pixel, the pixel value of the target pixel is considered to be extremely convex. This method of setting the threshold value is an example, and for example, threshold values as shown in FIGS. 12(a) to 12(d) may be set.

[0140] Hereinafter, taking M as 3 and Q as 3, a specific example of the process of the protrusion determination unit 143 will be described. FIG. 16 is a diagram schematically showing an example of pixel values in the correction process according to the present embodiment. FIG. 16 shows an example where the pixel P44 has an extremely large pixel value "8000". Taking the pixel P44 as the target pixel, the pixel values of a pixel region of three rows and three columns including its eight neighboring pixels are referred to. At this time, since the pixel value "8000" of the pixel P44 is three times or more larger than the second largest pixel value "1000" of the pixel P33 or the pixel P34, it is an extremely convex pixel value. Also, when the determination is made by the method described in the second embodiment, the connectivity number of the pixels P32, P33, P34, P35, and P44 is 5, which is larger than the threshold value M, so these are textures. In the present embodiment, in such a case, the pixels P32, P33, P34, and P35 are determined to be outside the correction target, but the pixel P44 is determined to be a correction target because it is an extremely convex pixel and is prominent.

[0141] As described above, according to this embodiment, an information processing apparatus and an information processing method that can obtain the same effects as those of the second embodiment are provided. Further, in this embodiment, abnormal pixels in the texture can be detected and corrected.

[0142] [Fourth Embodiment] In this embodiment, another example of the texture determination process in the second embodiment will be described. In this embodiment, descriptions of elements common to the first embodiment or the second embodiment may be omitted or simplified.

[0143] FIG. 17 is a functional block diagram of the correction processing function of the information processing apparatus 1 according to this embodiment. The information processing apparatus 1 further includes a density determination unit 144 instead of the second connection unit 140, the second pixel state holding unit 141, and the second connection number determination unit 142 in FIG. 9.

[0144] The function of the density determination unit 144 is realized, for example, by the information processing unit 129 in FIG. 1. The function of the density determination unit 144 may also be realized by the CPU 125 executing a correction processing program. This correction processing program may be held in advance in the data storage unit 122 or the ROM 127, or may be acquired from another device via the communication unit 128. The operations of these respective units will be described later.

[0145] FIG. 18 is a flowchart showing the correction processing executed by the information processing apparatus 1 according to this embodiment. The correction processing method of this embodiment will be described according to the flowchart of FIG. 18. In the flowchart of FIG. 18, steps S11, S12, S13, S14, and S15 are the same as those in FIG. 10, so the descriptions are omitted or simplified.

[0146] In step S41, the second pixel value determination unit 138 and the determination result holding unit 139 perform a determination process for pixels having a high pixel value (pixel value determination process). This process is substantially the same as the process of the second pixel value determination unit 138 and the determination result holding unit 139 described in the second embodiment. That is, the second pixel value determination unit 138 determines whether the pixel value of the target pixel is high based on a representative value determined by the pixel values of the pixel region of three rows and three columns including the target pixel and its eight neighboring pixels. Then, the determination result holding unit 139 holds information indicating the determination result by the second pixel value determination unit 138 for each pixel. Note that the pixel value determination process may also be one that determines pixels having a low pixel value as described in the second embodiment. That is, it can be said that the determination by the second pixel value determination unit 138 in the present embodiment determines that the difference between the pixel value of the target pixel and the representative value is outside a predetermined range (outside the fifth range).

[0147] In this embodiment, when it is determined in step S13 that the pixel value of the target pixel is an outlier (YES in step S13), the process proceeds to step S42. In step S42, the density determination unit 144 acquires the determination result held in the determination result holding unit 139, and calculates the number of pixels determined to have a high pixel value within a pixel region of five rows and five columns including the target pixel and its 24 neighboring pixels. In other words, the density determination unit 144 calculates the density of pixels determined to have a high pixel value within a predetermined range including the target pixel. The density determination unit 144 determines whether the target pixel is a texture or not based on whether the number of pixels determined to have a high pixel value is greater than a threshold value R (the fifth threshold value) (determination of the fifth condition). The threshold value R is a positive integer, and it is desirable that R be 2 or more from the viewpoint of detecting that pixels with high pixel values exist densely. When the number of pixels determined to have a high pixel value is greater than the threshold value R (YES in step S42), the target pixel is determined to be a texture, and the process proceeds to step S15. In this case, the target pixel is not a correction target. When the number of pixels determined to have a high pixel value is less than or equal to the threshold value R (NO in step S42), the target pixel is determined not to be a texture, and the process proceeds to step S14. In step S14 or step S15, the correction unit 136 performs the same pixel value output process as in the first embodiment.

[0148] FIG. 19 is a diagram schematically showing an example of pixel values in the correction process according to this embodiment. A specific example of the process of this embodiment will be described assuming that the threshold value R in the above-described process is 3. In the example of FIG. 19, similar to FIG. 11, four pixels P32, P33, P34, and P35 with hatches are determined to be pixels having high pixel values. In a pixel region R1 of five rows and five columns including the pixel P33, which is the target pixel, and its 24 neighboring pixels, there are four pixels determined to have a high pixel value. Since this number is greater than the threshold value R, the density determination unit 144 determines that the pixel P33 is a texture and excludes the pixel P33 from the correction target.

[0149] In this embodiment, for the sake of simplicity of explanation, it is assumed that the pixel data holding unit 131 and the determination result holding unit 139 hold data for one frame. However, as described above, since the density determination unit 144 refers to the data area of the pixel region R1 in the vicinity of the target pixel, it is sufficient to have data corresponding to the number of rows of the pixel region R1. Therefore, the determination result holding unit 139 can also be configured by a band memory having a storage capacity capable of holding data corresponding to the number of rows of the pixel region R1 referred to in the process of the density determination unit 144.

[0150] Also, the correction data generation unit 132 refers to the data areas of the pixels in a 3×3 range centered on the target pixel. And the second pixel value determination unit 138 refers to the data areas of the pixels in a 3×3 range centered on the pixel in the final row within the pixel region R1 in the vicinity of the target pixel. Therefore, the pixel data holding unit 131 can also be configured by a band memory having a storage capacity capable of holding data from one row before the target pixel to the row next to the final row of the pixel region R1 in the vicinity of the target pixel.

[0151] As described above, according to this embodiment, an information processing apparatus and an information processing method that can obtain the same effects as those of the first embodiment are provided. Also, in this embodiment, similar to the second embodiment, by determining the texture and excluding it from the correction target, the influence on the image quality due to incorrect correction of the texture can be reduced.

[0152] Note that the method of detecting and correcting abnormal pixels in the texture described in the third embodiment is also applicable to this embodiment.

[0153] [Fifth Embodiment] In this embodiment, a modified example of the correction process of the first embodiment will be described. In this embodiment, the description of the elements common to the first to fourth embodiments may be omitted or simplified.

[0154] In this embodiment, as an example of a method for determining the threshold value N of the linking number in the determination of isolated points in the first embodiment, a configuration example for determining the threshold value N of the linking number based on temperature information and the representative value of the neighboring pixel group is shown.

[0155] FIG. 20 is a functional block diagram regarding the correction processing function of the information processing apparatus 1 according to this embodiment. In addition to the same configuration as in the first embodiment, the information processing apparatus 1 further includes a representative value calculation unit 150, an information acquisition unit 151, a first table holding unit 152, and a first threshold determination unit 153.

[0156] The function of the information acquisition unit 151 is realized, for example, by the data input unit 121 in FIG. 1. The information acquisition unit 151 has a function of acquiring temperature information, and the data input unit 121 may include a temperature measurement device such as a temperature sensor for acquiring temperature information. The function of the first table holding unit 152 is realized, for example, by the data storage unit 122 in FIG. 1. The functions of the representative value calculation unit 150 and the first threshold determination unit 153 are realized, for example, by the information processing unit 129 in FIG. 1. The functions of the representative value calculation unit 150 and the first threshold determination unit 153 may be realized by the CPU 125 executing a correction processing program. This correction processing program may be held in advance in the data storage unit 122 or the ROM 127, or may be acquired from another device via the communication unit 128. The operations of these respective units will be described later.

[0157] The overall flow of the correction processing executed by the information processing apparatus 1 in this embodiment is substantially the same as that in the first embodiment except for the method of determining the threshold value N of the linking number, and thus the description thereof is omitted. FIG. 21 is a flowchart showing the determination processing of the threshold value N executed by the information processing apparatus 1 according to this embodiment.

[0158] In step S51, the information acquisition unit 151 acquires a frame start signal. The frame start signal is a signal that is "1" when the input pixel of interest is the first pixel of a frame, and "0" otherwise. Based on the frame start signal, the information acquisition unit 151 determines whether the input pixel of interest is the first pixel of a frame. If the input pixel of interest is the first pixel of a frame (YES in step S51), the process proceeds to step S52. In step S52, the information acquisition unit 151 acquires temperature information. The temperature information indicates the temperature of the photoelectric conversion device itself or the temperature of the environment in which the photoelectric conversion device is installed. If the input pixel of interest is not the first pixel of a frame (NO in step S51), the process proceeds to step S53 and the acquisition of temperature information is not performed.

[0159] In step S53, the representative value calculation unit 150 acquires the pixel values of a pixel region (neighboring pixel group) of three rows and three columns including the pixel of interest and eight neighboring pixels in its vicinity. Then, the representative value calculation unit 150 calculates a representative value (first representative value) from the pixel values of this pixel region. FIG. 22 is a diagram schematically showing the pixel of interest and neighboring pixels according to the present embodiment. Assuming that the pixel P22 in FIG. 22 is the pixel of interest, the pixels P11, P12, P13, P21, P23, P31, P32, and P33 in FIG. 22 are neighboring pixels. At this time, the representative value is, for example, the average value of the pixel values of the nine pixels P11 to P33.

[0160] Note that the range of the pixel region used for calculating the representative value is not limited to three rows and three columns, and may be, for example, a wider range. Also, the representative value may be a statistical value other than the average value, and may be, for example, a weighted average value, a median value, or a mode value of the pixel region. Further, the representative value may be calculated from the eight pixel values obtained by excluding the pixel of interest from the pixel region of three rows and three columns.

[0161] In step S54, the first threshold determination unit 153 acquires the temperature information acquired by the information acquisition unit 151 and the representative value calculated by the representative value calculation unit 150. The first threshold determination unit 153 refers to the two-dimensional look-up table held in the first table holding unit 152 and determines the threshold value N based on the temperature and the representative value. This threshold value N is used as the threshold value N for the number of connections in the determination of outliers described in the first embodiment.

[0162] FIG. 23 is an example of a two-dimensional look-up table used for determining the threshold value N according to the present embodiment. The first column of FIG. 23 shows an index based on the temperature range, and the first row of FIG. 23 shows an index based on the range of the representative value. The numerical values in the range from the second column to the seventh column and from the second row to the eleventh row of FIG. 23 indicate the values of the threshold value N determined based on the temperature and the representative value. For example, when the temperature is 60 degrees and the representative value is 200, the value of the threshold value N is 3 from the look-up table of FIG. 23. The first threshold determination unit 153 outputs the threshold value N determined in this way to the first connection number determination unit 137. Note that the temperature range, the range of the representative value, and the value of the threshold value N in FIG. 23 are examples and are not limited thereto. In this way, in the two-dimensional look-up table, the temperature information, the representative value, and the threshold value N are associated with each other.

[0163] An example of a method for preparing a two-dimensional look-up table will be described. First, the relationship between the temperature, the brightness of the background, and the occurrence probability of abnormal pixels is investigated under various temperatures. Then, based on the occurrence probability of abnormal pixels at each temperature, the probability that abnormal pixels are connected is calculated for each number of connections. Then, the number of connections at which this probability becomes a sufficiently small value is set as the threshold value N. For example, equations (1) to (4) can be used for the calculation of this probability.

[0164] Generally, the higher the temperature, the higher the probability of occurrence of abnormal pixels. Therefore, as shown in FIG. 23, a table is created such that the higher the temperature, the larger the threshold value N of the connectivity number. Also, when the background is bright (when the pixel value of neighboring pixels is large), the difference between the pixel value of the abnormal pixel and the pixel values of normal pixels in its vicinity becomes small, so the probability of misidentifying a normal pixel as being connected to an abnormal pixel increases. Therefore, as shown in FIG. 23, a table is created such that the larger the pixel value of neighboring pixels (i.e., the representative value), the smaller the threshold value N of the connectivity number. In creating this table, for example, when the pixel value of neighboring pixels is equal to or greater than a certain threshold value, an equation that subtracts a certain value from the threshold value of the connectivity number calculated from the dependency between the temperature and the occurrence rate of abnormal pixels may be used.

[0165] The storage medium constituting the first table holding unit 152 may be a ROM, a RAM, or a register. When the first table holding unit 152 is configured by a RAM or a register, the CPU 125 generates data for the lookup table and writes it to the RAM or the register, thereby holding the lookup table. The writing of the data for the lookup table may be performed prior to the processing in the information processing apparatus 1, or may be performed between the processing of one frame and the processing of another frame.

[0166] In this embodiment, as shown in FIG. 23, an example of determining the threshold value N of the connectivity number using a two-dimensional lookup table is shown, but the threshold value N of the connectivity number may be determined by another method. For example, the threshold value N of the connectivity number may be calculated using an equation with temperature information and the representative value as independent variables.

[0167] In this embodiment, for simplicity of explanation, it is assumed that the pixel data holding unit 131 and the first pixel state holding unit 134 hold data for one frame. However, this is not essential. The first linking number determination unit 137 refers to the data areas of the pixels in the range of (2N + 1) × (2N + 1) centered on the target pixel. Therefore, the pixel data holding unit 131 and the first pixel state holding unit 134 can also be configured by a band memory having a storage capacity capable of holding data from N rows before to N rows after the target pixel.

[0168] As described above, according to this embodiment, an information processing apparatus and an information processing method that obtain the same effects as those of the first embodiment are provided. Further, in this embodiment, the threshold value N of the linking number in the determination of the isolated point can be set based on the temperature information and the representative value calculated from the pixel values of the neighboring pixels. Thereby, since correction is performed in consideration of the change in the occurrence rate of abnormal pixels according to the temperature and the pixel values of the neighboring pixels, an information processing apparatus and an information processing method that can correct abnormal pixels more suitably are provided.

[0169] [Sixth Embodiment] In this embodiment, a modified example of the correction process of the second embodiment will be described. In this embodiment, the description of the elements common to the first to fifth embodiments may be omitted or simplified.

[0170] In this embodiment, as an example of a method for determining the threshold value M of the linking number in the determination of the texture of the second embodiment, a configuration example for determining the threshold value M of the linking number based on the temperature information and the representative value of the neighboring pixel group is shown.

[0171] FIG. 24 is a functional block diagram of the correction processing function of the information processing apparatus 1 according to this embodiment. The information processing apparatus 1 further includes a representative value calculation unit 150, an information acquisition unit 151, a first table holding unit 152, a first threshold determination unit 153, a second table holding unit 154, and a second threshold determination unit 155 in addition to the same configuration as that of the second embodiment. Since the operations of the representative value calculation unit 150, the information acquisition unit 151, the first table holding unit 152, and the first threshold determination unit 153 are substantially the same as those of the fifth embodiment, the description will be omitted or simplified as appropriate.

[0172] The function of the second table holding unit 154 is realized by, for example, the data storage unit 122 in FIG. 1. The function of the second threshold determination unit 155 is realized by, for example, the information processing unit 129 in FIG. 1. The function of the second threshold determination unit 155 may be realized by the CPU 125 executing a correction processing program. This correction processing program may be held in advance in the data storage unit 122 or the ROM 127, or may be acquired from another device via the communication unit 128. The operations of these respective units will be described later.

[0173] In this embodiment, the overall flow of the correction processing executed by the information processing apparatus 1 is substantially the same as that of the second embodiment except for the method of determining the threshold value M of the linking number, and thus the description thereof is omitted. FIG. 25 is a flowchart showing the determination processing of the threshold value N and the threshold value M executed by the information processing apparatus 1 according to this embodiment.

[0174] The processing from step S51 to step S54 is the same as that described in the fifth embodiment, and thus the description thereof is omitted. In step S55, the second threshold determination unit 155 acquires the temperature information acquired by the information acquisition unit 151 and the representative value calculated by the representative value calculation unit 150. The second threshold determination unit 155 refers to the two-dimensional look-up table held in the second table holding unit 154, and determines the threshold value M based on the temperature and the representative value. This threshold value M is used as the threshold value M of the linking number in the determination of the texture described in the second embodiment.

[0175] FIG. 26 is an example of a two-dimensional look-up table used to determine the threshold value M according to the present embodiment. The first column of FIG. 26 shows an index based on the temperature range, and the first row of FIG. 26 shows an index based on the range of representative values. The numerical values in the range from the second column to the seventh column and from the second row to the eleventh row of FIG. 26 indicate the values of the threshold value M determined based on the temperature and the representative value. For example, when the temperature is 60 degrees and the representative value is 200, the value of the threshold value M is 4 from the look-up table of FIG. 26. The second threshold determination unit 155 outputs the threshold value M determined in this way to the second connectivity number determination unit 142. Note that the temperature range, the range of representative values, and the values of the threshold value M in FIG. 26 are examples and are not limited thereto.

[0176] The above two-dimensional look-up table can be created in the same manner as the two-dimensional look-up table held in the first table holding unit 152 described in the fifth embodiment. Also, the configuration of the second table holding unit 154 can be the same as that of the first table holding unit 152.

[0177] In the present embodiment, as shown in FIG. 26, an example of determining the threshold value M of the connectivity number using a two-dimensional look-up table is shown, but the threshold value M of the connectivity number may be determined by another method. For example, the threshold value M of the connectivity number may be calculated using an equation with temperature information and the representative value as independent variables.

[0178] As described above, according to the present embodiment, an information processing device and an information processing method that can obtain the same effect as the second embodiment are provided. Also, in the present embodiment, a threshold M of the number of connections in determining the texture can be set based on temperature information and a representative value calculated from the pixel values ​​of the neighboring pixels. This allows the texture to be determined in consideration of the change in the occurrence rate of abnormal pixels according to the temperature and the pixel values ​​of the neighboring pixels. Also, similar to the fifth embodiment, a threshold N of the number of connections in determining the isolated point can be set based on the temperature information and a representative value calculated from the pixel values ​​of the neighboring pixels. This allows the correction to be performed in consideration of the change in the occurrence rate of abnormal pixels according to the temperature and the pixel values ​​of the neighboring pixels. Therefore, an information processing device and an information processing method that can more suitably correct abnormal pixels are provided.

[0179] In the present embodiment, the first table holding unit 152 and the second table holding unit 154 are described as separate elements, but one holding unit may serve both functions. For example, the threshold value N and the threshold value M may be determined by one lookup table. In addition, in the present embodiment, the first threshold value determining unit 153 and the second threshold value determining unit 155 are described as separate elements, but one threshold value determining unit may serve both functions. For example, one threshold value determining unit may determine both the threshold value N and the threshold value M, and the same threshold value may be supplied to the first linkage number determining unit 137 and the second linkage number determining unit 142.

[0180] The method of determining the threshold value M of the number of connections according to this embodiment is similarly applicable to the information processing device 1 according to the third embodiment.

[0181] [Seventh embodiment] In this embodiment, a modification of the correction process of the fourth embodiment will be described. In this embodiment, the description of elements common to the first to sixth embodiments may be omitted or simplified.

[0182] In this embodiment, as an example of a method for determining the pixel number threshold R in the density determination of the fourth embodiment, a configuration example will be shown in which the pixel number threshold R is determined based on temperature information and a representative value of a group of neighboring pixels.

[0183] FIG. 27 is a functional block diagram related to the correction processing function of the information processing apparatus 1 according to the present embodiment. The information processing apparatus 1 further includes a representative value calculation unit 150, an information acquisition unit 151, a first table holding unit 152, a first threshold determination unit 153, a second table holding unit 154, and a second threshold determination unit 155 in addition to the configuration similar to that of the fourth embodiment. Since the operations of the representative value calculation unit 150, the information acquisition unit 151, the first table holding unit 152, the first threshold determination unit 153, and the second table holding unit 154 are substantially the same as those of the sixth embodiment, the description thereof will be omitted or simplified as appropriate.

[0184] The overall flow of the correction process executed by the information processing apparatus 1 in the present embodiment is substantially the same as that of the fourth embodiment except for the method of determining the threshold value R of the number of pixels, and thus the description thereof will be omitted. Further, the determination process of the threshold value N and the threshold value R executed by the information processing apparatus 1 is the same as that of FIG. 25 except that the threshold value M is replaced with the threshold value R.

[0185] The second threshold determination unit 155 acquires the temperature information acquired by the information acquisition unit 151 and the representative value calculated by the representative value calculation unit 150 in the same manner as in the sixth embodiment. The second threshold determination unit 155 refers to the two-dimensional look-up table held in the second table holding unit 154 and determines the threshold value R based on the temperature and the representative value. This threshold value R is used as the threshold value R of the number of pixels in the density determination described in the fourth embodiment. The content of the two-dimensional look-up table can be the same as that of FIG. 26.

[0186] As described above, according to the present embodiment, an information processing apparatus and an information processing method that can obtain the same effects as those of the fourth embodiment are provided. Further, in the present embodiment, the threshold value R of the number of pixels in density determination can be determined based on the temperature information and the representative value calculated from the pixel values of neighboring pixels. Thereby, texture determination is performed in consideration of the change in the occurrence rate of abnormal pixels according to the temperature and the pixel values of neighboring pixels. Similarly to the fifth embodiment, the threshold value N of the number of connections in the determination of outliers can be set based on the temperature information and the representative value calculated from the pixel values of neighboring pixels. Thereby, correction is performed in consideration of the change in the occurrence rate of abnormal pixels according to the temperature and the pixel values of neighboring pixels. Therefore, an information processing apparatus and an information processing method that can correct abnormal pixels more suitably are provided.

[0187] [Eighth Embodiment] In the present embodiment, a modified example of the apparatus configuration of the seventh embodiment will be described. In the present embodiment, the description of elements common to the first to seventh embodiments may be omitted or simplified.

[0188] In the present embodiment, a modified example in which the information processing apparatus 1 receives temperature information from the outside is shown. In the configuration of the present embodiment, a temperature measuring device such as a temperature sensor that measures temperature is arranged outside the information processing apparatus 1.

[0189] FIG. 28 is a functional block diagram regarding the correction processing function of the information processing apparatus 1 according to the present embodiment. The information processing apparatus 1 has the same functional blocks as those in the seventh embodiment, but is different from the seventh embodiment in that the information acquisition unit 151 has a function of receiving temperature information from the outside of the information processing apparatus 1. The information acquisition unit 151 is communicatively connected to a temperature measuring device such as a temperature sensor arranged outside the information processing apparatus 1, and acquires temperature information from the temperature measuring device. The acquired temperature information is used for the processing in the first threshold determination unit 153 and the second threshold determination unit 155 in the same manner as in the seventh embodiment.

[0190] As described above, according to the present embodiment, an information processing apparatus and an information processing method that achieve the same effects as those of the seventh embodiment are provided. Further, in the present embodiment, since the thermometer can be arranged outside the information processing apparatus 1, the apparatus can be miniaturized.

[0191] Further, according to the present embodiment, an information processing system including the above-described information processing apparatus 1 and a temperature measuring apparatus that generates temperature information and transmits it to the information processing apparatus 1 is provided.

[0192] [Ninth Embodiment] In the present embodiment, a modified example of the apparatus configuration of the seventh embodiment will be described. In the present embodiment, descriptions of elements common to the first to eighth embodiments may be omitted or simplified.

[0193] In the present embodiment, a modified example in which the information processing apparatus 1 receives a two-dimensional look-up table from the outside is shown. In the configuration of the present embodiment, the apparatus that generates and stores the two-dimensional look-up table is arranged outside the information processing apparatus 1.

[0194] FIG. 29 is a functional block diagram related to the correction processing function of the information processing apparatus 1 according to the present embodiment. The information processing apparatus 1 has the same functional blocks as those of the seventh embodiment, but is different from the seventh embodiment in that the information acquisition unit 151 has a function of receiving a two-dimensional look-up table from outside the information processing apparatus 1. The information acquisition unit 151 is communicatively connected to an external information processing apparatus arranged outside the information processing apparatus 1, and acquires two two-dimensional look-up tables (a first table and a second table) from the external information processing apparatus. The first table is held in the first table holding unit 152, and the second table is held in the second table holding unit 154. The first table and the second table are respectively used for the processing in the first threshold determination unit 153 and the second threshold determination unit 155 in the same manner as in the seventh embodiment.

[0195] As described above, according to this embodiment, an information processing apparatus and an information processing method that achieve the same effects as those of the seventh embodiment are provided. Further, in this embodiment, since the two-dimensional look-up table can be supplied from outside the information processing apparatus 1, the correction process can be controlled from outside.

[0196] Also, according to this embodiment, an information processing system is provided that includes the above-described information processing apparatus 1 and an apparatus that generates at least one of the first table and the second table and transmits it to the information processing apparatus 1.

[0197] [Tenth Embodiment] In this embodiment, a modified example of the apparatus configuration of the ninth embodiment will be described. In this embodiment, descriptions of elements common to the first to ninth embodiments may be omitted or simplified.

[0198] In this embodiment, a modified example is shown in which the information processing apparatus 1 receives a one-dimensional look-up table from the outside. In the configuration of this embodiment, the apparatus that generates and stores the one-dimensional look-up table is arranged outside the information processing apparatus 1.

[0199] FIG. 30 is a functional block diagram regarding the correction processing function of the information processing apparatus 1 according to this embodiment. The information processing apparatus 1 has a configuration in which the first table holding unit 152 and the second table holding unit 154 are omitted from the same functional blocks as those in the ninth embodiment. Further, the information acquisition unit 151 has a function of receiving a one-dimensional look-up table from outside the information processing apparatus 1.

[0200] The information acquisition unit 151 is communicatively connected to an external information processing apparatus arranged outside the information processing apparatus 1, and acquires two one-dimensional look-up tables (the first table and the second table) from the external information processing apparatus. The first table and the second table are each used for the processing in the first threshold determination unit 153 and the second threshold determination unit 155, similar to the seventh embodiment.

[0201] The external information processing device has a temperature measuring device and a storage medium that stores a two-dimensional look-up table. The external information processing selects one row from the two-dimensional look-up table based on the temperature information acquired by the temperature measuring device, and generates a one-dimensional look-up table in which a representative value and a threshold value are associated. For example, in the example of FIG. 23, when the temperature is 60 degrees, a one-dimensional look-up table is generated by extracting only the row (the 7th row) where the temperature is "51-60" from the look-up table of FIG. 23.

[0202] FIG. 31 is a flowchart showing the determination process of the threshold value N and the threshold value R executed by the information processing device 1 according to the present embodiment.

[0203] The process of step S51 is the same as that described in the fifth embodiment, so the description is omitted. In step S56, the information acquisition unit 151 acquires a one-dimensional look-up table (the first table) for determining the threshold value N generated in the external information processing. In step S57, the information acquisition unit 151 acquires a one-dimensional look-up table (the second table) for determining the threshold value R generated in the external information processing. The process of step S53 is the same as that described in the fifth embodiment, so the description is omitted.

[0204] In step S54, the first threshold determination unit 153 acquires the first table acquired by the information acquisition unit 151 and the representative value calculated by the representative value calculation unit 150. The first threshold determination unit 153 refers to the first table and determines the threshold value N based on the representative value. This threshold value N is used as the threshold value N of the connection number in the determination of the isolated point described in the first embodiment. An example will be described with reference to FIG. 23. For example, when the temperature is 60 degrees, a one-dimensional look-up table obtained by extracting only the row (the 7th row) where the temperature is "51-60" from the look-up table of FIG. 23 is input. When the representative value is 200, the value of the threshold value N is 3 from this one-dimensional look-up table.

[0205] In step S58, the second threshold determination unit 155 acquires the second table acquired by the information acquisition unit 151 and the representative value calculated by the representative value calculation unit 150. The second threshold determination unit 155 refers to the second table and determines a threshold value R based on the representative value. This threshold value R is used as the threshold value R for the number of pixels in the density determination described in the fourth embodiment. An example will be described with reference to FIG. 26. For example, when the temperature is 60 degrees, a one-dimensional look-up table obtained by extracting only the row (the seventh row) in which the temperature in the look-up table of FIG. 26 is "51-60" is input. When the representative value is 200, the value of the threshold value R is 4 from this one-dimensional look-up table.

[0206] As described above, according to the present embodiment, there is provided an information processing apparatus and an information processing method that can obtain the same effects as those of the seventh embodiment. Further, in the present embodiment, since the look-up table can be supplied from the outside of the information processing apparatus 1, the correction process can be controlled from the outside. Further, in the present embodiment, since the operation can be performed without holding the look-up table inside the information processing apparatus 1, the storage capacity can be reduced.

[0207] Further, according to the present embodiment, there is provided an information processing system including the above-described information processing apparatus 1 and a device that generates at least one of the first table and the second table and transmits it to the information processing apparatus 1.

[0208] [Eleventh Embodiment] In the present embodiment, a specific configuration example of a photoelectric conversion device including an avalanche photodiode that can be applied to the data input unit 121 of the information processing apparatus 1 according to the first to tenth embodiments will be described. The configuration example of the present embodiment is an example, and the photoelectric conversion device applicable to the data input unit 121 is not limited to this.

[0209] FIG. 32 is a schematic diagram showing the overall configuration of the photoelectric conversion device 100 according to the present embodiment. The photoelectric conversion device 100 includes a sensor substrate 11 (first substrate) and a circuit substrate 21 (second substrate) laminated on each other. The sensor substrate 11 and the circuit substrate 21 are electrically connected to each other. The sensor substrate 11 has a pixel region 12 in which a plurality of pixel circuits 101 are arranged so as to form a plurality of rows and a plurality of columns. The circuit substrate 21 has a first circuit region 22 in which a plurality of pixel signal processing units 103 are arranged so as to form a plurality of rows and a plurality of columns, and a second circuit region 23 arranged on the outer periphery of the first circuit region 22. The second circuit region 23 may include a circuit for controlling the plurality of pixel signal processing units 103. The sensor substrate 11 has a light incident surface for receiving incident light and a connection surface facing the light incident surface. The sensor substrate 11 is connected to the circuit substrate 21 on the connection surface side. That is, the photoelectric conversion device 100 is a so-called back-illuminated type.

[0210] In this specification, "planar view" refers to viewing from a direction perpendicular to the surface opposite to the light incident surface. Further, the cross section refers to a surface in a direction perpendicular to the surface opposite to the light incident surface of the sensor substrate 11. Note that although the light incident surface may be a rough surface when viewed microscopically, in that case, the planar view is defined based on the light incident surface when viewed macroscopically.

[0211] Hereinafter, the sensor substrate 11 and the circuit substrate 21 will be described as diced chips, but the sensor substrate 11 and the circuit substrate 21 are not limited to chips. For example, the sensor substrate 11 and the circuit substrate 21 may be wafers. Further, when the sensor substrate 11 and the circuit substrate 21 are diced chips, the photoelectric conversion device 100 may be manufactured by dicing after being laminated in a wafer state, or may be manufactured by being laminated after being diced.

[0212] FIG. 33 is a schematic block diagram showing an arrangement example of the sensor substrate 11. In the pixel region 12, a plurality of pixel circuits 101 arranged in a plurality of rows and a plurality of columns are arranged. Each of the plurality of pixel circuits 101 has a photoelectric conversion unit 102 including an avalanche photodiode (hereinafter referred to as APD) as a photoelectric conversion element in the substrate.

[0213] The conductivity type of the charge used as the signal charge among the charge pairs generated in the APD is referred to as the first conductivity type. The first conductivity type refers to the conductivity type in which charges having the same polarity as the signal charge are majority carriers. Also, the conductivity type opposite to the first conductivity type, that is, the conductivity type in which charges having a polarity different from the signal charge are majority carriers is referred to as the second conductivity type. In the APD in the following description, the anode of the APD is set to a fixed potential, and a signal is taken out from the cathode of the APD. Therefore, the semiconductor region of the first conductivity type is an N-type semiconductor region, and the semiconductor region of the second conductivity type is a P-type semiconductor region. Note that a configuration in which the cathode of the APD is set to a fixed potential and a signal is taken out from the anode of the APD may also be used. In this case, the semiconductor region of the first conductivity type is a P-type semiconductor region, and the semiconductor region of the second conductivity type is an N-type semiconductor region. Also, in the following, the case where one node of the APD is set to a fixed potential will be described, but a configuration in which the potentials of both nodes fluctuate may also be used.

[0214] FIG. 34 is a schematic block diagram showing a configuration example of the circuit board 21. The circuit board 21 has a first circuit region 22 in which a plurality of pixel signal processing units 103 arranged in a plurality of rows and a plurality of columns are arranged.

[0215] Further, a vertical scanning circuit 110, a horizontal scanning circuit 111, a reading circuit 112, pixel output signal lines 113, an output circuit 114, and a control signal generation unit 115 are arranged on the circuit board 21. The plurality of photoelectric conversion units 102 shown in FIG. 33 and the plurality of pixel signal processing units 103 shown in FIG. 34 are electrically connected via connection wirings provided for each pixel circuit 101.

[0216] The control signal generation unit 115 is a control circuit that generates control signals for driving the vertical scanning circuit 110, the horizontal scanning circuit 111, and the readout circuit 112, and supplies these control signals to each of these units. Thereby, the control signal generation unit 115 controls the driving timing and the like of each unit.

[0217] Based on the control signal supplied from the control signal generation unit 115, the vertical scanning circuit 110 supplies a control signal to each of the plurality of pixel signal processing units 103. The vertical scanning circuit 110 supplies a control signal to each pixel signal processing unit 103 row by row via the driving lines provided for each row in the first circuit region 22. Note that, as will be described later, there may be a plurality of these driving lines for each row. Logic circuits such as a shift register and an address decoder may be used for the vertical scanning circuit 110. Thereby, the vertical scanning circuit 110 selects the row from which the pixel signal processing unit 103 outputs a signal.

[0218] The signal output from the photoelectric conversion unit 102 of the pixel circuit 101 is processed by the pixel signal processing unit 103. The pixel signal processing unit 103 acquires and holds a digital signal having a plurality of bits by counting the number of pulses output from the APD included in the photoelectric conversion unit 102.

[0219] The pixel signal processing unit 103 does not necessarily have to be provided one by one for all the pixel circuits 101. For example, one pixel signal processing unit 103 may be shared by a plurality of pixel circuits 101. In this case, the pixel signal processing unit 103 provides a signal processing function for each pixel circuit 101 by sequentially processing the signals output from the respective photoelectric conversion units 102.

[0220] The horizontal scanning circuit 111 supplies a control signal to the readout circuit 112 based on the control signal supplied from the control signal generation unit 115. The pixel signal processing unit 103 is connected to the readout circuit 112 via the pixel output signal lines 113 provided for each column in the first circuit region 22. The pixel output signal lines 113 for one column are shared by a plurality of pixel signal processing units 103 in the corresponding column. The pixel output signal lines 113 include a plurality of wirings and have at least a function of outputting a digital signal from each pixel signal processing unit 103 to the readout circuit 112 and a function of supplying a control signal for selecting a column for outputting a signal to the pixel signal processing unit 103. The readout circuit 112 outputs a signal to a storage unit or a signal processing unit outside the photoelectric conversion device 100 via the output circuit 114 based on the control signal supplied from the control signal generation unit 115.

[0221] The arrangement of the photoelectric conversion units 102 in the pixel region 12 may be arranged in a one-dimensional manner. Also, the functions of the pixel signal processing unit 103 do not necessarily have to be provided one by one in all the pixel circuits 101. For example, one pixel signal processing unit 103 may be shared by a plurality of pixel circuits 101. In this case, the pixel signal processing unit 103 provides a signal processing function for each pixel circuit 101 by sequentially processing the signals output from each photoelectric conversion unit 102.

[0222] As shown in FIGS. 33 and 34, a first circuit region 22 in which a plurality of pixel signal processing units 103 are arranged is arranged in a region overlapping the pixel region 12 in plan view. Then, in plan view, the vertical scanning circuit 110, the horizontal scanning circuit 111, the readout circuit 112, the output circuit 114, and the control signal generation unit 115 are arranged so as to overlap between the end of the sensor substrate 11 and the end of the pixel region 12. In other words, the sensor substrate 11 has a pixel region 12 and a non-pixel region arranged around the pixel region 12. Then, in the circuit substrate 21, a second circuit region 23 in which the vertical scanning circuit 110, the horizontal scanning circuit 111, the readout circuit 112, the output circuit 114, and the control signal generation unit 115 are arranged is arranged in a region overlapping the non-pixel region in plan view.

[0223] Note that the arrangements of the pixel output signal lines 113, the readout circuit 112, and the output circuit 114 are not limited to those shown in FIG. 34. For example, the pixel output signal lines 113 may be arranged to extend in the row direction and be shared by a plurality of pixel signal processing units 103 in the corresponding row. And the readout circuit 112 may be arranged so that the pixel output signal lines 113 of each row are connected.

[0224] FIG. 35 is a schematic block diagram showing a configuration example of one pixel of the photoelectric conversion unit 102 and the pixel signal processing unit 103 according to the present embodiment. FIG. 35 schematically shows a more specific configuration example including the connection relationship between the photoelectric conversion unit 102 arranged on the sensor substrate 11 and the pixel signal processing unit 103 arranged on the circuit substrate 21. Note that in FIG. 35, the drive lines between the vertical scanning circuit 110 and the pixel signal processing unit 103 in FIG. 34 are shown as drive lines 213 and 214.

[0225] The photoelectric conversion unit 102 has an APD 201. The pixel signal processing unit 103 has a quench element 202, a waveform shaping unit 210, a counter circuit 211, and a selection circuit 212. Note that the pixel signal processing unit 103 only needs to have at least one of the waveform shaping unit 210, the counter circuit 211, and the selection circuit 212.

[0226] The APD 201 generates charge pairs corresponding to incident light by photoelectric conversion. A voltage VL (first voltage) is supplied to the anode of the APD 201. Further, the cathode of the APD 201 is connected to the first terminal of the quench element 202 and the input terminal of the waveform shaping unit 210. A voltage VH (second voltage) higher than the voltage VL supplied to the anode is supplied to the cathode of the APD 201. Thereby, a reverse bias voltage is supplied between the anode and the cathode of the APD 201 such that the APD 201 performs an avalanche multiplication operation. In the APD 201 to which the reverse bias voltage is supplied, when charges are generated by incident light, these charges cause avalanche multiplication and an avalanche current is generated.

[0227] When a reverse bias voltage is supplied to the APD201, there are two operating modes: the Geiger mode and the linear mode. The Geiger mode is a mode in which the potential difference between the anode and the cathode is operated at a potential difference greater than the breakdown voltage, and the linear mode is a mode in which the potential difference between the anode and the cathode is operated near or below the breakdown voltage.

[0228] An APD operated in the Geiger mode is called a SPAD (Single Photon Avalanche Diode). At this time, for example, the voltage VL (first voltage) is -30V and the voltage VH (second voltage) is 1V. The APD201 may be operated in the linear mode or the Geiger mode. In the case of a SPAD, the potential difference becomes larger than that of a linear-mode APD, and the avalanche multiplication effect becomes prominent. Therefore, it is preferably a SPAD.

[0229] The quenching element 202 functions as a load circuit (quenching circuit) during signal multiplication by avalanche multiplication. The quenching element 202 suppresses the voltage supplied to the APD201 to suppress avalanche multiplication (quenching operation). Further, the quenching element 202 returns the voltage supplied to the APD201 to the voltage VH by flowing a current corresponding to the voltage drop due to the quenching operation (recharge operation). The quenching element 202 can be, for example, a resistive element.

[0230] The waveform shaping unit 210 shapes the potential change of the cathode of the APD201 obtained at the time of photon detection and outputs a pulse signal. As the waveform shaping unit 210, for example, an inverter circuit is used. FIG. 35 shows an example in which one inverter is used as the waveform shaping unit 210. However, the waveform shaping unit 210 may use a circuit in which a plurality of inverters are connected in series, or may be other circuits having a waveform shaping effect.

[0231] The counter circuit 211 counts the pulse signal output from the waveform shaping unit 210 and holds a digital signal indicating the count value. Also, when a control signal is supplied from the vertical scanning circuit 110 via the drive line 213, the counter circuit 211 resets the held signal.

[0232] A control signal is supplied to the selection circuit 212 from the vertical scanning circuit 110 shown in FIG. 34 via the drive line 214 shown in FIG. 35. In response to this control signal, the selection circuit 212 switches the electrical connection and disconnection between the counter circuit 211 and the pixel output signal line 113. The selection circuit 212 includes, for example, a buffer circuit or the like for outputting a signal corresponding to the value held in the counter circuit 211.

[0233] In the example of FIG. 35, the electrical connection and disconnection between the counter circuit 211 and the pixel output signal line 113 are switched in the selection circuit 212. However, the method of controlling the signal output to the pixel output signal line 113 is not limited to this. For example, a switch such as a transistor may be arranged at a node between the quenching element 202 and the APD 201, between the photoelectric conversion unit 102 and the pixel signal processing unit 103, etc., and the signal output to the pixel output signal line 113 may be controlled by switching the electrical connection and disconnection. Also, the signal output to the pixel output signal line 113 may be controlled by changing the value of the voltage VH or voltage VL supplied to the photoelectric conversion unit 102 using a switch such as a transistor.

[0234] FIG. 35 shows a configuration example using the counter circuit 211. However, instead of the counter circuit 211, a time - digital conversion circuit (Time to Digital Converter: hereinafter, TDC) and a memory may be used to obtain the timing of detecting the pulse. At this time, the generation timing of the pulse signal output from the waveform shaping unit 210 is converted into a digital signal by the TDC. In this case, a control signal (reference signal) may be supplied to the TDC from the vertical scanning circuit 110 in FIG. 34 via a drive line. The TDC acquires, as a digital signal, a signal indicating the relative time of the input timing of the pulse with respect to the control signal.

[0235] Figures 36(a), 36(b), and 36(c) are diagrams for explaining the operation of APD 201 according to this embodiment. Figure 36(a) is a diagram showing APD 201, quenching element 202, and waveform shaping unit 210 extracted from Figure 35. As shown in Figure 36(a), let the connection node of the input terminals of APD 201, quenching element 202, and waveform shaping unit 210 be nodeA. Also, as shown in Figure 36(a), let the output side of waveform shaping unit 210 be nodeB.

[0236] Figure 36(b) is a graph showing the time change of the potential of nodeA in Figure 36(a). Figure 36(c) is a graph showing the time change of the potential of nodeB in Figure 36(a). During the period from time t0 to time t1, a voltage of VH - VL is applied to APD 201 in Figure 36(a). When photons are incident on APD 201 at time t1, avalanche multiplication occurs in APD 201. As a result, an avalanche current flows through quenching element 202, and the potential of nodeA drops. Thereafter, the amount of potential drop becomes even larger, and the voltage applied to APD 201 gradually decreases. Then, the avalanche multiplication in APD 201 stops at time t2. As a result, the voltage level of nodeA no longer drops below a certain value. Thereafter, during the period from time t2 to time t3, a current that compensates for the voltage drop flows from the node with voltage VH to nodeA, and nodeA settles to its original potential at time t3.

[0237] In the above process, the potential of nodeB becomes high during the period when the potential of nodeA is lower than a certain threshold. In this way, the waveform of the potential drop of nodeA caused by the incidence of photons is shaped by waveform shaping unit 210 and output as a pulse to nodeB.

[0238] According to this embodiment, an optoelectronic conversion device using an avalanche photodiode that can be applied to the data input unit 121 of the information processing apparatus 1 of the first to tenth embodiments is provided. The optoelectronic conversion device using an avalanche photodiode may generate more abnormal pixels than an optoelectronic conversion device using a general photodiode without avalanche multiplication. Since the information processing apparatus 1 of the first to tenth embodiments can preferably correct abnormal pixels, it is effective for correcting the output signal of the optoelectronic conversion device using an avalanche photodiode.

[0239] [Embodiment 12] The information processing apparatus 1 in the above-described embodiment can be applied to various devices. Examples of the devices include digital still cameras, digital camcorders, camera heads, copiers, fax machines, mobile phones, in-vehicle cameras, observation satellites, surveillance cameras, and the like. FIG. 37 shows a block diagram of a digital still camera as an example of the device. FIG. 37 is an example in which the information processing apparatus 1 shown in FIG. 1 is applied to a digital still camera.

[0240] The device 70 shown in FIG. 37 includes a barrier 706, a lens 702, a diaphragm 704, and an imaging device 700 (an example of a photoelectric conversion device). The device 70 further includes a signal processing unit (processing device) 708, a timing generation unit 720, an overall control and arithmetic unit 718 (control device), a memory unit 710 (storage device), a recording medium control I / F unit 716, a recording medium 714, and an external I / F unit 712. At least one of the barrier 706, the lens 702, and the diaphragm 704 is an optical device corresponding to the device. The barrier 706 protects the lens 702, and the lens 702 forms an optical image of the subject on the imaging device 700. The diaphragm 704 makes the amount of light passing through the lens 702 variable. The imaging device 700 converts the optical image formed by the lens 702 into image data (image signal). The signal processing unit 708 performs various corrections, data compression, etc. on the imaging data output from the imaging device 700. The timing generation unit 720 outputs various timing signals to the imaging device 700 and the signal processing unit 708. The overall control and arithmetic unit 718 controls the entire digital still camera, and the memory unit 710 temporarily stores the image data. The recording medium control I / F unit 716 is an interface for recording or reading image data to / from the recording medium 714, and the recording medium 714 is a removable recording medium such as a semiconductor memory for recording or reading the imaging data. The external I / F unit 712 is an interface for communicating with an external computer or the like. The timing signal or the like may be input from outside the device. Further, the device 70 may include a display device (monitor, electronic viewfinder, etc.) for displaying the information obtained by the photoelectric conversion device. The device includes at least a photoelectric conversion device. Further, the device 70 includes at least any one of an optical device, a control device, a processing device, a display device, a storage device, and a mechanical device that operates based on the information obtained by the photoelectric conversion device. The mechanical device is a movable part (for example, a robot arm) that operates by receiving the signal of the photoelectric conversion device.

[0241] Each pixel may include a plurality of photoelectric conversion units (a first photoelectric conversion unit and a second photoelectric conversion unit). The signal processing unit 708 may be configured to process a pixel signal based on the charge generated in the first photoelectric conversion unit and a pixel signal based on the charge generated in the second photoelectric conversion unit, and acquire distance information from the imaging device 700 to the subject.

[0242] [Embodiment 13] FIGS. 38(a) and 38(b) are block diagrams of devices related to an in-vehicle camera in this embodiment. FIG. 38 shows an example in which the information processing device 1 shown in FIG. 1 is applied to a moving body such as a vehicle. The device 80 includes an imaging device 800 (an example of a photoelectric conversion device) and a signal processing device (processing device) that processes signals from the imaging device 800. The device 80 has an image processing unit 801 that performs image processing on a plurality of pieces of image data acquired by the imaging device 800, and a parallax calculation unit 802 that calculates parallax (phase difference of a parallax image) from the plurality of pieces of image data acquired by the device 80. Further, the device 80 has a distance measurement unit 803 that calculates the distance to an object based on the calculated parallax, and a collision determination unit 804 that determines whether there is a possibility of collision based on the calculated distance. Here, the parallax calculation unit 802 and the distance measurement unit 803 are an example of distance information acquisition means for acquiring distance information to an object. That is, the distance information is information related to parallax, defocus amount, distance to an object, and the like. The collision determination unit 804 may determine the possibility of collision using any of these distance information. The distance information acquisition means may be realized by dedicatedly designed hardware, or may be realized by a software module. Further, it may be realized by an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or a combination thereof.

[0243] Device 80 is connected to a vehicle information acquisition device 810 and can acquire vehicle information such as vehicle speed, yaw rate, and steering angle. Further, a control ECU 820, which is a control device that outputs a control signal for generating a braking force on the vehicle based on the determination result of the collision determination unit 804, is connected to the device 80. Further, the device 80 is also connected to an alarm device 830 that issues an alarm to the driver based on the determination result of the collision determination unit 804. For example, when the collision determination unit 804 determines that there is a high possibility of collision, the control ECU 820 performs vehicle control to avoid the collision and reduce the damage, such as applying the brakes, returning the accelerator, and suppressing the engine output. The alarm device 830 warns the user by sounding an alarm such as a sound, displaying alarm information on the screen of a car navigation system, etc., or applying vibration to the seat belt or steering wheel. The device 80 functions as a control means for controlling the operation of controlling the vehicle as described above.

[0244] In the present embodiment, the device 80 images the periphery of the vehicle, for example, the front or the rear. FIG. 38(b) shows the device when imaging the front of the vehicle (imaging range 850). The vehicle information acquisition device 810 as imaging control means sends an instruction to the device 80 or the imaging device 800 to perform the imaging operation. With such a configuration, the ranging accuracy can be further improved.

[0245] In the above, an example of controlling so as not to collide with other vehicles has been described, but it is also applicable to control for automatically driving following other vehicles, control for automatically driving so as not to deviate from the lane, etc. Further, the device is not limited to vehicles such as automobiles, and can be applied to moving bodies (moving devices) such as ships, airplanes, artificial satellites, industrial robots, and consumer robots. In addition, it can be applied not only to moving bodies but also to devices that widely utilize object recognition or biometric recognition, such as advanced road traffic systems (ITS) and monitoring systems.

[0246] [Modified Embodiment] The present invention is not limited to the above-described embodiments and can be variously modified. For example, an example in which a part of the configuration of any one of the embodiments is added to another embodiment, or an example in which a part of the configuration of any one of the embodiments is replaced with a part of the configuration of another embodiment is also an embodiment of the present invention.

[0247] The disclosure of this specification includes the complement of the concepts described in this specification. That is, for example, if this specification describes that "A is B" (A = B), even if the description that "A is not B" (A ≠ B) is omitted, this specification is considered to disclose or imply that "A is not B". This is because when the description that "A is B" is given, it is assumed that the case where "A is not B" is considered.

[0248] The disclosure of this specification includes the following configurations or methods. (Configuration 1) A first connection unit that determines that two adjacent pixels among a plurality of pixels arranged in a plurality of rows and a plurality of columns belong to the same first connected pixel group when the difference in pixel values between the two pixels is within a first range; A pixel value determination unit that determines that a first condition is satisfied when the rank of the pixel value of the target pixel is higher than the upper limit of a second range or lower than the lower limit of the second range in a neighborhood pixel group including the target pixel and neighborhood pixels arranged in the neighborhood of the target pixel; A first connection number determination unit that determines that a second condition is satisfied when the number of pixels belonging to the first connected pixel group including the target pixel is less than or equal to a first threshold; A correction unit that corrects the pixel value of the target pixel in at least a first case where the first condition and the second condition are satisfied, and does not perform the correction performed in the first case on the pixel value of the target pixel when the first condition is not satisfied or the second condition is not satisfied; An information acquisition unit that acquires temperature information; A representative value calculation unit that calculates a first representative value of the neighborhood pixel group; A first threshold determination unit that determines the first threshold based on the temperature information and the first representative value; An information processing apparatus characterized by having the above. (Configuration 2) The upper limit of the second range is the second digit, and the lower limit of the second range is the rank of the number obtained by subtracting 1 from the number of pixels included in the neighboring pixel group. The information processing apparatus according to Configuration 1, characterized in that. (Configuration 3) The first threshold value is 2 or more. The information processing apparatus according to Configuration 1 or 2, characterized in that. (Configuration 4) The neighboring pixels are arranged in the same row or an adjacent row as the target pixel, and in the same column or an adjacent column as the target pixel. The information processing apparatus according to any one of Configurations 1 to 3, characterized in that. (Configuration 5) The neighboring pixel group includes the target pixel and a plurality of the neighboring pixels arranged so as to surround the target pixel. The information processing apparatus according to Configuration 4, characterized in that. (Configuration 6) The first threshold value determination unit determines the first threshold value based on a look-up table showing the relationship between the temperature information, the first representative value, and the first threshold value. The information processing apparatus according to any one of Configurations 1 to 5, characterized in that. (Configuration 7) In two adjacent first pixels and second pixels, when the difference between the pixel value of the first pixel and the second representative value of the neighboring pixel group including the neighboring pixels arranged near the first pixel is outside the third range, and the difference between the pixel value of the second pixel and the third representative value of the neighboring pixel group including the neighboring pixels arranged near the second pixel is outside the fourth range, a second connection unit that determines that the first pixel and the second pixel belong to the same second connected pixel group; A second connection number determination unit that determines that the third condition is satisfied when the number of pixels belonging to the second connected pixel group including the target pixel is greater than the second threshold value; A second threshold value determination unit that determines the second threshold value based on the temperature information and the first representative value; and further has When the correction unit satisfies both the first condition and the second condition but does not satisfy the third condition in a first case, the correction unit corrects the pixel value of the target pixel. When the correction unit does not satisfy the first condition, does not satisfy the second condition, or satisfies the third condition, the correction unit does not perform the correction performed in the first case on the pixel value of the target pixel. The information processing apparatus according to Configuration 1, characterized in that. (Configuration 8) The second threshold value is 2 or more. The information processing apparatus according to Configuration 7, characterized in that. (Configuration 9) The second representative value and the third representative value are the median values of the pixel values of a plurality of pixels included in the neighboring pixel group. The information processing apparatus according to Configuration 7 or 8, characterized in that. (Configuration 10) The second threshold value determination unit determines the second threshold value based on a look-up table showing the relationship between the temperature information, the first representative value, and the second threshold value. The information processing apparatus according to any one of Configurations 7 to 9, characterized in that. (Configuration 11) In two adjacent first pixels and second pixels, when the difference between the pixel value of the first pixel and the second representative value of the neighboring pixel group including the neighboring pixels arranged near the first pixel is outside a third range, and the difference between the pixel value of the second pixel and the third representative value of the neighboring pixel group including the neighboring pixels arranged near the second pixel is outside a fourth range, a second connection unit that determines that the first pixel and the second pixel belong to the same second connected pixel group; A second connection number determination unit that determines that the third condition is satisfied when the number of pixels belonging to the second connected pixel group including the target pixel is greater than a second threshold value; A protrusion determination unit that determines that the fourth condition is satisfied when the pixel value of the target pixel is greater than or equal to a third threshold value or less than or equal to a fourth threshold value; A second threshold value determination unit that determines the second threshold value based on the temperature information and the first representative value; further comprising When the correction unit satisfies both the first condition and the second condition but does not satisfy the third condition, it corrects the pixel value of the target pixel. Even when the correction unit satisfies all of the first condition, the second condition, the third condition, and the fourth condition, it corrects the pixel value of the target pixel. When the correction unit does not satisfy the first condition, does not satisfy the second condition, or does not satisfy the fourth condition, it does not perform the correction performed in the first case on the pixel value of the target pixel. The information processing apparatus according to Configuration 1, characterized in that. (Configuration 12) The second threshold determination unit determines the second threshold based on a look-up table showing the relationship between the temperature information, the first representative value, and the second threshold. The information processing apparatus according to Configuration 11, characterized in that. (Configuration 13) In a plurality of pixels within a predetermined range including the target pixel, when the number of pixels whose difference between the pixel value of a certain pixel and the representative value of a neighborhood pixel group including neighborhood pixels arranged in its vicinity is outside the fifth range is greater than the fifth threshold, a density determination unit determines that the fifth condition is satisfied; A second threshold determination unit that determines the fifth threshold based on the temperature information and the first representative value; further comprising: When the correction unit satisfies both the first condition and the second condition but does not satisfy the fifth condition, it corrects the pixel value of the target pixel. When the correction unit does not satisfy the first condition, does not satisfy the second condition, or satisfies the fifth condition, it does not perform the correction performed in the first case on the pixel value of the target pixel. The information processing apparatus according to Configuration 1, characterized in that. (Configuration 14) The fifth threshold is 2 or more. The information processing apparatus according to Configuration 13, characterized in that. (Configuration 15) The second threshold determination unit determines the fifth threshold based on a look-up table indicating the relationship between the temperature information, the first representative value, and the fifth threshold. The information processing apparatus according to configuration 13 or 14, characterized by the above. (Configuration 16) The pixel value is generated based on a signal obtained by a photoelectric conversion element photoelectrically converting incident light. The information processing apparatus according to any one of configurations 1 to 15, characterized by the above. (Configuration 17) The photoelectric conversion element includes an avalanche photodiode. The information processing apparatus according to configuration 16, characterized by the above. (Configuration 18) The first representative value is an average value, a weighted average value, a median value, or a mode value of pixel values of a plurality of pixels included in the neighboring pixel group. The information processing apparatus according to any one of configurations 1 to 17, characterized by the above. (Configuration 19) The information acquisition unit includes a temperature measurement device that generates temperature information by measuring temperature. The information processing apparatus according to any one of configurations 1 to 18, characterized by the above. (Configuration 20) The information acquisition unit acquires the temperature information from a device external to the information processing apparatus. The information processing apparatus according to any one of configurations 1 to 18, characterized by the above. (Configuration 21) The information acquisition unit acquires a look-up table indicating the relationship between the temperature information, the first representative value, and the first threshold from a device external to the information processing apparatus, and the first threshold determination unit determines the first threshold based on the temperature information, the first representative value, and the look-up table. The information processing apparatus according to any one of configurations 1 to 20, characterized by the above. (Configuration 22) It further has a holding unit that holds a look-up table indicating the relationship between the temperature information, the first representative value, and the first threshold. The first threshold determination unit determines the first threshold based on the temperature information, the first representative value, and the look-up table. The information processing apparatus according to any one of Configurations 1 to 21, characterized in that. (Configuration 23) A first connection unit that determines that two adjacent pixels among a plurality of pixels arranged in a plurality of rows and a plurality of columns belong to the same first connected pixel group when the difference in pixel values between the two pixels is within a first range; A pixel value determination unit that determines that a first condition is satisfied when the rank of the pixel value of the target pixel is higher than the upper limit of a second range or lower than the lower limit of the second range in a neighborhood pixel group including the target pixel and a neighborhood pixel arranged in the vicinity of the target pixel; A first connection number determination unit that determines that a second condition is satisfied when the number of pixels belonging to the first connected pixel group including the target pixel is equal to or less than a first threshold; A correction unit that corrects the pixel value of the target pixel in at least a first case where the first condition and the second condition are satisfied, and does not perform the correction performed in the first case on the pixel value of the target pixel when the first condition is not satisfied or the second condition is not satisfied; A representative value calculation unit that calculates a first representative value of the neighborhood pixel group; An information acquisition unit that acquires a look-up table showing the relationship between the first representative value and the first threshold generated based on temperature information; A first threshold determination unit that determines the first threshold based on the first representative value and the look-up table; An information processing apparatus, characterized by comprising. (Configuration 24) A plurality of photoelectric conversion elements arranged in a plurality of rows and a plurality of columns; The information processing apparatus according to any one of Configurations 1 to 23, to which a pixel value based on a signal output from the plurality of photoelectric conversion elements is input; A photoelectric conversion device including the same. (Configuration 25) The photoelectric conversion device according to Configuration 24; An optical device corresponding to the photoelectric conversion device; A control device for controlling the photoelectric conversion device, A processing device for processing a signal output from the photoelectric conversion device, A display device for displaying information obtained by the photoelectric conversion device, A storage device for storing information obtained by the photoelectric conversion device, and At least one of a mechanical device that operates based on information obtained by the photoelectric conversion device, and a device characterized by comprising the same. (Configuration 26) The processing device acquires distance information from the photoelectric conversion device to the subject, and the device according to Configuration 25, characterized in that. (Method 27) When the difference between the pixel values of two adjacent pixels in a plurality of pixels arranged in a plurality of rows and a plurality of columns is within a first range, determining that the two pixels belong to the same first connected pixel group; In a neighborhood pixel group including a target pixel and neighborhood pixels arranged in the vicinity of the target pixel, determining that a first condition is satisfied when the rank of the pixel value of the target pixel is higher than the upper limit of a second range or lower than the lower limit of the second range; Obtaining temperature information; Calculating a first representative value of the neighborhood pixel group; Determining a first threshold value based on the temperature information and the first representative value; Determining that a second condition is satisfied when the number of pixels belonging to the first connected pixel group including the target pixel is less than or equal to the first threshold value; Correcting the pixel value of the target pixel in a first case where at least the first condition and the second condition are satisfied; Having, When the first condition is not satisfied or the second condition is not satisfied, the correction performed in the first case is not performed on the pixel value of the target pixel An information processing method characterized by the above. (Method 28) Determining that two adjacent pixels among a plurality of pixels arranged in a plurality of rows and a plurality of columns belong to the same first connected pixel group when the difference between the pixel values of the two pixels is within a first range; Determining that a first condition is satisfied when, among a group of neighboring pixels including a target pixel and neighboring pixels arranged in the vicinity of the target pixel, the rank of the pixel value of the target pixel is higher than the upper limit of a second range or lower than the lower limit of the second range; Calculating a first representative value of the group of neighboring pixels; Obtaining a look-up table showing the relationship between the first representative value and a first threshold value, which is generated based on temperature information; Determining the first threshold value based on the first representative value and the look-up table; Determining that a second condition is satisfied when the number of pixels belonging to the first connected pixel group including the target pixel is less than or equal to the first threshold value; Correcting the pixel value of the target pixel in a first case where at least the first condition and the second condition are satisfied; having When the first condition is not satisfied or the second condition is not satisfied, not performing the correction performed in the first case on the pixel value of the target pixel An information processing method characterized by the above. (Configuration 29) A program for causing a computer to execute the information processing method according to Method 27 or 28. (Configuration 30) A recording medium storing a program for causing a computer to execute the information processing method according to Method 27 or 28.

[0249] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. Further, it can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0250] Note that the above-described embodiments are merely examples of implementation when carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner by these. That is, the present invention can be implemented in various forms without departing from its technical idea or its main features.

Explanation of Reference Numerals

[0251] 1 Information processing apparatus 133 First pixel value determination unit 135 First connection unit 136 Correction unit 137 First connection number determination unit 150 Representative value calculation unit 151 Information acquisition unit 153 First threshold determination unit

Claims

1. A first connection unit that determines that two adjacent pixels among a plurality of pixels arranged in a plurality of rows and a plurality of columns belong to the same first connected pixel group when the difference between the pixel values of the two pixels is within a first range; A pixel value determination unit that determines that a first condition is satisfied when the rank of the pixel value of the target pixel is higher than the upper limit of a second range or lower than the lower limit of the second range among a group of neighboring pixels including the target pixel and neighboring pixels arranged in the vicinity of the target pixel; A first connection number determination unit that determines that a second condition is satisfied when the number of pixels belonging to the first connected pixel group including the target pixel is equal to or less than a first threshold; A correction unit that corrects the pixel value of the target pixel in a first case where at least the first condition and the second condition are satisfied, and does not perform the correction performed in the first case on the pixel value of the target pixel when the first condition is not satisfied or the second condition is not satisfied; An information acquisition unit that acquires temperature information; A representative value calculation unit that calculates a first representative value of the group of neighboring pixels; A first threshold determination unit that determines the first threshold based on the temperature information and the first representative value; An information processing apparatus, characterized by comprising the above.

2. The upper limit of the second range is the second rank, and the lower limit of the second range is the rank of the number obtained by subtracting 1 from the number of pixels included in the group of neighboring pixels The information processing apparatus according to claim 1, characterized by the above.

3. The first threshold is 2 or more The information processing apparatus according to claim 1, characterized by the above.

4. The neighboring pixels are arranged in the same row or an adjacent row as the target pixel, and in the same column or an adjacent column as the target pixel The information processing apparatus according to claim 1, characterized by the above.

5. The group of neighboring pixels includes the target pixel and a plurality of the neighboring pixels arranged so as to surround the target pixel The information processing apparatus according to claim 4, characterized by the above.

6. The first threshold determination unit determines the first threshold based on a look-up table showing the relationship between the temperature information, the first representative value, and the first threshold The information processing apparatus according to claim 1, characterized by the above.

7. In two adjacent first pixels and second pixels, when the difference between the pixel value of the first pixel and the second representative value of a group of neighboring pixels including neighboring pixels arranged near the first pixel is outside a third range, and the difference between the pixel value of the second pixel and the third representative value of a group of neighboring pixels including neighboring pixels arranged near the second pixel is outside a fourth range, a second connection unit that determines that the first pixel and the second pixel belong to the same second connected pixel group; A second connection number determination unit that determines that a third condition is satisfied when the number of pixels belonging to the second connected pixel group including the target pixel is greater than a second threshold; A second threshold determination unit that determines the second threshold based on the temperature information and the first representative value; further comprising: The correction unit corrects the pixel value of the target pixel in a first case where both the first condition and the second condition are satisfied and the third condition is not satisfied; The correction unit does not perform the correction performed in the first case on the pixel value of the target pixel when the first condition is not satisfied, the second condition is not satisfied, or the third condition is satisfied. The information processing apparatus according to claim 1, characterized in that.

8. The second threshold is 2 or more. The information processing apparatus according to claim 7, characterized in that.

9. The second representative value and the third representative value are median values of pixel values of a plurality of pixels included in the group of neighboring pixels. The information processing apparatus according to claim 7, characterized in that.

10. The second threshold determination unit determines the second threshold based on a look-up table showing the relationship between the temperature information, the first representative value, and the second threshold. The information processing apparatus according to claim 7, characterized in that.

11. In two adjacent first pixels and second pixels, when the difference between the pixel value of the first pixel and the second representative value of a group of neighboring pixels including neighboring pixels arranged near the first pixel is outside a third range, and the difference between the pixel value of the second pixel and the third representative value of a group of neighboring pixels including neighboring pixels arranged near the second pixel is outside a fourth range, a second connection unit that determines that the first pixel and the second pixel belong to the same second connected pixel group; A second connection number determination unit that determines that a third condition is satisfied when the number of pixels belonging to the second connected pixel group including the target pixel is greater than a second threshold; A protrusion determination unit that determines that a fourth condition is satisfied when the pixel value of the target pixel is greater than or equal to a third threshold or less than or equal to a fourth threshold. A second threshold determination unit that determines the second threshold based on the temperature information and the first representative value; further comprising; When the correction unit satisfies both the first condition and the second condition and does not satisfy the third condition, the correction unit corrects the pixel value of the target pixel. Even when the correction unit satisfies all of the first condition, the second condition, the third condition, and the fourth condition, the correction unit corrects the pixel value of the target pixel. When the correction unit does not satisfy the first condition, does not satisfy the second condition, or does not satisfy the fourth condition, the correction unit does not perform the correction performed in the first case on the pixel value of the target pixel. The information processing apparatus according to claim 1, characterized in that.

12. The second threshold determination unit determines the second threshold based on a look-up table showing the relationship between the temperature information, the first representative value, and the second threshold. The information processing apparatus according to claim 11, characterized in that.

13. In a plurality of pixels within a predetermined range including the target pixel, when the number of pixels whose difference between the pixel value of a certain pixel and the representative value of a neighborhood pixel group including the neighborhood pixels arranged in the vicinity thereof is outside the fifth range is greater than the fifth threshold, a density determination unit that determines that the fifth condition is satisfied; A second threshold determination unit that determines the fifth threshold based on the temperature information and the first representative value; further comprising; When the correction unit satisfies both the first condition and the second condition and does not satisfy the fifth condition, the correction unit corrects the pixel value of the target pixel. When the correction unit does not satisfy the first condition, does not satisfy the second condition, or satisfies the fifth condition, the correction unit does not perform the correction performed in the first case on the pixel value of the target pixel. The information processing apparatus according to claim 1, characterized in that.

14. The fifth threshold is 2 or more. The information processing apparatus according to claim 13, characterized in that.

15. The second threshold determination unit determines the fifth threshold based on a look-up table showing the relationship between the temperature information, the first representative value, and the fifth threshold. The information processing apparatus according to claim 13, characterized in that.

16. The pixel value is generated based on a signal obtained by a photoelectric conversion element photoelectrically converting incident light. The information processing apparatus according to claim 1, characterized in that.

17. The photoelectric conversion element includes an avalanche photodiode. The information processing apparatus according to claim 16, characterized in that.

18. The first representative value is an average value, a weighted average value, a median value, or a mode value of pixel values of a plurality of pixels included in the neighboring pixel group. The information processing apparatus according to claim 1, wherein the information processing apparatus is characterized in that.

19. The information acquisition unit includes a temperature measurement device that generates temperature information by measuring temperature. The information processing apparatus according to claim 1, wherein the information processing apparatus is characterized in that.

20. The information acquisition unit acquires the temperature information from a device external to the information processing apparatus. The information processing apparatus according to claim 1, wherein the information processing apparatus is characterized in that.

21. The information acquisition unit acquires a look-up table showing the relationship between the temperature information, the first representative value, and the first threshold value from a device external to the information processing apparatus. The first threshold determination unit determines the first threshold value based on the temperature information, the first representative value, and the look-up table. The information processing apparatus according to claim 1, wherein the information processing apparatus is characterized in that.

22. The apparatus further includes a holding unit that holds a look-up table showing the relationship between the temperature information, the first representative value, and the first threshold value. The first threshold determination unit determines the first threshold value based on the temperature information, the first representative value, and the look-up table. The information processing apparatus according to claim 1, wherein the information processing apparatus is characterized in that.

23. A first connection unit that determines that two adjacent pixels belong to the same first connected pixel group when the difference between the pixel values of the two adjacent pixels among a plurality of pixels arranged in a plurality of rows and a plurality of columns is within a first range. A pixel value determination unit that determines that a first condition is satisfied when the rank of the pixel value of the target pixel is higher than the upper limit of a second range or lower than the lower limit of the second range in a neighboring pixel group including the target pixel and neighboring pixels arranged in the vicinity of the target pixel. A first connection number determination unit that determines that a second condition is satisfied when the number of pixels belonging to the first connected pixel group including the target pixel is equal to or less than a first threshold value. A correction unit that corrects the pixel value of the target pixel in a first case where at least the first condition and the second condition are satisfied, and does not perform the correction performed in the first case on the pixel value of the target pixel when the first condition is not satisfied or the second condition is not satisfied. A representative value calculation unit that calculates a first representative value of the neighboring pixel group. An information acquisition unit that acquires a look-up table showing the relationship between the first representative value and the first threshold value, which is generated based on temperature information. A first threshold determination unit that determines the first threshold based on the first representative value and the look-up table; An information processing apparatus characterized by comprising the same.

24. A plurality of photoelectric conversion elements arranged to form a plurality of rows and a plurality of columns; The information processing apparatus according to any one of claims 1 to 23, wherein a pixel value based on a signal output from the plurality of photoelectric conversion elements is input; A photoelectric conversion device including the same.

25. The photoelectric conversion device according to claim 24; An optical device corresponding to the photoelectric conversion device; A control device that controls the photoelectric conversion device; A processing device that processes a signal output from the photoelectric conversion device; A display device that displays information obtained by the photoelectric conversion device; A storage device that stores information obtained by the photoelectric conversion device, and At least one of a mechanical device that operates based on information obtained by the photoelectric conversion device, characterized in that the device is provided.

26. The device according to claim 25, wherein the processing device acquires distance information from the photoelectric conversion device to a subject.

27. Determining that two adjacent pixels belong to the same first connected pixel group when the difference between the pixel values of the two adjacent pixels in a plurality of pixels arranged to form a plurality of rows and a plurality of columns is within a first range; Determining that a first condition is satisfied when the rank of the pixel value of the target pixel is higher than the upper limit of a second range or lower than the lower limit of the second range in a neighborhood pixel group including the target pixel and a neighborhood pixel arranged adjacent to the target pixel; Acquiring temperature information; Calculating a first representative value of the neighborhood pixel group; Determining a first threshold based on the temperature information and the first representative value; Determining that a second condition is satisfied when the number of pixels belonging to the first connected pixel group including the target pixel is less than or equal to the first threshold; Correcting the pixel value of the target pixel in a first case where at least the first condition and the second condition are satisfied; Having, When the first condition is not satisfied or the second condition is not satisfied, the correction performed in the first case is not performed on the pixel value of the target pixel An information processing method characterized by the above.

28. Determining that two adjacent pixels belong to the same first connected pixel group when the difference between the pixel values of the two adjacent pixels in a plurality of pixels arranged to form a plurality of rows and a plurality of columns is within a first range; In a group of neighboring pixels including a target pixel and neighboring pixels arranged in the vicinity of the target pixel, determining that a first condition is satisfied when the rank of the pixel value of the target pixel is higher than the upper limit of a second range or lower than the lower limit of the second range; calculating a first representative value of the group of neighboring pixels; obtaining a look-up table showing the relationship between the first representative value and a first threshold value, generated based on temperature information; determining the first threshold value based on the first representative value and the look-up table; determining that a second condition is satisfied when the number of pixels belonging to the first connected pixel group including the target pixel is equal to or less than the first threshold value; correcting the pixel value of the target pixel in a first case where at least the first condition and the second condition are satisfied; comprising when the first condition is not satisfied or the second condition is not satisfied, not performing the correction performed in the first case on the pixel value of the target pixel An information processing method characterized by the above.

29. A program for causing a computer to execute the information processing method according to claim 27 or 28.

30. A recording medium storing a program for causing a computer to execute the information processing method according to claim 27 or 28.

Images