Image signal processor

Abstract

The present invention provides an image signal processor capable of performing image conversion. [Solution] An image signal processor 100 according to one embodiment of the present invention may include: a kernel generation unit 210 that generates a target kernel including a pair of adjacent target pixels; a texture determination unit 220 that analyzes the texture of the target kernel and determines whether the target kernel corresponds to a flat region or a dark region; a threshold setting unit 230 that sets a threshold based on the determination result of the texture determination unit 220; a defective pixel determination unit 240 that detects whether a pair of target pixels are defective pixels based on the pixel data of a reference pixel having the same attributes as the pair of target pixels; and a defective pixel correction unit 300 that corrects one of the pair of target pixels if the pair of target pixels are defective pixels.

Description

[Technical Field]

[0001] This disclosure relates to an image signal processor capable of performing image conversion. [Background technology]

[0002] Image sensing devices are devices that capture optical images using the properties of light-sensitive semiconductor materials. With the development of industries such as automotive, medical, computer, and telecommunications, the demand for high-performance image sensing devices is increasing in various fields such as smartphones, digital cameras, gaming devices, the Internet of Things, robots, security cameras, and medical microcameras.

[0003] Typically, the original image captured by an image sensing device may contain fundamental defects or images of defective pixels that do not qualify as a normal image due to temporary factors. Generally, when a defective pixel exists in an image sensor, a device can be used to interpolate the data of the defective pixel using data from adjacent pixels. However, if two or more defective pixels exist adjacent to each other in an image sensing device, the data of other defective pixels may be used during the correction of the defective pixel, potentially leading to incorrect correction of the defective pixel. [Overview of the project] [Problems that the invention aims to solve]

[0004] The technical concept of the present invention is to provide an image signal processor that can effectively correct defective pixel data even when two or more defective pixels are adjacent to each other.

[0005] The technical problems of the present invention are not limited to those mentioned above, and other technical problems not mentioned can be clearly understood by those skilled in the art from the following description. [Means for solving the problem]

[0006] An image signal processor according to one embodiment of the present invention disclosed herein may include: a kernel generation unit that generates a target kernel including a pair of adjacent target pixels; a texture determination unit that analyzes the texture of the target kernel and determines whether the target kernel corresponds to a flat region or a dark region; a threshold setting unit that sets a threshold based on the determination result of the texture determination unit; a defective pixel determination unit that detects whether a pair of target pixels are defective pixels based on the pixel data of a reference pixel having the same attributes as the pair of target pixels; and a defective pixel correction unit that corrects one of the pair of target pixels if the pair of target pixels are defective pixels.

[0007] An image signal processor according to one embodiment of the present invention disclosed herein may include: a texture determination unit that analyzes the texture of a target kernel including a pair of adjacent target pixels to determine kernel type information; a threshold setting unit that sets a threshold based on the kernel type information; a defective pixel determination unit that detects whether a pair of target pixels are defective pixels based on the threshold; and a defective pixel correction unit that corrects one of the target pixels if the pair of target pixels are defective pixels. [Effects of the Invention]

[0008] The embodiments disclosed herein provide the effect of enabling more accurate detection and correction of defective pixels, even when two or more defective pixels are adjacent to each other. In addition, this document can provide various effects that can be understood directly or indirectly. [Brief explanation of the drawing]

[0009] [Figure 1]It is a block diagram showing an image signal processor according to an embodiment of the present disclosure. [Figure 2] It is a block diagram showing in more detail the defective pixel detection unit shown in FIG. 1. [Figure 3] It is a diagram exemplarily showing a kernel generated by the kernel generation unit shown in FIG. 2. [Figure 4] It is a diagram for explaining the operation of determining texture by the texture determination unit shown in FIG. 2. [Figure 5] It is a diagram for explaining the operation of determining defective pixels by the defective pixel determination unit shown in FIG. 2. [Figure 6] It is a diagram for explaining the operation of correcting defective pixels by the defective pixel correction unit shown in FIG. 1. [Figure 7] It is another embodiment for explaining the operation of correcting defective pixels by the defective pixel correction unit shown in FIG. 1. [Figure 8] It is a flowchart for explaining the operation of the image signal processor of FIG. 1. [Figure 9] It is a block diagram showing an example of a computing device corresponding to the image signal processor of FIG. 1.

Embodiments for Carrying Out the Invention

[0010] Hereinafter, various embodiments will be described with reference to the accompanying drawings. However, it should be understood that the present disclosure is not limited to specific embodiments and includes various modifications, equivalents, and / or alternatives of the embodiments. Embodiments of the present disclosure can provide various effects directly or indirectly recognizable by the present disclosure.

[0011] FIG. 1 is a block diagram showing an image signal processor according to an embodiment of the present disclosure. Referring to FIG. 1, an Image Signal Processor (hereinafter referred to as "ISP") 100 can perform at least one image signal processing on image data IDATA and generate corrected image data IDATA_P.

[0012] The ISP 100 can reduce noise in the image data IDATA and perform image signal processing for image quality improvement such as demosaicing, defect pixel correction, gamma correction, color filter array interpolation, color matrix, color correction, color enhancement, lens distortion correction, etc.

[0013] In addition, the ISP 100 can compress the image data on which the image signal processing for image quality improvement has been performed to generate an image file, or can restore the image data from the image file. The compression format of the image may be a reversible format or an irreversible format. As an example of the compression format, in the case of a still image, formats such as JPEG (Joint Photographic Experts Group) format and JPEG2000 format can be used. Also, in the case of a video, a video file can be generated by compressing a plurality of frames according to the MPEG (Moving Picture Experts Group) standard.

[0014] Image data IDATA can be generated by an image sensing device that captures an optical image of a scene, but the scope of the present invention is not limited thereto. The image sensing device may include a pixel array containing multiple pixels for detecting light incident from the scene, a control circuit for controlling the pixel array, and a readout circuit that converts the analog pixel signals received from the pixel array into digital image data IDATA and outputs it. In this disclosure, the description assumes that the image data IDATA is generated by an image sensing device.

[0015] A pixel array can include a color filter array (CFA) in which color filters are arranged according to a specific pattern (e.g., Bayer pattern, quad Bayer pattern, nona Bayer pattern, RGBW pattern, etc.) so that each pixel can detect light in a predetermined wavelength band. The pattern of the image data IDATA can be determined according to the type of pattern of the CFA.

[0016] An ISP100 according to one embodiment of the present invention may include a defective pixel detector 200 and a defective pixel corrector 300.

[0017] The defective pixel detection unit 200 can detect defective pixels based on the image data IDATA. The defective pixel detection unit 200 can detect defective pixels and output defective pixel information DPD to the defective pixel correction unit 300.

[0018] A defective pixel can refer to a pixel that is unable to properly generate pixel data corresponding to the intensity of incident light. A defective pixel may be a fixed defective pixel predetermined according to its pixel attributes (e.g., a PDAF (phase detection auto focusing) pixel, a defective pixel with a defect due to manufacturing process limitations), or a defective pixel that is temporarily unable to generate normal pixel data due to environmental or structural causes. Here, a PDAF pixel is a pixel that acquires phase difference information to realize the autofocus function, and from the viewpoint of image data processing, it can be classified as a defective pixel.

[0019] The defective pixel detection unit 200 can detect the location information of a defective pixel from the image data IDATA. For example, the defective pixel detection unit 200 can detect the pixel data of a defective pixel.

[0020] For the sake of clarity, this disclosure defines pixel data as the digital data corresponding to the pixel signal of each pixel, and image data (IDATA) as the set of pixel data for pixels corresponding to a certain unit (e.g., a frame or kernel). Here, a frame may correspond to the entire pixel array, and a kernel may mean a unit for image signal processing. In this specification, a pixel being "included" in a kernel may mean that the pixel "corresponds" to a kernel that corresponds to a particular unit of operation.

[0021] The defective pixel detection unit 200 receives location information of defective pixels that has been stored in advance from the image sensing device that generates the image data IDATA, and can determine whether or not a target pixel is a defective pixel based on the location information of the defective pixels. The image sensing device can store fixed location information of defective pixels in its internal storage (e.g., OTP (One Time Programmable) memory) for process reasons, and can provide the location information of defective pixels to the ISP 100. A more detailed description of the operation of the defective pixel detection unit 200 will be given later with reference to Figure 2.

[0022] If the defective pixel detection unit 200 determines that a target pixel is a defective pixel, the defective pixel correction unit 300 can correct the pixel data of the target pixel based on the kernel image data that contains the target pixel. Here, the pixel data of the target pixel may refer to the normal color pixel data that could be obtained if the target pixel was not a defective pixel.

[0023] As an example, the defective pixel correction unit 300 can correct the pixel data of a target pixel using the pixel data of a pixel that has the same attributes as the target pixel among the pixels included in the kernel. As another example, the defective pixel correction unit 300 can perform defective pixel correction in units of a mask having a predetermined size. Here, the defective pixel correction may be an operation in which the pixel data of at least one pixel that is the same type (and / or different type) as the target pixel is calculated (for example, by linear interpolation) within a mask in which the target pixel to be corrected is placed in the center, and the pixel data corresponding to the target pixel is calculated. A more detailed operation of the defective pixel correction unit 300 will be described later with reference to Figures 6 and 7.

[0024] Figure 2 is a block diagram showing the defective pixel detection unit shown in Figure 1 in more detail. Referring to Figure 2, the defective pixel detection unit 200 can include a kernel generation unit 210, a texture determination unit 220, a threshold setting unit 230, and a defective pixel determination unit 240.

[0025] The kernel generation unit 210 can generate a kernel for determining defective pixels from the pixel data contained in the image data IDATA. The kernel generated by the kernel generation unit 210 is movable within the input image. For example, the kernel generation unit 210 can generate an operational kernel for detecting defective pixels while moving through the pixels in two units from the beginning to the end of the pixel. An example of a kernel generated by the kernel generation unit 210 will be explained in more detail with reference to Figure 3, which will be described later.

[0026] The texture determination unit 220 can analyze textures on a kernel-by-kernel basis based on the image data IDATA. The image data IDATA corresponding to a single frame may contain textures of various sizes and shapes. A texture refers to a collection of pixels that have similarity; for example, a subject with a unified color included in a scene can be recognized as a texture.

[0027] The texture may also be one of the properties indicating whether the target kernel is a flat region, an edge (or corner) region, or a pattern region more complex than an edge region. Here, the target kernel may mean a unit for image signal processing that contains the target pixels to be corrected. A flat region may mean a region in which the target kernel has no particular directionality and has pixel data that is very similar overall, and may mean a texture region that is simpler than an edge region. The texture may also be one of the properties indicating whether the target kernel is a dark region or not. A dark region may mean a texture region in which the target kernel is darker than the brightness already set.

[0028] The texture determination unit 220 can analyze the texture and determine whether the target kernel containing the target pixel is a flat region or a dark region. For example, the texture determination unit 220 may be divided into a flat kernel determination unit or a dark kernel determination unit. Here, the target kernel can be a target kernel for determining whether a region is flat or dark, from a set of pixel data of a certain unit that contains the pixel data of the target pixel.

[0029] The texture determination unit 220 can determine a target kernel as a flat region if the target kernel does not correspond to a certain pattern shape. Here, the certain pattern shape can refer to a corner pattern, an edge pattern, etc. For example, the texture determination unit 220 can determine a target kernel as a flat region if the standard deviation of the pixel data of each pixel included in the target kernel is less than a set value. The set value can correspond to a value pre-stored in the ISP 100 for determining the kernel type. As another example, the texture determination unit 220 can determine a flat region by setting a threshold based on the median value of the pixels included in the target kernel.

[0030] The texture determination unit 220 can determine the brightness of a kernel based on the pixel data of each target kernel. For example, the texture determination unit 220 can determine whether or not the target kernel is in a dark region by calculating the median value using the pixel values ​​of green pixels surrounding the target pixel.

[0031] The texture determination unit 220 can transmit kernel type information of the target kernel to the threshold setting unit 230. For example, the texture determination unit 220 can transmit kernel type information indicating whether the target kernel is a flat region or a dark region to the threshold setting unit 230. An example of how the texture determination unit 220 determines the texture will be explained in more detail with reference to Figure 4, which will be described later.

[0032] The threshold setting unit 230 receives kernel type information from the texture determination unit 220 and can set a threshold value that serves as a reference value for detecting defective pixels. For example, the threshold setting unit 230 can set thresholds in different ways based on the kernel type information of the target kernel.

[0033] The threshold setting unit 230 can set a first threshold if the target kernel is determined to be a flat region. For example, if the target kernel is a flat region, the first threshold can be set based on the brightness of the target pixel. That is, the threshold setting unit 230 may determine it as a specific percentage of the current kernel brightness value (for example, the average value of green pixels). As another example, the threshold setting unit 230 can set the first threshold based on the standard deviation of pixels located within the target kernel. As yet another example, the threshold setting unit 230 can set the first threshold by comparing the standard deviation of the pixel data of pixels located in the same channel within the target kernel with the standard deviation of the pixel data of pixels located within the target kernel. As yet another example, the threshold setting unit 230 can set the first threshold based on the median value obtained by the texture determination unit 220. As yet another example, the threshold setting unit 230 can set the first threshold based on the pixel values ​​of similar pixels that have the same attributes as the target pixel.

[0034] Furthermore, the threshold setting unit 230 can set a second threshold if the target kernel is determined to be in a dark region. For example, if the threshold setting unit 230 determines that the target kernel is in a dark region, it can set the second threshold to a fixed constant.

[0035] The threshold setting unit 230 can transmit threshold information, including the set threshold, to the defective pixel determination unit 240. For example, the threshold setting unit 230 can transmit threshold information corresponding to a target kernel that is a flat region (first threshold) or threshold information corresponding to a target kernel that is a dark region (second threshold) to the defective pixel determination unit 240.

[0036] The defective pixel determination unit 240 receives threshold information and can determine whether the target pixel is a "couplet defective pixel". Here, a couplet defective pixel may mean a pair of defective pixels that are adjacent to each other and located in a continuous sequence.

[0037] As an example, the defective pixel determination unit 240 can determine a defective pixel using the brightness difference between a "target pixel" and a "reference pixel" having the same color as the target pixel. Here, the "target pixel" can indicate the object to be determined as to whether or not it is a defective pixel. The "reference pixel" can correspond to multiple surrounding pixels other than the target pixel channel that is subject to defect correction.

[0038] The defective pixel determination unit 240 can compare the pixel data of the target pixel with the pixel data of the reference pixel. The defective pixel determination unit 240 can determine whether the target pixel is a couplet defective pixel by comparing the difference value obtained by comparing the pixel data of the target pixel with the pixel data of the reference pixel with the first threshold or second threshold described above.

[0039] If the defective pixel determination unit 240 determines that a target pixel is a couplet defective pixel, it can generate defective pixel information DPD, which includes the coordinate information and pixel data of the target pixel. For example, the defective pixel information DPD may include information indicating whether the defective pixel is a defective pixel contained in a target kernel that is a flat region or a defective pixel contained in a target kernel that is a dark region. A detailed explanation of the process for determining defective pixels will be described later with reference to Figure 5.

[0040] Figure 3 is an illustrative diagram showing a kernel generated by the kernel generation unit shown in Figure 2. Referring to Figure 3, in this disclosure, the kernel generated by the kernel generation unit 210 may include M × N pixels arranged in a matrix. Here, M and N are distinct natural numbers, and M may be a natural number greater than N. As the kernel size increases, the resources required for detecting and correcting defective pixels may increase. Therefore, in this disclosure, a kernel with a horizontal size greater than its vertical size can be used to reduce resources. In other words, the kernel according to the embodiment of this disclosure may be an asymmetric kernel with a horizontal size greater than its vertical size.

[0041] For example, let's assume that the kernel generated by the kernel generation unit 210 is a 10x5 kernel unit having 10 rows and 5 columns. In the target kernel, green (G), red (R), and blue (B) color filters can be arranged in a quad Bayer pattern. One color value from green (G), red (R), and blue (B) is matched for each of the four unit pixel groups UPG arranged in a 2x2 matrix, thereby constructing a 10x5 kernel.

[0042] This disclosure uses a quad Bayer pattern kernel as an example, but the technical concept of this disclosure can also be applied to kernels in which color pixels are arranged in other ways, such as nona Bayer pattern, hexa Bayer pattern, RGBW pattern, and mono pattern. In this disclosure, kernels of sizes other than 10x5 may be used depending on the performance of the image signal processor, the required correction accuracy, and the color pixel arrangement method, and the unit of the kernel is not limited.

[0043] In Figure 3, embodiments shown in (A) and (B) can illustrate a case where the target kernel includes a target pixel T corresponding to a green (G) color filter. Furthermore, embodiments shown in Figure 3, (C) and (D), can illustrate a case where the target kernel includes a target pixel T corresponding to a specific color filter. For example, the specific color filter may be a red (R) color filter, a blue (B) color filter, etc., but in this disclosure, the red (R) color filter is described exemplarily.

[0044] In Figures 3(A) to (D), pixels P00 to P49 can form a 10x5 kernel. Pixels P00 to P49 included in the target kernel can be grouped into multiple unit pixel groups. For example, each of these unit pixel groups can correspond to the same color filter and include adjacent pixels. For instance, adjacent pixels P30, P31, P40, and P41, corresponding to the green (G) color filter, can be grouped into one unit pixel group. Adjacent pixels P32, P33, P42, and P43, corresponding to the red (R) color filter, can be grouped into another unit pixel group. Adjacent pixels P10, P11, P20, and P21, corresponding to the blue (B) color filter, can be grouped into another unit pixel group. The remaining pixels included in the target kernel can be grouped in a similar manner.

[0045] In the target kernel, the target pixel T can correspond to a pair of pixels P04 and P05 located in the center of the bottom row. The target pixel T may also be a pair of pixels P04 and P05 that are adjacent to each other. Of the pair of pixels P04 and P05, the target pixel located on the left can be defined as the left pixel CL, and the target pixel located on the right can be defined as the right pixel CR.

[0046] In the case of kernel (A) in Figure 3, the target pixel T can be located in the 2x1 unit pixel group UPG1 corresponding to the green (G) color filter. In the case of kernel (B) in Figure 3, the target pixel T can be located in the lower row pixels of the 4x4 unit pixel group UPG2 corresponding to the green (G) color filter. In the case of kernel (C) in Figure 3, the target pixel T can be located in the 2x1 unit pixel group UPG3 corresponding to the red (R) color filter. In the case of kernel (D) in Figure 3, the target pixel T can be located in the lower row pixels of the 4x4 unit pixel group UPG4 corresponding to the red (R) color filter.

[0047] Figure 4 is a diagram illustrating the operation by which the texture determination unit shown in Figure 2 determines the texture. Figures 4(A) and (B) can show the case where the target kernel corresponds to a green (G) color filter and includes a target pixel T located on the top row line, as described in Figure 3(A) above. Figures 4(C) and (D) can show the case where the target kernel corresponds to a green (G) color filter and includes a target pixel T located on the bottom row line, as described in Figure 3(B) above.

[0048] Furthermore, Figures 4(E) and (F) can show the case where the target kernel corresponds to a red (R) color filter and includes a target pixel T located on the top row line, as described in Figure 3(C) above. Figures 4(G) and (H) can show the case where the target kernel corresponds to a red (R) color filter and includes a target pixel T located on the bottom row line, as described in Figure 3(D) above.

[0049] In this disclosure, the texture determination unit 220 can determine whether the target kernel corresponds to a flat region or a dark region by using the median value of a plurality of green color pixels, including the target pixel T, that are located adjacent to the target pixel T, among the pixels included in the target kernel.

[0050] For example, the texture determination unit 220 can group multiple green color pixels into a specific number of pixels arranged adjacently in a particular direction and calculate the median. Here, the method for calculating the median can be done by arranging the pixel values ​​in the target kernel in order of size and extracting the intermediate value, but the calculation method is not limited to this. As an example, if there are a total of four pixels to be calculated, the median can be obtained by calculating the average of the second largest pixel value and the third largest pixel value among the four pixel values.

[0051] The texture determination unit 220 can group pixels in at least two different directions so that no two defective pixels exist together within a grouped set of pixels, since two defective pixels can exist together within a grouped set of pixels. In other words, the texture determination unit 220 can group pixels in the horizontal (hor) and vertical (ver) directions, respectively, and perform calculation operations for each pixel group.

[0052] In this disclosure, as an example, a total of 16 green pixels from rows R1 to R4 are used in the calculation. However, this disclosure is not limited to this, and the number of pixels used in the calculation can be easily changed.

[0053] Figures 4(A), (C), (E), and (G) show the case where each green pixel is grouped horizontally. That is, the texture determination unit 220 can use the green pixels located in rows R1 to R4 out of the multiple rows R1 to R5 for calculation.

[0054] The texture determination unit 220 can group four green pixels G00, G01, G02, and G03 located in the center region of row R1 in the target kernel and calculate the median. The median value med1 is calculated based on the horizontally grouped green pixels G00, G01, G02, and G03 in row R1. hor This is as shown in [Equation 1] below.

[0055] [Formula 1]

number

[0056] The texture determination unit 220 can group four green pixels G10, G11, G12, and G13 located in the center region of row R2 in the target kernel and calculate the median. The median med2 is calculated based on the horizontally grouped green pixels G10, G11, G12, and G13 in row R2. hor This is as shown in [Equation 2] below.

[0057] [Formula 2]

number

[0058] The texture determination unit 220 can group the four green pixels G20, G21, G22, and G23 located in the center region of the green pixels placed in row R3 in the target kernel and calculate the median. The median med3 is calculated based on the green pixels G20, G21, G22, and G23 that are grouped horizontally in row R3. hor This is as shown in [Equation 3] below.

[0059] [Formula 3]

Number

[0060] In the target kernel, the texture determination unit 220 can group four green pixels G30, G31, G32, and G33 arranged in the center area among the green pixels arranged in row R4 and calculate the median value. In row R4, the median value med4 calculated based on the horizontally grouped green pixels G30, G31, G32, and G33 hor is as shown in the following [Equation 4].

[0061] [Equation 4]

Number

[0062] The texture determination unit 220 uses the horizontally calculated median values med1 hor , med2 hor , med3 hor , med4 hor calculated by the above [Equation 1] to [Equation 4] to set the maximum value as the representative horizontal median value med hor .

[0063] On the other hand, (B), (D), (F), and (H) of FIG. 4 can show the case of grouping each green pixel in the vertical direction. That is, the texture determination unit 220 can use the green pixels arranged in columns C2 to C9 among the plurality of columns C1 to C10 for calculation.

[0064] In the target kernel, the texture determination unit 220 can group four green pixels G00, G10, G20, and G30 among the green pixels in columns C2 and C3 and calculate the median value. In columns C2 and C3, the median value med1 calculated based on the vertically grouped green pixels G00, G10, G20, and G30 verThis is as shown in [Equation 5] below.

[0065] [Formula 5]

number

[0066] The texture determination unit 220 can group four green pixels G00, G10, G20, and G30 from the green pixels of columns C4 and C5 in the target kernel and calculate the median. The median med2 is calculated based on the vertically grouped green pixels G01, G11, G21, and G31 in columns C4 and C5. ver This is as shown in [Equation 6] below.

[0067] [Formula 6]

number

[0068] The texture determination unit 220 can group four green pixels G00, G10, G20, and G30 from the green pixels of columns C6 and C7 in the target kernel and calculate the median. The median med3 is calculated based on the vertically grouped green pixels G02, G12, G22, and G32 in columns C6 and C7. ver This is as shown in [Equation 7] below.

[0069] [Formula 7]

number

[0070] The texture determination unit 220 can group four green pixels G00, G10, G20, and G30 from the green pixels of columns C8 and C9 in the target kernel and calculate the median. The median med4 is calculated based on the vertically grouped green pixels G03, G13, G23, and G33 in columns C8 and C9. ver This is as shown in [Equation 8] below.

[0071] [Formula 8]

number

[0072] The texture determination unit 220 calculates the vertical median value med1 calculated by the aforementioned [Equations 5] to [8]. ver med2 ver med3 ver med4 ver The maximum value among them is the vertical representative median med ver It can be set to that.

[0073] The texture determination unit 220 determines the horizontal representative value med in the target kernel to prevent two defective pixels from occurring within a grouped set of pixels. hor and the representative median in the vertical direction med ver Among these, the median with the smaller value can be selected as the representative median (med). The texture determination unit 220 can determine whether or not the target kernel falls within the dark region based on the representative median (med).

[0074] On the other hand, the texture determination unit 220 can determine whether or not the target kernel falls within a flat region using the aforementioned representative median value. The method used by the texture determination unit 220 to determine the flat region is as shown in [Equation 9] below.

[0075] [Formula 9]

number

[0076] As shown in [Equation 9] above, the texture determination unit 220 determines a threshold (th) based on the representative median (med). dyn This can be set. Then, the texture determination unit 220 determines that the difference (absolute value abs) between the green pixel (Gi) and the representative median (med) within the target kernel is the threshold th dyn If the number of pixels that are less than or equal to 2 is 2, the target kernel can be determined to be a flat region.

[0077] When determining textures, a sorting process is necessary if dynamic range (the difference between the maximum and minimum values) is used to determine the deviation between pixels. However, when determining textures using a comparison method like the one disclosed here, a separate sorting process can be omitted, thereby reducing the number of calculations.

[0078] Figure 5 is a diagram illustrating the operation by which the defective pixel detection unit shown in Figure 2 determines defective pixels. Figures 5(A) and (B) can show the case where the target kernel corresponds to a green (G) color filter and includes a target pixel T located on the top row line, as described in Figure 3(A) above. Figures 5(C) and (D) can show the case where the target kernel corresponds to a green (G) color filter and includes a target pixel T located on the bottom row line, as described in Figure 3(B) above.

[0079] Furthermore, Figures 5(E) and (F) can show the case where the target kernel corresponds to a red (R) color filter and includes a target pixel T located on the top row line, as described in Figure 3(C) above. Figures 5(G) and (H) can show the case where the target kernel corresponds to a red (R) color filter and includes a target pixel T located on the bottom row line, as described in Figure 3(D) above.

[0080] The defective pixel determination unit 240 can determine whether a target pixel is a defective pixel based on the first threshold and second threshold applied by the threshold setting unit 230. Specifically, if the target kernel is a flat region, the defective pixel determination unit 240 can determine whether a target pixel is a defective pixel by comparing the difference between the pixel data of the target pixel and the average value of the pixel data of the reference pixel with the aforementioned first threshold. Conversely, if the target kernel is a dark region, the defective pixel determination unit 240 can determine whether a target pixel is a defective pixel by comparing the difference between the pixel data of the target pixel and the average value of the pixel data of the reference pixel with the aforementioned second threshold. In the following explanation, the first threshold and the second threshold will be collectively referred to as "thresholds".

[0081] The defective pixel determination unit 240 can compare the pixel data of the target pixel, which is the target of the determination of whether or not it is a defective pixel, with the average value of the pixel data of the reference pixels in the target kernel. If the difference value of the comparison result is greater than or equal to a threshold, the defective pixel determination unit 240 can determine that the target pixel is a couplet defective pixel that does not have normal pixel data.

[0082] As shown in Figure 3 above, of the pair of target pixels P04 and P05, the target pixel located on the left can be defined as the left pixel CL (first target pixel), and the target pixel located on the right can be defined as the right pixel CR (second target pixel). The defective pixel determination unit 240 can determine that a pixel is a "couplet defective pixel" if both the left pixel CL and the right pixel CR are defective pixels.

[0083] For example, as shown in Figure 5(A), the defective pixel determination unit 240 can compare the pixel data of the left pixel CL with the average value of the pixel data of the reference pixels P00, P01, P08, P12, P13, P16, P17, P22, P23, P26, P27, P30, P31, P34, P35, P38, P40, P41, P44, and P45, which are adjacent to the left pixel CL and correspond to the green color. The defective pixel determination unit 240 can then determine that the left pixel CL is a defective pixel if the difference value obtained from the comparison is greater than or equal to a threshold.

[0084] As shown in Figure 5(B), the defective pixel determination unit 240 can compare the pixel data of the right pixel CR with the average value of the pixel data of the reference pixels P01, P08, P09, P12, P13, P16, P17, P22, P23, P26, P27, P31, P34, P35, P38, P39, P44, P45, P48, and P49, which are adjacent to the right pixel CR and correspond to the green color. The defective pixel determination unit 240 can then determine that the right pixel CR is a defective pixel if the difference value obtained from the comparison is greater than or equal to a threshold. Since the remaining target kernels in Figure 5 (C) to (H) can be understood to determine defective pixels in a similar manner, redundant explanations will be omitted.

[0085] In the embodiment shown in Figure 5, an example was described in which there are 20 green reference pixels and 12 red reference pixels for comparison with the target pixel. However, this disclosure is not limited to this, and the number of reference pixels can be easily changed.

[0086] Figure 6 is a diagram illustrating the operation in which defective pixels are corrected by the defective pixel correction unit shown in Figure 1. In Figure 6, embodiments shown in (A) and (B) can illustrate the case where the target kernel includes a target pixel T corresponding to a green (G) color filter. And in Figure 6, embodiments shown in (C) and (D) can illustrate the case where the target kernel includes a target pixel T corresponding to a red (R) color filter.

[0087] The defective pixel correction unit 300 can correct a couplet defective pixel if the defective pixel determination unit 240 determines that a couplet defective pixel is detected. For example, the defective pixel correction unit 300 can correct only the pixel value of the remaining defective pixel (e.g., the left pixel CL) among the couplet defective pixels, excluding the pixel value of one defective pixel (e.g., the right pixel CR).

[0088] For example, as shown in Figure 6(A), the left pixel CL(P04) can be corrected by averaging the pixel values ​​of the P05 pixel and the reference pixel P13. Here, the reference pixel P13 may belong to different unit pixel groups from the left pixel CL, be of the same type, and be the closest pixel to the reference pixel. That is, the correction value of the left pixel CL(P04) can be calculated as shown in [Equation 10] below.

[0089] [Formula 10]

number

[0090] As shown in Figure 6(B), the left pixel CL(P04) can be corrected by averaging the pixel values ​​of the P05 pixel and the reference pixel P14. Here, the reference pixel P14 may be a pixel that belongs to the same unit pixel group as the left pixel CL, is of the same type, and is located at the closest distance. That is, the correction value of the left pixel CL(P04) can be obtained as shown in [Equation 11] below.

[0091] [Formula 11]

number

[0092] As shown in Figure 6(C), the left pixel CL(P04) can be corrected by averaging the pixel values ​​of the P05 pixel and the reference pixels P01 and P34. Here, the reference pixels P01 and P34 may belong to different unit pixel groups from the left pixel CL, be of the same type, and be pixels located at the closest distance. That is, the correction value of the left pixel CL(P04) can be determined as shown in [Equation 12] below.

[0093] [Formula 12]

number

[0094] As shown in Figure 6(D), the left pixel CL(P04) can be corrected by averaging the pixel values ​​of the P05 pixel and the reference pixel P14. Here, the reference pixel P4 may be a pixel that is included in the same unit pixel group as the left pixel CL, is of the same type, and is located at the closest distance. That is, the correction value of the left pixel CL(P04) can be determined as shown in [Equation 13] below.

[0095] [Formula 13]

number

[0096] In the embodiment shown in Figure 6, one example of how defective pixels are corrected is described, based on the average value of reference pixels of the same type as the target pixel. However, this disclosure is not limited to this, and defective pixels may also be corrected using the median value as described above, and the method of correcting defective pixels is not limited to this.

[0097] For example, the defective pixel correction unit 300 can replace the pixel value of a defective pixel with the median value among the pixel values ​​of the reference pixels. For instance, the reference pixels can be sequentially aligned according to the magnitude of their pixel values, and the pixel value of a defective pixel can be replaced with a pixel value that has the median value among the pixel values ​​of the reference pixels.

[0098] Figure 7 is another embodiment illustrating the operation in which defective pixels are corrected by the defective pixel correction unit shown in Figure 1. Figure 7 is an embodiment illustrating the operation in which the defective pixel correction unit 300 corrects defective pixels when there are other defective pixels in the target kernel besides the target pixels to be corrected.

[0099] In Figure 7, embodiments (A) to (D) can show the case where the target kernel includes a target pixel T corresponding to a green (G) color filter. And in Figure 7, embodiments (E) to (I) can show the case where the target kernel includes a target pixel T corresponding to a red (R) color filter. In Figure 7, defective pixels where defects occur are indicated by "×".

[0100] For example, as shown in Figure 7(A), the left pixel CL(P04) can be corrected by averaging the pixel values ​​of the P05 pixel and the reference pixel P12. Here, the reference pixel P12 and the left pixel CL belong to different unit pixel groups and are of the same type. The closest same-type pixel to the P04 pixel is the P13 pixel, but since the P13 pixel is a defective pixel, the next closest reference pixel, P12, can be used. That is, the correction value for the left pixel CL(P04) can be calculated as shown in [Equation 14] below.

[0101] [Formula 14]

number

[0102] As shown in Figure 7(B), the left pixel CL(P04) can be corrected using the pixel value of pixel P13. Here, the reference pixel P13 may belong to different unit pixel groups from the left pixel CL, be of the same type, and be the closest pixel to it. The closest same-type pixel to pixel P04 is pixel P05, but since pixel P05 is a defective pixel, only the reference pixel P13 can be used. That is, the correction value for the left pixel CL(P04) can be calculated as shown in [Equation 15] below.

[0103] [Formula 15]

number

[0104] As shown in Figure 7(C), the left pixel CL(P04) can be corrected by averaging the pixel values ​​of the P05 pixel and the reference pixel P15. Here, the reference pixel P15 and the left pixel CL belong to the same unit pixel group and are of the same type. The closest pixel of the same type to the P04 pixel is the P14 pixel, but since the P14 pixel is a defective pixel, the next closest reference pixel, P15, can be used. That is, the correction value for the left pixel CL(P04) can be calculated as shown in [Equation 16] below.

[0105] [Formula 16]

number

[0106] As shown in Figure 7(D), the left pixel CL(P04) can be corrected by averaging the pixel values ​​of reference pixels P14 and P15. Here, reference pixel P14 may be a pixel that is included in the same unit pixel group as the left pixel CL, is of the same type, and is located at the closest distance. Similarly, reference pixel P15 may be a pixel that is included in the same unit pixel group as the left pixel CL, is of the same type, and is located at the second closest distance. The closest pixel of the same type to pixel P04 is pixel P05, but since pixel P05 is a defective pixel, the next closest reference pixels P14 and P15 can be used. That is, the correction value for the left pixel CL(P04) can be calculated as shown in [Equation 17] below.

[0107] [Formula 17]

number

[0108] As shown in Figure 7(E), the left pixel CL(P04) can be corrected by averaging the pixel values ​​of the P05 pixel and the reference pixel P34. Here, the reference pixel P34 belongs to a different unit pixel group from the left pixel CL and is a homogeneous pixel. The closest homogeneous pixels to the P04 pixel are the P01 pixel and the P34 pixel, but since the P01 pixel is a defective pixel, the reference pixel P34 can be used. That is, the correction value for the left pixel CL(P04) can be calculated as shown in [Equation 18] below.

[0109] [Formula 18]

number

[0110] As shown in Figure 7(F), the left pixel CL(P04) can be corrected by averaging the pixel values ​​of reference pixels P01 and P34. Here, reference pixels P01 and P34 belong to different unit pixel groups from the left pixel CL and are of the same type. The closest pixel of the same type to pixel P04 is pixel P05, but since pixel P05 is a defective pixel, the next closest reference pixels P01 and P34 can be used. That is, the correction value for the left pixel CL(P04) can be calculated as shown in [Equation 19] below.

[0111] [Formula 19]

number

[0112] As shown in Figure 7(G), the left pixel CL(P04) can be corrected by averaging the pixel values ​​of the P05 pixel and the reference pixel P01. Here, the reference pixel P01 and the left pixel CL belong to different unit pixel groups and are of the same type. The closest pixels of the same type to the P04 pixel are the P01 pixel and the P34 pixel, but since the P34 pixel is a defective pixel, the reference pixel P01 can be used. In other words, the correction value of the left pixel CL(P04) can be calculated as shown in [Equation 20] below.

[0113] [Formula 20]

number

[0114] The remaining target kernels in Figure 7(H) and (I) can be understood to correct defective pixels in a similar manner, so redundant explanations are omitted.

[0115] Figure 8 is a flowchart illustrating the operation of the image signal processor shown in Figure 1. Referring to Figure 8, the kernel generation unit 210 can generate a kernel for determining defective pixels (step S1). The kernel generated by the kernel generation unit 210 is movable within the input image. For example, the kernel generation unit 210 can scan the image data IDATA, which is arranged in a matrix array, from the top left to the bottom right, and generate a kernel while moving two pixels to the right at a time.

[0116] Subsequently, the texture determination unit 220 targets the kernel generated by the kernel generation unit 210 and can determine whether the target kernel corresponds to a flat region or a dark region (step S2).

[0117] Next, the texture determination unit 220 can determine whether or not the target kernel corresponds to a flat region (step S3). Here, a target kernel being a flat region may mean that the target kernel is a kernel included in a flat region.

[0118] The threshold setting unit 230 can set a first threshold if the target kernel is determined to be a flat region (step S5). For example, if the target kernel is a flat region, the threshold setting unit 230 can set a first threshold based on the brightness of the target pixel.

[0119] The threshold setting unit 230, when the target pixel is a green pixel, uses the aforementioned representative median (med) and threshold th obtained by the texture determination unit 220. dyn The first threshold can be set by summing the values ​​of the first threshold and the offset value.

[0120] Furthermore, if the target pixel is a color pixel (red or blue), the threshold setting unit 230 can set a first threshold using the pixel data of similar characteristic pixels, which are pixels in the target kernel that have the same attributes as the target pixel. Here, pixels that have the same attributes as the target pixel can be pixels that correspond to the same color filter or the same channel as the target pixel. In this disclosure, the same channel may mean the position of pixels that are the same relative to the center point of the microlens. For example, the threshold setting unit 230 can set a first threshold by finding the minimum value among the pixel values ​​of a specific number (e.g., 18) of similar characteristic pixels and then summing the offset values.

[0121] The texture determination unit 220 can then determine whether or not the target kernel falls within the dark region (step S4). Here, the statement that the target kernel is in the dark region may mean that the target kernel is a kernel that is included in the dark region.

[0122] The threshold setting unit 230 can set a second threshold if the target kernel is determined to be in a dark region (step S6). For example, since the threshold setting unit 230 operates only on dark textures when the target kernel is in a dark region, it can set the second threshold to a specific constant value, taking into account analog gain or digital gain, etc.

[0123] Next, the defective pixel determination unit 240 can determine within the target kernel the unit pixel group containing the target pixel and the reference pixel group containing the reference pixel to be compared. The defective pixel determination unit 240 can compare the pixel data of the target pixel with the pixel data of the reference pixel. By comparing the difference value obtained by comparing the pixel data of the target pixel and the pixel data of the reference pixel with the threshold value mentioned above, the defective pixel determination unit 240 can determine whether or not the target pixel is a couplet defective pixel (step S7).

[0124] For example, the defective pixel determination unit 240 can determine a target pixel as a defective pixel if the difference between the threshold value included in the threshold information and the aforementioned difference value is greater than or equal to a preset difference value. For example, if the target kernel is in a flat region, the defective pixel determination unit 240 can determine a target pixel as a defective pixel if the difference between the first threshold value and the aforementioned difference value is greater than or equal to a preset difference value. For example, if the target kernel is in a dark region, the defective pixel determination unit 240 can determine a target pixel as a defective pixel if the difference between the second threshold value and the aforementioned difference value is greater than or equal to a preset difference value.

[0125] If the defective pixel determination unit 240 determines that the target pixel is not a defective pixel (No. in S7), the process can proceed to step S9. Then, the kernel generation unit 210 can move two unit pixels and generate an operation kernel for detecting a defective pixel (step S9).

[0126] Next, if the defective pixel determination unit 240 determines that a couplet defective pixel is detected, the defective pixel correction unit 300 can interpolate the pixel data of the couplet defective pixel using the pixel data of the pixels included in the target kernel (step S8). In this disclosure, the pixel value of one defective pixel among the couplet defective pixels can be removed without correction and not included in the image data IDATA_P. Then, only the pixel value of the remaining defective pixel can be corrected to generate corrected image data IDATA_P that includes the interpolated target pixel data.

[0127] Figure 9 is a block diagram showing an example of a computing device corresponding to the image signal processor in Figure 1. Referring to Figure 9, the computing device 900 can be shown as one embodiment of the hardware configuration for performing the operation of the image signal processor 100 in Figure 1.

[0128] The computing device 900 can be mounted on a chip independent of the chip on which the image sensing device is mounted. According to one embodiment, the chip on which the image sensing device is mounted and the chip on which the computing device 900 is mounted can be realized in a single package, for example, an MCP (multi-chip package), but the scope of the present invention is not limited thereto.

[0129] Furthermore, the internal configuration or arrangement of the computing device 900 and the image sensing device may vary depending on the embodiment. For example, at least a portion of the configuration of the image sensing device may be included in the computing device 900. Alternatively, at least a portion of the configuration of the computing device 900 may be included in the image sensing device. In this case, at least a portion of the configuration of the computing device 900 may be mounted together with the chip on which the image sensing device is installed.

[0130] The computing device 900 may include a processor 910, memory 920, an input / output interface 930, and a communication interface 940.

[0131] The processor 910 can process the data and / or instructions necessary to perform the operation of the configurations 200 and 300 of the image signal processor 100 as described in Figure 1. That is, the processor 910 may refer to the image signal processor 100 itself, but the scope of the present invention is not limited thereto.

[0132] Memory 920 can store data and / or instructions necessary for the operation of configurations 200 and 300 of the image signal processor 100, and can be accessed by processor 910. For example, memory 920 can be implemented as volatile memory (e.g., DRAM (Dynamic Random Access Memory), SRAM (Static Random Access Memory), etc.) or non-volatile memory (e.g., PROM (Programmable Read Only Memory), EPROM (Erasable PROM), EEPROM (Electrically Erasable PROM), flash memory, etc.).

[0133] In other words, the computer program for operating the image signal processor 100 disclosed in this document is recorded in memory 920 and executed and processed by processor 910, thereby realizing the operation of the image signal processor 100.

[0134] The input / output interface 930 can provide an interface that connects an external input device (e.g., keyboard, mouse, touch panel, etc.) and / or an external output device (e.g., display) with the processor 910, enabling the transmission and reception of data.

[0135] The communication interface 940 is configured to send and receive various types of data with external devices (e.g., application processor, external memory, etc.) and may be a device that supports wired or wireless communication. [Explanation of symbols]

[0136] 100 Image Signal Processors 200 Defect Pixel Detection Unit 210 Kernel generation unit 220 Texture determination unit 230 Threshold setting section 240 Defect Pixel Detection Unit 300 Defective Pixel Correction Unit 900 Computing Devices 910 Processor 920 memory 930 Input / Output Interface 940 Communication Interface

Claims

1. A kernel generation unit that generates a target kernel containing a pair of adjacent target pixels, A texture determination unit analyzes the texture of the target kernel and determines whether the target kernel corresponds to a flat region or a dark region. A threshold setting unit sets a threshold based on the determination result of the texture determination unit, A defective pixel determination unit detects whether a pair of target pixels are defective pixels based on the pixel data of reference pixels having the same attributes as the pair of target pixels, If the pair of target pixels is a defective pixel, a defective pixel correction unit corrects one of the pair of target pixels. An image signal processor, including...

2. The aforementioned target kernel is It includes multiple pixels arranged in an M × N matrix containing M rows and N columns (where M and N are distinct natural numbers), The image signal processor according to claim 1, wherein the number of M is greater than the number of N in the asymmetric kernel.

3. The kernel generation unit, The image signal processor according to claim 1, which generates the target kernel by moving pixels in units of two pixels within the input image.

4. The image signal processor according to claim 1, wherein the multiple pixels included in the target kernel correspond to the same color filter and are grouped together with adjacent pixels.

5. The aforementioned pair of target pixels are The image signal processor according to claim 1, wherein each corresponds to the same color filter and is located adjacent to each other within the same unit pixel group.

6. The texture determination unit, An image signal processor according to claim 1, which determines the flat region or the dark region based on the median of the pair of target pixels and a specific number of reference pixels.

7. The texture determination unit, A first representative median is set by grouping a specific number of pixels adjacent in the first direction from among the multiple pixels included in the target kernel. A second representative median is set by grouping a specific number of pixels adjacent to each other in a second direction different from the first direction among the aforementioned plurality of pixels, The image signal processor according to claim 6, which determines the dark region based on the smaller representative median among the first representative median and the second representative median.

8. The texture determination unit, A threshold is set based on the aforementioned representative median. The image signal processor according to claim 7, wherein in the target kernel, if the number of pixels in which the difference between the pixel corresponding to the green color filter and the representative median is less than or equal to the threshold is two or less, the target kernel is determined to be the flat region.

9. The threshold setting unit is, If the target kernel is the flat region, the threshold is set to a first threshold based on the brightness of the pair of target pixels. The image signal processor according to claim 1, wherein if the target kernel is the dark region, the threshold is set to a second threshold having a fixed constant value.

10. The threshold setting unit is, The image signal processor according to claim 9, wherein, if the pair of target pixels are pixels corresponding to a green color filter, the first threshold is set based on the median value of the reference pixels, the threshold set by the texture determination unit, and the offset value.

11. The aforementioned reference pixel is The image signal processor according to claim 1, having a color filter of the same color as the color filter corresponding to the target pixels, and located in a different unit pixel group from the pair of target pixels.

12. The aforementioned defective pixel determination unit, If the difference between the pixel data of the first target pixel and the average of the pixel data of each of the first reference pixels surrounding the first target pixel is greater than or equal to the threshold, the first target pixel is determined to be the defective pixel. The image signal processor according to claim 1, wherein if the difference between the pixel data of the second target pixel and the average of the pixel data of each of the second reference pixels surrounding the second target pixel is greater than or equal to the threshold, the second target pixel is determined to be the defective pixel.

13. The aforementioned defective pixel determination unit, The image signal processor according to claim 12, which determines that both the first target pixel and the second target pixel are defective pixels, to be a couplet defective pixel.

14. The aforementioned defective pixel correction unit, The image signal processor according to claim 13, wherein if a pixel is determined to be a couplet defect, the pixel value of the second target pixel is removed and the pixel value of the first target pixel is corrected.

15. The aforementioned defective pixel correction unit, The image signal processor according to claim 13, which corrects the first target pixel by averaging the pixel values ​​of the reference pixel.

16. The aforementioned defective pixel correction unit, The image signal processor according to claim 13, which corrects the first target pixel using the median value of the reference pixel.

17. A texture determination unit analyzes the texture of a target kernel containing a pair of adjacent target pixels to determine kernel type information, A threshold setting unit that sets a threshold based on the kernel type information, A defective pixel determination unit that detects whether the pair of target pixels are defective pixels based on the threshold, If the pair of target pixels is a defective pixel, a defective pixel correction unit corrects one of the pair of target pixels. An image signal processor, including...

18. The texture determination unit, The image signal processor according to claim 17, which determines the kernel type information based on whether the target kernel corresponds to a flat region or a dark region.

19. The aforementioned pair of target pixels are The image signal processor according to claim 17, wherein each corresponds to the same color filter and is located adjacent to each other within the same unit pixel group.

20. The aforementioned defective pixel determination unit, If the difference between the pixel data of the first target pixel and the average of the pixel data of each of the first reference pixels surrounding the first target pixel is greater than or equal to the threshold, the first target pixel is determined to be the defective pixel. The image signal processor according to claim 17, wherein if the difference between the pixel data of the second target pixel and the average of the pixel data of each of the second reference pixels surrounding the second target pixel is greater than or equal to the threshold, the second target pixel is determined to be the defective pixel.

Images