Image signal processor and method for processing an image signal

Abstract

The present application relates to an image signal processor and a method for processing an image signal. An image signal processor includes a defective pixel detector configured to detect a defective pixel from among similar characteristic pixels, the similar characteristic pixels representing pixels having the same characteristic as a characteristic of a target pixel included in a target kernel; a pattern determiner configured to determine a pattern of the target kernel based on pixel data of pixels included in the target kernel when the defective pixel is present; and a pixel interpolator configured to interpolate the target pixel based on pixel data of pixels located in a texture region of the pattern from among dissimilar characteristic pixels and pixel data of pixels located in the texture region from among the similar characteristic pixels when the defective pixel is located in the texture region.

Description

Technical Field

[0001] The embodiments of this disclosure generally relate to an image signal processor capable of performing image conversion and a method for processing image signals. Background Technology

[0002] An image sensor is a device used to capture optical images by converting light into electrical signals using photosensitive semiconductor materials that react to light. With the development of the automotive, medical, computer, and communications industries, the demand for high-performance image sensing devices is constantly increasing in various devices such as smartphones, digital cameras, game consoles, the Internet of Things (IoT), robots, security cameras, and medical miniature cameras.

[0003] In image sensing devices, pixel arrays that directly capture optical images may include defective pixels that cannot properly acquire color images due to manufacturing errors. To achieve autofocus functionality, there is a growing demand for image sensing devices manufactured such that the pixel array includes phase difference detection pixels. Phase difference detection pixels, capable of acquiring phase difference-related information, cannot acquire color images in the same way as defective pixels, making them potentially detrimental from the perspective of the color image. Summary of the Invention

[0004] Various embodiments of this disclosure relate to image signal processors and image signal processing methods that can improve the accuracy of correction of defective pixels.

[0005] When the target pixel is a defective pixel, its pixel data can be interpolated based on the pixel data of the surrounding pixels. However, some of the surrounding pixels may also be defective pixels. Various embodiments of this disclosure relate to a method for interpolating the pixel data of a target pixel even when both the target pixel and the surrounding pixels are defective pixels.

[0006] According to embodiments of this disclosure, an image signal processor may include: a defective pixel detector configured to detect defective pixels from similar characteristic pixels, the similar characteristic pixels representing pixels having the same characteristics as target pixels included in the target kernel; a pattern determiner configured to determine a pattern of the target kernel based on pixel data of pixels included in the target kernel when the defective pixel is present; and a pixel interpolator configured to interpolate the target pixel based on pixel data of pixels in the texture region among dissimilar characteristic pixels and pixel data of pixels in the texture region among similar characteristic pixels when the defective pixel is located in a texture region of the pattern, the dissimilar characteristic pixels representing pixels having characteristics different from those of the target pixel.

[0007] According to another embodiment of this disclosure, an image signal processor may include: a pattern determiner configured to determine a pattern of the target kernel based on pixel data including pixels in the target kernel of the target pixel; a pixel interpolator configured to generate dissimilar characteristic pixel data including at least one of gradients or ratios between dissimilar characteristic pixels located in the texture region when a pixel with similar characteristic defects having the same characteristics as the target pixel is located in a texture region of the pattern, the dissimilar characteristic pixels representing pixels having characteristics different from the target pixel; and a pixel interpolator configured to interpolate the target pixel based on the dissimilar characteristic pixel data.

[0008] According to another embodiment of this disclosure, a method for processing an image signal may include the following steps: determining defective pixels from similar characteristic pixels, wherein the similar characteristic pixels represent pixels having the same characteristics as target pixels included in the target kernel; when the defective pixel exists, determining a pattern of the target kernel based on pixel data of the pixels included in the target kernel; and when the defective pixel is located in a texture region of the pattern, interpolating the target pixel based on pixel data of pixels in the texture region among dissimilar characteristic pixels and pixel data of pixels in the texture region among similar characteristic pixels, wherein the dissimilar characteristic pixels represent pixels having characteristics different from those of the target pixel.

[0009] It should be understood that the above overview and the following detailed description of this disclosure are illustrative and descriptive, and are intended to provide a further description of the claimed disclosure. Attached Figure Description

[0010] The above and other features and advantages of embodiments of this disclosure will become apparent when considered in conjunction with the accompanying drawings and the following detailed description.

[0011] Figure 1 This is a block diagram illustrating an image signal processor according to an embodiment of the present disclosure.

[0012] Figure 2 It is shown Figure 1 The diagram shows a detailed block diagram of the defect pixel corrector.

[0013] Figure 3 This is a flowchart illustrating the operation of an image signal processor according to an embodiment of the present disclosure.

[0014] Figure 4A This is a diagram showing the target kernel where the pixel corresponding to the red filter is located at the center.

[0015] Figure 4B This is a diagram illustrating the operation of an image signal processor according to an embodiment of the present disclosure when the pixel located at the center of the target kernel corresponds to a red color filter.

[0016] Figure 5A This is a diagram showing the target kernel where the pixel corresponding to the green filter is located at the center.

[0017] Figure 5B This is a diagram illustrating the operation of an image signal processor according to an embodiment of the present disclosure when the pixel located at the center of the target kernel corresponds to a green filter.

[0018] Figure 6A This is a diagram showing the target kernel where the pixel corresponding to the blue filter is located at the center.

[0019] Figure 6B This is a diagram illustrating the operation of an image signal processor according to an embodiment of the present disclosure when the pixel located at the center of the target kernel corresponds to a blue filter.

[0020] Figure 7 This illustrates an embodiment according to the present disclosure. Figure 1 A block diagram of the computing device corresponding to the image signal processor. Detailed Implementation

[0021] This disclosure provides embodiments of an image signal processor capable of performing image transformation and a method for processing image signals, which can be used in configuration to substantially solve one or more technical or engineering problems and mitigate limitations or disadvantages encountered in some image signal processors in the art. Some embodiments of this disclosure relate to image signal processors and image signal processing methods that can improve the correction accuracy of defective pixels, etc. Some embodiments of this disclosure relate to a method for interpolating pixel data of a target pixel even when both the target pixel and surrounding pixels are defective pixels. Recognizing the above problems, according to embodiments of this disclosure, even when the target pixel and the pixels surrounding the target pixel are defective pixels, the correction accuracy of the target pixel can be improved by providing a standard for selecting interpolation pixels for interpolating the pixel data of the target pixel.

[0022] Reference will now be made in detail to some embodiments of the present disclosure illustrated in the accompanying drawings. Wherever possible, the same reference numerals will be used throughout the drawings to refer to the same or similar parts. While embodiments of the present disclosure are readily adaptable to various modifications and alternatives, specific embodiments are shown in the drawings. However, these embodiments should not be construed as limiting to those described herein.

[0023] In the following description, various embodiments will be illustrated with reference to the accompanying drawings. However, it should be understood that the embodiments are not limited to the specific embodiments, but include various modifications, equivalents, and / or substitutions of the embodiments. The embodiments of this disclosure can provide various advantageous effects that can be directly or indirectly recognized by those skilled in the art.

[0024] Figure 1 This is a block diagram illustrating an image signal processor 100 according to an embodiment of the present disclosure.

[0025] Reference Figure 1 The image signal processor (ISP) 100 can perform at least one image signal processing on image data (IDATA) to generate processed image data (IDATA_P). The ISP 100 can reduce noise in the image data (IDATA) and can perform various image signal processing operations (e.g., demosaic, defect pixel correction, gamma correction, color filter array interpolation, color matrix, color correction, color enhancement, lens distortion correction, etc.) to improve the image quality of the image data. Furthermore, the ISP 100 can compress the image data created by performing image signal processing for image quality improvement, allowing the ISP 100 to create image files using the compressed image data. Alternatively, the ISP 100 can recover image data from an image file. In this case, the scheme used to compress such image data can be a reversible or irreversible format. As representative examples of such compression formats, when using still images, the Joint Photographic Experts Group (JPEG) format, JPEG 2000 format, etc., can be used. Furthermore, when using moving images, multiple frames can be compressed according to the Moving Picture Experts Group (MPEG) standard, enabling the creation of moving image files.

[0026] Image data (IDATA) can be generated by an image sensing device that captures an optical image of a scene, but the scope of this disclosure is not limited thereto. The image sensing device may include: a pixel array comprising a plurality of pixels configured to sense incident light received from the scene; control circuitry configured to control the pixel array; and readout circuitry configured to output digital image data (IDATA) by converting analog pixel signals received from the pixel array into digital image data (IDATA). In some embodiments of this disclosure, image data (IDATA) may be generated by the image sensing device.

[0027] The pixel array of the image sensing device may include defective pixels that cannot properly capture color images due to process limitations or temporary noise intrusion. Additionally, the pixel array may include phase difference detection pixels configured to acquire phase difference-related information for autofocus functionality. Phase difference detection pixels cannot acquire color images in the same way as defective pixels, such that, from the perspective of the color image, phase difference detection pixels can be considered defective pixels. In some embodiments, for ease of description and better understanding of this disclosure, the defective pixels and phase difference detection pixels that cannot properly acquire color images will be collectively referred to as "defective pixels" hereinafter.

[0028] To improve the quality of color images, it is crucial to enhance the accuracy of defective pixel correction. Therefore, an image signal processor 100 based on some embodiments of this disclosure may include a defective pixel detector 200, a pattern determiner 300, and a defective pixel corrector 400.

[0029] The defective pixel detector 200 can detect pixel data of pixels identified as having defects from image data (IDATA). In this disclosure, for simplicity, digital data corresponding to the pixel signal of each pixel will be defined as pixel data, and the set of pixel data corresponding to a predetermined unit (e.g., a frame or a kernel) will be defined as image data (IDATA). Here, a frame corresponds to the entire pixel array, and a kernel can represent a unit used for image signal processing. In this disclosure, the statement that a pixel is "included" in a kernel can mean that the pixel "corresponds" to a kernel corresponding to a specific operating unit. The operation of the defective pixel detector 200 can be performed based on a target pixel as the pixel to be corrected and a target kernel including the target pixel.

[0030] The target pixel may correspond to a pixel identified as a defective pixel. For example, a target pixel may correspond to a pixel identified as a defective pixel based on the difference between the pixel data of the target pixel and the pixel data of each pixel included in the target kernel. Alternatively, a target pixel may correspond to a pixel identified as a defective pixel based on the difference between the pixel data of the target pixel and the average of the pixel data of the pixels included in the target kernel. Whether a target pixel is a defective pixel can be determined by the defective pixel detector 200, but the implementation is not limited thereto. It is also possible to independently determine whether peripheral pixels included in the target kernel are defective pixels.

[0031] The defective pixel detector 200 can determine whether each peripheral pixel included in the target kernel is a defective pixel. Here, peripheral pixels can refer to pixels included in the target kernel other than the target pixel. The defective pixel detector 200 can determine whether a peripheral pixel is a defective pixel based on the pixel data difference between the peripheral pixel and another peripheral pixel in the kernel. For example, when the pixel data difference between the peripheral pixel with the largest pixel data and the peripheral pixel with the second largest pixel data in the kernel is greater than or equal to a threshold, the defective pixel detector 200 can determine that the peripheral pixel with the largest pixel data is a defective pixel that does not have normal pixel data. Alternatively, when the pixel data difference between the peripheral pixel with the smallest pixel data and the peripheral pixel with the second smallest pixel data in the kernel is greater than or equal to a threshold, the defective pixel detector 200 can determine that the peripheral pixel with the smallest pixel data is a defective pixel. Here, the pixel data difference can be calculated based on the pixel data of peripheral pixels similar to the target pixel.

[0032] The threshold used in this disclosure may be a preset value or a fixed constant, or a specific ratio of the brightness value of the target kernel (e.g., the average value of green). In one embodiment, the ISP 100 may set the threshold based on the standard deviation of pixel data corresponding to pixels located in the target kernel. For example, the ISP 100 may set the threshold by comparing the standard deviation of pixels located in the same channel within the target kernel with the pixel data of pixels located in the target kernel. In this disclosure, a channel may refer to the relative position of a pixel with respect to the center of the microlens or the relative position of a pixel in a pixel arrangement. From different viewing angles, a channel may refer to a group of pixels having the same pixel characteristic type, such as a color filter type (e.g., red, green, or blue), a corresponding signal path, or a processing path for the pixel.

[0033] According to one embodiment, the defective pixel detector 200 can determine whether each of the similar characteristic pixels among the pixels included in the target kernel that have the same characteristics as the target pixel is a defective pixel. That is, a similar characteristic pixel refers to a pixel having the same characteristics, and the characteristics may include at least one characteristic factor. For example, the characteristics may include at least one characteristic factor of at least one of the color filter or channel corresponding to the pixel.

[0034] When the features selectively include a feature factor of one of the color filters or channels corresponding to a pixel, a pixel having the same features as the target pixel can be a pixel corresponding to the same color filter or the same channel as the target pixel. For example, when the target pixel is a pixel corresponding to a red color filter, the defective pixel detector 200 can determine whether each of the peripheral pixels included in the target kernel and corresponding to the red color filter is a defective pixel.

[0035] When the characteristics include both the color filter and the channel corresponding to the pixel as two characteristic factors, a pixel having the same characteristics as the target pixel can be a pixel that corresponds to the same color filter and the same channel as the target pixel. For example, when the target pixel corresponds to the red color filter and the first channel among multiple channels, the defect pixel detector 200 can determine whether each of the peripheral pixels included in the target kernel that corresponds to the red color filter and the first channel is a defect pixel.

[0036] In this disclosure, a similar characteristic pixel can refer to a pixel having the same characteristics as the target pixel, and a dissimilar characteristic pixel can refer to a pixel having different characteristics from the target pixel.

[0037] In one implementation, when the feature includes multiple feature factors, a similar feature pixel can be a pixel whose feature factors are all the same as those of the target pixel, and a dissimilar feature pixel can be a pixel that has at least one feature factor that is different from the target pixel.

[0038] In another embodiment, when the feature includes multiple feature factors, a similar feature pixel can be a pixel having at least one feature factor that is the same as the feature factor of the target pixel, and a dissimilar feature pixel can be a pixel whose feature factors are all different from the feature factors of the target pixel.

[0039] Regarding the target pixel, a pixel that is a pixel with similar characteristics and has defective pixel data can correspond to a pixel with similar characteristics and defects. Furthermore, in this disclosure, the same channel can refer to the position of a pixel that is in the same relative position as the center of the microlens, or the position of a pixel that is in the same relative position in a specific pattern (e.g., a 2×2 matrix, a 3×3 matrix) in which pixels are arranged adjacent to each other.

[0040] In another embodiment, the defective pixel detector 200 can receive pre-stored location information about defective pixels from the image sensing device that generates image data (IDATA), and can determine whether a target pixel is a defective pixel based on the location information about the defective pixels. For process-related reasons, the image sensing device can store the location information about fixed defective pixels in internal memory (e.g., one-time programmable (OTP) memory) and provide the location information about defective pixels to the ISP 100.

[0041] The defective pixel detector 200 can send defective pixel data (DPD) to the pattern determiner 300. The defective pixel data includes pixel data of defective pixels determined by the defective pixel detector 200 or pre-stored defective pixels received from the image sensing device, location information about the defective pixels, and / or the presence / absence of the defective pixels.

[0042] Pattern determiner 300 can identify pixel data of defective pixels, location information about defective pixels, or the presence / absence of defective pixels. Pattern determiner 300 can determine the pattern of a target core based on pixel data of pixels included in the target core. In one embodiment, when a defective pixel is found among similar characteristic pixels of a target pixel based on DPD, pattern determiner 300 can determine the pattern of the target core based on pixel data of pixels included in the target core.

[0043] For example, the pattern determiner 300 may determine the pattern of the target kernel based on at least one of data obtained by comparing pixel data between pixels located in the texture region of the pattern or data obtained by comparing pixel data between pixels located in the texture region and pixels located outside the texture region. In one embodiment, the pattern may include at least one of a horizontal pattern, a vertical pattern, a diagonal pattern from the lower left to the upper right, or a reverse diagonal pattern from the upper left to the lower right.

[0044] A texture is a set of similar pixels. For example, an object with a uniform color in a scene can be identified as a texture. The boundaries of a texture can extend in a specific direction, and the difference between pixel data within the boundaries and pixel data outside the boundaries can be greater than the difference between pixel data. In this disclosure, similar pixels of a texture can be considered to have a different technical meaning than pixels with similar characteristics.

[0045] Pattern determiner 300 can send pattern data (PTD) to defect pixel corrector 400. The pattern data includes the type of pattern of the target core determined by pattern determiner 200, as well as the pixel data of the defect pixel, the location information of the defect pixel, or the presence / absence of the defect pixel.

[0046] The defective pixel corrector 400 can identify defective pixel data, location information about the defective pixel, presence / absence of the defective pixel, or pattern type based on PTD. When a defective pixel exists among similar characteristic pixels located in the texture region of a pattern, the defective pixel corrector 400 can interpolate the target pixel based on pixel data of a dissimilar characteristic pixel that has characteristics different from those of the target pixel.

[0047] The defect pixel corrector 400 can generate processed image data (IDATA_P) by interpolating the target pixels based on image data (IDATA). See below for further details. Figure 2 Provides a detailed description of the operation of the defect pixel corrector 400.

[0048] Figure 1The detailed configuration of the ISP 100 shown and the information transmission according to that configuration are one implementation, and other configurations and information transmissions may be used in other implementations. The detailed configuration of the ISP 100 and the information transmission according to that configuration are not limited to this disclosure. For example, the defective pixel detector 200 and the pattern determiner 300 may be configured as a single module. Furthermore, for example, the defective pixel detector 200 may send defective pixel data (DPD) to the defective pixel corrector 400.

[0049] Figure 2 It is shown Figure 1 The diagram shows a detailed block diagram of the defect pixel corrector.

[0050] Reference Figure 2 The defective pixel corrector 400 may include a dissimilar characteristic pixel determiner 410 and a pixel interpolator 420.

[0051] The dissimilarity pixel determiner 410 may include at least one of a gradient determiner 412 or a ratio determiner 414. Based on the PTD, the dissimilarity pixel determiner 410 can detect pixel data of dissimilarity pixels located in a texture region of a pattern determined to correspond to a target kernel. For example, referring to... Figure 1 When the pattern determiner 300 determines that the pattern of the target kernel corresponds to the horizontal pattern, the dissimilar characteristic pixel determiner 410 can detect the pixel data of dissimilar characteristic pixels in the texture region of the horizontal pattern located within the target kernel.

[0052] The gradient determiner 412 can determine the gradient of pixel data between dissimilar feature pixels located in the texture region of the pattern. For example, the gradient determiner 412 can calculate the pixel data difference between dissimilar feature pixels located in the texture region of the pattern that are adjacent to pixels symmetrically positioned relative to the target pixel and similar feature defect pixels and dissimilar feature pixels adjacent to the target pixel.

[0053] The ratio determiner 414 can determine the ratio of pixel data of dissimilar feature pixels located in the texture region of the pattern. For example, the ratio determiner 414 can calculate a value obtained by dividing the pixel data of dissimilar feature pixels adjacent to pixels that are symmetrically positioned relative to the target pixel and similar feature defect pixels by the image data of dissimilar feature pixels adjacent to the target pixel.

[0054] The dissimilar characteristic pixel determiner 410 can send dissimilar characteristic pixel data (UPD) to the pixel interpolator 420, which includes at least one of pixel data gradient or ratio between dissimilar characteristic pixels located in the texture region of the pattern, as determined by the gradient determiner 412 or the ratio determiner 414.

[0055] Pixel interpolator 420 can interpolate the pixel data of a target pixel based on dissimilar characteristic pixel data (UPD). See below for further details. Figures 4A to 6B Provides a more detailed description of the operation of the pixel interpolator that interpolates the target pixel.

[0056] Figure 3 This is a flowchart illustrating the operation of an image signal processor according to an embodiment of the present disclosure.

[0057] Reference Figures 1 to 3 The defective pixel detector 200 can determine whether a defective pixel exists among similar characteristic pixels located around the target pixel (S100). When the pixel data difference between the pixel with the largest pixel data and the pixel with the second largest pixel data among similar characteristic pixels is greater than or equal to a threshold, the defective pixel detector 200 can identify the pixel with the largest pixel data as a defective pixel. Alternatively, when the pixel data difference between the pixel with the smallest pixel data and the pixel with the second smallest pixel data among similar characteristic pixels is greater than or equal to a threshold, the pixel with the smallest pixel data can be identified as a defective pixel.

[0058] When it is determined that a defective pixel exists among similar characteristic pixels surrounding the target pixel (S100: Yes), the pattern determiner 300 can determine the pattern of the target kernel based on the pixel data of the pixels included in the target kernel (S110). The pattern determiner 300 can determine the pattern of the target kernel based on at least one of the following: data obtained by comparing pixel data between pixels located in the texture region of the pattern, or data obtained by comparing pixel data between pixels located in the texture region and pixels located outside the texture region.

[0059] When it is determined that no defective pixels exist among similar characteristic pixels surrounding the target pixel (S100: No), the ISP 100 can determine that interpolating the pixel data of the target pixel based on the pixel data of dissimilar characteristic pixels is unnecessary and can terminate the process. In one embodiment, when it is determined that no defective pixels exist among similar characteristic pixels surrounding the target pixel (S100: No), the ISP 100 can interpolate the pixel data of the target pixel based on the pixel data of similar characteristic pixels.

[0060] In one implementation, the operation of determining whether a defective pixel exists among similar characteristic pixels located around the target pixel can be performed by a defective pixel detector 200. For example, based on the determination by the defective pixel detector 200 of the presence / absence of defective pixels among similar characteristic pixels located around the target pixel, the defective pixel detector 200 can send defective pixel data (DPD) to a pattern determiner 300.

[0061] In another embodiment, the operation of determining whether there are defective pixels among similar characteristic pixels located around the target pixel can be performed by the pattern determiner 300. For example, based on DPD, the pattern determiner 300 can determine whether there are defective pixels among similar characteristic pixels located around the target pixel. Furthermore, when defective pixels exist, the pattern determiner 300 can initiate the operation of determining the pattern of the target kernel.

[0062] The defect pixel corrector 400 can determine whether a defect pixel exists in the texture region of the pattern (S120). The defect pixel corrector 400 can determine whether a defect pixel with similar characteristics identified as a defect pixel in operation S100 is included in the texture region of the pattern as determined in operation S110.

[0063] When it is determined that a defective pixel exists in the texture region of the pattern (S120: Yes), the pixel interpolator 420 can interpolate the target pixel based on the pixel data of the dissimilar characteristic pixels (S130). The pixel interpolator 420 can interpolate the target pixel based on the pixel data of the dissimilar characteristic pixels located in the texture region and the pixel data of the similar characteristic pixels located in the texture region.

[0064] When it is determined that no defective pixels exist in the texture region of the pattern (S120: No), the ISP 100 can determine that interpolating the pixel data of the target pixel based on dissimilar characteristic pixels is unnecessary and can terminate the process. In one embodiment, when it is determined that no defective pixels exist in the texture region of the pattern (S120: No), the ISP 100 can interpolate the pixel data of the target pixel based on the pixel data of similar characteristic pixels.

[0065] In one implementation, the operation of determining whether defective pixels exist in the texture region of a pattern can be performed by a pattern determiner 300. For example, based on the determination of the presence / absence of defective pixels in the texture region of a pattern, the pattern determiner 300 can send pattern data (PTD) to a defective pixel corrector 400.

[0066] In another embodiment, the operation of determining whether defective pixels exist in the texture region of the pattern can be performed by the defective pixel corrector 400. For example, based on at least one of DPD or PTD, the defective pixel corrector 400 can determine whether defective pixels exist in the texture region of the pattern. When defective pixels exist in the texture region, the defective pixel corrector 400 can initiate interpolation of the target pixel.

[0067] Figure 4A This is a diagram showing the target kernel where the pixel corresponding to the red filter is located at the center.

[0068] Reference Figure 1 , Figure 2 and Figure 4A The target core 40 may include red pixels corresponding to a red filter, green pixels corresponding to a green filter, and blue pixels corresponding to a blue filter. The target core 40 may have red, green, and blue pixels arranged in a Bayer pattern array. Red pixels can detect red light and generate red pixel data, green pixels can detect green light and generate green pixel data, and blue pixels can detect blue light and generate blue pixel data. Each of the red, green, and blue pixel data ranges from 0 to 1023.

[0069] In the following description, a core arranged in a Bayer pattern will be provided. However, the technical concept of this disclosure can also be applied to cores having pixels arranged in other patterns, such as: a quad Bayer pattern, in which pixels of the same color are arranged in a 2x2 matrix to form a Bayer pattern; a nona Bayer pattern, in which pixels of the same color are arranged in a 3x3 matrix to form a Bayer pattern; a QxQ pattern (or a 16-in-1 Bayer pattern), in which pixels of the same color are arranged in a 4x4 matrix to form a Bayer pattern; an RGBW pattern, in which a green pixel in the Bayer pattern is replaced by a white pixel; a quad full 2PD pattern or a quad full 4 coupled PD pattern, in which multiple (2 or 4) pixels of the same color in the quad Bayer pattern correspond to a single microlens; a semi-shielded PD pattern, in which a light-shielding layer is placed on some of the green pixels in the Bayer pattern; a paired PD pattern, in which green pixels are arranged adjacent to each other in a Bayer pattern such that a single microlens corresponds to an adjacent green pixel; or a monochrome pattern, in which pixels without color filters are arranged.

[0070] For example, for a 4-in-1 fully coupled PD pattern, when the features include color filters and channels as feature factors, the similar feature pixels of the target pixels arranged in the upper left of a 2x2 matrix composed of green pixels (i.e., the position where the first row and the first column of the matrix intersect) can correspond to the pixels arranged in the upper left of another 2x2 matrix composed of green pixels (i.e., the position where the first row and the second column of the matrix intersect).

[0071] Furthermore, depending on the performance of the ISP 100, the required correction accuracy, and the arrangement of color pixels, a core of a different size than the 5x5 core (e.g., a 10x10 core) can be used.

[0072] In this disclosure, defective pixel correction is performed by the ISP 100 in a 5x5 kernel with 5 rows and 5 columns. The characteristics and relative positions of the pixels included in the target kernel 40 are described based on the row, column, and color filter of the pixels. For example, the center of the target kernel 40 may be occupied by the red pixel R22.

[0073] The target pixel can be located at the center of the target kernel 40. In one embodiment, the target kernel 40 may include a target pixel located at the center, such as a red pixel R22. In this case, pixels with similar characteristics, i.e., pixels having the same characteristics as the target pixels included in the target kernel 40, may correspond to red pixels R00, R02, R04, R20, R24, R40, R42, and R44. Green and blue pixels included in the target kernel 40 may correspond to pixels with dissimilar characteristics.

[0074] Therefore, for the target kernel 40, the defective pixel detector 200 can determine whether each of the similar characteristic pixels (i.e., red pixels R00, R02, R04, R20, R24, R40, R42, and R44) is a defective pixel. Pixels determined by the defective pixel detector 200 to be similar characteristic pixels can correspond to similar characteristic defective pixels.

[0075] Furthermore, for the target kernel 40, the pixel interpolator 420 can interpolate the target pixel based on pixel data of at least one of the dissimilar characteristic pixels (i.e., green pixels G01, G03, G10, G12, G14, G21, G23, G30, G32, G34, G41 and G43, and blue pixels B11, B13, B31 and B33).

[0076] Figure 4B This is a diagram illustrating the operation of an image signal processor according to an embodiment of the present disclosure when a pixel located at the center of the target kernel corresponds to a red color filter.

[0077] Reference Figure 1 , Figure 2 and Figure 4B The pixel interpolator 420 can interpolate the target pixel based on the horizontal left (HL) target kernel 41, the horizontal right (HR) target kernel 42, the vertical up (VU) target kernel 43, the vertical down (VD) target kernel 44, the diagonal up (SU) target kernel 45, the diagonal down (SD) target kernel 46, the back diagonal up (BSU) target kernel 47, or the back diagonal down (BSD) target kernel 48.

[0078] HL target kernel 41 and HR target kernel 42 can correspond to the case where the pattern determiner 300 determines the target kernel as a horizontal pattern. For example, in HL target kernel 41 and HR target kernel 42, the region corresponding to the pixels in the second to fourth rows of the kernel can be a texture region. The pattern and texture region of the target kernel can also be described below. Figure 5B and Figure 6B The same explanation is given in Chinese.

[0079] VU target kernel 43 and VD target kernel 44 can correspond to the case where the pattern determiner 300 determines the target kernel as a vertical pattern. For example, in VU target kernel 43 and VD target kernel 44, the region corresponding to the pixels located in the second to fourth columns of the kernel can be a texture region. The pattern and texture region of the target kernel can also be described below. Figure 5B and Figure 6B The same explanation is given in Chinese.

[0080] The SU target core 45 and SD target core 46 can correspond to the case where the pattern determiner 300 determines the target core as a diagonal pattern. For example, in the SU target core 45 and SD target core 46, the areas corresponding to green pixel G03, red pixel R04, green pixel G12, blue pixel B13, green pixel G14, green pixel G21, red pixel R22, green pixel G23, green pixel G30, blue pixel B31, green pixel G32, red pixel R40, and green pixel G41 can be textured areas. The pattern and textured areas of the target core can also be described below. Figure 5B and Figure 6B The same explanation is given in Chinese.

[0081] BSU target core 47 and BSD target core 48 can correspond to the case where the pattern determiner 300 determines the target core as a pattern in the reverse diagonal direction. For example, in BSU target core 47 and BSD target core 48, the areas corresponding to red pixel R00, green pixel G01, green pixel G10, blue pixel B11, green pixel G12, green pixel G21, red pixel R22, green pixel G23, green pixel G32, blue pixel B33, green pixel G34, green pixel G43, and red pixel R44 can be textured areas. Patterns and textured areas can also be described below. Figure 5B and Figure 6B The same explanation is given in Chinese.

[0082] When the target pixel corresponds to the red filter, the pixel with similar characteristic defects is located to the right of the target pixel, and the pattern determiner 300 determines the target kernel as a horizontal pattern, the pixel interpolator 420 can interpolate the target pixel with reference to the HL target kernel 41.

[0083] The HL target kernel 41 may include red pixel R22 as a target pixel and corresponds to a horizontal pattern. The HL target kernel 41 may also correspond to a target kernel including red pixel R24 as a pixel with similar characteristic defects. Since the pixel with similar characteristic defects is located to the right of the target pixel in the HL target kernel 41, the pixel interpolator 420 can interpolate the target pixel based on pixel data of similar or dissimilar characteristic pixels located on the left side of the HL target kernel 41 in the horizontal direction.

[0084] In the HL target kernel 41, pixels marked with dashed circles can correspond to defective pixels, and pixels marked with solid circles can correspond to pixels used for interpolation of target pixels. When the pixel interpolator 420 interpolates target pixels with reference to the HL target kernel 41, the pixel data of the target pixels can be calculated using equations such as Equation 1 or Equation 2 below.

[0085] In Equation 1 or Equation 2, the pixel data of the interpolated target pixel can correspond to r22, and the pixel data of the similar characteristic pixel R20 can correspond to r20. Furthermore, the pixel data of the dissimilar characteristic pixels G10, G12, G30, and G32 can correspond to g10, g12, g30, and g32, respectively. Additionally, in the equation, G1 can correspond to the value obtained by dividing the average pixel data of the red pixels included in the HL target kernel 41 by the average pixel data of the green pixels included in the HL target kernel 41. In one embodiment, G1 can be any set value or a preset threshold.

[0086] [Formula 1]

[0087] r22 = r20 + G1 × {(g12 + g32) – (g10 + g30)} / 2

[0088] [Equation 2]

[0089] r22 = r20 × G1 × {(g12 + g32) / (g10 + g30)}

[0090] In one implementation, the difference between dissimilar pixel data in Equation 1 can be determined by gradient determiner 412, and the ratio between dissimilar pixel data in Equation 2 can be determined by ratio determiner 414. The difference or ratio between dissimilar pixel data described in the equations given below can be understood in a similar manner.

[0091] When the target pixel corresponds to the red filter, the pixel with similar characteristic defects is located to the left of the target pixel, and the pattern determiner 300 determines the target kernel as a horizontal pattern, the pixel interpolator 420 can interpolate the target pixel with reference to the HR target kernel 42.

[0092] The HR target kernel 42 may include a red pixel R22 as a target pixel and corresponds to a horizontal pattern. The HR target kernel 42 may also correspond to a target kernel that includes a red pixel R20 as a pixel with similar characteristic defects. Since the pixel with similar characteristic defects is located to the left of the target pixel in the HR target kernel 42, the pixel interpolator 420 can interpolate the target pixel based on pixel data of similar or dissimilar characteristic pixels located to the right of the HR target kernel 42 in the horizontal direction.

[0093] In the HR target kernel 42, pixels marked with dashed circles can correspond to defective pixels, and pixels marked with solid circles can correspond to pixels used for interpolation of target pixels. When the pixel interpolator 420 interpolates target pixels with reference to the HR target kernel 42, the pixel data of the target pixels can be calculated using equations such as Equation 3 or Equation 4 below.

[0094] In Equation 3 or Equation 4, the pixel data of the interpolated target pixel can correspond to r22, and the pixel data of the similar characteristic pixel R24 can correspond to r24. Furthermore, the pixel data of the dissimilar characteristic pixels G12, G14, G32, and G34 can correspond to g12, g14, g32, and g34, respectively. Additionally, in the equation, G1 can correspond to the value obtained by dividing the average pixel data of the red pixels included in the HR target kernel 42 by the average pixel data of the green pixels included in the HR target kernel 42. In one embodiment, G1 can be any set value or a preset threshold.

[0095] [Formula 3]

[0096] r22 = r24 + G1 × {(g12 + g32) – (g14 + g34)} / 2

[0097] [Formula 4]

[0098] r22 = r24 × G1 × {(g12 + g32) / (g14 + g34)}

[0099] When the target pixel corresponds to the red filter, the pixel with similar characteristic defects is located below the target pixel, and the pattern determiner 300 determines the target kernel as a vertical pattern, the pixel interpolator 420 can interpolate the target pixel with reference to the VU target kernel 43.

[0100] VU target kernel 43 may include red pixel R22 as a target pixel and corresponds to a vertical pattern. VU target kernel 43 may correspond to a target kernel including red pixel R42 as a pixel with similar characteristic defects. Since the pixel with similar characteristic defects is located below the target pixel in VU target kernel 43, pixel interpolator 420 can interpolate the target pixel based on pixel data of similar or dissimilar characteristic pixels located at the top of VU target kernel 43 in the vertical direction.

[0101] In the VU target kernel 43, pixels marked with dashed circles can correspond to defective pixels, and pixels marked with solid circles can correspond to pixels used for interpolation of target pixels. When the pixel interpolator 420 interpolates target pixels with reference to the VU target kernel 43, the pixel data of the target pixels can be calculated using equations such as Equation 5 or Equation 6 below.

[0102] In Equation 5 or Equation 6, the pixel data of the interpolated target pixel can correspond to r22, and the pixel data of the similar characteristic pixel R02 can correspond to r02. Furthermore, the pixel data of the dissimilar characteristic pixels G01, G03, G21, and G23 can correspond to g01, g03, g21, and g23, respectively. Additionally, in the equation, G1 can correspond to the value obtained by dividing the average pixel data of the red pixels included in the VU target kernel 43 by the average pixel data of the green pixels included in the VU target kernel 43. In one embodiment, G1 can be any set value or a preset threshold.

[0103] [Formula 5]

[0104] r22 = r02 + G1 × {(g21 + g23) – (g01 + g03)} / 2

[0105] [Formula 6]

[0106] r22 = r02 × G1 × {(g21 + g23) / (g01 + g03)}

[0107] When the target pixel corresponds to the red filter, the pixel with similar characteristic defects is located above the target pixel, and the pattern determiner 300 determines the target kernel as a vertical pattern, the pixel interpolator 420 can interpolate the target pixel with reference to the VD target kernel 44.

[0108] VD target kernel 44 may include red pixel R22 as a target pixel and corresponds to a vertical pattern. VD target kernel 44 may correspond to a target kernel including red pixel R02 as a pixel with similar characteristic defects. Since the pixel with similar characteristic defects is located above the target pixel in VD target kernel 44, pixel interpolator 420 can interpolate the target pixel based on pixel data of similar or dissimilar characteristic pixels located at the lower part of VD target kernel 44 in the vertical direction.

[0109] In the VD target kernel 44, pixels marked with dashed circles can correspond to defective pixels, and pixels marked with solid circles can correspond to pixels used for interpolation of target pixels. When the pixel interpolator 420 interpolates target pixels with reference to the VD target kernel 44, the pixel data of the target pixels can be calculated using equations such as Equation 7 or Equation 8 below.

[0110] In Equation 7 or Equation 8, the pixel data of the interpolated target pixel can correspond to r22, and the pixel data of the similar characteristic pixel R42 can correspond to r42. Furthermore, the pixel data of the dissimilar characteristic pixels G21, G23, G41, and G43 can correspond to g21, g23, g41, and g43, respectively. Additionally, in the equation, G1 can correspond to the value obtained by dividing the average pixel data of the red pixels included in the VD target kernel 44 by the average pixel data of the green pixels included in the VD target kernel 44. In one embodiment, G1 can be any set value or a preset threshold.

[0111] [Formula 7]

[0112] r22 = r42 + G1 × {(g21 + g23) – (g41 + g43)} / 2

[0113] [Formula 8]

[0114] r22 = r42 × G1 × {(g21 + g23) / (g41 + g43)}

[0115] When the target pixel corresponds to the red filter, the similar characteristic defect pixel is located at the lower left corner relative to the target pixel, and the pattern determiner 300 determines the target kernel as a diagonal pattern, the pixel interpolator 420 can interpolate the target pixel with reference to the SU target kernel 45.

[0116] The SU target kernel 45 may include red pixel R22 as a target pixel and corresponds to a diagonal pattern. The SU target kernel 45 may also correspond to a target kernel including red pixel R40 as a pixel with similar characteristic defects. Since the pixel with similar characteristic defects is located at the lower left corner of the SU target kernel 45 relative to the target pixel, the pixel interpolator 420 can interpolate the target pixel based on pixel data of similar or dissimilar characteristic pixels located at the upper part of the SU target kernel 45 in the diagonal direction.

[0117] In the SU target kernel 45, pixels marked with dashed circles can correspond to defective pixels, and pixels marked with solid circles can correspond to pixels used for interpolation of target pixels. When the pixel interpolator 420 interpolates target pixels with reference to the SU target kernel 45, the pixel data of the target pixels can be calculated using equations such as Equation 9 or Equation 10 below.

[0118] In Equation 9 or Equation 10, the pixel data of the interpolated target pixel can correspond to r22, and the pixel data of the similar characteristic pixel R04 can correspond to r04. Furthermore, the pixel data of the dissimilar characteristic pixels G12, G23, G03, and G14 can correspond to g12, g23, g03, and g14, respectively. Additionally, in the equation, G1 can correspond to the value obtained by dividing the average pixel data of the red pixels included in the SU target kernel 45 by the average pixel data of the green pixels included in the SU target kernel 45. In one embodiment, G1 can be any set value or a preset threshold.

[0119] [Formula 9]

[0120] r22 = r04 + G1 × {(g12 + g23) – (g03 + g14)} / 2

[0121] [Formula 10]

[0122] r22 = r04 × G1 × {(g12 + g23) / (g03 + g14)}

[0123] When the target pixel corresponds to the red filter, the similar characteristic defect pixel is located at the upper right corner relative to the target pixel, and the pattern determiner 300 determines the target kernel as a diagonal pattern, the pixel interpolator 420 can interpolate the target pixel with reference to the SD target kernel 46.

[0124] The SD target kernel 46 may include red pixel R22 as a target pixel and corresponds to a diagonal pattern. The SD target kernel 46 may also include red pixel R04 as a target kernel for a pixel with similar characteristic defects. Since the pixel with similar characteristic defects is located at the upper right corner of the SD target kernel 46 relative to the target pixel, the pixel interpolator 420 can interpolate the target pixel based on pixel data of similar or dissimilar characteristic pixels located at the lower part of the SD target kernel 46 in the diagonal direction.

[0125] In the SD target kernel 46, pixels marked with dashed circles can correspond to defective pixels, and pixels marked with solid circles can correspond to pixels used for interpolation of target pixels. When the pixel interpolator 420 interpolates target pixels with reference to the SD target kernel 46, the pixel data of the target pixels can be calculated using equations such as Equation 11 or Equation 12 below.

[0126] In Equation 11 or Equation 12, the pixel data of the interpolated target pixel can correspond to r22, and the pixel data of the similar characteristic pixel R40 can correspond to r40. Furthermore, the pixel data of the dissimilar characteristic pixels G21, G32, G30, and G41 can correspond to g21, g32, g30, and g41, respectively. Additionally, in the equation, G1 can correspond to the value obtained by dividing the average pixel data of the red pixels included in the SD target kernel 46 by the average pixel data of the green pixels included in the SD target kernel 46. In one embodiment, G1 can be any set value or a preset threshold.

[0127] [Equation 11]

[0128] r22 = r40 + G1 × {(g21 + g32) – (g30 + g41)} / 2

[0129] [Equation 12]

[0130] r22 = r40 × G1 × {(g21 + g32) / (g30 + g41)}

[0131] When the target pixel corresponds to the red filter, the similar characteristic defect pixel is located at the lower right corner relative to the target pixel, and the pattern determiner 300 determines the target kernel as a reverse diagonal pattern, the pixel interpolator 420 can interpolate the target pixel with reference to the BSU target kernel 47.

[0132] BSU target kernel 47 may include red pixel R22 as a target pixel and corresponds to a pattern in the reverse diagonal direction. BSU target kernel 47 may correspond to a target kernel including red pixel R44 as a pixel with similar characteristic defects. Since the pixel with similar characteristic defects is located at the lower right corner of the BSU target kernel 47 relative to the target pixel, pixel interpolator 420 can interpolate the target pixel based on pixel data of similar or dissimilar characteristic pixels located at the upper part of the BSU target kernel 47 in the reverse diagonal direction.

[0133] In the BSU target core 47, pixels marked with dashed circles can correspond to defective pixels, and pixels marked with solid circles can correspond to pixels used for interpolation of target pixels. When the pixel interpolator 420 interpolates target pixels with reference to the BSU target core 47, the pixel data of the target pixels can be calculated using equations such as Equation 13 or Equation 14 below.

[0134] In Equation 13 or Equation 14, the pixel data of the interpolated target pixel can correspond to r22, and the pixel data of the similar characteristic pixel R00 can correspond to r00. Furthermore, the pixel data of the dissimilar characteristic pixels G01, G10, G12, and G21 can correspond to g01, g10, g12, and g21, respectively. Additionally, in the equation, G1 can correspond to the value obtained by dividing the average pixel data of the red pixels included in the BSU target kernel 47 by the average pixel data of the green pixels included in the BSU target kernel 47. In one embodiment, G1 can be any set value or a preset threshold.

[0135] [Equation 13]

[0136] r22 = r00 + G1 × {(g12 + g21) – (g01 + g10)} / 2

[0137] [Formula 14]

[0138] r22 = r00 × G1 × {(g12 + g21) / (g01 + g10)}

[0139] When the target pixel corresponds to the red filter, the similar characteristic defect pixel is located at the upper left corner relative to the target pixel, and the pattern determiner 300 determines the target kernel as a reverse diagonal pattern, the pixel interpolator 420 can interpolate the target pixel with reference to the BSD target kernel 48.

[0140] BSD target kernel 48 may include red pixel R22 as a target pixel and corresponds to a pattern in the reverse diagonal direction. BSD target kernel 48 may also correspond to a target kernel including red pixel R00 as a pixel with similar characteristic defects. Since the pixel with similar characteristic defects is located at the upper left corner of the BSD target kernel 48 relative to the target pixel, pixel interpolator 420 can interpolate the target pixel based on pixel data of similar or dissimilar characteristic pixels located at the lower part of the BSD target kernel 48 in the reverse diagonal direction.

[0141] In the BSD target kernel 48, pixels marked with dashed circles can correspond to defective pixels, and pixels marked with solid circles can correspond to pixels used for interpolation of target pixels. When the pixel interpolator 420 interpolates target pixels with reference to the BSD target kernel 48, the pixel data of the target pixels can be calculated using equations such as Equation 15 or Equation 16 below.

[0142] In Equation 15 or Equation 16, the pixel data of the interpolated target pixel can correspond to r22, and the pixel data of the similar characteristic pixel R44 can correspond to r44. Furthermore, the pixel data of the dissimilar characteristic pixels G23, G32, G34, and G44 can correspond to g23, g32, g34, and g44, respectively. Additionally, in the equation, G1 can correspond to the value obtained by dividing the average pixel data of the red pixels included in the BSD target kernel 48 by the average pixel data of the green pixels included in the BSD target kernel 48. In one embodiment, G1 can be any set value or a preset threshold.

[0143] [Formula 15]

[0144] r22 = r44 + G1 × {(g23 + g32) – (g34 + g43)} / 2

[0145] [Formula 16]

[0146] r22 = r44 × G1 × {(g23 + g32) / (g34 + g43)}

[0147] Figure 5A This is a diagram showing the target kernel where the pixel corresponding to the green filter is located at the center.

[0148] Reference Figure 1 , Figure 2 and Figure 5AThe target pixel can be located at the center of the target kernel 50. In one embodiment, the target kernel 50 may include a target pixel located at the center, which is a green pixel G22. In this case, pixels with similar characteristics, that is, pixels having the same characteristics as the target pixels included in the target kernel 50, may correspond to green pixels G00, G02, G04, G11, G13, G20, G22, G24, G31, G33, G40, G42, and G44. Red and blue pixels included in the target kernel 50 may correspond to pixels with dissimilar characteristics.

[0149] Therefore, for the target kernel 50, the defective pixel detector 200 can determine whether each of the similar characteristic pixels (i.e., green pixels G00, G02, G04, G11, G13, G20, G22, G24, G31, G33, G40, G42, and G44) is a defective pixel. Pixels determined by the defective pixel detector 200 to be similar characteristic pixels can correspond to similar characteristic defective pixels.

[0150] Furthermore, for the target kernel 50, the pixel interpolator 420 can interpolate the target pixel based on pixel data of at least one of the dissimilar characteristic pixels (i.e., red pixels R01, R03, R21, R23, R41 and R43, and blue pixels B10, B12, B14, B30, B32 and B34).

[0151] Figure 5B This is a diagram illustrating the operation of an image signal processor according to an embodiment of the present disclosure when a pixel located at the center of the target kernel corresponds to a green filter.

[0152] Reference Figure 1 , Figure 2 and Figure 5B When the target pixel corresponds to the green filter, the pixel with similar characteristic defects is located to the right of the target pixel, and the pattern determiner 300 determines the target kernel as a horizontal pattern, the pixel interpolator 420 can interpolate the target pixel with reference to the HL target kernel 51.

[0153] The HL target kernel 51 may include green pixel G22 as a target pixel and corresponds to a horizontal pattern. The HL target kernel 51 may also correspond to a target kernel including green pixel G24 as a pixel with similar characteristic defects. Since the pixel with similar characteristic defects is located to the right of the target pixel in the HL target kernel 51, the pixel interpolator 420 can interpolate the target pixel based on pixel data of similar or dissimilar characteristic pixels located on the left side of the HL target kernel 51 in the horizontal direction.

[0154] In the HL target kernel 51, pixels marked with dashed circles can correspond to defective pixels, and pixels marked with solid circles can correspond to pixels used for interpolation of target pixels. When the pixel interpolator 420 interpolates target pixels with reference to the HL target kernel 51, the pixel data of the target pixels can be calculated using equations such as Equation 17 or Equation 18 below.

[0155] In Equation 17 or Equation 18, the pixel data of the interpolated target pixel can correspond to g22, and the pixel data of the similar characteristic pixel G20 can correspond to g20. Furthermore, the pixel data of the dissimilar characteristic pixels B10, B12, B30, and B32 can correspond to b10, b12, b30, and b32, respectively. Additionally, in the equation, G2 can correspond to the value obtained by dividing the average pixel data of the green pixels included in the HL target kernel 51 by the average pixel data of the blue pixels included in the HL target kernel 51. In one embodiment, G2 can be any set value or a preset threshold.

[0156] [Equation 17]

[0157] g22 = g20 + G2 × {(b12 + b32) – (b10 + b30)} / 2

[0158] [Formula 18]

[0159] g22 = g20 × G2 × {(b12 + b32) / (b10 + b30)}

[0160] When the target pixel corresponds to the green filter, the pixel with similar characteristic defects is located to the left of the target pixel, and the pattern determiner 300 determines the target kernel as a horizontal pattern, the pixel interpolator 420 can interpolate the target pixel with reference to the HR target kernel 52.

[0161] The HR target kernel 52 may include a green pixel G22 as a target pixel and corresponds to a horizontal pattern. The HR target kernel 52 may also correspond to a target kernel that includes a green pixel G20 as a pixel with similar characteristic defects. Since the pixel with similar characteristic defects is located to the left of the target pixel in the HR target kernel 52, the pixel interpolator 420 can interpolate the target pixel based on pixel data of similar or dissimilar characteristic pixels located to the right of the HR target kernel 52 in the horizontal direction.

[0162] In the HR target kernel 52, pixels marked with dashed circles can correspond to defective pixels, and pixels marked with solid circles can correspond to pixels used for interpolation of target pixels. When the pixel interpolator 420 interpolates target pixels with reference to the HR target kernel 52, the pixel data of the target pixels can be calculated using equations such as Equation 19 or Equation 20 below.

[0163] In Equation 19 or Equation 20, the pixel data of the interpolated target pixel can correspond to g22, and the pixel data of the similar characteristic pixel G24 can correspond to g24. Furthermore, the pixel data of dissimilar characteristic pixels B12, B14, B32, and B34 can correspond to b12, b14, b32, and b34, respectively. Additionally, in the equation, G2 can correspond to the value obtained by dividing the average pixel data of the green pixels included in the HR target kernel 52 by the average pixel data of the blue pixels included in the HR target kernel 52. In one embodiment, G2 can be any set value or a preset threshold.

[0164] [Formula 19]

[0165] g22 = g24 + G2 × {(b12 + b32) – (b14 + b34)} / 2

[0166] [Formula 20]

[0167] g22 = g24 × G2 × {(b12 + b32) / (b14 + b34)}

[0168] When the target pixel corresponds to the green filter, the pixel with similar characteristic defects is located below the target pixel, and the pattern determiner 300 determines the target kernel as a vertical pattern, the pixel interpolator 420 can interpolate the target pixel with reference to the VU target kernel 53.

[0169] VU target kernel 53 may include green pixel G22 as target pixel and corresponds to a vertical pattern. VU target kernel 53 may also correspond to a target kernel including green pixel G42 as a pixel with similar characteristic defects. Since the pixel with similar characteristic defects is located below the target pixel in VU target kernel 53, pixel interpolator 420 can interpolate the target pixel based on pixel data of similar or dissimilar characteristic pixels located at the top of VU target kernel 53 in the vertical direction.

[0170] In the VU target kernel 53, pixels marked with dashed circles can correspond to defective pixels, and pixels marked with solid circles can correspond to pixels used for interpolation of target pixels. When the pixel interpolator 420 interpolates target pixels with reference to the VU target kernel 53, the pixel data of the target pixels can be calculated using equations such as Equation 21 or Equation 22 below.

[0171] In Equation 21 or Equation 22, the pixel data of the interpolated target pixel can correspond to g22, and the pixel data of the similar characteristic pixel G02 can correspond to g02. Furthermore, the pixel data of the dissimilar characteristic pixels R01, R03, R21, and R23 can correspond to r01, r03, r21, and r23, respectively. Additionally, in the equation, G3 can correspond to the value obtained by dividing the average pixel data of the green pixels included in the VU target kernel 53 by the average pixel data of the red pixels included in the VU target kernel 53. In one embodiment, G3 can be any set value or a preset threshold.

[0172] [Equation 21]

[0173] g22 = g02 + G3 × {(r21 + r23) – (r01 + r03)} / 2

[0174] [Equation 22]

[0175] g22 = g02 × G3 × {(r21 + r23) / (r01 + r03)}

[0176] When the target pixel corresponds to the green filter, the pixel with similar characteristic defects is located above the target pixel, and the pattern determiner 300 determines the target kernel as a vertical pattern, the pixel interpolator 420 can interpolate the target pixel with reference to the VD target kernel 54.

[0177] VD target kernel 54 may include green pixel G22 as a target pixel and corresponds to a vertical pattern. VD target kernel 54 may also correspond to a target kernel including green pixel G02 as a pixel with similar characteristic defects. Since the pixel with similar characteristic defects is located above the target pixel in VD target kernel 54, pixel interpolator 420 can interpolate the target pixel based on pixel data of similar or dissimilar characteristic pixels located at the lower part of VD target kernel 54 in the vertical direction.

[0178] In the VD target kernel 54, pixels marked with dashed circles can correspond to defective pixels, and pixels marked with solid circles can correspond to pixels used for interpolation of target pixels. When the pixel interpolator 420 interpolates target pixels with reference to the VD target kernel 54, the pixel data of the target pixels can be calculated using equations such as Equation 23 or Equation 24 below.

[0179] In Equation 23 or Equation 24, the pixel data of the interpolated target pixel can correspond to g22, and the pixel data of the similar characteristic pixel G42 can correspond to g42. Furthermore, the pixel data of the dissimilar characteristic pixels R21, R23, R41, and R43 can correspond to r21, r23, r41, and r43, respectively. Additionally, in the equation, G3 can correspond to the value obtained by dividing the average pixel data of the green pixels included in the VD target kernel 54 by the average pixel data of the red pixels included in the VD target kernel 54. In one embodiment, G3 can be any set value or a preset threshold.

[0180] [Equation 23]

[0181] g22 = g42 + G3 × {(r21 + r23) – (r41 + r43)} / 2

[0182] [Equation 24]

[0183] g22 = g42 × G3 × {(r21 + r23) / (r41 + r43)}

[0184] When the target pixel corresponds to the green filter, the similar characteristic defect pixel is located at the lower left corner relative to the target pixel, and the pattern determiner 300 determines the target kernel as a diagonal pattern, the pixel interpolator 420 can interpolate the target pixel with reference to the SU target kernel 55.

[0185] The SU target kernel 55 may include green pixel G22 as a target pixel and corresponds to a diagonal pattern. The SU target kernel 55 may also correspond to a target kernel including green pixel G31 as a pixel with similar characteristic defects. The pixel interpolator 420 may interpolate the target pixel based on pixel data of similar characteristic pixels located on the line along the diagonal direction within the SU target kernel 55.

[0186] In the SU target kernel 55, pixels marked with dashed circles can correspond to defective pixels, and pixels marked with solid circles can correspond to pixels used for interpolation of target pixels. When the pixel interpolator 420 interpolates target pixels with reference to the SU target kernel 55, the pixel data of the target pixels can be calculated using equations such as Equation 25 below.

[0187] In Equation 25, the pixel data of the interpolation target pixel can correspond to g22, and the pixel data of similar characteristic pixels G04, G13 and G40 can correspond to g04, g13 and g40 respectively.

[0188] [Equation 25]

[0189] g22 = g13 + (g40 – g04) / 2

[0190] In one implementation, when the target pixel corresponds to a green filter, a pixel with similar characteristic defects is located at the lower left corner of the target pixel, and the pattern determiner 300 determines the target kernel as a diagonal pattern, the pixel interpolator 420 can interpolate the pixel data of the target pixel based on the average of the pixel data of the green pixel G04 and the pixel data of the green pixel G40.

[0191] When the target pixel corresponds to the green filter, the pixel with similar characteristic defects is located at the upper left corner of the target pixel, and the pattern determiner 300 determines the target kernel as a diagonal pattern, the pixel interpolator 420 can interpolate the target pixel with reference to the SD target kernel 56.

[0192] The SD target kernel 56 may include green pixel G22 as a target pixel and corresponds to a diagonal pattern. The SD target kernel 56 may also correspond to a target kernel including green pixel G13 as a pixel with similar characteristic defects. The pixel interpolator 420 may interpolate the target pixel based on pixel data of similar characteristic pixels located on the line along the diagonal direction within the SD target kernel 56.

[0193] In the SD target kernel 56, pixels marked with dashed circles can correspond to defective pixels, and pixels marked with solid circles can correspond to pixels used for interpolation of target pixels. When the pixel interpolator 420 interpolates target pixels with reference to the SD target kernel 56, the pixel data of the target pixels can be calculated using equations such as Equation 26 below.

[0194] In Equation 26, the pixel data of the interpolation target pixel can correspond to g22, and the pixel data of similar characteristic pixels G04, G31 and G40 can correspond to g04, g31 and g40 respectively.

[0195] [Equation 26]

[0196] g22 = g31 + (g04 – g40) / 2

[0197] In one implementation, when the target pixel corresponds to a green filter, a pixel with similar characteristic defects is located at the upper right corner of the target pixel, and the pattern determiner 300 determines the target kernel as a diagonal pattern, the pixel interpolator 420 can interpolate the pixel data of the target pixel based on the average of the pixel data of the green pixel G04 and the pixel data of the green pixel G40.

[0198] When the target pixel corresponds to the green filter, the pixel with similar characteristic defects is located at the lower right corner of the target pixel, and the pattern determiner 300 determines the target kernel as a reverse diagonal pattern, the pixel interpolator 420 can interpolate the target pixel with reference to the BSU target kernel 57.

[0199] BSU target core 57 may include green pixel G22 as a target pixel and corresponds to a reverse diagonal pattern. BSU target core 57 may correspond to a target core including green pixel G33 as a pixel with similar characteristic defects. Pixel interpolator 420 may interpolate the target pixel based on pixel data of similar characteristic pixels located on the line along the diagonal direction within BSU target core 57.

[0200] In the BSU target core 57, pixels marked with dashed circles can correspond to defective pixels, and pixels marked with solid circles can correspond to pixels used for interpolation of target pixels. When the pixel interpolator 420 interpolates target pixels with reference to the BSU target core 57, the pixel data of the target pixels can be calculated using equations such as Equation 27 below.

[0201] In Equation 27, the pixel data of the interpolation target pixel can correspond to g22, and the pixel data of similar characteristic pixels G00, G11 and G44 correspond to g00, g11 and g44 respectively.

[0202] [Equation 27]

[0203] g22 = g11 + (g44 – g00) / 2

[0204] In one implementation, when the target pixel corresponds to a green filter, a pixel with similar characteristic defects is located at the lower right corner of the target pixel, and the pattern determiner 300 determines the target kernel as a reverse diagonal pattern, the pixel interpolator 420 can interpolate the pixel data of the target pixel based on the average of the pixel data of the green pixel G00 and the pixel data of the green pixel G44.

[0205] When the target pixel corresponds to the green filter, the pixel with similar characteristic defects is located at the upper left corner of the target pixel, and the pattern determiner 300 determines the target kernel as a reverse diagonal pattern, the pixel interpolator 420 can interpolate the target pixel with reference to the BSD target kernel 58.

[0206] BSD target kernel 58 may include green pixel G22 as a target pixel and corresponds to a reverse diagonal pattern. BSD target kernel 58 may correspond to a target kernel including green pixel G11 as a pixel with similar characteristic defects. Pixel interpolator 420 may interpolate the target pixel based on pixel data of similar characteristic pixels located on the line along the reverse diagonal direction of BSD target kernel 58.

[0207] In the BSD target kernel 58, pixels marked with dashed circles can correspond to defective pixels, and pixels marked with solid circles can correspond to pixels used for interpolation of target pixels. When the pixel interpolator 420 interpolates target pixels with reference to the BSD target kernel 58, the pixel data of the target pixels can be calculated using equations such as Equation 28 below.

[0208] In Equation 28, the pixel data of the interpolation target pixel can correspond to g22, and the pixel data of similar characteristic pixels G00, G33 and G44 can correspond to g00, g33 and g44 respectively.

[0209] [Equation 28]

[0210] g22 = g33 + (g00 – g44) / 2

[0211] In one implementation, when the target pixel corresponds to a green filter, a pixel with similar characteristic defects is located at the upper left corner of the target pixel, and the pattern determiner 300 determines that the target kernel is a reverse diagonal pattern, the pixel interpolator 420 can interpolate the pixel data of the target pixel based on the average of the pixel data of the green pixel G00 and the pixel data of the green pixel G44.

[0212] Figure 6A This is a diagram showing the target kernel where the pixel corresponding to the blue filter is located at the center.

[0213] Reference Figure 1 , Figure 2 and Figure 6A The target pixel can be located at the center of the target core 60. In one embodiment, the target core 60 may include a target pixel located at the center, such as blue pixel B22. In this case, pixels with similar characteristics, i.e., pixels having the same characteristics as the target pixels included in the target core 60, may correspond to blue pixels B00, B02, B04, B20, B24, B40, B42, and B44. Red and green pixels included in the target core 60 may correspond to pixels with dissimilar characteristics.

[0214] Therefore, for the target kernel 60, the defective pixel detector 200 can determine whether each of the similar characteristic pixels (i.e., blue pixels B00, B02, B04, B20, B24, B40, B42, and B44) is a defective pixel. Pixels determined by the defective pixel detector 200 to be similar characteristic pixels can correspond to similar characteristic defective pixels.

[0215] Furthermore, for the target kernel 60, the pixel interpolator 420 can interpolate the target pixel based on pixel data of at least one of the dissimilar characteristic pixels (i.e., green pixels G01, G03, G10, G12, G14, G21, G23, G30, G32, G34, G41 and G43, and red pixels R11, R13, R31 and R33).

[0216] Figure 6B This is a diagram illustrating the operation of an image signal processor according to an embodiment of the present disclosure when a pixel located at the center of the target kernel corresponds to a blue color filter.

[0217] Reference Figure 2 , Figure 4B and Figure 6B When the target pixel corresponds to the blue filter, it can be compared with... Figure 4B The case where the target pixel corresponds to the red filter can be understood similarly. For example, Figure 6B The pixel interpolator 420, referring to HL target kernel 61, HR target kernel 62, VU target kernel 63, VD target kernel 64, SU target kernel 65, SD target kernel 66, BSU target kernel 67, or BSD target kernel 68, performs interpolation operations on target pixels in a manner similar to those described above. Figure 4B To understand this, we need to describe the operation of the pixel interpolator 420. When... Figure 6B The blue pixels in the target kernel shown match the red pixels, and Figure 6B When the red pixel of the target kernel shown matches the red pixel, Figure 6B The target kernel shown can be similar to Figure 4B The target kernel shown is used for understanding.

[0218] also, Figure 6B The pixel interpolator 420 can be referenced in the above text. Figure 4B Equations 1 through 16 describe interpolation of the target pixels in a similar manner. Therefore, redundant descriptions are omitted.

[0219] Figure 7 This illustrates an embodiment according to the present disclosure. Figure 1 A block diagram of the computing device corresponding to the image signal processor.

[0220] Reference Figure 7 The computing device 700 can represent a device for performing... Figure 1 An implementation of the hardware configuration for the operation of the image signal processor 100.

[0221] The computing device 700 can be mounted on a separate chip from the chip on which the image sensing device is mounted. According to one embodiment, the chip on which the image sensing device is mounted and the chip on which the computing device 700 is mounted can be implemented in a single package (e.g., a multi-chip package (MCP)), but the scope of this disclosure is not limited thereto.

[0222] Furthermore, the internal configuration or arrangement of the computing device 700 and the image sensing device can vary depending on the implementation. For example, at least a portion of the image sensing device may be included in the computing device 700. Alternatively, at least a portion of the computing device 700 may be included in the image sensing device. In this case, at least a portion of the computing device 700 may be mounted together on a chip on which the image sensing device is mounted.

[0223] The computing device 700 may include a processor 710, a memory 720, an input / output (I / O) interface 730, and a communication interface 740.

[0224] Processor 710 can process the execution Figure 1 The data and / or instructions required for the operation of the components of the image signal processor 100 described herein. That is, processor 710 may refer to image signal processor 100, but the scope of this disclosure is not limited thereto.

[0225] The memory 720 may store data and / or instructions required for performing operations of components 200 and 300 of the image signal processor 100, and may be accessed by the processor 710. For example, the memory 720 may be volatile memory (e.g., dynamic random access memory (DRAM), static random access memory (SRAM), etc.) or non-volatile memory (e.g., programmable read-only memory (PROM), erasable PROM (EPROM), etc.), EEPROM (electrically erasable PROM), flash memory, etc.).

[0226] That is, a computer program for performing the operations of the image signal processor 100 described in this disclosure can be recorded in the memory 720 and executed and processed by the processor 710 to realize the operation of the image signal processor 100.

[0227] Input / output (I / O) interface 730 is an interface that connects external input devices (e.g., keyboard, mouse, touch panel, etc.) and / or external output devices (e.g., display) to processor 710 to allow sending and receiving data.

[0228] The communication interface 740 is a component capable of sending various data to and receiving various data from external devices (e.g., application processors, external memory, etc.), and can be a device that supports wired or wireless communication.

[0229] As can be clearly seen from the above description, according to the embodiments disclosed in this disclosure, even when the target pixel and the pixels surrounding the target pixel are defective pixels, the correction accuracy of the target pixel can be improved by providing a standard for selecting interpolation pixels for interpolating the pixel data of the target pixel.

[0230] The embodiments disclosed herein can provide various beneficial effects that can be directly or indirectly recognized by those skilled in the art.

[0231] Those skilled in the art will understand that embodiments of this disclosure can be implemented in other specific ways than those described herein. Furthermore, claims not expressly stated in the appended claims may be proposed as combinations of embodiments, or may be included as new claims through subsequent amendments after the filing of this application.

[0232] While several illustrative embodiments have been described, it should be understood that modifications and enhancements to the disclosed embodiments and other embodiments can be designed based on the description and / or illustrations in this disclosure. Furthermore, embodiments can be combined to form additional embodiments.

[0233] Cross-reference to related applications

[0234] This application claims priority and benefit to Korean Patent Application No. 10-2024-0178904, filed on December 4, 2024, the disclosure of which is incorporated herein by reference in its entirety.

Claims

1. An image signal processor, the image signal processor comprising: A defective pixel detector that detects defective pixels from pixels with similar characteristics, wherein the pixels with similar characteristics are pixels that have the same characteristics as the target pixels included in the target kernel; A pattern determiner, when the defective pixel exists, determines the pattern of the target kernel based on the pixel data of the pixels included in the target kernel; as well as A pixel interpolator, when the defective pixel is located in the texture region of the pattern, interpolates the target pixel based on the pixel data of pixels with dissimilar characteristics located in the texture region and the pixel data of pixels with similar characteristics located in the texture region, wherein the dissimilar characteristic pixels represent pixels having characteristics different from those of the target pixel.

2. The image signal processor according to claim 1, wherein, The feature includes at least one of a color filter or a channel.

3. The image signal processor according to claim 1, wherein, When the pixel data difference between the pixel with the largest pixel data and the pixel with the second largest pixel data among the similar characteristic pixels is greater than or equal to a threshold, the defective pixel detector determines the pixel with the largest pixel data as the defective pixel.

4. The image signal processor according to claim 1, wherein, When the pixel data difference between the pixel with the smallest pixel data and the pixel with the second smallest pixel data among the similar characteristic pixels is greater than or equal to a threshold, the defective pixel detector determines the pixel with the smallest pixel data as the defective pixel.

5. The image signal processor according to claim 1, wherein, The pattern determiner determines the pattern based on at least one of a first comparison data obtained by comparing pixel data of pixels located in the texture region, or a second comparison data obtained by comparing pixel data of pixels located in the texture region and pixel data of pixels located outside the texture region.

6. The image signal processor according to claim 5, wherein, The pattern includes at least one of the following: a horizontal pattern, a vertical pattern, a diagonal pattern, or a reverse diagonal pattern.

7. The image signal processor according to claim 1, wherein, The pixel interpolator interpolates the target pixel based on the pixel data of two pixels among the dissimilar characteristic pixels, wherein the two pixels are a pixel adjacent to a pixel symmetrically positioned relative to the target pixel and the defective pixel, and a pixel adjacent to the target pixel.

8. The image signal processor according to claim 7, wherein, The pixel interpolator interpolates the target pixel based on a value obtained by subtracting the pixel data of the pixel adjacent to the target pixel from the pixel data of the pixel adjacent to the target pixel from the pixel data of the pixel adjacent to the target pixel, and a value obtained by adding the pixel data of the pixel symmetrically positioned to the defective pixel and the pixel data of the pixel adjacent to the target pixel.

9. The image signal processor according to claim 7, wherein, The pixel interpolator interpolates the target pixel based on a value obtained by dividing the pixel data of the pixel adjacent to the pixel positioned symmetrical to the defective pixel by the pixel data of the pixel adjacent to the target pixel, and a value obtained by multiplying the pixel data of the pixel positioned symmetrical to the defective pixel by the pixel data of the pixel adjacent to the target pixel.

10. The image signal processor according to claim 1, wherein, The pixel interpolator interpolates the target pixel based on pixel data of pixels located centered on the target pixel along a direction corresponding to the pattern.

11. The image signal processor according to claim 1, wherein, The target core has a monochrome pattern, a four-in-one pattern, a nine-in-one pattern, a QxQ pattern, or an RGBW pattern.

12. An image signal processor, the image signal processor comprising: A pattern determiner that determines the pattern of the target kernel based on pixel data including pixels in the target kernel of the target pixel; A pixel interpolator that, when a pixel with a similar characteristic defect having the same characteristic as the target pixel is located in a texture region of the pattern, generates dissimilar characteristic pixel data including at least one of gradients or ratios between dissimilar characteristic pixels located in the texture region, the dissimilar characteristic pixels representing pixels having characteristics different from those of the target pixel; as well as A pixel interpolator that interpolates the target pixel based on the dissimilar pixel data.

13. The image signal processor according to claim 12, wherein, The feature includes at least one of a color filter or a channel.

14. The image signal processor according to claim 12, wherein, The pixel interpolator interpolates the target pixel based on pixel data of pixels adjacent to pixels symmetrically positioned relative to the target pixel and pixels with similar characteristic defects, and pixel data of pixels adjacent to the target pixel.

15. The image signal processor according to claim 14, wherein, The pixel interpolator interpolates the target pixel based on a value obtained by subtracting the pixel data of the pixel adjacent to the target pixel from the pixel data of the pixel adjacent to the target pixel from the pixel data of the pixel adjacent to the target pixel, and a value obtained by adding the pixel data of the pixel symmetrically positioned to the pixel of the similar defect and the pixel data of the pixel adjacent to the target pixel.

16. The image signal processor according to claim 14, wherein, The pixel interpolator interpolates the target pixel based on a value obtained by dividing the pixel data of the pixel adjacent to the pixel located symmetrical to the pixel with the same characteristic defect by the pixel data of the pixel adjacent to the target pixel, and a value obtained by multiplying the pixel data of the pixel located symmetrical to the pixel with the same characteristic defect by the pixel data of the pixel adjacent to the target pixel.

17. A method for processing an image signal, the method comprising the following steps: Defective pixels are determined from similar characteristic pixels, wherein the similar characteristic pixels are pixels that have the same characteristics as the target pixels included in the target kernel; When the defective pixel exists, the pattern of the target kernel is determined based on the pixel data of the pixels included in the target kernel; as well as When the defective pixel is located in the texture region of the pattern, the target pixel is interpolated based on the pixel data of the pixels in the texture region among the dissimilar characteristic pixels and the pixel data of the pixels in the texture region among the similar characteristic pixels. The dissimilar characteristic pixels represent pixels with characteristics different from those of the target pixel.

18. The method according to claim 17, wherein, The steps for determining the defective pixels include the following: When the pixel data difference between the pixel with the largest pixel data and the pixel with the second largest pixel data among the similar characteristic pixels is greater than or equal to a threshold, the pixel with the largest pixel data is determined as the defective pixel.

19. The method of claim 17, wherein, The steps for determining the pattern include the following: The pattern is determined based on at least one of a first comparison data obtained by comparing pixel data of pixels located in the texture region of the pattern, or a second comparison data obtained by comparing pixel data of pixels located in the texture region and pixel data of pixels located outside the texture region.

20. The method of claim 17, wherein, The interpolation step for the target pixel includes the following steps: The target pixel is interpolated based on the pixel data of two pixels among the dissimilar characteristic pixels, wherein the two pixels are a pixel adjacent to a pixel symmetrically positioned relative to the target pixel and the defective pixel, and a pixel adjacent to the target pixel.

Images