Image signal processor and image signal processing method

Abstract

To increase accuracy of correction for defective pixels and the like.SOLUTION: An image signal processor 100 according to one embodiment of the present invention includes: a directional component extraction unit 121 that extracts directional components of a target kernel by performing a convolution operation between the target kernel including a target pixel 430 and each of a plurality of directional kernels; an interpolation kernel determination unit 122 that determines an interpolation kernel based on the directional components; and a pixel interpolation unit 123 that interpolates the target pixel using data included in the interpolation kernel.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to an image signal processor and a method of processing an image signal capable of performing image transformations. [Background technology]

[0002] Image sensing devices are devices that capture optical images using the properties of light-sensitive semiconductor materials that react to light. With the development of industries such as automobiles, medicine, computers, and communications, there is an increasing demand for high-performance image sensing devices in various fields such as smartphones, digital cameras, game consoles, the Internet of Things, robots, security cameras, and medical micro cameras.

[0003] In an original image captured by an image sensing device, pixels corresponding to different colors (e.g., red, green, blue, white) are usually arranged according to a certain color pattern (e.g., RGBW pattern), and in order to convert the original image into a complete image (e.g., RGBW image), an interpolation operation is performed according to a predetermined algorithm. Such an algorithm basically includes an operation of interpolating pixels with missing information using information of surrounding pixels. Summary of the Invention [Problem to be solved by the invention]

[0004] The technical idea of ​​the present invention is to provide an image signal processor and an image signal processing method capable of improving the accuracy of correction for defective pixels and the like.

[0005] The technical problems of the present invention are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description. [Means for solving the problem]

[0006] An image signal processor according to an embodiment of the present invention disclosed in this document may include a directional component extraction unit that extracts directional components of the target kernel by convolution of a target kernel including a target pixel with each of a plurality of directional kernels, an interpolation kernel determination unit that determines an interpolation kernel based on the directional components, and a pixel interpolation unit that interpolates the target pixel using data included in the interpolation kernel.

[0007] According to another embodiment of the present invention, an image signal processor may include a directional component extraction unit that extracts directional components of the target kernel by performing a first convolution operation on a target kernel including a target pixel and each of a first plurality of kernels, an interpolation kernel determination unit that determines an interpolation kernel based on a value obtained by performing a second convolution operation on the directional components and each of a second plurality of kernels, and a pixel interpolation unit that interpolates the target pixel using data included in the interpolation kernels.

[0008] An image signal processing method according to an embodiment of the present invention may include the steps of: extracting a first directional component of the first color kernel by convolving a first color kernel including a first target pixel with each of a plurality of directional kernels; determining a first interpolation kernel based on the first directional component; and interpolating the first target pixel using data included in the first interpolation kernel. Effect of the Invention

[0009] According to an embodiment disclosed herein, by analyzing directional components contained in a target kernel and interpolating a target pixel based on an interpolation kernel corresponding to the target kernel, it is possible to improve accuracy in correcting defective pixels even when the target kernel contains multiple directional components. In addition, various other benefits may be provided by this document, either directly or indirectly. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram illustrating an image signal processor according to one embodiment of the present disclosure. [Diagram 2] 2 is a block diagram illustrating a defective pixel correction unit shown in FIG. 1. [Diagram 3] 3 is a flowchart for explaining the operation of the defective pixel correction unit shown in FIG. 2. [Figure 4a] FIG. 2 is an exemplary diagram illustrating multiple directional kernels. [Figure 4b] FIG. 4 is a diagram for explaining an example of how step S100 in FIG. 3 is performed. [Figure 5a] 1 is a formula showing an example of a number of cost kernels; [Figure 5b] FIG. 2 is an exemplary diagram illustrating multiple cost kernels. [Figure 5c] FIG. 4 is a diagram for explaining an example of how step S120 of FIG. 3 is performed. [Figure 6a] FIG. 2 is an exemplary diagram illustrating multiple interpolation kernels. [Figure 6b] FIG. 4 is a diagram for explaining an example of how step S150 in FIG. 3 is performed. [Figure 7] 1 is a formula showing an example of the overall process of interpolating one pixel. [Figure 8] FIG. 2 is a diagram illustrating an example of the overall process of interpolating one pixel. [Figure 9] 11 is a flowchart illustrating an operation of classifying kernels by color and interpolating a target pixel. [Figure 10] 2 is a block diagram of an example of a computing device corresponding to the image signal processor of FIG. 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Various embodiments will be described below with reference to the accompanying drawings. However, it should be understood that the present disclosure is not limited to specific embodiments, and includes various modifications, equivalents, and / or alternatives of the embodiments. The embodiments of the present disclosure can provide various effects that are directly or indirectly recognizable by the present disclosure.

[0012] FIG. 1 is a block diagram illustrating an image signal processor according to one embodiment of the present disclosure. Referring to FIG. 1, an image signal processor (ISP) 100 may perform at least one image signal processing on image data (IDATA) to generate processed image data (IDATA_P). The ISP 100 may perform image signal processing for reducing noise on the image data (IDATA) and improving image quality, such as demosaicing, defect pixel correction, gamma correction, color filter array interpolation, color matrix, color correction, color enhancement, and lens distortion correction. The ISP 100 may also compress image data that has been subjected to image signal processing for improving image quality to generate an image file, or may restore image data from the image file. The image compression format may be a lossless format or a lossy format. As an example of the compression format, in the case of a still image, a JPEG (Joint Photographic Experts Group) format or a JPEG2000 format may be used. Additionally, for video, multiple frames can be compressed according to the Moving Picture Experts Group (MPEG) standard to generate a video file.

[0013] The image data (IDATA) may be generated by an image sensing device that captures an optical image of a scene, but the scope of the present invention is not limited thereto. The image sensing device may include a pixel array including a plurality of pixels for sensing light incident from a scene, a control circuit for controlling the pixel array, and a readout circuit for converting analog pixel signals received from the pixel array into digital image data (IDATA) and outputting the converted signal. In the present disclosure, it is assumed that the image data (IDATA) is generated by the image sensing device.

[0014] A pixel array of an image sensing device may include defective pixels that are unable to capture a color image normally due to process limitations or the inflow of temporary noise. The pixel array may also include phase difference detection pixels that obtain phase difference related information to realize an autofocusing function. The phase difference detection pixels are unable to obtain a color image like defective pixels, and can be treated as defective pixels from the viewpoint of color imaging. In this disclosure, defective pixels and phase difference detection pixels that are unable to obtain a color image normally are commonly referred to as "defective pixels."

[0015] In order to improve the quality of a color image, it is essential to improve the accuracy of correction for defective pixels. To this end, the ISP 100 according to one embodiment of the present invention may include a defective pixel detector 110 and a defective pixel corrector 120.

[0016] The defective pixel detector 110 can detect pixel data of defective pixels from the image data (IDATA). For convenience of explanation, in this disclosure, digital data corresponding to a pixel signal of each pixel is defined as pixel data, and a set of pixel data corresponding to a certain unit (e.g., a frame or a kernel) is defined as image data (IDATA). Here, a frame corresponds to the entire pixel array, and a kernel may mean a unit for image signal processing.

[0017] According to an embodiment, the defective pixel detection unit 110 may detect pixel data of a defective pixel based on image data (IDATA). Exemplarily, the defective pixel detection unit 110 may determine whether the target pixel is a defective pixel based on a difference between the pixel data of the target pixel and an average value of pixel data in a kernel. That is, the defective pixel detection unit 110 may determine that the target pixel is a defective pixel that does not have normal pixel data when a difference between the pixel data of the target pixel and the average value of pixel data in a kernel is equal to or greater than a predetermined threshold. The predetermined threshold may be a fixed constant or a specific percentage of a brightness value (e.g., green average value) of the current kernel.

[0018] According to another embodiment, the defective pixel detector 110 may receive pre-stored defective pixel position information from an image sensing device that generates image data (IDATA) and determine whether a target pixel is defective based on the defective pixel position information. The image sensing device may store fixed defective pixel position information in an internal storage (e.g., a one time programmable (OTP) memory) for process reasons, and may provide the defective pixel position information to the ISP 100.

[0019] If the target pixel is determined to be a defective pixel by the defective pixel detector 110, the defective pixel corrector 120 can correct pixel data of the target pixel based on image data of a kernel including the target pixel.

[0020] Referring to FIG. 2, the defective pixel correction unit 120 may include a directional component extraction unit 121, an interpolation kernel determination unit 122, and a pixel interpolation unit 123.

[0021] Image data (IDATA) corresponding to one frame may include directional components of various sizes and shapes. The directional component refers to a set of pixels having similarity among the surrounding pixels of the target pixel centered on the target pixel, which is quantified in size and direction. For example, an object having a uniform color included in a scene may be recognized as a directional component. For example, if pixels having a uniform color among the surrounding pixels of the target pixel are arranged horizontally and their boundaries are clear, the directional component may be understood to have a horizontal direction and a high directional value. The directional component may include a horizontal direction, a vertical direction, or a diagonal direction, and may not include the target pixel. For example, the directional component may be located within a target kernel including the target pixel and may include a stripe pattern, a corner pattern, etc. that does not include the target pixel.

[0022] The directional component extraction unit 121 may extract a directional component corresponding to a target pixel included in the target kernel. For example, the directional component extraction unit 121 may extract a directional component corresponding to a target pixel included in the target kernel by performing a convolution operation on the target kernel and each of a plurality of directional kernels.

[0023] The directional component corresponding to the target pixel included in the target kernel may reflect the characteristics of various directional kernels. For example, the diagonal directional component may be extracted by convolving the target kernel with a diagonal directional kernel. Furthermore, various directional components may be further extracted by combining the directional kernels. For example, if the target kernel includes both diagonal and horizontal components, the diagonal and horizontal kernels may be selectively combined and multiplied by a preset weight to extract a cost of a new directional component reflecting the diagonal and horizontal components.

[0024] The interpolation kernel determination unit 122 can extract a cost due to a combination of directional components based on the extracted directional components. As an example, the interpolation kernel determination unit 122 can extract a cost due to a combination of directional components by performing a convolution operation of the directional components and each of a plurality of cost kernels.

[0025] The interpolation kernel determination unit 122 may determine an interpolation kernel corresponding to a target pixel included in the target kernel based on the extracted cost. For example, the interpolation kernel determination unit 122 may determine an interpolation kernel corresponding to a cost kernel from which a maximum or minimum value of the extracted costs is derived as an interpolation kernel corresponding to the target pixel.

[0026] The pixel interpolation unit 123 may interpolate pixel data of a target pixel included in the target kernel based on the interpolation kernel determined by the interpolation kernel determination unit 122. As an example, the pixel interpolation unit 123 may associate data located at the center of the interpolation kernel determined by the interpolation kernel determination unit 122 with the target pixel, and determine a value obtained by convolving the target kernel and the interpolation kernel as pixel data of the target pixel.

[0027] FIG. 3 is a flowchart for explaining the operation of the defective pixel correction unit shown in FIG. Referring to FIG. 3, the defective pixel correction unit 120 may include a step (S100) of extracting a directional component for a target kernel including a target pixel (DP), a step (S140) of determining an interpolation kernel based on the extracted directional component and a plurality of cost kernels, and a step (S150) of interpolating the target pixel (DP) using data included in the determined interpolation kernel.

[0028] 3 to 9, a method for correcting a defective pixel will be described by taking as an example a kernel in which red, green, blue, and white pixels are arranged in an RGBW pattern. It is also assumed that the target pixel is located at the center of the target kernel. For example, if the target kernel includes a first pixel (not shown) and a 25th pixel (not shown) and is a kernel having a size of 5×5 arranged in 5 columns starting from the first pixel (not shown), the target pixel may correspond to a pixel located at the 13th pixel (not shown). The red pixel may sense red light to generate red pixel data, the green pixel may sense green light to generate green pixel data, the blue pixel may sense blue light to generate blue pixel data, and the white pixel may sense white light to generate white pixel data. It is also assumed that the red pixel data, the green pixel data, the blue pixel data, and the white pixel data each have a range of 0 to 1023.

[0029] Although a kernel arranged in an RGBW pattern will be described below as an example, the technical idea of ​​the present invention can also be applied to kernels in which color pixels are arranged in other ways, such as a Bayer pattern, a quad Bayer pattern, a nona Bayer pattern, a hexa Bayer pattern, a mono pattern, etc. Also, kernels of different sizes (e.g., 3×3, 5×5, 10×10) may be used depending on the performance of the ISP 100, the required correction accuracy, the arrangement of color pixels, etc.

[0030] The directional component extraction unit 121 may extract directional components of the target kernel by performing a convolution operation on the target kernel and each of a plurality of directional kernels (S100). For example, the directional component extraction unit 121 may perform a convolution operation on the target kernel and a first directional kernel to extract a first directional component, perform a convolution operation on the target kernel and a second directional kernel to extract a second directional component, and perform a convolution operation on the target kernel and a third directional kernel to extract a third directional component.

[0031] The interpolation kernel determination unit 122 may multiply each cost kernel by a weight reflecting the sensor characteristics (S110), and may extract costs by performing a point-wise convolution operation on each of the direction components extracted in step S100 and the multiple cost kernels multiplied by the weights in step S110 (S120). For example, the interpolation kernel determination unit 122 may multiply the first cost kernel by a first weight, and multiply the second cost kernel by a second weight. For example, the interpolation kernel determination unit 122 may perform a point-wise convolution operation on the first cost kernel multiplied by the first to third direction components and the first weight to extract a first cost, and may perform a point-wise convolution operation on the second cost kernel multiplied by the first to third direction components and the second weight to extract a second cost.

[0032] The interpolation kernel determination unit 122 can extract a cost index number by applying an Arg max or Arg min function to the cost extracted in step S120 (S130), and can determine an interpolation kernel corresponding to the index number extracted in step S130 (S140). For example, the interpolation kernel determination unit 122 can extract index number 1 corresponding to the first cost by applying an Arg max or Arg min function to the first cost and the second cost, and can determine a first interpolation kernel corresponding to index number 1.

[0033] The pixel interpolation unit 123 may interpolate the target pixel using data included in the interpolation kernel determined in operation S140 (S150). For example, the pixel interpolation unit 123 may interpolate the target pixel by performing a convolution operation on the target kernel and data included in the first interpolation kernel determined in operation S140.

[0034] According to an embodiment, the ISP 100 may include a directional component extraction unit 121 that extracts directional components of the target kernel by convolving a target kernel including a target pixel with each of a plurality of directional kernels, an interpolation kernel determination unit 122 that determines an interpolation kernel based on the directional components, and a pixel interpolation unit 123 that interpolates the target pixel using data included in the interpolation kernels.

[0035] A more detailed operation of the defective pixel correction unit 120 will be described later with reference to FIG. 4a onwards. FIG. 4a is an exemplary diagram showing multiple directional kernels.

[0036] Referring to FIG. 4a, examples of directional kernels 400-425 are shown. As an example, the illustrated directional kernels can be convolved with a target kernel such that the center of the kernel corresponds to a target pixel (DP). As an example, the sizes of the directional kernels can be assumed to be consistent with each other, with zero-padding on edges of the directional kernels that are smaller than the size of the largest directional kernel. As an example, the size of the target kernel can be assumed to be consistent with the size of the largest directional kernel.

[0037] As an example, the first direction kernel 400 to the eighth direction kernel 407 include data corresponding to a corner portion or an edge portion centered on the kernel center, and may correspond to direction kernels capable of extracting components of a corner portion or an edge portion around a target pixel (DP). As an example, the ninth direction kernel 408 to the twelfth direction kernel 411 include data corresponding to a corner portion excluding a vertex portion centered on the kernel center, and may correspond to direction kernels capable of extracting components of a corner portion excluding a vertex portion around a target pixel (DP). As an example, the thirteenth direction kernel 412 to the twenty-fourth direction kernel 423 include data corresponding to a linear line, and may correspond to direction kernels capable of extracting linear components around a target pixel (DP). As an example, the twenty-fifth direction kernel 424 and the twenty-sixth direction kernel 425 include data corresponding to a linear line crossing the kernel, and may correspond to direction kernels capable of extracting linear components crossing the target pixel (DP). The direction kernels shown in FIG. 4a are exemplary, and various kernels filled with points, lines, faces, or patterns may exist.

[0038] In addition, although the present disclosure assumes a 3×3, 5×5, or 7×7 kernel, various other directional kernels may exist for kernels larger than those illustrated, and the convolution operation according to an embodiment of the present invention may be applied to such directional kernels in substantially the same manner.

[0039] FIG. 4b is a diagram for explaining an example in which step S100 of FIG. 3 is performed. 4b, the directional component extraction unit 121 may extract a directional component 450 by convoluting the target kernel and each of a plurality of directional kernels (e.g., 400, 401, 402, 403, or 408). For example, in order to extract the directional component 450 around the target pixel 430, the directional component extraction unit 121 may extract a first directional component 440 by convoluting the target pixel 430 and the first directional kernel 400. In this case, a value obtained by convoluting pixel data of a surrounding pixel 431 corresponding to the size of the first directional kernel 400 around the target pixel 430 and the first directional kernel 400 corresponds to 0x1+1x1+0x0+4x1+0x0+0x0+3x0+0x0=5, and the data of the first directional component 440 may be defined as 5.

[0040] For example, to extract the directional component 450 centered on the target pixel 430, the directional component extraction unit 121 may extract the second directional component 441 by convoluting the target pixel 430 with the second directional kernel 401. In this case, a value obtained by convoluting the pixel data of the surrounding pixel 431 corresponding to the size of the second directional kernel 401 centered on the target pixel 430 with the second directional kernel 401 corresponds to 0x0+1x0+0x0+4x1+0x0+0x1+3x1+0x0=7, and the data of the second directional component 441 may be defined as 7.

[0041] For example, in order to extract the directional component 450 centered on the target pixel 430, the directional component extraction unit 121 may extract the ninth directional component 448 by performing a convolution operation on the target pixel 430 and the ninth directional kernel 408. In this case, a value obtained by performing a convolution operation on the pixel data of the surrounding pixel 432 corresponding to the size of the ninth directional kernel 408 centered on the target pixel 430 and the ninth directional kernel 408 corresponds to 0x0+3x1+0x0+3x0+0x0+2x1+0x1+1x0+0x0+5x0+0x0+4x0+0x0+0x0+1x0+0x0+3x0+0x0+2x0+0x0+5x0+0x0+1x0+0x0=5, and thus the data of the ninth directional component 448 may be defined as 5.

[0042] The directional component extraction unit 121 may stack each of the directional components (e.g., 440, 441, 442, 443, or 448) and represent it as a directional component in the form of a 3D stack kernel 450. Each of the directional components (e.g., 440, 441, 442, 443, or 448) may correspond to each of the directional kernels (e.g., 400, 401, 402, 403, or 408).

[0043] According to an embodiment, the size of the target kernel may be set to the same size as the size of the largest directional kernel among the plurality of directional kernels.

[0044] FIG. 5a shows an exemplary mathematical formula for multiple cost kernels. 5a, the cost kernels according to one embodiment may be expressed in the form of a kernel corresponding to a 1×n matrix. For example, the first cost kernel corresponds to k=0 in the table, and may be expressed in the form of a kernel corresponding to a matrix that is (1 1 0 0 0 1 0 0 0 0...) / 3-(0 0 1 1 1 0 1 1 0 0...) / 5=(1 / 3 1 / 3 -1 / 5 -1 / 5 -1 / 5 1 / 3 -1 / 5 -1 / 5 0 0...) according to the Bn formula. For example, the second cost kernel corresponds to k=1 in the table and can be expressed in kernel form as the matrix (0 0 1 1 0 0 0 1 0 0...) / 3-(1 1 0 0 1 1 1 0 0 0...) / 5=(-1 / 5 -1 / 5 1 / 3 1 / 3 -1 / 5 -1 / 5 -1 / 5 0 0 0...) according to the Bn formula. For example, the 17th cost kernel corresponds to k=16 in the table and can be represented in kernel form as a matrix corresponding to (0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1...) / 3-(0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0...) / 4=(0 0 0 0 1 / 3 0 0 0 1 / 3 -1 / 4 -1 / 4 -1 / 4 -1 / 4 0 0 0 0 0 0 0 1 / 3...).

[0045] As an example, the multiple cost kernels may be multiplied by weights. For example, the first cost kernel may be expressed as a kernel form obtained by multiplying (1 / 3 1 / 3 -1 / 5 -1 / 5 -1 / 5 1 / 3 -1 / 5 -1 / 5 0 0...) by -1, which corresponds to a matrix of (-1 / 3 -1 / 3 1 / 5 1 / 5 1 / 5 -1 / 3 1 / 5 1 / 5 0 0...). For example, the second cost kernel may be expressed as a kernel form obtained by multiplying (-1 / 5 -1 / 5 1 / 3 1 / 3 -1 / 5 -1 / 5 -1 / 5 0 0 0...) by 0, which corresponds to a matrix of (0 0 0 0 0 0 0 0 0 0...). The weights may be preset values ​​to reflect at least one of the following factors: lens, sensor, color, alignment, imbalance, and crosstalk.

[0046] FIG. 5b is an exemplary diagram showing multiple cost kernels. Referring to Fig. 5b, examples of cost kernels 500 to 523 corresponding to the formulas of Fig. 5a are shown. The illustrated cost kernels may be expressed in the form of a kernel corresponding to a 1 x n matrix, where n corresponds to the number of directional components included in the directional components of Fig. 4b (e.g., 450 in Fig. 4b). As an example, the first cost kernel 500 corresponds to k = 0 in the table of Fig. 5a, and the number of directional components included in the directional components 450 of Fig. 4b is 26, which corresponds to the number of directional kernels of Fig. 4a, so that it may be expressed in the form of a kernel corresponding to a matrix that is (1 / 3 1 / 3 -1 / 5 -1 / 5 -1 / 5 1 / 3 -1 / 5 -1 / 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0) according to the formula Bn. As an example, the 17th cost kernel 516 corresponds to k=16 in the table of FIG. 5a, and the number of directional components included in the directional components 450 of FIG. 4b is 26, which corresponds to the number of directional kernels in FIG. 4a, so it can be expressed in the form of a kernel corresponding to a matrix which is (0 0 0 0 1 / 3 0 0 0 1 / 3 -1 / 4 -1 / 4 -1 / 4 -1 / 4 0 0 0 0 0 0 0 1 / 3 0 0 0 0 0).

[0047] As an example, the first cost kernel 500 may correspond to a cost kernel that extracts the first direction component 440, the second direction component 441, and the sixth direction component (not shown) of FIG. 4b as positive average values, and extracts the third direction component 442 to the fifth direction component (not shown), the seventh direction component (not shown), and the eighth direction component (not shown) of FIG. 4b as negative average values. The first cost kernel 500 may be multiplied by a first weighting factor reflecting sensor characteristics, etc. As an example, the second cost kernel 501 may correspond to a cost kernel that extracts the third direction component 442, the fourth direction component 443, and the eighth direction component (not shown) of FIG. 4b as positive average values, and extracts the first direction component 440, the second direction component 441, and the fifth direction component (not shown) to the seventh direction component (not shown) of FIG. 4b as negative average values. The second cost kernel 501 may be multiplied by a second weighting factor reflecting sensor characteristics, etc.

[0048] In addition, in the present disclosure, a cost kernel (e.g., 500) is assumed to have a kernel type corresponding to a 1×26 matrix, but a larger kernel than the illustrated kernel may reflect more various directional components. In addition, in the present disclosure, a cost kernel of 24 is assumed to have a kernel type corresponding to a 1×26 matrix, but a larger number of cost kernels than the illustrated cost kernel may extract costs by combining more various directional components. The convolution operation according to an embodiment of the present invention may be applied to such cost kernels in substantially the same manner.

[0049] FIG. 5c is a diagram for explaining an example in which step S120 of FIG. 3 is performed. 5c, the interpolation kernel determination unit 122 may extract costs 530 to 553 by performing a convolution operation on the directional component 450 of FIG. 4b and a plurality of cost kernels 500, 501, 512, or 534. For example, in order to extract the costs 530 to 553 from the directional component 450, the interpolation kernel determination unit 122 may extract a first cost 530 by performing a convolution operation on the directional component 450 and a first cost kernel 500. In this case, the interpolation kernel determination unit 122 may perform a point-wise convolution operation on the directional component 450 and the first cost kernel 500, in which the directional component 450 corresponds to a channel and the first cost kernel 500 corresponds to a unit kernel. For example, the value obtained by convolving (point-wise convolution) the directional component 450 and the first cost kernel 500 is 5×(1 / 3)+7×(1 / 3)+3×(-1 / 5)+1×(-1 / 5)+1×(-1 / 5)+4×(1 / 3)+3×(-1 / 5)+0×(-1 / 5)+5×0+6×0...=3.7, and the data of the first cost 530 can be defined as 3.7.

[0050] As an example, in order to extract costs 530 to 553 from the directional component 450, the interpolation kernel determination unit 122 may perform a convolution operation on the directional component 450 and the second cost kernel 501 to extract the second cost 531. In this case, the interpolation kernel determination unit 122 may perform a point-wise convolution operation on the directional component 450 and the second cost kernel 501, in which the directional component 450 corresponds to a channel and the second cost kernel 501 corresponds to a unit kernel. For example, the value obtained by convolving (point-wise convolution) directional component 450 and second cost kernel 501 is 5×(-1 / 5)+7×(-1 / 5)+3×(1 / 3)+1×(1 / 3)+1×(-1 / 5)+4×(-1 / 5)+3×(-1 / 5)+0×(1 / 3)+5×0+6×0...=-2.7, and the data of second cost 531 can be defined as -2.7.

[0051] As an example, in order to extract the costs 530 to 553 from the directional component 450, the interpolation kernel determination unit 122 may perform a convolution operation on the directional component 450 and the 13th cost kernel 512 to extract the 13th cost 542. In this case, the interpolation kernel determination unit 122 may perform a point-wise convolution operation on the directional component 450 and the 13th cost kernel 512, in which the directional component 450 corresponds to a channel and the 13th cost kernel 512 corresponds to a unit kernel. For example, the value obtained by convolving (point-wise convolution) directional component 450 and 13th cost kernel 512 is 5×0+7×0+3×0+1×0+1×0+4×0+3×(-1 / 5)+0×0+5×(1 / 2)+6×(-1 / 5)+3×(-1 / 5)+8×(-1 / 5)+2×(-1 / 5)+3×0+3×0+5×0+1×0+5×0+7×0+2×0+1×(1 / 2)+2×0+5×0+1×0=-1.4, and the data of 13th cost 542 can be defined as -4.2.

[0052] The interpolation kernel determination unit 122 can extract an index value by applying an Arg max function or an Arg min function to the first cost 530 to the 24th cost 553 (for example, S130 in FIG. 3). The index value can correspond to the number of each cost kernel. For example, when the first cost 530 has the maximum value among the first cost 530 to the 24th cost 553, the interpolation kernel determination unit 122 can extract 1 as the index number by applying an Arg max function to the first cost 530 to the 24th cost 553, and can identify the first cost kernel 500. For example, when the 13th cost 542 has the minimum value among the first cost 530 to the 24th cost 553, the interpolation kernel determination unit 122 can extract 13 as the index number by applying an Arg min function to the first cost 530 to the 24th cost 553, and can identify the 13th cost kernel 512.

[0053] According to an embodiment, the interpolation kernel determination unit 122 may determine the interpolation kernel based on a value obtained by convoluting the directional component 450 with each of the multiple cost kernels 500 to 523. As an example, each of the multiple cost kernels 500 to 523 may include a preset weight as data. As an example, the value obtained by convoluting the directional component 450 with each of the multiple cost kernels 500 to 523 may be a result of point-wise convolution with the directional component 450 corresponding to a channel and each of the multiple cost kernels 500 to 523 corresponding to a unit kernel. As an example, the interpolation kernel may be an interpolation kernel corresponding to a cost kernel that derives a maximum or minimum value among values ​​obtained by convoluting the directional component 450 with each of the multiple cost kernels 500 to 523.

[0054] FIG. 6a is an exemplary diagram showing multiple interpolation kernels. Referring to FIG. 6a, examples of interpolation kernels 600 to 623 are shown. The pixel interpolation unit 123 may perform a convolution operation of one of the illustrated interpolation kernels 600 to 623 with the target kernel so that the center of the kernel of the illustrated interpolation kernels 600 to 623 corresponds to the target pixel (DP). As an example, the pixel interpolation unit 123 may perform a convolution operation of the target kernel with the interpolation kernel so that the center of the kernel of the illustrated interpolation kernel corresponds to the target pixel (DP). As an example, the interpolation kernel for interpolating the target pixel (DP) may correspond to an index value obtained by taking an Arg max function or an Arg min function for the first cost 530 to the 24th cost 553 in FIG. 5c (for example, S140 in FIG. 3). For example, the index value corresponds to the number of each cost kernel, and one interpolation kernel may correspond to each cost kernel. For example, when the first cost 530 among the first cost 530 to the 24th cost 553 has the maximum value, the interpolation kernel determination unit 122 can extract 1 as an index number by applying an Arg max function to the first cost 530 to the 24th cost 553, and can determine the first interpolation kernel 600 corresponding to the first cost kernel 500 as the interpolation kernel corresponding to the target pixel (DP). For example, when the 13th cost 542 among the first cost 530 to the 24th cost 553 has the minimum value, the interpolation kernel determination unit 122 can extract 13 as an index number by applying an Arg min function to the first cost 530 to the 24th cost 553, and can determine the 13th interpolation kernel 612 corresponding to the 13th cost kernel 512 as the interpolation kernel corresponding to the target pixel (DP). The interpolation kernel shown in FIG. 6a is exemplary, and the present disclosure will be described assuming a 3×3 kernel, but various other interpolation kernels may exist for kernels larger than the illustrated kernel. The interpolation operation according to an embodiment of the present invention can be applied to such an interpolation kernel in substantially the same manner.

[0055] FIG. 6b is a diagram for explaining an example in which step S150 of FIG. 3 is performed. 6b, the pixel interpolation unit 123 may perform a convolution operation of the neighboring pixels 631 of the defective pixel 630 and the interpolation kernel 623 to interpolate the defective pixel 630. The pixel interpolation unit 123 may perform the convolution operation such that the center of the interpolation kernel corresponds to the target pixel (DP). For example, in order to interpolate the defective pixel 630, the pixel interpolation unit 123 may perform a convolution operation of the neighboring pixels 631 of the defective pixel 630 and the interpolation kernel 623 to obtain the interpolated pixel data 640 of the defective pixel 630. In this case, the value obtained by performing a convolution operation of the neighboring pixels 631 of the defective pixel 630 and the interpolation kernel 623 corresponds to 0x0+1x1+0x0+4x1+0x1+0x0+3x1+0x0=8, and the interpolated pixel data 640 of the defective pixel 630 may be defined as 8.

[0056] As an example, the number of neighboring pixels 631 of defective pixel 630 used for convolution may vary depending on the size of the interpolation kernel. For example, if the interpolation kernel has a kernel size of 5×5, the neighboring pixels of defective pixel 630 used for convolution may correspond to 0, 3, 0, 3, 0, 2, 0, 1, 0, 5, 0, 4, 0, 0, 1, 0, 3, 0, 2, 0, 5, 0, 1, 0 from the upper left, which corresponds to a kernel size of 5×5.

[0057] According to an embodiment, the pixel interpolation unit 123 may correspond data located at the center of the interpolation kernel to a target pixel, and may determine a convolution value of the target kernel and the interpolation kernel as pixel data of the target pixel.

[0058] FIG. 7 is a diagram showing an example of the overall process of interpolating one pixel. 7, the interpolated pixel data of the defective pixel according to an example can be derived. In the formula according to an example, x can be defined as an input image, a as a direction kernel, b as a cost kernel, w as a weight kernel including weights as data, y as a cost, c as an interpolation kernel, and o as an output image. In the formula according to an example, l can be defined as a cost kernel identification number, m as a direction kernel position number, n as a direction kernel number, t as a cost kernel identification number (l) derived by the argmax{Y} function, and i and j as pixel coordinates.

[0059] For example, x^i,j may be defined as pixel data of a pixel located at coordinates i, j of an input image. For example, the input image may correspond to a target kernel. For example, if x^i,j corresponds to a defective pixel, the ISP100 may set x^i,j to 0. For example, if x^i,j corresponds to a defective pixel, the ISP100 may not use x^i,j in the calculation process. For example, x^1,2 may be defined as pixel data of a pixel located at row 1, column 1 of the input image. For example, referring again to FIG. 4b, when a kernel including the target pixel 430 is assumed to be the input image, x^1,1 may be defined as 0. For example, referring again to FIG. 4b, when a kernel including the target pixel 430 is assumed to be the input image, the target pixel 430 may correspond to x^4,4. In this case, the ISP100 may set x^4,4 to 0. Alternatively, the ISP100 may not use x^4,4 in the calculation process.

[0060] For example, a^n_i,j may be defined as data located at i, j coordinates of the nth direction kernel. For example, if x^i,j correspond to defective pixels, the ISP 100 may set a^n_i,j to 0. For example, if x^i,j correspond to defective pixels, the ISP 100 may not use a^n_i,j in the calculation process. For example, a^1_1,1 may be defined as data located at row 1, column 1 of the 1st direction kernel. Referring again to FIG. 4a, for example, the first direction kernel 400 may correspond to a^1_i,j. In this case, a^1_1,1 may be defined as 1. Also, a^1_3,3 may be defined as 0. For example, the 24th direction kernel 423 may correspond to a^24_i,j. In this case, a^24_4,7 may be defined as 1. Also, a^24_5,7 may be defined as 0.

[0061] For example, a^n_i,j*x^i,j may be defined as a value obtained by convoluting pixel data of a pixel located at i, j coordinates of the input image with data located at i, j coordinates of the direction kernel. Referring again to FIG. 4b, for example, a^1_i,j*x^i,j corresponds to 0x1+1x1+0x0+4x1+0x0+0x0+3x0+0x0=5, which is a value obtained by convoluting pixel data of a surrounding pixel 431 corresponding to the size of the first direction kernel 400 with the first direction kernel 400 centered on the target pixel 430, and a^1_i,j*x^i,j may be defined as a first direction component 440. For example, a^9_i,j*x^i,j is a value obtained by convolving pixel data of surrounding pixels 432 corresponding to the size of the ninth direction kernel 408 around the target pixel 430 with the ninth direction kernel 408, which corresponds to 0x0+3x1+0x0+3x0+0x0+2x1+0x1+1x0+0x0+5x0+0x0+4x0+0x0+0x0+1x0+0x0+3x0+0x0+2x0+0x0+5x0+0x0+1x0+0x0=5, and a^9_i,j*x^i,j can be defined as the ninth direction component 448.

[0062] As an example, b_l^m may be defined as the mth data of the lth cost kernel. For example, b_1^1 may be defined as the first data of the first cost kernel (e.g., 500 in FIG. 5b). Referring again to FIG. 5b, for example, the first cost kernel 500 may correspond to b_1^m. In this case, b_1^1 may be defined as 1 / 3. Also, b_1^3 may be defined as -1 / 5. For example, referring again to FIG. 5b, the 24th cost kernel 523 may correspond to b_24^m. In this case, b_24^1 may be defined as -1 / 4. Also, b_24^9 may be defined as 0.

[0063] As an example, w_mn may be defined as a weight corresponding to the m-th data of the cost kernel or the n-th data of the directional component among the data of the weight kernel. For example, w_11 may be defined as a weight corresponding to the first data of the cost kernel and the first data of the directional component. For example, referring to FIG. 5c again, w_11 may be defined as a weight multiplied by 1 / 3, -1 / 5, (...), 0, (...), -1 / 4, which are the first data of the first cost kernel 500 to the 24th cost kernel 523. As an example, w_11 may be defined as a weight multiplied by the first directional component 440. In the present disclosure, the directional component 450 and the cost kernels 500 to 523 are each subjected to a point-wise convolution operation, so that the case where the number of data digits m of the cost kernel and the number of data digits n of the directional component are the same will be described. However, m and n may be different from each other in other convolution operations other than the point-wise convolution operation illustrated.

[0064] As an example, y_l may be defined as the l-th cost. For example, y_1 may be defined as a value obtained by performing a point-wise convolution operation on the directional component 450 and the first cost kernel 500. Referring again to FIG. 5c, for example, y_1 corresponds to 5×(1 / 3)+7×(1 / 3)+3×(-1 / 5)+1×(-1 / 5)+1×(-1 / 5)+4×(1 / 3)+3×(-1 / 5)+0×(-1 / 5)+5×0+6×0...=3.7, which is a value obtained by performing a point-wise convolution operation on the directional component 450 and the first cost kernel 500, and y_1 may be defined as the first cost 530. For example, y_13 may be defined as a value obtained by performing a point-wise convolution operation on the directional component 450 and the 13th cost kernel. For example, y_13 corresponds to 5×0+7×0+3×0+1×0+1×0+4×0+3×(-1 / 5)+0×0+5×(1 / 2)+6×(-1 / 5)+3×(-1 / 5)+8×(-1 / 5)+2×(-1 / 5)+3×0+3×0+5×0+1×0+5×0+7×0+2×0+1×(1 / 2)+2×0+5×0+1×0=-1.4, which is the value obtained by convolving (point-wise convolution) the directional component 450 and the 13th cost kernel 512, and y_13 can be defined as the 13th cost 542.

[0065] As an example, the index number can be defined as a result of applying the Argmax or Argmin function to a set having y_l as an element. For example, when y_13 has the maximum value among y_1 to y_24, the interpolation kernel determination unit 122 can apply the Argmax function to the set including y_1 to y_24 and extract 13 as the index number. For example, when y_24 has the minimum value among y_1 to y_24, the interpolation kernel determination unit 122 can apply the Argmin function to y_1 to y_24 and extract 24 as the index number.

[0066] For example, c^t_i,j may be defined as data located at i,j coordinates of the tth interpolation kernel. t may be defined as an index number extracted by the Argmax or Argmin function. For example, c^1_1,1 may be defined as data located at row 1, column 1 of the 1st interpolation kernel. Referring again to FIG. 6a, for example, the first interpolation kernel 600 may correspond to c^1_i,j. In this case, c^1_1,1 may be defined as 0. Also, c^1_1,2 may be defined as 1. For example, the 12th interpolation kernel 611 may correspond to c^12_i,j. In this case, c^12_1,3 may be defined as 1. Also, c^12_3,1 may be defined as 1.

[0067] As an example, o may be defined as an output image. It may be defined as a value obtained by convoluting pixel data of a pixel located at i, j coordinates of an input image with data located at i, j coordinates of an interpolation kernel. Referring again to FIG. 6b, for example, c^24_i,j*x^i,j corresponds to 0x0+1x1+0x0+4x1+0x1+0x0+3x1+0x0=8, which is a value obtained by convoluting pixel data of a surrounding pixel 631 corresponding to the size of the 24th interpolation kernel 623 with the first interpolation kernel 623, with the target pixel 430 as the center, and o may be defined as c^t_i,j*x^i,j. As an example, the interpolation of defective pixels may be performed in a general convolution operation order, where one defective pixel is interpolated according to the interpolation method according to the present disclosure, and then the next defective pixel is interpolated according to the interpolation method according to the present disclosure.

[0068] FIG. 8 is a diagram showing an example of the overall process of interpolating one pixel. 8, when a target pixel 801 included in a target kernel 800 corresponds to a defective pixel, the target pixel 801 may be interpolated to an interpolated pixel data 815 according to an interpolation method according to the present disclosure. After the defective pixel is interpolated according to the interpolation method according to the present disclosure, the ISP 100 may change the target kernel and the target pixel. For example, after the target pixel 801 is interpolated to the interpolated pixel data 815, the ISP 100 may set a second target pixel 803 and may set a second target kernel 802 including the second target pixel 803.

[0069] As an example, the target kernel 800 may include a target pixel 801. The target pixel 801 may be a pixel located at the center of the target kernel 800. The ISP 100 may extract a directional component 805 of the target kernel 800 to interpolate the target pixel 801. For example, the directional component extraction unit 121 may extract the directional component 805 by performing a convolution operation on the target kernel 800 and each of the first plurality of kernels 804. The first plurality of kernels 804 may include the first directional kernel 400 to the 26th directional kernel 425 of FIG. 4a. For example, the directional component extraction unit 121 may extract a first directional component (not shown) by performing a convolution operation on the target kernel 800 and a first directional kernel (not shown), and may extract a second directional component (not shown) by performing a convolution operation on the target kernel 800 and a second directional kernel (not shown). As an example, the directional component extraction unit 121 may sequentially stack the directional components 805 and store them as data in the form of a 3D stack kernel. For example, when the number of directional kernels is n, the directional component extraction unit 121 may store the n directional components as data in the form of a 3D stack kernel.

[0070] As an example, the ISP 100 may extract costs (e.g., 809, 810, 811) of a target kernel 800 to interpolate a target pixel 801. For example, the interpolation kernel determination unit 122 may extract costs (e.g., 809, 810, 811) by convolving the directional component 805 with each of the second plurality of kernels (e.g., 806, 807, 808). As another example, the second plurality of kernels may include the first cost kernel 500 to the 24th cost kernel 523 of FIG. 5b. For example, the interpolation kernel determination unit 122 may extract a first cost 809 by point-wise convolution of the directional component 805 with the first cost kernel 806, and extract a second cost 810 by point-wise convolution of the directional component 805 with the second cost kernel 807.

[0071] As an example, the ISP 100 may determine an interpolation kernel (e.g., 812) corresponding to the target kernel 800 in order to interpolate the target pixel 801. For example, the interpolation kernel determination unit 122 may extract an index number by taking an Arg max or Arg min function on the costs (e.g., 809, 810, 811). For example, if the third cost 811 corresponds to the highest value among the extracted costs (e.g., 809, 810, 811) and the interpolation kernel determination unit 122 takes an Arg max function on the extracted costs (e.g., 809, 810, 811), the interpolation kernel determination unit 122 may extract 3, which is an index number corresponding to the third cost. In this case, the interpolation kernel determination unit 122 may determine the third interpolation kernel 812, which corresponds to the index number 3, as the interpolation kernel corresponding to the target kernel 800.

[0072] For example, the ISP 100 may perform a convolution operation of the target kernel and the interpolation kernel 812 to interpolate the target pixel 801. The defective pixel 813 shown in FIG. 8 may be understood to correspond to the target pixel 801 included in the target kernel 800. For example, the pixel interpolation unit 123 may perform a convolution operation such that the center of the interpolation kernel 812 corresponds to the defective pixel 813. For example, the pixel interpolation unit 123 may perform a convolution operation of the neighboring pixels 814 of the defective pixel 813 and the interpolation kernel 812 to interpolate the defective pixel 813, thereby obtaining the interpolated pixel data 815 of the defective pixel 813. In this case, the value obtained by performing a convolution operation of the neighboring pixels 814 of the defective pixel 813 and the interpolation kernel 812 corresponds to 0x0+2x0+0x0+1x1+2x1+0x0+3x0+0x0=3, and the interpolated pixel data 815 of the defective pixel 813 may be defined as 3.

[0073] According to an embodiment, the ISP 100 may include a directional component extracting unit 121 that extracts a directional component 805 of the target kernel by performing a first convolution operation on a target kernel including a target pixel and each of the first plurality of kernels, an interpolation kernel determining unit 122 that determines an interpolation kernel based on a value obtained by performing a second convolution operation on the directional component 805 and each of the second plurality of kernels, and a pixel interpolating unit 123 that interpolates the target pixel using data included in the interpolation kernel. As an example, the first convolution operation may be an operation performed in a state in which the target pixel corresponds to data located at the center of each of the first plurality of kernels. As an example, the second convolution operation may be a point-wise convolution operation in which the directional component 805 corresponds to a channel and each of the second plurality of kernels corresponds to a unit kernel. As an example, the interpolation kernel may be an interpolation kernel corresponding to a kernel that derives a maximum or minimum value among the values ​​obtained by performing the second convolution operation among the kernels included in the second plurality of kernels.

[0074] FIG. 9 is a flowchart for explaining an operation of classifying kernels by color and interpolating a target pixel. 9, the ISP 100 may separate pixels having a first color filter, pixels having a second color filter, or pixels having a third color filter from an input image, and classify the target kernel into a first color kernel, a second color kernel, or a third color kernel. For example, when pixels corresponding to different colors (e.g., red, green, blue, and white) are arranged according to an RGBW color pattern in an original image captured by an image sensing device, the ISP 100 may classify the original image into an R pixel kernel including only pixels having a red filter, a G pixel kernel including only pixels having a green filter, a B pixel kernel including only pixels having a blue filter, and a W pixel kernel including only pixels having a white filter. For example, the ISP 100 may separate the W pixel kernel, which has the largest number of pixels, from the RGB pixel kernel (S200).

[0075] As an example, the ISP 100 may use the W pixel kernel as a target kernel and perform an interpolation of a defective pixel included in the W pixel kernel. In this case, the ISP 100 may set pixel data located at a position corresponding to a pixel having an RGB color filter in the W pixel kernel to 0. Alternatively, the ISP 100 may set pixel data of a direction kernel located at a position corresponding to a pixel having an RGB color filter in the W pixel kernel to 0. For example, the ISP 100 may use the W pixel kernel as a target kernel and perform the interpolation operations disclosed in FIGS. 4a to 8 to interpolate a defective pixel included in the W pixel kernel (S210).

[0076] As an example, the ISP 100 may use a W pixel kernel as a target kernel, perform interpolation of defective pixels included in the W pixel kernel, and then combine the interpolated W pixel kernel with the separated RGB pixel kernel. For example, the ISP 100 may interpolate all defective pixels included in the W pixel kernel, and then combine the interpolated W pixel kernel with the RGB pixel kernel separated in step S200 to generate an RGBW pixel kernel (S220). As an example, pixels having an R, G, or B color filter among the pixels included in the RGBW pixel kernel may include defective pixels.

[0077] As an example, the ISP 100 may use an RGBW pixel kernel as a target kernel and perform an interpolation of a defective pixel included in the RGBW pixel kernel. For example, the ISP 100 may use an RGBW pixel kernel as a target kernel and perform the interpolation operations disclosed in Figures 4a to 8 to interpolate a defective pixel included in the RGBW pixel kernel (S230).

[0078] As an example, the ISP 100 may use the W pixel kernel as a target kernel, and use data of a directional component (e.g., 450 in FIG. 4b) or data of a cost (e.g., 530 to 533 in FIG. 5c) used when interpolating a defective pixel included in the W pixel kernel, to use an RGB pixel kernel as a target kernel and perform interpolation of a defective pixel included in the RGB pixel kernel.

[0079] As an example, the ISP 100 may use an RGBW pixel kernel as a target kernel, perform interpolation of defective pixels included in the RGBW pixel kernel, and generate a Bayer pattern image, which is an RGB pattern, based on the interpolated RGBW pixel kernel. For example, the ISP 100 may use an RGBW pixel kernel as a target kernel, perform the interpolation operations disclosed in Figs. 4a to 8 to interpolate defective pixels included in the RGBW pixel kernel, and generate a Bayer pattern image, which is an RGB pattern, based on the interpolated RGBW pixel kernel.

[0080] According to an embodiment, the directional component extraction unit 121 may extract a directional component using pixel data of pixels having a first color filter among pixels included in the target kernel. As an example, the directional component extraction unit 121 may set pixel data of pixels having a color filter different from the first color filter among pixels included in the target kernel to 0. As an example, data corresponding to pixel positions having a color filter different from the first color filter among data included in a plurality of directional kernels may be set to 0.

[0081] According to an embodiment, the image signal processing method may include the steps of: extracting a first directional component of the first color kernel by convolving a first color kernel including a first target pixel with each of a plurality of directional kernels; determining a first interpolation kernel based on the first directional component; and interpolating the first target pixel using data included in the first interpolation kernel. As an example, the image signal processing method may further include blending an interpolated first color kernel including the interpolated first target pixel with a second color kernel to generate an interpolated target kernel. As an example, the image signal processing method may further include the steps of convolving an interpolated target kernel including a second target pixel with each of a plurality of directional kernels to extract a second directional component of the interpolated target kernel; determining a second interpolation kernel based on the second directional component; and interpolating the second target pixel using data included in the second interpolation kernel. As an example, the image signal processing method may include the step of determining a first interpolation kernel based on a value obtained by convolving the first directional component with each of a plurality of cost kernels. For example, the image signal processing method may further include multiplying each of the plurality of cost kernels by a preset weight. For example, the image signal processing method may include performing point-wise convolution by associating the first directional component with a channel and each of the plurality of cost kernels with an identity kernel.

[0082] FIG. 10 is a block diagram of an example of a computing device corresponding to the image signal processor of FIG. Referring to FIG. 10, a computing device 1000 may represent one embodiment of a hardware configuration for performing the operations of the image signal processor 100 of FIG.

[0083] The computing device 1000 may be mounted on a chip separate from a chip on which an image sensing device is mounted. According to an embodiment, the chip on which an image sensing device is mounted and the chip on which the computing device 1000 is mounted may be implemented in a single package, for example, a multi-chip package (MCP), but the scope of the present invention is not limited thereto.

[0084] In addition, the internal configuration or arrangement of the computing device 1000 and the image sensing device may vary depending on the embodiment. For example, at least a part of the configuration of the image sensing device may be included in the computing device 1000. Or, at least a part of the configuration of the computing device 1000 may be included in the image sensing device. In this case, at least a part of the configuration of the computing device 1000 may be mounted on a chip on which the image sensing device is mounted.

[0085] Computing device 1000 may include a processor 1010, a memory 1020, an input / output interface 1030, and a communication interface 1040.

[0086] The processor 1010 can process data and / or instructions necessary to perform the operations of the components 110, 120 of the image signal processor 100 described in Fig. 1. That is, the processor 1010 may refer to the image signal processor 100 itself, although the scope of the present invention is not limited thereto.

[0087] The memory 1020 can store data and / or instructions necessary to perform the operations of the components 110, 120 of the image signal processor 100 and can be accessed by the processor 1010. For example, the memory 1020 can be implemented with volatile memory (e.g., Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), etc.) or non-volatile memory (e.g., Programmable Read Only Memory (PROM), Erasable PROM (EPROM), Electrically Erasable PROM (EEPROM), flash memory, etc.).

[0088] That is, a computer program for performing the operations of the image signal processor 100 disclosed in this document can be recorded in the memory 1020 and executed and processed by the processor 1010 to realize the operations of the image signal processor 100.

[0089] The input / output interface 1030 can provide an interface to connect an external input device (e.g., a keyboard, a mouse, a touch panel, etc.) and / or an external output device (e.g., a display) to the processor 1010 and enable data to be transmitted and received.

[0090] The communication interface 1040 is configured to be capable of transmitting and receiving various data to and from an external device (eg, an application processor, an external memory, etc.), and may be a device capable of supporting wired or wireless communication. [Explanation of symbols]

[0091] 100 Image Signal Processor (ISP) 110 Defective pixel detection unit 120 Defective pixel correction unit 121 Direction component extraction part 122 Interpolation kernel determination unit 123 pixel interpolation section

Claims

1. a direction component extracting unit for extracting direction components of the target kernel by performing a convolution operation on a target kernel including a target pixel and each of a plurality of direction kernels; an interpolation kernel determination unit that determines an interpolation kernel based on the directional component; a pixel interpolator for interpolating the target pixel using data included in the interpolation kernel; an image signal processor including:

2. The image signal processor according to claim 1 , wherein the interpolation kernel determination unit determines the interpolation kernel based on a value obtained by convolving the directional component with each of a plurality of cost kernels.

3. 3. The image signal processor of claim 2, wherein each of the plurality of cost kernels includes a preset weight as data.

4. 3. The image signal processor of claim 2, wherein the value obtained by convolving the directional component with each of the plurality of cost kernels is a result of performing point-wise convolution by associating the directional component with a channel and by associating each of the plurality of cost kernels with an identity kernel.

5. 3. The image signal processor according to claim 2, wherein the interpolation kernel corresponds to a cost kernel obtained by deriving a maximum or minimum value among values ​​obtained by convoluting the directional component with each of the plurality of cost kernels.

6. 2. The image signal processor of claim 1, wherein the directional component extractor extracts the directional component using pixel data of pixels having a first color filter among pixels included in the target kernel.

7. 7. The image signal processor of claim 6, wherein the directional component extractor sets pixel data of pixels having a color filter different from the first color filter among the pixels included in the target kernel to 0.

8. 7. The image signal processor of claim 6, wherein the directional component extractor sets data included in the plurality of directional kernels, which corresponds to a pixel position having a color filter different from the first color filter, to 0.

9. The image signal processor of claim 1 , wherein the size of the target kernel is the same as the size of a largest directional kernel among the plurality of directional kernels.

10. 2. The image signal processor of claim 1, wherein the pixel interpolation unit associates data located at a center of the interpolation kernel with the target pixel, and determines a convolution value of the target kernel and the interpolation kernel as pixel data of the target pixel.

11. a directional component extracting unit for extracting directional components of the target kernel by performing a first convolution operation on a target kernel including a target pixel and each of the first plurality of kernels; an interpolation kernel determination unit that determines an interpolation kernel based on a value obtained by performing a second convolution operation on the directional component and each of the second plurality of kernels; a pixel interpolator for interpolating the target pixel using data included in the interpolation kernel; an image signal processor including:

12. 12. The image signal processor of claim 11, wherein the first convolution operation is performed while the target pixel corresponds to data located at a center of each of the first plurality of kernels.

13. 12. The image signal processor of claim 11, wherein the second convolution operation is a point-wise convolution operation in which the directional components correspond to channels and each of the second plurality of kernels corresponds to an identity kernel.

14. 12. The image signal processor of claim 11, wherein the interpolation kernel is an interpolation kernel corresponding to a kernel that derives a maximum or minimum value from among the second convolution operation values, among the kernels included in the second plurality of kernels.

15. extracting a first directional component of the first color kernel by convolving a first color kernel including a first target pixel with each of a plurality of directional kernels; determining a first interpolation kernel based on the first directional component; Interpolating the first target pixel using data contained in the first interpolation kernel; 13. A method for processing an image signal, comprising:

16. 16. The method of claim 15, further comprising blending an interpolated first color kernel and a second color kernel comprising the interpolated first target pixel to generate an interpolated target kernel.

17. performing a convolution operation on the interpolated target kernel including a second target pixel and each of the plurality of directional kernels to extract a second directional component of the interpolated target kernel; determining a second interpolation kernel based on the second directional component; 17. The method of claim 16, further comprising: interpolating the second target pixel using data contained in the second interpolation kernel.

18. The step of determining a first interpolation kernel based on the first directional component includes:

16. The image signal processing method according to claim 15, further comprising the step of determining the first interpolation kernel based on a value obtained by convolving the first directional component with each of a plurality of cost kernels.

19. The step of determining the first interpolation kernel comprises:

20. The method of claim 18, further comprising the step of multiplying each of the plurality of cost kernels by a preset weight.

20. The step of determining the first interpolation kernel based on a value obtained by convolving the first directional component with each of a plurality of cost kernels includes:

20. The method of claim 17, further comprising the step of performing point-wise convolution by associating the first directional component with a channel and by associating each of the plurality of cost kernels with an identity kernel.

Images