Image signal processor and image signal processing method

Abstract

The invention relates to an image signal processor and an image signal processing method. An image signal processor includes: a directional component extractor configured to extract a directional component of a target kernel by performing a convolution operation between the target kernel and each of a plurality of directional kernels, the target kernel including a target pixel; an interpolation kernel determiner configured to determine an interpolation kernel based on the directional component; and a pixel interpolator configured to interpolate a target pixel using the data included in the interpolation kernel.

Description

Technical Field

[0001] Embodiments of the present disclosure generally relate to an image signal processor capable of performing image conversion and an image signal processing method thereof. Background Art

[0002] Image sensing devices are devices that capture optical images by converting light into electrical signals using photosensitive semiconductor materials that react to light. With the development of the automotive, medical, computer, and communication industries, the demand for high-performance image sensing devices is increasing in various fields such as smartphones, digital cameras, game consoles, the Internet of Things (IoT), robots, surveillance cameras, and medical micro cameras.

[0003] The original image captured by the image sensing device may include a plurality of pixels corresponding to different colors (e.g., red, blue, and green). The plurality of pixels included in the original image may be arranged according to a specific color pattern (e.g., an RGBW pattern). In order to convert the original image into a complete image (e.g., an RGBW image), an operation of interpolating pixels may be performed according to a predetermined algorithm. Such an algorithm basically includes an operation of interpolating pixels having missing (or omitted) information using information of neighboring pixels. Summary of the invention

[0004] According to an embodiment of the present disclosure, an image signal processor may include: a directional component extractor, which is configured to extract a directional component of a target kernel by performing a convolution operation between the target kernel and each of a plurality of directional kernels, the target kernel including a target pixel; an interpolation kernel determiner, which is configured to determine the interpolation kernel based on the directional component; and a pixel interpolator, which is configured to interpolate the target pixel using data included in the interpolation kernel.

[0005] According to another embodiment of the present disclosure, an image signal processor may include: a directional component extractor, which is configured to extract a directional component of a target kernel by performing a first convolution operation between the target kernel and each of a plurality of first kernels, the target kernel including a target pixel; an interpolation kernel determiner, which is configured to obtain a convolution result value by performing a second convolution operation between each of a plurality of second kernels and the directional component; and a pixel interpolator, which is configured to interpolate the target pixel using data included in the interpolation kernel.

[0006] According to another embodiment of the present disclosure, an image signal processing method may include the following steps: extracting a first directional component of a first color kernel by performing a convolution operation between a first color kernel and each of a plurality of directional kernels, the first color kernel including a first target pixel; determining a first interpolation kernel based on the first directional component; and interpolating the first target pixel using data included in the first interpolation kernel. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The above and other features and advantageous aspects of the disclosed technology will become readily apparent with reference to the following detailed description when considered in conjunction with the accompanying drawings.

[0008] Figure 1 is a block diagram showing an image signal processor according to some embodiments of the present disclosure.

[0009] Figure 2 Some embodiments of the present disclosure are shown Figure 1 Block diagram of the defective pixel corrector shown.

[0010] Figure 3 Some embodiments of the present disclosure are shown Figure 2 A flow chart of the operation of the defective pixel corrector is shown.

[0011] Figure 4A is a diagram showing multiple directional kernels based on some embodiments of the present disclosure.

[0012] Figure 4B It is a diagram showing some embodiments of the present disclosure for executing Figure 3 FIG. 1 is a diagram of the method of operation S100 shown.

[0013] Figure 5A Formulas showing multiple cost kernels based on some embodiments of the present disclosure are illustrated.

[0014] Figure 5B is a diagram illustrating multiple cost kernels according to some embodiments of the present disclosure.

[0015] Figure 5C It is a diagram showing some embodiments of the present disclosure for executing Figure 3 FIG. 1 is a diagram of the method of operation S120 shown.

[0016] Fig. 6A is a diagram illustrating multiple interpolation kernels according to some embodiments of the present disclosure.

[0017] Figure 6B It is a diagram showing some embodiments of the present disclosure for executing Figure 3 FIG. 1 is a diagram of the method of operation S150 shown.

[0018] Figure 7 Formulas showing the entire process of interpolating one pixel based on some embodiments of the present disclosure are illustrated.

[0019] Figure 8 is a diagram showing the entire process of interpolating one pixel according to some embodiments of the present disclosure.

[0020] Fig. 9 is a flow chart illustrating a method of interpolating a target kernel by classifying the kernels by color according to some embodiments of the present disclosure.

[0021] Fig.10 It is a diagram showing some embodiments of the present disclosure. Figure 1 A block diagram of a computing device corresponding to an image signal processor. DETAILED DESCRIPTION

[0022] Embodiments of the present disclosure provide implementations and examples of image signal processors and image signal processing methods that can increase the accuracy of correction of defective pixels, etc., which can be used to substantially solve one or more technical or engineering problems and alleviate the configuration of limitations or shortcomings encountered in some image signal processors in the art. Some embodiments of the present disclosure relate to an image signal processor and an image signal processing method that can increase the accuracy of correction of defective pixels, etc. Recognizing the above problems, an image signal processor and an image signal processing method based on some embodiments of the present disclosure can analyze the directional components included in the target kernel, and can interpolate the target kernel based on the interpolation kernel corresponding to the target kernel, so that the accuracy of correction of defective pixels can be increased even when multiple directional components are included in the target kernel. Generally, in image processing, interpolation is a technique for estimating the value of pixels at non-integer coordinates within an image. When scaling or transforming an image, the resulting pixel position may not be aligned with the original pixel grid. Interpolation helps to estimate the intensity values ​​of these non-integer pixel positions based on the known values ​​of neighboring or other pixels.

[0023] Reference will now be made in detail to some embodiments of the present disclosure, examples of which are shown in the accompanying drawings. Whenever possible, the same reference numerals will be used throughout the accompanying drawings to represent the same or similar parts. Although the present disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof are shown as examples in the accompanying drawings. However, the present disclosure should not be construed as being limited to the embodiments set forth herein.

[0024] Hereinafter, various embodiments will be described with reference to the accompanying drawings. However, it should be understood that the disclosed technology is not limited to a specific embodiment, but includes various modifications, equivalents and / or substitutions of the embodiments. The embodiments of the present disclosure can provide various effects that can be directly or indirectly recognized by the disclosed technology.

[0025] Various embodiments of the present disclosure relate to an image signal processor capable of increasing the accuracy of correction of a defective pixel or the like and an image signal processing method thereof.

[0026] It is to be understood that both the foregoing general description and the following detailed description of the present disclosure are exemplary and descriptive and are intended to provide further description of the claimed embodiments.

[0027] Figure 1 is a block diagram showing an image signal processor 100 according to some embodiments of the present disclosure.

[0028] Reference Figure 1 , the image signal processor (ISP) 100 may perform at least one image signal processing on the image data (IDATA) to generate processed image data (IDATA_P). The image signal processor 100 may reduce noise of the image data (IDATA) and may perform various types of image signal processing for image quality improvement of the image data (e.g., demosaicing, defective pixel correction, gamma correction, color filter array interpolation, color matrix, color correction, color enhancement, lens distortion correction, etc.). In addition, the image signal processor 100 may compress the image data created by performing the image signal processing for image quality improvement so that the image signal processor 100 may create an image file using the compressed image data. Alternatively, the image signal processor 100 may restore the image data from the image file. In this case, the scheme for compressing such image data may be a reversible format or an irreversible format. As a representative example of such a compression format, in the case of using a still image, a Joint Photographic Experts Group (JPEG) format, a JPEG 2000 format, etc. may be used. In addition, in the case of using a moving image, a plurality of frames may be compressed according to the Moving Picture Experts Group (MPEG) standard so that a moving image file may be created.

[0029] The image data (IDATA) may be generated by an image sensing device that captures an optical image of a scene, but the scope of the disclosed technology is not limited thereto. In an embodiment, the image sensing device may include: a pixel array including a plurality of pixels, configured to sense incident light received from a scene; a control circuit configured to control the pixel array; and a readout circuit configured to output digital image data (IDATA) by converting an analog pixel signal received from the pixel array into digital image data (IDATA). In some embodiments of the present disclosure, the image data (IDATA) is generated by an image sensing device.

[0030] The pixel array of the image sensing device may include defective pixels that cannot normally capture color images due to process limitations or temporary noise influx. In addition, the pixel array may include phase difference detection pixels configured to obtain phase difference related information to implement an autofocus function. In the same manner as the defective pixels, the phase difference detection pixels cannot obtain color images, so that from the perspective of the color image, the phase difference detection pixels can be regarded as defective pixels. In some embodiments, for the convenience of description and better understanding of the disclosed technology, defective pixels and phase difference detection pixels that are each unable to normally obtain color images will be collectively referred to as "defective pixels" below.

[0031] In order to increase the quality of color images, it is essential to improve the correction accuracy of defective pixels. To this end, the image signal processor 100 according to some embodiments of the present disclosure may include a defective pixel detector 110 and a defective pixel corrector 120.

[0032] The defective pixel detector 110 may detect pixel data of a defective pixel from the image data (IDATA). In some embodiments of the present disclosure, for convenience of description, digital data corresponding to a pixel signal of each pixel will be defined as pixel data below, and a collection (or aggregation) of pixel data corresponding to a predetermined unit (e.g., a frame or a kernel) will be defined as image data (IDATA) below. Here, a frame may correspond to an entire pixel array, and a kernel may refer to a unit for image signal processing.

[0033] In some embodiments, the defective pixel detector 110 may detect pixel data of a defective pixel based on the image data (IDATA). For example, the defective pixel detector 110 may calculate the difference between the pixel data of the target pixel and the average value of the pixel data. Here, the pixel data of the target pixel will be used as a target for determining whether the corresponding pixel is a defective pixel, and the average value of the pixel data belongs to the kernel. In addition, the defective pixel detector 110 may determine whether the target pixel is a defective pixel based on the calculated difference. That is, when the difference between the pixel data of the target pixel and the average value of the pixel data belonging to the kernel is greater than or equal to a predetermined threshold, the defective pixel detector 110 may determine that the target pixel is a defective pixel without normal pixel data. In some embodiments, the predetermined threshold may be a fixed constant, or may be a specific ratio of the brightness value (e.g., green average) of the current kernel.

[0034] In some other implementations, the defective pixel detector 110 may receive pre-stored position information of the defective pixel from the image sensing device that generates the image data (IDATA). In addition, the defective pixel detector 110 may determine whether the target pixel is a defective pixel based on the position information of the defective pixel. Due to manufacturing process reasons, the image sensing device may store the position information of the fixed defective pixel in an internal storage unit. In some embodiments, the internal storage unit may be a one-time programmable (OTP) memory. In addition, the image sensing device may provide the position information of the defective pixel to the image signal processor 100.

[0035] When the defective pixel detector 110 determines that the target pixel is a defective pixel, the defective pixel corrector 120 may correct pixel data of the target pixel based on image data including a kernel of the target pixel.

[0036] Reference Figure 2, the defective pixel corrector 120 may include a directional component extractor 121, an interpolation kernel determiner 122, and a pixel interpolator 123.

[0037] The image data (IDATA) corresponding to a frame may include directional components of various sizes and shapes. The directional component may mean to quantify a set (or aggregation) of pixels with similarity among the neighboring pixels of the target pixel according to size and / or direction. For example, a subject with a uniform color included in a captured scene may be identified as a directional component. For example, when the pixels with a uniform color among the neighboring pixels around the target pixel are arranged in the horizontal direction and the boundaries between the pixels with the uniform color are clear, this means that the corresponding directional component has a horizontal direction and a high directional value. In some embodiments, the directional component may include a horizontal direction, a vertical direction, or a diagonal direction, and may not include the target pixel. For example, the directional component may be located in a target kernel including the target pixel, and may include a stripe pattern, an angle pattern, etc., each of which does not include the target pixel. The directional component extractor 121 may extract the directional component corresponding to the target pixel included in the target kernel. In one example, the directional component extractor 121 may perform a convolution operation between the target kernel and each of the plurality of directional kernels to extract the directional component corresponding to the target pixel included in the target kernel. In the context of image processing, convolution operations are often used for tasks such as edge detection, blurring, sharpening, and feature extraction. Convolution works by defining a small matrix (also called a kernel or filter). The matrix is ​​usually much smaller than the input image. The kernel slides or convolves across the entire input image. At each location, the element-by-element multiplication of the overlapping portion of the kernel and the image is calculated. The results of the element-by-element multiplication are then summed to generate a single value. This result is usually placed in a new image (often called an output or feature map). The convolution operation is usually repeated for each position in the input image, and the result is a new image (feature map) that can emphasize specific features according to the selected kernel. The directional component corresponding to the target pixel included in the target kernel can reflect the characteristics of various directional kernels. For example, the directional component extractor 121 can extract the directional component of the diagonal direction by performing a convolution operation between the target kernel and the diagonal directional kernel. In addition, the directional component extractor 121 can extract more diverse directional components by combining the directional kernels. For example, when the target kernel includes both the diagonal directional component and the horizontal directional component, the directional component extractor 121 can selectively combine the diagonal directional kernel and the horizontal directional kernel. For example, a kernel including 1 value included in a horizontal direction kernel and 1 value included in a diagonal direction kernel as 1 value may correspond to a kernel that can extract a direction having an intermediate direction between the horizontal direction and the diagonal direction. In addition, the directional component extractor 121 may multiply the combined kernel by a preset weight, and may extract a cost of a new directional component reflecting the diagonal direction component and the horizontal direction component. The cost associated with the directional component may be associated with various image processing tasks, and often involves optimization or energy minimization problems.Some examples of costs in image processing are computational cost, memory cost, filter design, feature redundancy, noise sensitivity, computational efficiency, parameter tuning (e.g., size of the convolution kernel), etc.

[0038] The interpolation kernel determiner 122 may extract the cost according to the combination of the directional components based on the extracted directional components. In one example, the interpolation kernel determiner 122 may perform a convolution operation between each directional component and each of the plurality of cost kernels to extract the cost according to the combination of the directional components.

[0039] The interpolation kernel determiner 122 may determine an interpolation kernel corresponding to a target pixel included in the target kernel based on the extracted cost. In one example, the interpolation kernel determiner 122 may determine an interpolation kernel corresponding to a cost kernel that derives a maximum value or a minimum value (i.e., a maximum cost value or a minimum cost value) among the extracted costs as the interpolation kernel corresponding to the target pixel.

[0040] The pixel interpolator 123 may interpolate the pixel data of the target pixel included in the target kernel based on the interpolation kernel determined by the interpolation kernel determiner 122. In one example, the pixel interpolator 123 may match the data located at the center of the interpolation kernel determined by the interpolation kernel determiner 122 with the target pixel. In addition, the pixel interpolator 123 may perform a convolution operation between the target kernel and the interpolation kernel, and may determine the convolution result value as the pixel data of the target pixel.

[0041] Figure 3 Some embodiments of the present disclosure are shown Figure 2 A flow chart of the operation of the defective pixel corrector 120 is shown.

[0042] Reference Figure 3 , the operation of the defective pixel corrector 120 may include: extracting a directional component of a target kernel including a target pixel DP (S100); determining an interpolation kernel based on the extracted directional component and a plurality of cost kernels (S140); and interpolating the target pixel DP using data included in the determined interpolation kernel (S150).

[0043] exist Figures 3 to 9 In the following, a method for correcting defective pixels using a kernel indicating an RGBW pattern as an example will be described in detail. For example, the RGBW pattern may be a kernel including red pixels, green pixels, blue pixels, and white pixels. In addition, the target pixel is located at the center of the target kernel. Figure 4BAs shown in the example of , when the target kernel is a (5×5) kernel including the first to 25th pixels, the target pixel may correspond to the pixel located at the position of the thirteenth pixel. The red pixel (R) may generate red pixel data by detecting red light, the green pixel (G) may generate green pixel data by detecting green light, the blue pixel (B) may generate blue pixel data by detecting blue light, and the white pixel may generate white pixel data by detecting white light. In addition, each of the red pixel data, the green pixel data, the blue pixel data, and the white pixel data has a range of 0 to 1023.

[0044] Although for the convenience of description, Figure 4B As shown, the embodiment of the present disclosure discloses that the image is a (5×5) size kernel with 25 pixels arranged in an RGBW pattern, but the technical concept of the disclosed technology can also be applied to another kernel, in which the color pixels are arranged in other patterns such as Bayer pattern, quaternary Bayer pattern, nine-element Bayer pattern, sextuple Bayer pattern, monochrome pattern, etc. The type of image pattern is not limited thereto and can also be fully changed as needed. In addition, a kernel having another size other than (5×5) size (for example, (3×3) size and (10×10) size) can be used according to the performance of the image signal processor 100, the required correction accuracy, the arrangement method of the color pixels, etc.

[0045] The directional component extractor 121 may extract the directional component of the target kernel by performing a convolution operation between the target kernel and each of the plurality of directional kernels (S100). For example, the directional component extractor 121 may extract the first directional component by performing a convolution operation between the target kernel and the first directional kernel, may extract the second directional component by performing a convolution operation between the target kernel and the second directional kernel, and may extract the third directional component by performing a convolution operation between the target kernel and the third directional kernel.

[0046] The interpolation kernel determiner 122 may multiply each cost kernel by a weight reflecting the characteristics of each sensor (S110). In addition, the interpolation kernel determiner 122 may extract multiple costs (S120) by performing point-by-point convolution between each cost kernel multiplied by the weight in operation S110 and each directional component extracted in operation S100. Generally, point-by-point convolution is a convolution operation in image processing that uses a 1×1 filter kernel instead of a larger filter kernel. This means that the filter considers only one pixel at a time along each dimension. For example, the interpolation kernel determiner 122 may multiply a first cost kernel by a first weight and may multiply a second cost kernel by a second weight. For example, the interpolation kernel determiner 122 may extract a first cost by performing a point-by-point convolution operation between a first cost kernel multiplied by a first weight and each of the first to third directional components, and may extract a second cost by performing a point-by-point convolution operation between a second cost kernel multiplied by a second weight and each of the first to third directional components.

[0047] The interpolation kernel determiner 122 may extract a cost index number (S130) by applying a function "Arg max" or "Arg min" to the cost extracted in operation S120. The Arg max function may correspond to a function for extracting an index number of the highest value among a plurality of costs extracted as a result of convolution. For example, if costs 1, 2, 3, 0, 1, 4, 2 are extracted as a result of convolution, 6 may be derived when the Arg max function is applied to 1, 2, 3, 0, 1, 4, and 2. This is because cost 4 is a maximum value and its digit is 6. The Arg min function is a function for extracting an index number of a low value, and a specific operation corresponds to Argmax. In addition, the interpolation kernel determiner 122 may determine an interpolation kernel corresponding to the index number extracted in operation S130 (S140). For example, the interpolation kernel determiner 122 may extract an index number "1" corresponding to the first cost by applying a function "Arg max" or "Arg min" to the first cost and the second cost, and may determine a first interpolation kernel corresponding to the index number "1".

[0048] The pixel interpolator 123 may interpolate the target pixel using the data included in the interpolation kernel determined in operation S140 (S150). For example, the pixel interpolator 123 may interpolate the target pixel by performing a convolution operation between the target kernel and the data included in the first interpolation kernel determined in operation S140.

[0049] In some embodiments, in the image signal processor (ISP) 100, the directional component extractor 121 may extract the directional component of the target kernel by performing a convolution operation between the target kernel including the target pixel and each of the plurality of directional kernels. The interpolation kernel determiner 122 may determine the interpolation kernel based on the directional component. The pixel interpolator 123 may interpolate the target pixel using the data included in the interpolation kernel.

[0050] Will refer to it later Figure 4A The following figures describe the operation of the defective pixel corrector 120 in more detail.

[0051] Figure 4A is a diagram showing multiple directional kernels (400-425) according to some embodiments of the present disclosure.

[0052] Reference Figure 4A , showing an example of directional kernels (400-425). A_n is the symbol of multiple directional kernels, for example, A_5 corresponds to the sixth directional kernel 405. In one example, Figure 4A The directional kernel may perform a convolution operation between the target kernel and each directional kernel so that the center of the kernel may correspond to the target pixel (DP). In some embodiments, the target kernel and the directional kernel may be located at Figure 4AThe position of the pixel at the center of the directional kernel shown is matched with the position of the DP of the target kernel to perform a convolution operation. In one example, the sizes of the directional kernels can be matched with each other by zero-filling the edge portions of the directional kernels each having a size smaller than the maximum directional kernel. In one example, the size of the target kernel can match the size of the maximum directional kernel. In one example, the first to eighth directional kernels (400-407) may include data corresponding to the corner portion or edge portion relative to the center of the kernel, so that the first to eighth directional kernels (400-407) may correspond to the directional kernel that can extract the component of the corner or edge portion located around the target pixel (DP). In one example, the ninth to twelfth directional kernels (408-411) may include data corresponding to the corner portion other than the vertex portion relative to the center of the kernel, and may correspond to the directional kernel that can extract the component of the corner portion other than the vertex portion located around the target pixel (DP). In one example, the thirteenth to twenty-fourth directional kernels (412-423) may include data corresponding to a line shape, so that the thirteenth to twenty-fourth directional kernels (412-423) may correspond to a directional kernel capable of extracting a linear component (i.e., a linear component) located around the target pixel (DP). In one example, each of the twenty-fifth directional kernel 424 and the twenty-sixth directional kernel 425 may include data corresponding to a line shape intersecting the kernel, so that the twenty-fifth and twenty-sixth directional kernels (424, 425) may correspond to a directional kernel capable of extracting a linear component intersecting the target pixel (DP). Figure 4A The directional kernel shown is only an example, and there may be various kernels filled with points, lines, surfaces, or patterns.

[0053] Although for convenience of description, the kernel is implemented as a (3×3) kernel, a (5×5) kernel, or a (7×7) kernel in the present embodiment, there may be more diverse directional kernels in kernels larger than the above example kernels. The convolution operation according to the embodiments of the present disclosure may be applied in substantially the same manner as these directional kernels.

[0054] Figure 4B It is a diagram showing some embodiments of the present disclosure for executing Figure 3 FIG. 1 is a diagram of the method of operation S100 shown.

[0055] Reference Figure 4B, the directional component extractor 121 may extract the directional component 450 by performing a convolution operation between the target kernel and each directional kernel (e.g., 400, 401, 402, 403, or 408). For example, in order to extract the directional component 450 centered at the target pixel 430, the directional component extractor 121 may extract the first directional component 440 by performing a convolution operation between the target pixel 430 and the first directional kernel 400. At this time, the convolution result value obtained by performing a convolution operation between the first directional kernel 400 and the pixel data of the neighboring pixel 431 corresponding to the size of the first directional kernel 400 with respect to the target pixel 430 may be represented by "0×1+1×1+0×0+4×1+0×0+0×0+3×0+0×0=5", and the data of the first directional component 440 may be defined as "5".

[0056] In one example, in order to extract the directional component 450 centered on the target pixel 430, the directional component extractor 121 may extract the second directional component 441 by performing a convolution operation between the target pixel 430 and the second directional kernel 401. At this time, the convolution result value obtained by performing a convolution operation between the second directional kernel 401 and the pixel data of the neighboring pixel 431 corresponding to the size of the second directional kernel 401 with respect to the target pixel 430 may be represented by "0×0+1×0+0×0+4×1+0×0+0×1+3×1+0×0=7", and the data of the second directional component 441 may be defined as "7".

[0057] In one example, in order to extract the directional component 450 centered on the target pixel 430, the directional component extractor 121 may extract the ninth directional component 448 by performing a convolution operation between the target pixel 430 and the ninth directional kernel 408. At this time, the convolution result value obtained by performing a convolution operation between the ninth directional kernel 408 and the pixel data of the neighboring pixel 432 corresponding to the size of the ninth directional kernel 408 with respect to the target pixel 430 may be represented by “0×0+3×1+0×0+3×0+0×0+2×1+0×1+1×0+0×0+5×0+0×0+4×0+0×0+0×0+1×0+0×0+

[0058] 3×0+0×0+2×0+0×0+5×0+0×0+1×0+0×0=5”, and the data of the ninth direction component 448 can be defined as “5”.

[0059] The directional component extractor 121 may stack the respective directional components (e.g., 440, 441, 442, 443, or 448), and may express the stacked result as a directional component 450 formed into a three-dimensional (3D) stacked kernel shape. The respective directional components (e.g., 440, 441, 442, 443, or 448) may correspond to the respective directional kernels (e.g., 400, 401, 402, 403, or 408).

[0060] In some embodiments, the size of the target kernel may be set to be the same size as the size of the largest directional kernel among the plurality of directional kernels.

[0061] Figure 5A Formulas showing multiple cost kernels based on some embodiments of the present disclosure are illustrated.

[0062] Reference Figure 5A , multiple cost kernels based on some embodiments of the present disclosure may be expressed as kernels corresponding to a (1×n) matrix. For example, the first cost kernel may correspond to Figure 5A In some embodiments, B n may correspond to the equation used to compute the cost kernel, and e may be the equation for B n The determinant of the equation, i can be used to identify the n The determinant of the left side of the e determinant, j can be used to identify the n The determinant of the e determinant on the right. In some embodiments, based on B n Formula, at k=0, using 0, 5, 1 as i values, the sum of e_i corresponds to (1 0 0 0 0 ...) + (0 00 0 0 ...) + (0 1 0 0 0 0 ...) = (1 1 0 0 0 1 ...), and the result divided by the number of i (for example, 3) can be located at B n In some embodiments, based on B n Formula, at k=0, using 4, 6, 3, 2, 7 as j values, the sum of e_j corresponds to (0 0 0 0 1 0 0 0 ...)+(0 0 0 0 0 0 01 0 ...)+(0 0 0 1 0 0 0 0 ...)+(0 0 1 00 0 0 0 ...)+(0 0 1 00 0 0 0 ...)+(0 0 0 0 0 0 0 1 ...)=(0 0 1 1 1 0 1 1 ...), and the result divided by the number of j (e.g., 5) can be located at B n According to the instruction B nThe first cost kernel may be expressed as a kernel corresponding to the matrix represented by “(1 1 0 0 01 0 0 0 0..) / 3-(0 0 1 1 1 0 1 1 0 0..) / 5=(1 / 3 1 / 3 -1 / 5 -1 / 5-1 / 5 1 / 3 -1 / 5 -1 / 5 0 0..)”. For example, the second cost kernel may correspond to Figure 5A According to the indication B n The second cost kernel may be expressed as a kernel corresponding to the matrix represented by “(0 0 11 0 0 0 1 0 0..) / 3-(1 1 0 0 1 1 1 0 0 0..) / 5=(-1 / 5 -1 / 5 1 / 3 1 / 3 -1 / 5 -1 / 5 -1 / 5 0 0 0..)”. For example, the 17th cost kernel may correspond to Figure 5A According to the instruction B n , the 17th cost kernel can be expressed as the kernel corresponding to the matrix represented by “(0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 00 01..) / 3-(0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0..) / 4=(0 0 0 0 1 / 3 0 0 0 1 / 3 -1 / 4 -1 / 4 -1 / 4-1 / 4 0 0 0 0 0 0 0 1 / 3..)”.

[0063] In one example, each of the plurality of cost kernels may be multiplied by a weight. For example, a first cost kernel may be obtained by multiplying a value of "-1" by (1 / 3 1 / 3 -1 / 5 -1 / 5 -1 / 5 1 / 3 -1 / 5 -1 / 5 0 0..), and may be expressed as a kernel corresponding to a matrix represented by "(-1 / 3 -1 / 3 1 / 5 1 / 5 1 / 5 -1 / 3 1 / 5 1 / 5 0 0..)". For example, a second cost kernel may be obtained by multiplying a value of "0" by (-1 / 5 -1 / 5 1 / 3 1 / 3 -1 / 5 -1 / 5 -1 / 5 0 0 0..), and may be expressed as a kernel corresponding to a matrix represented by "(0 0 0 0 0 0 0 0 0 0 0..)". The weight value may be a preset value to reflect an element including at least one of a lens, a sensor, a color, an arrangement, an imbalance, a crosstalk, and the like.

[0064] Figure 5B is a diagram illustrating multiple cost kernels according to some embodiments of the present disclosure.

[0065] Reference Figure 5B , showing that Figure 5A Example of the cost kernel (500-523) corresponding to the formula of . Figure 5B The cost kernel shown can be expressed as a kernel corresponding to a (1×n) matrix, where “n” can correspond to the number of cells included in Figure 4B The directional component of Figure 4B In one example, the first cost kernel 500 may correspond to Figure 5A "k = 0" as depicted in the table of Figure 4B The number of directional components included in the directional component 450 is equal to Figure 4A The number of directions of the kernel corresponds to 26. Therefore, according to the instruction B n , the first cost kernel 500 may be expressed as a kernel corresponding to the matrix represented by “(1 / 31 / 3 -1 / 5 -1 / 5 -1 / 5 1 / 3 -1 / 5 -1 / 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0)”. In one example, the 17th cost kernel 516 may correspond to Figure 5A "k=16" as depicted in the table of Figure 4B The number of directional components included in the directional component 450 is equal to Figure 4A The number of directions of the kernel corresponds to 26. Therefore, according to the instruction B n , the 17th cost kernel can be expressed as the kernel corresponding to the matrix represented by “(0 0 0 0 1 / 3 0 0 0 1 / 3 -1 / 4 -1 / 4 -1 / 4 -1 / 4 0 0 0 0 0 0 0 1 / 3 00 0 0 0)”.

[0066] In one example, the first cost kernel 500 may correspond to extracting the first direction component 440 (see Figure 4B ), the second direction component 441 (see Figure 4B ) and the sixth directional component (not shown) as a positive average, while extracting the third to fifth directional components (442, etc.) (see Figure 4B ), the seventh direction component (not shown) and the eighth direction component (not shown) as the cost kernel of the negative average. The first cost kernel 500 may be multiplied by the first weight after being affected by the sensor characteristics, etc. In one example, the second cost kernel 501 may correspond to extracting the third direction component 442 (see Figure 4B ), the fourth direction component 443 (see Figure 4B ) and the eighth direction component (not shown) as a positive average, while extracting the first direction component 440 (see Figure 4B ), the second direction component 441 (see Figure 4B) and the fifth to seventh directional components (not shown) as the cost kernel of the negative average. The second cost kernel 501 may be multiplied by the second weight after being affected by the sensor characteristics and the like.

[0067] In addition, although in this embodiment, for the convenience of description, the cost kernel (e.g., 500) is implemented as a kernel corresponding to a (1×26) matrix, the scope or spirit of the disclosed technology is not limited thereto, and it should be noted that more diverse directional components can be reflected in a kernel having a larger size than the example kernel. In addition, although in this embodiment, 24 cost kernels are used for the convenience of description, when more cost kernels than the number of example cost kernels are used, the cost can be extracted by combining the directional components in a more diverse manner. Convolution operations based on some embodiments of the present disclosure can also be applied substantially in the same manner as these cost kernels.

[0068] Figure 5C It is a diagram showing some embodiments of the present disclosure for executing Figure 3 FIG. 1 is a diagram of the method of operation S120 shown.

[0069] Reference Figure 5C , the interpolation kernel determiner 122 may be configured to determine the interpolation kernel values ​​between the respective cost kernels (eg, 500, 501, 512, 523) and Figure 4B The interpolation kernel determiner 122 may perform a convolution operation between the first cost kernel 500 and the directional component 450, and may extract the cost (530-553). For example, in order to extract the cost (530-553) from the directional component 450, the interpolation kernel determiner 122 may extract the first cost 530 by performing a convolution operation between the first cost kernel 500 and the directional component 450. At this time, with respect to the directional component 450 and the first cost kernel 500, the interpolation kernel determiner 122 may perform a point-by-point convolution operation, wherein the directional component 450 is matched to the channel, and the first cost kernel 500 is matched to the unit kernel. For example, the convolution result value obtained by performing a convolution operation between the first cost kernel 500 and the directional component 450 may correspond to "5×(1 / 3)+7×(1 / 3)+3×(-1 / 5)+1×(-1 / 5)+1×(-1 / 5)+4×(1 / 3)+3×(-1 / 5)+0×(-1 / 5)+5×0+6×0..=3.7", and the data of the first cost 530 may be defined as "3.7".

[0070] In one example, in order to extract the cost (530-553) from the directional component 450, the interpolation kernel determiner 122 may extract the second cost 531 by performing a convolution operation between the second cost kernel 501 and the directional component 450. At this time, with respect to the directional component 450 and the second cost kernel 501, the interpolation kernel determiner 122 may perform a point-by-point convolution operation, wherein the directional component 450 is matched to the channel and the second cost kernel 501 is matched to the unit kernel. For example, the convolution result value obtained by performing the convolution operation between the second cost kernel 501 and the directional component 450 may correspond to "5×(-1 / 5)+7×(-1 / 5)+3×(1 / 3)+1×(1 / 3)+1×(-1 / 5)+4×(-1 / 5)+3×(-1 / 5)+0×(1 / 3)+5×0+6×0..=-2.7", and the data of the second cost 531 may be defined as "-2.7".

[0071] In one example, in order to extract the cost (530-553) from the directional component 450, the interpolation kernel determiner 122 may extract the 13th cost 542 by performing a convolution operation between the 13th cost kernel 512 and the directional component 450. At this time, with respect to the directional component 450 and the 13th cost kernel 512, the interpolation kernel determiner 122 may perform a point-by-point convolution operation, wherein the directional component 450 is matched to the channel and the 13th cost kernel 512 is matched to the unit kernel. For example, the convolution result value obtained by performing a convolution operation between the 13th cost kernel 512 and the directional component 450 may correspond to “5×0+7×0+3×0+1×0+1×0+4×0+3×(-1 / 5)+0×0+5×(1 / 2)+6×(-1 / 5)+3×(-1 / 5)+8×(-1 / 5)+2×(-1 / 5)+3×0+3×0+5×0+1×0+5×0+7×0+2×0+1×(1 / 2)+2×0+5×0+1×0=-1.4”, and the data of the 13th cost 542 may be defined as “-4.2”.

[0072] The interpolation kernel determiner 122 may extract the index value (eg, Figure 3S130 of ). The index value may correspond to the number of each cost kernel. For example, when the first cost 530 among the first to 24th costs (530-553) has a maximum value, the interpolation kernel determiner 122 may extract a value "1" as an index number by applying the function "Arg max" to the 24th cost 553, and may identify the first cost kernel 500. For example, when the 13th cost 542 among the first to 24th costs (530-553) has a minimum value, the interpolation kernel determiner 122 may extract a value 13 as an index number by applying the function "Arg min" to the first to 24th costs (530-553).

[0073] In some embodiments, the interpolation kernel determiner 122 may determine the interpolation kernel based on the value obtained by performing a convolution operation between each cost kernel (500-523) and the directional component 450. In one example, each cost kernel (500-523) may include a preset weight as data. In one example, the value obtained by performing a convolution operation between each cost kernel (500-523) and the directional component 450 may be a result value of a point-by-point convolution operation, in which the directional component 450 is matched to a channel and each cost kernel (500-523) is matched to a unit kernel. In one example, the interpolation kernel may be an interpolation kernel corresponding to a cost kernel that derives a maximum or minimum value among the convolution result values ​​obtained by performing a convolution operation between each cost kernel (500-523) and the directional component 450.

[0074] Fig. 6A is a diagram illustrating multiple interpolation kernels according to some embodiments of the present disclosure.

[0075] Reference Fig. 6A , showing an example of an interpolation kernel (600~623). C_l is a symbol for multiple interpolation kernels, for example, C_0 corresponds to the first interpolation kernel 600. The pixel interpolator 123 may perform a convolution operation between the target kernel and one of the interpolation kernels (600~623) so that the center of the interpolation kernel (600~623) corresponds to the target pixel (DP). In one example, the pixel interpolator 123 may perform a convolution operation between the target kernel and the interpolation kernel so that the center of the interpolation kernel corresponds to the target pixel (DP). In one example, the interpolation kernel used to interpolate the target pixel (DP) may correspond to an index value (for example, Figure 3S140). For example, the index value may correspond to the number of each cost kernel. One interpolation kernel may correspond to each cost kernel. For example, when the first cost 530 among the first to 24th costs (530-553) has a maximum value, the interpolation kernel determiner 122 may extract a value of 1 as an index number by applying the function "Arg max" to the first to 24th costs (530-553), and may determine the first interpolation kernel 600 corresponding to the first cost kernel 500 as the interpolation kernel corresponding to the target pixel (DP). For example, when the 13th cost 542 among the first to 24th costs (530-553) has a minimum value, the interpolation kernel determiner 122 may extract a value of 13 as an index number by applying the function "Arg min" to the first to 24th costs (530-553), and may determine the 13th interpolation kernel 612 corresponding to the 13th cost kernel 512 as the interpolation kernel corresponding to the target pixel (DP). Fig. 6A The interpolation kernel shown is only an example, and in this embodiment the kernel is a (3×3) kernel, but there may be more diverse interpolation kernels in kernels with a larger size than the example kernel. Interpolation operations based on some embodiments of the present disclosure may be applied in substantially the same manner as these interpolation kernels.

[0076] Figure 6B It is a diagram showing some embodiments of the present disclosure for executing Figure 3 FIG. 1 is a diagram of the method of operation S150 shown.

[0077] Reference Figure 6B , the pixel interpolator 123 may interpolate the defective pixel 630 by performing a convolution operation between the interpolation kernel 623 and the neighboring pixels 631 of the defective pixel 630. The pixel interpolator 123 may perform the convolution operation so that the center of the interpolation kernel corresponds to the target pixel (DP). For example, in order to interpolate the defective pixel 630, the pixel interpolator 123 may perform a convolution operation between the interpolation kernel 623 and the neighboring pixels 631 of the defective pixel 630, and may obtain interpolated pixel data 640 of the defective pixel 630. At this time, the convolution result value obtained by performing the convolution operation between the interpolation kernel 623 and the neighboring pixels 631 of the defective pixel 630 may correspond to "0×0+1×1+0×0+4×1+0×1+0×0+3×1+0×0=8", and the interpolated pixel data 640 of the defective pixel 630 may be defined as "8".

[0078] In one example, the number of neighboring pixels 631 of the defective pixel 630 used for convolution may vary according to the size of the interpolation kernel.

[0079] For example, when the interpolation kernel has a (5×5) kernel, the neighboring pixels of the defective pixel 630 to be used for convolution may correspond, in sequence, to “0, 3, 0, 3, 0, 2, 0, 1, 0, 5, 0, 4, 0, 0, 1, 0, 3, 0, 2, 0, 5, 0, 1, 0” arranged in sequence from the upper left side of the defective pixel 630, and may correspond to the size of the (5×5) kernel.

[0080] In some embodiments, the pixel interpolator 123 may enable the target pixel to correspond to data located at the center of the interpolation kernel. In addition, the pixel interpolator 123 may determine the convolution result value obtained by performing a convolution operation between the target kernel and the interpolation kernel as the pixel data of the target pixel.

[0081] Figure 7 Formulas showing the entire process of interpolating one pixel based on some embodiments of the present disclosure are illustrated.

[0082] Reference Figure 7 , the interpolated pixel data of the defective pixel according to an example can be derived. In the formula according to an example, "χ" can be defined as an input image, "a" can be defined as a directional kernel, "b" can be defined as a cost kernel, "w" can be defined as a weight kernel including weights as data, "y" can be defined as a cost, "c" can be defined as an interpolation kernel, and "o" can be defined as an output image. In the formula according to an example, "l" can be defined as a cost kernel number, "m" can be defined as a directional kernel position number, "n" can be defined as a directional kernel number, "t" can be defined as a cost kernel ID number (l) to be derived as a function "argmax{Y}", and (i,j) can be defined as pixel coordinates. In an example, "l" can be defined as a cost kernel identification (ID) number, and "m" can be defined as a directional kernel position number.

[0083] In one example, x i,j can be defined as the pixel data of the pixel located at coordinate (i, j) of the input image. In one example, the input image can correspond to the target kernel. In one example, when x i,j When the pixel corresponds to a defect, the image signal processor 100 may i,j is set to a value of "0". In one example, when x i,j When the pixel corresponds to a defect, the image signal processor 100 may not use x in the calculation process. i,j For example, x 1,2 can be defined as the pixel data of the pixel located in the first row and second column of the input image. Figure 4B , when the kernel including the target pixel 430 is the input image, x 1,1 can be defined as zero "0". For example, referring to Figure 4B , when the kernel including the target pixel 430 is the input image, the target pixel 430 may correspond to x 4,4 At this time, the image signal processor 100 can 4,4 The value of is set to zero "0". Alternatively, the image signal processor 100 may not use x in the calculation process. 4,4 The value of .

[0084] In one example, can be defined as the data at the coordinate (i, j) of the kernel in the nth direction. In one example, when x i,j When a defective pixel corresponds to a defective pixel, the image signal processor 100 may The value of is set to zero "0". In one example, when x i,j When a defective pixel corresponds to a defective pixel, the image signal processor 100 may not use For example, Can be defined as the data located at the first row and first column position of the kernel in the first direction. Return to reference Figure 4A , for example, the first direction kernel 400 may correspond to at this time, can be defined as the value "1". In addition, can be defined as zero "0". For example, the 24th direction kernel 423 can correspond to at this time, can be defined as the value "1". In addition, Can be defined as zero "0".

[0085] In one example, It can be defined as the convolution result value obtained by performing a convolution operation between the pixel data of the pixel located at the coordinate (i, j) and the data located at the coordinate (i, j) of the directional kernel. Return Reference Figure 4B ,For example, may correspond to a convolution result value obtained by performing a convolution operation between the first direction kernel 400 and the pixel data of the neighboring pixel 431 corresponding to the size of the first direction kernel 400 with respect to the target pixel 430 (i.e., 0×1+1×1+0×0+4×1+0×0+0×0+3×0+0×0=5), and It can be defined as the first direction component 440. For example, The convolution result value (ie, the value obtained by performing a convolution operation between the ninth direction kernel 408 and the pixel data of the neighboring pixel 432 corresponding to the size of the ninth direction kernel 408 with respect to the target pixel 430) may correspond to Figure 4B0×0+3×1+0×0+3×0+0×0+2×1+0×1+1×0+0×0+5×0+0×0+4×0+0×0+0×0+1×0+0×0+3×0+0×0+2×0+0×0+5×0+0×0+1×0+0×0=5), and Can be defined as a ninth directional component 448 .

[0086] In one example, It can be defined as the mth data of the lth cost kernel. For example, can be defined as the first cost kernel (e.g., Figure 5B Return to reference Figure 5B , for example, the first cost kernel 500 may correspond to at this time, can be defined as "1 / 3". In addition, can be defined as "-1 / 5". For example, Figure 5B In the example, the 24th cost kernel 523 may correspond to at this time, Can be defined as "-1 / 4". In addition, Can be defined as zero "0".

[0087] In one example, w mn It can be defined as the weight corresponding to the mth data of the cost kernel or the nth data of the direction component among the data of the weight kernel. 11 can be defined as the weight corresponding to the first data of the cost kernel and the first data of the direction component. Figure 5C , w 11 can be defined as a weight multiplied by a plurality of first data (i.e., 1 / 3, -1 / 5, (..), 0, (..), -1 / 4) of the first to 24th cost kernels (500-523). In one example, w 11 Can be defined as a weight multiplied by the first direction component 440. In some embodiments, for convenience of description, since each cost kernel (500-523) and the direction component 450 are subjected to a point-by-point convolution operation, the number of data bits (m) of the cost kernel is equal to the number of data bits (n) of the direction component. However, in other convolution operations other than the point-by-point convolution operation, "m" may be different from "n".

[0088] In one example, y l can be defined as the lth cost (cost #1). For example, y 1 It can be defined as the point-by-point convolution result value obtained by performing a point-by-point convolution operation between the first cost kernel 500 and the direction component 450. Return to reference Figure 5C , for example, y 1may correspond to a convolution (ie, point-by-point convolution) result value obtained by performing a convolution (ie, point-by-point convolution) operation between the first cost kernel 500 and the direction component 450, and y 1 can be defined as the first cost 530. For example, y 13 y may be defined as a point-by-point convolution result value obtained by performing a point-by-point convolution operation between the 13th cost kernel and the direction component 450. For example, y 13 The convolution (ie, point-by-point convolution) result value (ie, reference value) obtained by performing a convolution (ie, point-by-point convolution) operation between the 13th cost kernel 512 and the direction component 450 may correspond to the convolution (ie, point-by-point convolution) result value (ie, reference value Figure 5C 450 and 512, 5×0+7×0+3×0+1×0+1×0+4×0+3×(-1 / 5)+0×0+5×(1 / 2)+6×(-1 / 5)+3×(-1 / 5)+8×(-1 / 5)+2×(-1 / 5)+3×0+3×0+5×0+1×0+5×0+7×0+2×0+1×(1 / 2)+2×0+5×0+1×0=-1.4), and y 13 Can be defined as the 13th cost 542.

[0089] In one example, the index number may be defined as the l Acts as the result of applying the function "Argmax" or "Arg min" to the collection (or aggregate) of elements. For example, when y 1 To 24 The y 13 When the interpolation kernel determiner 122 has the maximum value, the interpolation kernel determiner 122 can 1 ~y 24 The collection (or aggregation) of y applies the function "Arg max" to extract "13" as the index number. For example, when y 1 To 24 The y 24 When y has the minimum value, the interpolation kernel determiner 122 can 1 ~y 24 Apply the function "Arg min" to extract "24" as the index number.

[0090] In one example, can be defined as the data at coordinates (i, j) of the t-th interpolation kernel, where "t" is defined as the index number extracted from the function "Arg max" or "Arg min". For example, Can be defined as the data located at the first row and first column position of the first interpolation kernel (ie, interpolation kernel #1). Return to reference Fig. 6A , for example, the first interpolation kernel 600 may correspond to at this time, Can be defined as zero "0", can be defined as a value of "1". For example, the 12th interpolation kernel 611 can correspond to at this time, can be defined as the value "1", Can be defined as the value "1".

[0091] In one example, "o" may be defined as an output image. "o" may be defined as a convolution result value obtained by performing a convolution operation between pixel data of a pixel located at coordinates (i, j) of an input image and data corresponding to coordinates (i, j) of an interpolation kernel. Return to Reference Figure 6B ,For example, may correspond to a convolution result value obtained by performing a convolution operation between the 24th interpolation kernel 623 and the pixel data of the neighboring pixel 631 corresponding to the size of the 24th interpolation kernel 623 (i.e., 0×0+1×1+0×0+4×1+0×1+0×0+3×1+0×0=8), and “o” may be defined as In one example, interpolation of defective pixels may be performed according to the order of general convolution operations. In more detail, one defective pixel may be interpolated according to the interpolation method of the present embodiment, and then the next defective pixel may be interpolated according to the interpolation method of the present embodiment.

[0092] Figure 8 is a diagram showing the entire process of interpolating one pixel according to some embodiments of the present disclosure.

[0093] Reference Figure 8 , when the target pixel 801 included in the target kernel 800 corresponds to a defective pixel, the target pixel 801 may be interpolated using the pixel data 815 interpolated according to the interpolation method of the present embodiment. After the defective pixel is interpolated according to the interpolation method of the present embodiment, the image signal processor 100 may change the target kernel and the target pixel. For example, after the target pixel 801 is interpolated using the interpolated pixel data 815, the image signal processor 100 may set the second target pixel 803, and may set the second target pixel 802 (including the second target pixel 803).

[0094] In one example, the target kernel 800 may include a target pixel 801. The target pixel 801 may be a pixel located at the center of the target kernel 800. The image signal processor 100 may extract a direction component 805 of the target kernel 800 to interpolate the target pixel 801. For example, the direction component extractor 121 may extract the direction component 805 by performing a convolution operation between the target kernel 800 and each of the first kernels 804. The first kernel 804 may include Figure 4AThe first to 26th directional kernels (400-425) of the target kernel 800 are obtained. For example, the directional component extractor 121 may extract the first directional component by performing a convolution operation between the target kernel 800 and the first directional kernel, and may extract the second directional component by performing a convolution operation between the target kernel 800 and the second directional kernel. In one example, the directional component extractor 121 may stack the directional components 805 in sequence, and may store the stacking results as data formed into a 3D stacked kernel shape. For example, when the number of directional kernels is "n", the directional component extractor 121 may store N directional components as data formed into a 3D stacked kernel shape.

[0095] In one example, the image signal processor 100 may extract the cost (e.g., 809, 810, 811) of the target kernel 800 to interpolate the target pixel 801. For example, the interpolation kernel determiner 122 may extract the cost (e.g., 809, 810, 811) by performing a convolution operation between each directional component 805 and each second kernel (e.g., 806, 807, 808). In another example, the second kernel may include Figure 5B For example, the interpolation kernel determiner 122 may extract the first cost 809 by performing a point-by-point convolution operation between the first cost kernel 806 and the direction component 805. In addition, the interpolation kernel determiner 122 may extract the second cost 810 by performing a point-by-point convolution operation between the second cost kernel 807 and the direction component 805.

[0096] In one example, the image signal processor 100 may determine an interpolation kernel (e.g., 812) corresponding to the target kernel 800 in order to interpolate the target pixel 801. For example, the interpolation kernel determiner 122 may extract an index number by applying a function "Arg max" or "Arg min" to the cost (e.g., 809, 810, 811). For example, when the third cost 811 among the extracted costs (e.g., 809, 810, 811) corresponds to the highest value and the interpolation kernel determiner 122 applies the function "Arg max" to the extracted costs (e.g., 809, 810, 811), the interpolation kernel determiner 122 may extract "3" as the index number corresponding to the third cost. At this time, the interpolation kernel determiner 122 may determine the third interpolation kernel 812 corresponding to the index number "3" as the interpolation kernel corresponding to the target kernel 800.

[0097] In one example, the image signal processor 100 may perform a convolution operation between the target kernel and the interpolation kernel 812 to interpolate the target pixel 801 . Figure 8The defective pixel 813 shown may be understood to correspond to the target pixel 801 included in the target kernel 800. For example, the pixel interpolator 123 may perform a convolution operation so that the center of the interpolation kernel 812 corresponds to the defective pixel 813. For example, in order to interpolate the defective pixel 813, the pixel interpolator 123 may perform a convolution operation between the interpolation kernel 812 and the neighboring pixels 814 of the defective pixel 813, and may obtain interpolated pixel data 815 of the defective pixel 813. At this time, the convolution result value obtained by performing the convolution operation between the interpolation kernel 812 and the neighboring pixels 814 of the defective pixel 813 may correspond to "0×0+2×0+0×0+1×1+2×1+0×0+3×0+0×0=3", and the interpolated pixel data 815 of the defective pixel 813 may be defined as "3".

[0098] In some embodiments, Figure 2 As shown, the image signal processor 100 may include a directional component extractor 121, an interpolation kernel determiner 122, and a pixel interpolator 123. The directional component extractor 121 may extract the directional component 805 of the target kernel by performing a first convolution operation between the target kernel including the target pixel and each first kernel. The interpolation kernel determiner 122 may determine the interpolation kernel based on the convolution result value obtained by performing a second convolution operation between each second kernel and the directional component 805. The pixel interpolator 123 may interpolate the target pixel using the data included in the interpolation kernel. In an example, the first convolution operation may refer to an operation to be performed in a state where the target pixel is matched to the data located at the center of each first kernel. In an example, the second convolution operation may refer to a point-by-point convolution operation in which the directional component 805 is matched to a channel and each second kernel is matched to a unit kernel. In an example, the interpolation kernel may refer to an interpolation kernel corresponding to a kernel that derives a maximum value or a minimum value among the kernels included in the second kernel that derive the second convolution result value.

[0099] Fig. 9 is a flow chart illustrating a method of interpolating a target kernel by classifying the kernels by color according to some embodiments of the present disclosure.

[0100] Reference Fig. 9 , the image signal processor 100 may divide the input image into pixels having a first color filter, pixels having a second color filter, and pixels having a third color filter, so that the target kernel may be classified into the first color kernel, the second color kernel, and the third color kernel.

[0101] For example, when pixels corresponding to different colors (e.g., red, green, blue, white) in the original image captured by the image sensing device are arranged according to the RGBW color pattern, the image signal processor 100 may classify the kernels of the original image into a red pixel (R pixel) kernel including pixels each having only a red color filter, a green pixel (G pixel) kernel including pixels each having only a green color filter, a blue pixel (B pixel) kernel including pixels each having a blue color filter, and a white pixel (W pixel) kernel including pixels each having a white color filter. For example, the image signal processor 100 may separate the W pixel kernel having the largest number of pixels from the RGB pixel kernel (S200).

[0102] In one example, the image signal processor 100 may interpolate defective pixels included in the W pixel kernel by using the W pixel kernel as the target kernel. At this time, the image signal processor 100 may determine the pixel data located at the position corresponding to the pixel with the RGB color filter among the W pixel kernel as zero "0". Alternatively, at this time, the image signal processor 100 may determine the pixel data of the directional kernel located at the position corresponding to the pixel with the RGB color filter among the W pixel kernel as zero "0". For example, in order to interpolate defective pixels included in the W pixel kernel by using the W pixel kernel as the target kernel, the image signal processor 100 may perform FIG. 4A to FIG. 8 The interpolation operation (S210) disclosed in .

[0103] In one example, the image signal processor 100 may perform interpolation of defective pixels included in the W pixel kernel by using the W pixel kernel as a target kernel, and may combine the interpolated W pixel kernel with the isolated RGB pixel kernel. For example, the image signal processor 100 may interpolate all defective pixels included in the W pixel kernel, may combine the interpolated W pixel kernel with the RGB pixel kernel separated in operation S200, and may generate an RGBW pixel kernel (S220). In one example, among the pixels included in the RGBW pixel kernel, a pixel having an R, G, or B color filter may include a defective pixel.

[0104] In one example, the image signal processor 100 may interpolate defective pixels included in the RGBW pixel kernel by using the RGBW pixel kernel as a target kernel. For example, in order to interpolate defective pixels included in the RGBW pixel kernel by using the RGBW pixel kernel as a target kernel, the image signal processor 100 may perform FIG. 4A to FIG. 8 The interpolation operation (S230) is shown.

[0105] In one example, the image signal processor 100 may use a directional component (eg, Figure 4B 450) data or costs (e.g. Figure 5C 530 to 533) and the defective pixels included in the RGB pixel kernel can be interpolated by using the RGB pixel kernel as the target kernel.

[0106] In one example, the image signal processor 100 may interpolate defective pixels included in the RGBW pixel kernel using the RGBW pixel kernel as a target kernel. In addition, the image signal processor 100 may generate a Bayer pattern image as an RGB pattern based on the interpolated RGBW pixel kernel. For example, in order to interpolate defective pixels included in the RGBW pixel kernel using the RGBW pixel kernel as a target kernel, the image signal processor 100 may perform FIG. 4A to FIG. 8 The interpolation operation shown is performed, and a Bayer pattern image as an RGB pattern can be generated based on the interpolated RGBW pixel kernel.

[0107] In some embodiments, the directional component extractor 121 may extract the directional component using pixel data of pixels each having a first color filter among pixels included in the target kernel. In one example, the directional component extractor 121 may determine the pixel data of pixels each having a color filter different from the first color filter among pixels included in the target kernel as zero "0". In one example, among the data included in the plurality of directional kernels, data corresponding to the position of the pixel having a color filter different from the first color filter may be set to zero "0".

[0108] In some embodiments, the image signal processing method may include: extracting a first directional component of the first color kernel by performing a convolution operation between a first color kernel having a first target kernel and each of a plurality of directional kernels; determining a first interpolation kernel based on the first directional component; and interpolating the first target pixel using data included in the first interpolation kernel. In one example, the image signal processing method may further include: generating an interpolation target kernel by mixing an interpolation first color kernel including the interpolated first target pixel with a second color kernel. In one example, the image signal processing method may include: extracting a second directional component of the interpolation target kernel by performing a convolution operation between an interpolation target kernel including a second target pixel and each of a plurality of directional kernels; determining a second interpolation kernel based on the second directional component; and interpolating the second target pixel using data included in the second interpolation kernel. The image signal processing method may further include: determining the first interpolation kernel based on a convolution result value obtained by performing a convolution operation between the first directional component and each cost kernel. In one example, the image signal processing method may further include: multiplying each cost kernel by a preset weight. In one example, an image signal processing method may include performing a point-by-point convolution operation, wherein a first direction component corresponds to a channel and each cost kernel corresponds to a unit kernel.

[0109] Fig.10 is shown with Figure 1 A block diagram of a computing device 1000 corresponding to an image signal processor.

[0110] Reference Fig.10 , the computing device 1000 may represent a computer program product for executing Figure 1 An embodiment of a hardware configuration of an operation of the image signal processor 100 is shown.

[0111] The computing device 1000 may be mounted on a chip independent of a chip on which an image sensing device is mounted. According to one embodiment, a chip on which an image sensing device is mounted and a chip on which the computing device 1000 is mounted may be implemented in one package (e.g., a multi-chip package (MCP)), but the scope of the disclosed technology is not limited thereto.

[0112] in addition, Figure 1 The internal configuration or arrangement of the image sensing device and the image signal processor 100 described in the foregoing may vary according to the implementation. For example, at least a portion of the image sensing device may be included in the image signal processor 100. Alternatively, at least a portion of the computing device 1000 may be included in the image sensing device. In this case, at least a portion of the computing device 1000 may be mounted together on a chip on which the image sensing device is mounted.

[0113] The computing device 1000 may include a processor 1010 , a memory 1020 , an input / output interface 1030 , and a communication interface 1040 .

[0114] Processor 1010 can process execution Figure 1 The processor 1010 may be data and / or instructions required for the operation of the components (110, 120) of the image signal processor 100 described in the specification. That is, the processor 1010 may refer to the image signal processor 100, but the scope of the disclosed technology is not limited thereto.

[0115] The memory 1020 may store data and / or instructions required to perform operations of the components (110, 120) of the image signal processor 100, and may be accessed by the processor 1010. For example, the memory 1020 may be a volatile memory (e.g., a dynamic random access memory (DRAM), a static random access memory (SRAM), etc.) or a non-volatile memory (e.g., a programmable read-only memory (PROM), an erasable PROM (EPROM), an EEPROM (electrically erasable PROM), a flash memory, etc.).

[0116] That is, a computer program for executing the operation of the image signal processor 100 disclosed in this document is recorded in the memory 1020 , and is executed and processed by the processor 1010 , thereby realizing the operation of the image signal processor 100 .

[0117] The input / output interface 1030 is an interface that connects an external input device (eg, keyboard, mouse, touch panel, etc.) and / or an external output device (eg, display) to the processor 1010 to allow transmission and reception of data.

[0118] The communication interface 1040 is a component that can transmit and receive various data with an external device (eg, an application processor, an external memory, etc.), and may be a device that supports wired or wireless communication.

[0119] It is obvious from the above description that the image signal processor and the image signal processing method based on some embodiments of the present disclosure can analyze the directional components included in the target kernel, and can interpolate the target kernel based on the interpolation kernel corresponding to the target kernel, so that even when multiple directional components are included in the target kernel, the correction accuracy of defective pixels can be increased.

[0120] The embodiments of the present disclosure may provide various effects that can be recognized directly or indirectly.

[0121] Although a number of exemplary embodiments have been described, it should be understood that modifications and enhancements to the disclosed embodiments and other embodiments may be envisioned based on what is described and / or shown in this disclosure. Therefore, the scope of the present disclosure should not be limited to the above-described embodiments, but should include their equivalents.

[0122] In the above-mentioned embodiments, all operations may be selectively performed, or some operations may be omitted. In various embodiments, the operations may not be performed in the order described, and may be rearranged. The embodiments disclosed in this specification and the accompanying drawings are merely examples to facilitate understanding of the present disclosure, and the present disclosure is not limited thereto. That is, it should be apparent to those skilled in the art that various modifications may be made based on the technical scope of the present disclosure.

[0123] Embodiments of the present disclosure are described in the drawings and the specification. Although specific terms are used here, these terms are only for describing the embodiments of the present disclosure. Therefore, the present disclosure is not limited to the above-mentioned embodiments, and many changes can be made within the scope of the present disclosure. It should be obvious to those skilled in the art that, in addition to the embodiments disclosed herein, various modifications can be made based on the technical scope of the present disclosure. In addition, the embodiments can be combined to form additional embodiments.

[0124] CROSS-REFERENCE TO RELATED APPLICATIONS

[0125] This patent application claims priority to and the benefit of Korean Patent Application No. 10-2023-0123383, filed on September 15, 2023, the disclosure of which is incorporated herein by reference in its entirety.

Claims

1. An image signal processor, comprising: a directional component extractor that extracts a directional component of a target kernel by performing a convolution operation between the target kernel and each of a plurality of directional kernels, the target kernel including the target pixel; an interpolation kernel determiner that determines an interpolation kernel based on the direction component; as well as A pixel interpolator interpolates the target pixel using the data included in the interpolation kernel.

2. The image signal processor according to claim 1, wherein: The interpolation kernel determiner obtains a convolution result value by performing a convolution operation between each of a plurality of cost kernels and the direction component, and determines the interpolation kernel based on the convolution result value.

3. The image signal processor according to claim 2, wherein: Each of the plurality of cost kernels includes a preset weight.

4. The image signal processor according to claim 2, wherein: The convolution result value includes a result value obtained by performing a point-by-point convolution operation in which the direction component is matched to a channel and each of the plurality of cost kernels is matched to a unit kernel.

5. The image signal processor according to claim 2, wherein: The interpolation kernel includes an interpolation kernel corresponding to a cost kernel that derives a maximum value or a minimum value among the convolution result values.

6. The image signal processor according to claim 1, wherein: The directional component extractor extracts the directional component by using pixel data of each pixel having a first color filter among pixels included in the target kernel.

7. The image signal processor according to claim 6, wherein: The directional component extractor determines pixel data of each pixel having a color filter different from the first color filter among pixels included in the target kernel as a specific value.

8. The image signal processor according to claim 6, wherein: The directional component extractor determines data corresponding to positions of respective pixels having a color filter different from the first color filter among the data included in the plurality of directional kernels as a specific value.

9. The image signal processor according to claim 1, wherein: The target kernel has the same size as a maximum directional kernel among the plurality of directional kernels.

10. The image signal processor according to claim 1, wherein: The pixel interpolator: matching the data located at the center of the interpolation kernel to the target pixel; and A convolution result value between the target kernel and the interpolation kernel is determined as pixel data of the target pixel.

11. An image signal processor, the image signal processor comprising: a directional component extractor that extracts a directional component of a target kernel by performing a first convolution operation between the target kernel and each of a plurality of first kernels, the target kernel including the target pixel; an interpolation kernel determiner that obtains a convolution result value by performing a second convolution operation between each of a plurality of second kernels and the direction component, and determines an interpolation kernel based on the convolution result value; as well as A pixel interpolator interpolates the target pixel using the data included in the interpolation kernel.

12. The image signal processor according to claim 11, wherein: The first convolution operation is performed in a state where the target pixel is matched to data located at the center of each of the plurality of first kernels.

13. The image signal processor according to claim 11, wherein: The second convolution operation is a point-by-point convolution operation in which the directional component is matched to a channel and each of the plurality of second kernels is matched to a unit kernel.

14. The image signal processor according to claim 11, wherein: The interpolation kernel refers to an interpolation kernel corresponding to a kernel that derives a maximum value or a minimum value among the convolution result values ​​obtained through the second convolution operation, among kernels included in the plurality of second kernels.

15. A method for processing an image signal, the method comprising the following steps: extracting a first directional component of a first color kernel by performing a convolution operation between the first color kernel and each of a plurality of directional kernels, the first color kernel including the first target pixel; determining a first interpolation kernel based on the first direction component; as well as The first target pixel is interpolated using data included in the first interpolation kernel.

16. The image signal processing method according to claim 15, further comprising the following steps: An interpolated target kernel is generated by blending an interpolated first color kernel with a second color kernel, the interpolated first color kernel including an interpolated first target pixel.

17. The image signal processing method according to claim 16, further comprising the following steps: extracting a second directional component of the interpolation target kernel by performing a convolution operation between each of the directional kernels and the interpolation target kernel including the second target pixel; determining a second interpolation kernel based on the second direction component; as well as The second target pixel is interpolated using the data included in the second interpolation kernel.

18. The image signal processing method according to claim 15, wherein: The step of determining the first interpolation kernel comprises the following steps: obtaining a convolution result value by performing a convolution operation between each of a plurality of cost kernels and the first direction component, and The first interpolation kernel is determined based on the convolution result value.

19. The image signal processing method according to claim 18, wherein: The step of determining the first interpolation kernel further comprises the following steps: Each of the plurality of cost kernels is multiplied by a preset weight.

20. The image signal processing method according to claim 18, wherein: The step of obtaining the convolution result value comprises the following steps: A point-wise convolution operation is performed by matching the first direction component to a channel and matching each of the plurality of cost kernels to a unit kernel.

Images