Image signal processor and image signal processing method

Abstract

The invention relates to an image signal processor and an image signal processing method. An image signal processor includes a first determiner and a pixel interpolator. A first determiner compares a target gradient combination of a target kernel including a target pixel with a reference gradient combination of each of a plurality of angle patterns to calculate a similarity score, and determines at least one angle pattern based on the calculated similarity scores. A pixel interpolator interpolates a target pixel using pixel data of an interpolated pixel determined based on the angle pattern.

Description

Technical Field

[0001] The techniques and implementations disclosed in this patent document generally relate to an image signal processor capable of performing image conversion and an image signal processing method thereof. Background Art

[0002] Image sensing devices are devices that capture optical images by converting light into electrical signals using photosensitive semiconductor materials that react to light. With the development of the automotive, medical, computer, and communications industries, there is an increasing demand for high-performance image sensing devices in various fields such as smartphones, digital cameras, game consoles, the Internet of Things (IoT), robots, surveillance cameras, and medical micro cameras.

[0003] The original image captured by the image sensing device may include a plurality of pixels corresponding to different colors (e.g., red, blue, and green). The plurality of pixels included in the original image may be arranged according to a specific arrangement of a color pattern (e.g., a Bayer pattern arrangement). In order to convert the original image into a complete image (e.g., an RGB image), an operation of interpolating pixels may be performed according to a predetermined algorithm. Since the algorithm basically includes an operation of interpolating pixels having missing (or omitted) information using information of adjacent pixels, severe noise may appear in an image having a specific pattern due to limitations of the algorithm. Summary of the invention

[0004] According to an embodiment of the disclosed technology, an image signal processor may include: a first determiner, which is configured to compare a target gradient combination including a target kernel of a target pixel with a reference gradient combination of each of a plurality of angular patterns to calculate a similarity score, and determine at least one angular pattern based on the calculated similarity score; and a pixel interpolator, which is configured to interpolate the target pixel using pixel data of an interpolated pixel determined based on the angular pattern.

[0005] According to another embodiment of the disclosed technology, an image signal processing method may include: comparing a target gradient combination of a target kernel including a target pixel with a reference gradient combination of each of a plurality of angular patterns to calculate a similarity score; determining an angular pattern corresponding to the target kernel based on the similarity score; and interpolating the target pixel using pixel data of an interpolated pixel determined based on the angular pattern. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The above and other features and advantageous aspects of the disclosed technology will become apparent with reference to the following detailed description when considered in conjunction with the accompanying drawings.

[0007] Figure 1 is a block diagram illustrating an example of an image signal processor based on some implementations of the disclosed technology.

[0008] Figure 2 Some implementations based on the disclosed technology are shown Figure 1 A block diagram of an example of a defective pixel corrector is shown.

[0009] Figure 3 is a schematic diagram showing examples of angular patterns based on some implementations of the disclosed technology.

[0010] Figure 4 Some implementations based on the disclosed technology are shown Figure 2 A flow chart of example operations of a pattern determiner is shown.

[0011] Figure 5 Some implementations based on the disclosed technology are shown Figure 4 FIG. 1 is a flowchart of an example of a detailed operation of operation S100 .

[0012] Fig. 6A and Figure 6B Some implementations based on the disclosed technology are shown Figure 5 FIG. 1 is a schematic diagram of an example of a plurality of pixel pairs used in the reference gradient combination of operation S200 .

[0013] Figure 7 Some implementations based on the disclosed technology are shown Figure 5 FIG. 2 is a schematic diagram of an example of detailed operation of operation S200 .

[0014] Figure 8 Some implementations based on the disclosed technology are shown Figure 4 FIG. 1 is a flowchart of an example of a detailed operation of operation S110 .

[0015] Fig.9A and Fig. 9B Some implementations based on the disclosed technology are shown Figure 8 FIG. 1 is a schematic diagram of an example of pixel combination used in the complex gradient combination of operation S300 .

[0016] Fig.10 Some implementations based on the disclosed technology are shown Figure 4 FIG. 1 is a flowchart of an example of a detailed operation of operation S120 .

[0017] Fig.11A and Fig. 11B Some implementations based on the disclosed technology are shown Fig.10 A schematic diagram of an example of pixel combination used in the Laplacian filter of operation S400.

[0018] Fig.12Some implementations based on the disclosed technology are shown Figure 4 FIG. 1 is a schematic diagram of an example of a detailed operation of operation S140 .

[0019] Fig.13 Some implementations based on the disclosed technology are shown Figure 4 FIG. 1 is a schematic diagram of an example of a detailed operation of operation S130 .

[0020] Fig.14 is a diagram showing some implementations based on the disclosed technology. Figure 1 A block diagram of an example of a computing device corresponding to an image signal processor. DETAILED DESCRIPTION

[0021] This patent document provides implementations and examples of image signal processors and image signal processing methods capable of improving the correction accuracy of defective pixels, etc., and their configurations can be used to fully solve one or more technical problems or engineering problems, and alleviate limitations or shortcomings encountered in some image signal processors in the field. Some implementations of the disclosed technology relate to image signal processors and image signal processing methods capable of improving the correction accuracy of defective pixels, etc. Recognizing the above problems, image signal processors based on some implementations of the disclosed technology can interpolate target pixels by determining an angular pattern corresponding to a target kernel, and can improve the correction accuracy of defective pixels even when the target kernel corresponds to the angular pattern.

[0022] Reference will now be made in detail to some embodiments of the disclosed technology, examples of which are shown in the accompanying drawings. In all drawings, the same reference numerals are used to represent the same or similar parts as much as possible. Although the present disclosure is susceptible to various modifications and alternative forms, its specific embodiments are shown in the accompanying drawings by way of example. However, the present disclosure should not be construed as being limited to the embodiments set forth herein.

[0023] In the following, various embodiments will be described with reference to the accompanying drawings. However, it should be understood that the disclosed technology is not limited to a specific embodiment, but includes various modifications, equivalents and / or alternatives of the embodiments. The embodiments of the disclosed technology can provide various effects that can be directly or indirectly recognized by the disclosed technology.

[0024] Various embodiments of the disclosed technology relate to an image signal processor and an image signal processing method thereof that can improve the correction accuracy of defective pixels and the like.

[0025] It is to be understood that both the foregoing general description and the following detailed description of the disclosed technology are illustrative and explanatory and are intended to provide further explanation of the disclosure as claimed.

[0026] Figure 1is a block diagram illustrating an example of an image signal processor 100 based on some implementations of the disclosed technology. Figure 2 Some implementations based on the disclosed technology are shown Figure 1 A block diagram of an example of a defective pixel corrector is shown. Figure 3 is a schematic diagram showing examples of angular patterns based on some implementations of the disclosed technology.

[0027] Reference Figure 1 , the image signal processor (ISP) 100 may perform at least one image signal processing on the image data (IDATA) to generate processed image data (IDATA_P). The image signal processor 100 may reduce the noise of the image data (IDATA), and may perform various image signal processing (e.g., demosaicing, defective pixel correction, gamma correction, color filter array interpolation, color matrix, color correction, color enhancement, lens distortion correction, etc.) to improve the image quality of the image data. In addition, the image signal processor 100 may compress the image data created by performing the image signal processing for image quality improvement, so that the image signal processor 100 can create an image file using the compressed image data. Alternatively, the image signal processor 100 may restore the image data from the image file. In this case, the scheme for compressing such image data may be a reversible format or an irreversible format. As a representative example of such a compression format, in the case of using a still image, a Joint Photographic Experts Group (JPEG) format, a JPEG 2000 format, etc. may be used. In addition, in the case of using a moving image, a plurality of frames may be compressed according to the Moving Picture Experts Group (MPEG) standard, so that a moving image file can be created.

[0028] The image data (IDATA) may be generated by an image sensing device that captures an optical image of a scene, but the scope of the disclosed technology is not limited thereto. The image sensing device may include a pixel array, a control circuit, and a readout circuit. The pixel array may include a plurality of pixels configured to sense incident light provided from a scene. The control circuit may be configured to control the pixel array to generate an analog pixel signal. The readout circuit may be configured to output digital image data (IDATA) by converting the analog pixel signal into digital image data (IDATA). In some implementations of the disclosed technology, it is assumed that the image data (IDATA) is generated by an image sensing device.

[0029] The pixel array of the image sensing device may include at least one defective pixel that cannot normally capture a color image due to process limitations or temporary noise inflow. In addition, the pixel array may include at least one phase difference detection pixel. As is well known, the phase difference detection pixel can be configured to obtain phase difference related information to improve autofocus performance. For example, the phase difference detection pixel may include a shape different from the pixel. The phase difference detection pixel cannot sense the color image in the same way as the defective pixel, so that from the perspective of the color image, the phase difference detection pixel can be regarded as a defective pixel. In some implementations, for ease of description and better understanding of the disclosed technology, defective pixels and phase difference detection pixels that each cannot normally sense the color image will be collectively referred to as "defective pixels" hereinafter.

[0030] In order to improve the quality of color images, the image signal processor 100 is required to accurately correct defective pixels. To this end, based on some implementations of the disclosed technology, the image signal processor 100 may include a defective pixel detector 200 and a defective pixel corrector 300.

[0031] In an exemplary embodiment, the defective pixel detector 200 can detect pixel data of defective pixels among multiple pixels of a pixel array based on image data (IDATA). In some implementations of the disclosed technology, for ease of description, digital data corresponding to the output signal of each pixel (hereinafter referred to as a pixel signal) will be defined as pixel data hereinafter, and a group of pixel data corresponding to a predetermined unit (e.g., a frame or a kernel) will be defined as image data (IDATA) hereinafter. Here, a frame may correspond to an entire pixel array including a plurality of pixels. A kernel may refer to a unit for image signal processing. For example, a kernel may refer to a group of pixels on which image signal processing is performed at one time. In addition, the actual value of the pixel data may be defined as a "pixel value".

[0032] In some implementations, the defective pixel detector 200 may detect pixel data of a defective pixel based on the image data (IDATA). For example, the defective pixel detector 200 may compare the pixel data of the target pixel with an average value of pixel data of pixels in the kernel (hereinafter, the average value of pixel data of the kernel). The defective pixel detector 200 may determine whether the target pixel is a defective pixel based on a difference between the pixel data of the target pixel and the average value of pixel data of the kernel. For example, when the difference is equal to or greater than a threshold value, the defective pixel detector 200 may determine that the target pixel is a defective pixel.

[0033] The threshold value described later may be a fixed constant, or may be a specific ratio of the brightness value of the kernel (e.g., green average value). In addition, in some implementations, it is assumed that when a measurement value (e.g., a difference value) is compared with a threshold value, if the measurement value is equal to or greater than the threshold value, then for ease of description, the condition is hereinafter represented by "large", and if the measurement value is less than the threshold value, then for ease of description, the condition is hereinafter represented by "small".

[0034] In some other embodiments, the defective pixel detector 200 may receive pre-stored position information of a defective pixel obtained based on a previous process or a pixel test process for correcting a defective pixel. In addition, the defective pixel detector 200 may determine whether the target pixel is a defective pixel based on the position information of the defective pixel. For example, the image sensing device may determine the position information of an inherent defective pixel as the position information of the defective pixel. In addition, the image sensing device may store the position information of the defective pixel in an internal storage unit (e.g., a one-time programmable (OTP) memory), and may provide the position information of the defective pixel to the image signal processor 100.

[0035] When the defective pixel detector 200 determines the target pixel as a defective pixel, the defective pixel corrector 300 may correct the pixel data of the target pixel based on the image data including the kernel of the target pixel.

[0036] Reference Figure 2 , the defective pixel corrector 300 may include an angle pattern determiner 310 and a pixel interpolator 350 .

[0037] The angle pattern determiner 310 may determine an angle pattern corresponding to a target kernel including a target pixel.

[0038] The image data IDATA corresponding to one kernel (or one frame) may include textures of various sizes and shapes. Texture may refer to a group (or set) of pixels having similarity. For example, an object having a uniform color included in a captured scene may be identified as a texture. The boundary of a texture may include at least one corner. For example, pixels inside a corner (hereinafter referred to as internal pixels) and pixels outside a corner (hereinafter referred to as external pixels). The difference between the pixel data of an internal pixel and the pixel data of an external pixel may be greater than the difference between the pixel data of adjacent pixels not restricted by the corner.

[0039] exist Figure 3, a corner pattern is shown as an example. A corner may refer to two line segments located on a horizontal line and a vertical line passing through a kernel and contacting each other at an intersection where the two lines intersect each other. In this way, a pattern in which pixels included in a kernel are distinguished from each other based on an angle used as a boundary may be defined as a corner pattern. For example, a first line of the two lines may be a vertical line that divides the kernel in a vertical direction, and a second line of the two lines may be a horizontal line that divides the kernel in a horizontal direction. The first line may intersect with the second line. For example, a corner may contact all of the first line, the second line, and the intersection. A corner pattern may be defined as at least one pixel that forms the same texture as a pixel located at a corner.

[0040] In this case, when a target pixel of the kernel is a defective pixel, the image signal processor 100 may correct the target pixel DP based on pixel data of adjacent pixels arranged according to an angular pattern.

[0041] Reference Figure 3 , the kernels K1 to K8 may include first to twenty-fifth pixels (P1 to P25) arranged in a 5×5 matrix form. For example, the target pixel DP may be located at the center of the kernels K1 to K8, respectively.

[0042] also, Figure 3 The shadow pixels may refer to pixels constituting an angle pattern. Here, the pixel data of the target pixel DP may represent normal color pixel data that may be obtained when the target pixel DP is a normal pixel.

[0043] Reference Figure 3 , examples of corner patterns each including two corners are shown as represented by PT_A, PT_B, PT_C, and PT_D, respectively. Figure 3 In each of PT_A to PT_D of FIG. 1 , various types of corner patterns each including two corners contacting each other at one vertex of the target pixel DP are shown. In addition, examples of corner patterns each including one corner are shown as represented by PT_E to PT_H. Figure 3 In each of PT_E to PT_H, various corner patterns each including the target pixel DP are shown, so that the corner patterns share one vertex of the target pixel DP in the target kernel as a vertex of the corner. Figure 3 The corner patterns shown in are disclosed for illustration purposes only, and there may be various corner patterns each filled with a textured area and a non-textured area, which are distinguished from each other based on horizontal and vertical lines used as boundaries when passing through the kernel. That is, the size and shape of the textured area may vary. Therefore, the position and number of corner patterns may also vary.

[0044] In more detail, kernels K1 to K8 may include at least one corner pattern PT_A to PT_H. For example, kernels K1 to K4 may include two corner patterns PT_A to PT_D based on the intersection of the corners. Kernels K5 to K8 may include one corner pattern PT_E to PT_H. Corner patterns PT_A, PT_D, PT_E, PT_F, PT_G, and PT_H may include a target pixel DP located at each vertex thereof. In addition, the target pixel DP may not be located within corner patterns PT_B and PT_C.

[0045] Although for ease of description, the embodiments of the disclosed technology assume that there are eight corner patterns in the (5×5) kernel, other implementations may also be used, and it should be noted that more diverse corner patterns may exist in kernels larger than the (5×5) kernel as needed. The defective pixel correction method based on some embodiments of the disclosed technology may also be applied to these corner patterns in substantially the same manner.

[0046] Return to reference Figure 2 , the angle pattern determiner 310 may include a first determiner 320 , a second determiner 330 , and a third determiner 340 .

[0047] The first determiner 320 may compare the target gradient combination with the reference gradient combination. The target gradient combination may be associated with a target kernel including the target pixel DP. The reference gradient combination may be associated with each angular pattern. The first determiner 320 may generate a similarity score through the comparison result. In addition, the first determiner 320 may determine the angular pattern corresponding to the target kernel based on the similarity score.

[0048] In some implementations, the reference gradient combination lists a reference gradient, which is a result of comparing a difference between pixel data of each of a plurality of pixel pairs with a predetermined threshold. Here, the reference gradient may indicate that the reference gradient corresponds to a plurality of pixel pair arrangements. For example, a pixel pair may be two pixels of the same color but located at different positions in the corner pattern.

[0049] In some implementations, the target gradient combination lists the target gradients. The target gradients may be arranged corresponding to a plurality of pixel pairs. Here, the target gradients may be obtained by using the positions of the pixels of the plurality of pixel pairs used in the reference gradient combination without change, by comparing the difference between the pixel data of the pixels of each of the plurality of pixel pairs with a predetermined threshold value based on the image data (IDATA) input to the target kernel. That is, the positions of the pixel pairs in the target kernel may be the same as the positions of the pixel pairs used to obtain the reference gradient combination. The plurality of pixel pairs will be described in more detail below.

[0050] When the first determiner 320 determines a plurality of angular patterns, the second determiner 330 may determine an angular pattern corresponding to the target kernel based on a complex gradient combination corresponding to each of the angular patterns determined by the first determiner 320 with respect to the target kernel.

[0051] In some implementations, the complex gradient combination may refer to a comparison result between average values ​​of pixel data difference values ​​between two pixels assigned to each pixel combination.

[0052] When the second determiner 330 determines a plurality of angular patterns, the third determiner 340 may determine an angular pattern corresponding to the target kernel based on a result of applying a Laplacian filter corresponding to each of the angular patterns determined by the second determiner 330 to the target kernel.

[0053] In some implementations, for each pixel combination, the Laplacian filter may represent a difference between a value obtained by doubling pixel data of one pixel among three pixels and a value corresponding to a sum of pixel data values ​​of the remaining two pixels except the one pixel.

[0054] When the angle pattern determiner 310 determines an angle pattern corresponding to the target kernel, the pixel interpolator 350 may interpolate the target pixel DP using the pixel data of the pixel based on the angle pattern determined by the angle pattern determiner 310 .

[0055] The detailed operation of the defective pixel corrector 300 will be described later with reference to the accompanying drawings. Figure 4 Give a description.

[0056] Figure 4 Some implementations based on the disclosed technology are shown Figure 2 A flow chart of example operations of a pattern determiner is shown. Figure 5 Some implementations based on the disclosed technology are shown Figure 4 FIG. 1 is a flowchart of an example of a detailed operation of operation S100 . Fig. 6A and Figure 6B Some implementations based on the disclosed technology are shown Figure 5 FIG. 1 is a schematic diagram of an example of a plurality of pixel pairs used in the reference gradient combination of operation S200 . Figure 7 Some implementations based on the disclosed technology are shown Figure 5 FIG. 2 is a schematic diagram of an example of detailed operation of operation S200 .

[0057] Reference Figure 4, the defective pixel corrector 300 may determine the type of the angle pattern by determining whether a target kernel including the target pixel DP satisfies a predetermined condition, and may interpolate the target pixel DP using an interpolation method corresponding to the determined type of the angle pattern.

[0058] exist Figures 4 to 13 In the example of a method of correcting defective pixels using a (5×5) kernel indicating a Bayer pattern will be described in detail below as an example. For example, the Bayer pattern may be a (5×5) kernel including blue pixels (B), red pixels (R), and green pixels (G). The red pixel (R) may generate red pixel data by detecting red light, the green pixel (G) may generate green pixel data by detecting green light, and the blue pixel (B) may generate blue pixel data by detecting blue light. In addition, it is assumed that each of the red pixel data, the green pixel data, and the blue pixel data has a range of 0 to 1023.

[0059] Although for ease of description, the embodiment of the disclosed technology has disclosed that the image is a (5×5) size kernel with 25 pixels arranged in a Bayer pattern, the technical idea of ​​the disclosed technology can also be applied to another kernel in which color pixels are arranged in other patterns such as a four-Bayer pattern, a nine-Bayer pattern, a six-Bayer pattern, an RGBW pattern, a monochrome pattern, etc., and the type of image pattern is not limited thereto and can also be fully changed as needed. In addition, according to the performance of the image signal processor 100, the required correction accuracy, the arrangement method of the color pixels, etc., a kernel having another size (for example, a (10×10) size) different from the (5×5) size can be used.

[0060] The first determiner 320 may compare the reference gradient combination with the target gradient combination of the target kernel, wherein the target pixel DP determined as a defective pixel by the defective pixel detector 200 is located at the center of the target kernel (S100). The first determiner 320 may determine at least one angular pattern corresponding to the target kernel based on the comparison result.

[0061] When the first determiner 320 determines that there is only one corner pattern corresponding to the target kernel (for example, the number of corner patterns in S100 = odd), the pixel interpolator 350 can interpolate the target pixel DP based on the pixel data of some pixels having the same color as the target pixel DP and being included in the same area as the texture area or non-texture area of ​​the corner pattern including the target pixel DP (S140). For example, when the corner pattern includes the target pixel DP as a texture area and the target pixel DP is a blue pixel, the pixel interpolator 350 can set the average value of the pixel data of the blue pixels located in the texture area of ​​the corner pattern belonging to the target kernel as the pixel data of the target pixel DP.

[0062] When the first determiner 320 determines that there are a plurality of angular patterns corresponding to the target kernel (e.g., the number of angular patterns in S100 = a complex number), the second determiner 330 may calculate a complex gradient combination of the target kernel to determine whether the number of angular patterns corresponds to the target kernel (S110). For example, the second determiner 330 may determine whether the number of angular patterns of the target kernel is 1 based on the complex gradient combination.

[0063] When the second determiner 330 determines that there is only one angular pattern corresponding to the target kernel (eg, the number of angular patterns in S110 = odd), the pixel interpolator 350 may interpolate the target pixel DP in the same manner as before ( S140 ).

[0064] When the second determiner 330 determines that there are a plurality of angular patterns corresponding to the target kernel (e.g., the number of angular patterns in S110=plural number), the third determiner 340 may apply a Laplacian filter to the target kernel, wherein the target pixel DP determined as a defective pixel by the defective pixel detector 200 is located at the center of the target kernel (S120). The third determiner 340 may determine whether the number of angular patterns corresponding to the target kernel is 1.

[0065] When the third determiner 340 determines that there is only one angular pattern corresponding to the target kernel (eg, the number of angular patterns in S120 = odd), the pixel interpolator 350 may interpolate the target pixel DP in the same manner as before ( S140 ).

[0066] When the third determiner 340 determines that there are a plurality of angular patterns corresponding to the target kernel (e.g., the number of angular patterns in S120 = a plurality), it can be seen that there is no angular pattern corresponding to the target kernel, so that the pixel interpolator 350 can interpolate the target pixel DP based on the pixel data of the pixel having the same color as the target pixel DP belonging to the target kernel (S130). For example, when the target pixel DP is a blue pixel, the pixel interpolator 350 can determine the average value of the pixel data of all blue pixels belonging to the target kernel as the pixel data of the target pixel DP.

[0067] Reference Figure 5 , the first determiner 320 may calculate a similarity score by comparing the reference gradient combination with the target gradient combination of the target kernel (S200). The similarity score may refer to the number of pixel pairs where the target gradient and the reference gradient are identical to each other by comparing the target gradient combination with the reference gradient combination based on each of the pixel pairs. For example, the similarity score may be set by comparing the target gradient of the pixel pair with the reference gradient of the pixel pair. The number of pixel pairs where the target gradient and the reference gradient match each other may be the similarity score.

[0068] The first determiner 320 may determine an angular pattern corresponding to the target kernel based on the similarity score. In addition, when the number of angular patterns corresponding to the target kernel is 1 based on the similarity score (i.e., "yes" in S210), the first determiner 320 may send pixel data of the angular pattern including the target pixel DP to the pixel interpolator 350. The target pixel DP may be located at a texture area or a non-texture area of ​​the angular pattern.

[0069] For example, when the target pixel DP is located at the texture area of ​​the corner pattern, the first determiner 320 may transmit pixel data of the pixel of the texture area to the pixel interpolator 350 .

[0070] When it is determined based on the similarity score that the number of angle patterns corresponding to the target kernel is plural (ie, “No” in S210 ), the first determiner 320 may transmit identification information for identifying the angle pattern to the second determiner 330 .

[0071] For example, when the angle patterns are PT_A, PT_B, PT_C, and PT_D, the first determiner 320 may transmit identification information of the angle patterns PT_A, PT_B, PT_C, and PT_D to the second determiner 330. The second determiner 330 may select at least one of the angle patterns PT_A, PT_B, PT_C, PT_D as a candidate angle pattern.

[0072] Fig. 6A is a schematic diagram showing an example of a plurality of pixel pairs (1) to (22) to be used in the reference gradient combination. When the reference gradient combination is calculated for an arbitrary kernel, the difference between the pixel data of the pixels of each pixel pair can be calculated based on the image data (IDATA) input to the kernel by using the positions of the pixel pairs used in the reference gradient without change. Therefore, as Fig. 6A As shown in FIG. 1 , 22 results may be listed. For each kernel, the input image data (IDATA) may be different. Therefore, even when the reference gradient combination obtained using the same pixel pair is used, the result of calculating the reference gradient combination for each kernel may be different. In this case, the target pixel DP may correspond to a blue pixel or a red pixel. One embodiment of calculating the reference gradient combination according to the disclosed technology may use a kernel in which the image data (IDATA) input to the target kernel corresponds to Figure 3 Example case of 8 corner patterns shown.

[0073] Figure 6B 2 is a schematic diagram showing an example of a plurality of pixel pairs (1) to (22) to be used in reference gradient combination. In this case, the target pixel DP may correspond to a green pixel G.

[0074] Figure 7 is a conceptual diagram illustrating an example of a method of calculating a similarity score by applying first to twenty-second pixel pairs to a reference gradient combination 500 and comparing the reference gradient combination 500 with a target gradient combination 510 based on each of the pixel pairs.

[0075] The reference gradient combination 500 of each of the corner patterns may refer to a comparison result of a difference between pixel data of each of a plurality of pixel pairs and a preset threshold. The reference gradient combination 500 may be obtained by combining the results of a plurality of pixel pairs. In addition, a pixel pair to be used in the reference gradient may include two pixels having the same color but located at different positions in the kernel.

[0076] The reference gradient combination 500 may indicate that a plurality of different pixel pairs located at predetermined positions relative to the kernel are selected, and the difference between the pixel data of the pixels of each of the selected pixel pairs is compared with the threshold value, thereby listing one of the two results represented by "large" and "small" according to the pixel pair. In addition, the reference gradient combination 500 may vary according to the type and number of angular patterns. For example, based on the pixel pair (1), the result value of the PT_A kernel is large because it is abs(P11-P15)>=threshold value, while the result value of the 400 kernel is abs(P11-P15)<threshold value, so its corresponding value is small.

[0077] The target gradient combination 510 of the target kernel 400 may represent a result of the reference gradient of the plurality of pixel pairs. The reference gradient may be obtained by comparing the difference between the pixel data of the pixels of each of the plurality of pixel pairs with a predetermined threshold value based on the image data (IDATA) input to the target kernel 400, using the positions of the pixels of the plurality of pixel pairs used in the reference gradient combination 500 without change.

[0078] For example, in the case where the eleventh pixel P11 and the twenty-third pixel P23, the third pixel P3, and the fifteenth pixel P15 are set as a pixel pair to be used in the reference gradient combination 500, results such as "large" and "small" may be calculated based on the result of the combination. Here, the reference gradient may be obtained by comparing a threshold value with each of a first difference value between pixel data at the position of the eleventh pixel P11 and pixel data of the twenty-third pixel P23 and a second difference value between pixel data at the position of the third pixel P3 and pixel data of the fifteenth pixel P15.

[0079] For example, when using in the reference gradient combination 500 Fig. 6A The 22 pairs of pixels are from Figure 3When determining the angle pattern corresponding to the target kernel 400 from the eight angle patterns PT_A to PT_H shown, for each of the angle patterns PT_A to PT_H, the difference between the pixel data of the pixels of each pixel pair can be compared with the threshold value, and one of the two results represented by "large" and "small" can be listed based on the 22 pixel pairs, so that the two result lists expressing "large" and "small" can be defined as the reference gradient combination 500.

[0080] In addition, the target gradient combination 510 of the target kernel 400 may refer to a result expressed as one of two results of "large" and "small" by comparing a difference between pixel data of each of the plurality of pixel pairs with a threshold value based on image data (IDATA) input into the target kernel 400, using pixel positions of the plurality of pixel pairs used in the reference gradient combination 500. The results "large" and "small" may be listed according to 22 pairs of pixels.

[0081] exist Figure 7 , the same gradient between the corner pattern PT_A and the target kernel 400 can be obtained in a total of 11 pixel pairs (i.e., pixel pairs (3), (6), (7), (9), (12), (14), (16), (18), (20), (21), and (22)), so that the similarity score can be defined as a score of 11. In this way, the similarity score between the target kernel 400 and each of the corner patterns PT_B to PT_H can be calculated, and the corner patterns each having the highest similarity score of 12 are the plurality of corner patterns PT_F and PT_H, so that the first determiner 320 can determine the pattern corresponding to the target kernel 400, and can send identification information of each of the corner patterns PT_F and PT_H to the second determiner 330.

[0082] Figure 8 Some implementations based on the disclosed technology are shown Figure 4 FIG. 1 is a flowchart of an example of a detailed operation of operation S110 . Fig.9A and Fig. 9B Some implementations based on the disclosed technology are shown Figure 8 FIG. 1 is a schematic diagram of an example of pixel combination used in the complex gradient combination of operation S300 .

[0083] Reference Figure 8, the second determiner 330 may calculate a complex gradient combination corresponding to the angular pattern provided from the first determiner 320 with respect to the target kernel (S300). The complex gradient combination may refer to a comparison result between average values ​​of the difference between pixel data of two pixels of the same color for each pixel combination. Each pixel combination may include at least one or more pixel groups paired with each other. For example, the complex gradient combination is based on the similarity score provided from the first determiner 320.

[0084] The second determiner 330 may determine an angular pattern corresponding to the target kernel based on a result of the complex gradient combination calculated with respect to the target kernel. In addition, when the number of angular patterns corresponding to the target kernel determined based on the complex gradient combination is only one (1) ("Yes" in S310), the second determiner 330 may send pixel data of pixels belonging to the same area as the texture area or the non-texture area of ​​the angular pattern including the target pixel DP to the pixel interpolator 350.

[0085] For example, when the target pixel DP is included in the determined texture area of ​​the angle pattern, the second determiner 330 may transmit pixel data of the pixel included in the determined texture area of ​​the angle pattern to the pixel interpolator 350 .

[0086] When it is determined that the number of angular patterns corresponding to the target kernel is a plural number based on the complex gradient combination (No in S310 ), the second determiner 330 may transmit identification information of each of the determined angular patterns to the third determiner 340 .

[0087] For example, when the angle patterns are PT_B and PT_C, the second determiner 330 may transmit identification information of the angle patterns PT_B and PT_C to the third determiner 340, and the third determiner 340 may determine the angle pattern using the angle patterns PT_B and PT_C as candidate angle patterns.

[0088] Fig.9A is a schematic diagram showing an example of pixel combinations used in complex gradient combination. When a complex gradient combination is calculated for an arbitrary kernel, the average value of the difference between the pixel data of the pixels of each pixel combination can be calculated based on the image data (IDATA) input to the kernel by using the positions of the pixels of the pixel combination used in the complex gradient combination without change.

[0089] Fig.9A Each cell of the table can include two pixel combinations. Fig.9A , the left pixel combination may correspond to the angular pattern corresponding to the row of the cell of the table, and the right pixel combination may correspond to the angular pattern corresponding to the column of the cell of the table. Both pixel combinations may correspond to the pixels indicated by the arrows. For example, referring to Fig.9APT_A and PT_B, two pixel combinations of PT_A may correspond to (P8, P12) and (P14, P18), and two pixel combinations of PT_B may correspond to (P8, P14) and (P12, P18). The pixel combination included in the cells located at the row of pattern PT_A and the column of pattern PT_B may correspond to the pixel combination used in the complex gradient combination for comparing patterns PT_A and PT_B.

[0090] For each kernel, the input image data (IDATA) may be different. Therefore, even when a complex gradient combination obtained using the same pixel combination is used, the result of calculating the complex gradient combination for each kernel may be different. In this case, the target pixel DP may correspond to a blue pixel or a red pixel.

[0091] Fig. 9B DP is a schematic diagram showing an example of pixel combinations used when angle patterns are compared with each other. At this time, the target pixel DP may correspond to a green pixel.

[0092] The complex gradient combination may refer to a comparison result between average values ​​of different values ​​between pixel data of two pixels for each pixel combination corresponding to the angle pattern. In addition, the complex gradient combination may refer to comparing the average value of the difference between pixel data of a pair of pixels for each pixel combination corresponding to the angle pattern with a predetermined threshold value to indicate one of two results represented by "large" and "small".

[0093] A pixel combination used in a complex gradient may refer to a pixel pair in which pixels of the same color are paired with each other, and a pixel combination to be used in a complex gradient combination corresponding to each angular pattern may be predetermined for each angular pattern.

[0094] For example, common reference Figure 3 , a pair of the eleventh pixel P11 and the twenty-third pixel P23 and a pair of the third pixel P3 and the fifteenth pixel P15 can be determined as a pixel combination corresponding to the first angular pattern. A pair of the twelfth pixel P12 and the eighteenth pixel P18 and a pair of the eighth pixel P8 and the fourteenth pixel P14 can be determined as a pixel combination corresponding to the second angular pattern.

[0095] At this time, when the first corner pattern and the second corner pattern are candidate corner patterns, when calculating the complex gradient combination corresponding to the first corner pattern and the second corner pattern for any kernel, only one result can be obtained from the results represented by "avg{abs(P11 - P23)+abs(P3 - P15)} < avg{abs(P12 - P18)+abs(P8 - P14)}", "avg{abs(P11 - P23)+abs(P3 - P15)} = avg{abs(P12 - P18)+abs(P8 - P14)}", and "avg{abs(P11 - P23)+abs(P3 - P15)} > avg{abs(P12 - P18)+abs(P8 - P14)}". Here, "avg" is an operator for calculating the average value and "abs" is an operator for calculating the absolute value.

[0096] In addition, the second determiner 330 can calculate the complex gradient combination, and can determine the corner pattern corresponding to the pixel combination with the same or smaller average value as the corner pattern corresponding to the target kernel. Therefore, when the first corner pattern and the second corner pattern are used as candidate patterns, when the result of calculating the complex gradient combination corresponding to the first corner pattern and the second corner pattern with respect to the target kernel is expressed as "avg{abs(P11 - P23)+abs(P3 - P15)} < avg{abs(P12 - P18)+abs(P8 - P14)}", the first corner pattern can be determined as the corner pattern corresponding to the target kernel, and when the result of calculating the complex gradient combination corresponding to the first corner pattern and the second corner pattern with respect to the target kernel is expressed as "avg{abs(P11 - P23)+abs(P3 - P15)} = avg{abs(P12 - P18)+abs(P8 - P14)}", the first corner pattern and the second corner pattern can be determined as the corner patterns corresponding to the target kernel.

[0097] Return reference Fig.9A , for example, when the corner patterns determined by the first determiner 320 are Figure 3 PT_B and PT_C, the pair of the eleventh pixel P11 and the twenty - first pixel P21 and the pair of the twenty - first pixel P21 and the twenty - third pixel P23 can be determined as the pixel combinations corresponding to PT_B, and the pair of the third pixel P3 and the fifth pixel P5 and the pair of the fifth pixel P5 and the fifteenth pixel P15 can be determined as the pixel combinations corresponding to PT_C.

[0098] When calculating the complex gradient combinations corresponding to PT_B and PT_C, only one result can be obtained from the results represented by "avg{abs(P11 - P21)+abs(P21 - P23)} < avg{abs(P3 - P5)+abs(P5 - P15)}", "avg{abs(P11 - P21)+abs(P21 - P23)} = avg{abs(P3 - P5)+abs(P5 - P15)}", and "avg{abs(P11 - P21)+abs(P21 - P23)} > avg{abs(P3 - P5)+abs(P5 - P15)}".

[0099] At this time, the second determiner 330 can calculate the complex gradient combinations and can determine the corner pattern corresponding to the pixel combination with the same or smaller average value as the corner pattern corresponding to the target kernel. Therefore, in the case where the corner pattern PT_B and the corner pattern PT_C are used as candidate patterns, when the result of calculating the complex gradient combinations corresponding to PT_B and PT_C with respect to the target kernel is expressed as "avg{abs(P11 - P21)+abs(P21 - P23)} < avg{abs(P3 - P5)+abs(P5 - P15)}", the corner pattern PT_B can be determined as the corner pattern corresponding to the target kernel, and when the result of calculating the complex gradient combinations corresponding to PT_B and PT_C with respect to the target kernel is expressed as "avg{abs(P11 - P21)+abs(P21 - P23)} = avg{abs(P3 - P5)+abs(P5 - P15)}", the corner patterns PT_B and PT_C can be determined as the corner patterns corresponding to the target kernel.

[0100] Return reference Fig.9A , for example, when the corner pattern determined by the first determiner 320 is represented by Figure 3 PT_A, PT_C, and PT_H, the complex gradient combinations corresponding to (PT_A and PT_C), (PT_A and PT_H), and (PT_C and PT_H) can be calculated.

[0101] When calculating the complex gradient combinations corresponding to PT_A and PT_C, only one result can be obtained from the multiple results represented by "avg{abs(P8 - P12)+abs(P14 - P18)} < avg{abs(P12 - P18)+abs(P8 - P14)}", "avg{abs(P8 - P12)+abs(P14 - P18)} = avg{abs(P12 - P18)+abs(P8 - P14)}", and "avg{abs(P8 - P12)+abs(P14 - P18)} > avg{abs(P12 - P18)+abs(P8 - P14)}".

[0102] When calculating the complex gradient combination corresponding to PT_A and PT_H, only one result can be obtained from multiple results represented by "avg{abs(P23 - P25)+abs(P25 - P15)} < avg{abs(P1 - P11)+abs(P1 - P3)+abs(P23 - P25)+abs(P25 - P15)}", "avg{abs(P23 - P25)+abs(P25 - P15)} = avg{abs(P1 - P11)+abs(P1 - P3)+abs(P23 - P25)+abs(P25 - P15)}", and "avg{abs(P23 - P25)+abs(P25 - P15)} > avg{abs(P1 - P11)+abs(P1 - P3)+abs(P23 - P25)+abs(P25 - P15)}".

[0103] When calculating the complex gradient combination corresponding to PT_C and PT_H, only one result can be obtained from multiple results represented by "avg{abs(P3 - P5)+abs(P5 - P8)} < avg{abs(P1 - P11)+abs(P1 - P3)+abs(P23 - P25)+abs(P25 - P15)}", "avg{abs(P3 - P5)+abs(P5 - P8)} = avg{abs(P1 - P11)+abs(P1 - P3)+abs(P23 - P25)+abs(P25 - P15)}", and "avg{abs(P3 - P5)+abs(P5 - P8)} > avg{abs(P1 - P11)+abs(P1 - P3)+abs(P23 - P25)+abs(P25 - P15)}".

[0104] At this time, the second determiner 330 may calculate a complex gradient combination and may determine a corner pattern corresponding to a pixel combination having the same or a smaller average value as the corner pattern corresponding to the target kernel. Accordingly, in the case where the corner patterns (PT_A, PT_C, PT_H) are used as candidate patterns, i) when the result of calculating the complex gradient combinations corresponding to PT_A and PT_C with respect to the target kernel is expressed as "avg{abs(P8 - P12)+abs(P14 - P18)} < avg{abs(P12 - P18)+abs(P8 - P14)}", ii) when the result of calculating the complex gradient combinations corresponding to PT_A and PT_H with respect to the target kernel is expressed as "avg{abs(P23 - P25)+abs(P25 - P15)} = avg{abs(P1 - P11)+abs(P1 - P3)+abs(P23 - P25)+abs(P25 - P15)}", and iii) when the result of calculating the complex gradient combinations corresponding to PT_C and PT_H with respect to the target kernel is expressed as "avg{abs(P3 - P5)+abs(P5 - P8)} > avg{abs(P1 - P11)+abs(P1 - P3)+abs(P23 - P25)+abs(P25 - P15)}", the corner patterns PT_A and PT_H may be determined as the corner patterns corresponding to the target kernel.

[0105] At this time, the second determiner 330 may calculate a complex gradient combination, and may determine the corner patterns (PT_A, PT_H) corresponding to pixel combinations each having a "small" value as the corner patterns corresponding to the target kernel, and may transmit identification information of the corner patterns (PT_A, PT_H) to the third determiner 340.

[0106] Fig.10 is a flowchart showing an example of detailed operations of Figure 4 operation S120 based on some implementations of the disclosed technology. Fig.11A and Fig. 11B is a schematic diagram showing an example of pixel combinations used in a Laplacian filter of Fig.10 operation S400 based on some implementations of the disclosed technology.

[0107] Referring to Fig.10 , the third determiner 340 may apply a Laplacian filter corresponding to each corner pattern provided from the second determiner 330 to the target kernel (S400). The Laplacian filter may represent a difference between a value obtained by doubling the pixel data of one pixel among three pixels of each pixel combination and a value corresponding to the sum of the pixel data values of the remaining two pixels other than the one pixel. For example, the Laplacian filter may be applied based on the complex gradient combination.

[0108] The third determiner 340 may determine the angular pattern corresponding to the target kernel based on the result of applying the Laplace filter to the target kernel. In addition, when the number of angular patterns corresponding to the target kernel determined based on the Laplace filter is only 1 ("Yes" in S410), the third determiner 340 may send pixel data of pixels belonging to the same area as the texture area or non-texture area of ​​the angular pattern including the target pixel DP (S420).

[0109] For example, when the target pixel DP is included in the texture area of ​​one determined angle pattern, pixel data of the pixel included in the texture area of ​​the determined angle pattern may be transmitted to the pixel interpolator 350 (S420).

[0110] When it is determined based on the Laplacian filter that the number of angular patterns corresponding to the target kernel is a plural number (No in S410 ), the third determiner 340 may transmit information indicating that the angular pattern cannot be specified to the pixel interpolator 350 .

[0111] For example, when the determined angle pattern is PT_B and PT_C, the third determiner 340 may transmit information indicating that the angle pattern cannot be specified to the pixel interpolator 350 .

[0112] Fig.11A An example of eight pixel combinations corresponding to each corner pattern used in the Laplace filter is shown. When the Laplace filter is applied to an arbitrary kernel, the difference between a value obtained by doubling the pixel data of one pixel of three pixels in each pixel combination and a value corresponding to the sum of the pixel data values ​​of the remaining two pixels can be calculated based on the image data (IDATA) input to the kernel by using the pixel positions of the pixel combinations used in the Laplace filter without change.

[0113] Fig.11A Each cell of the table shown may include a pixel combination corresponding to the angular pattern. A pixel at the center of each arrow and the pixels pointed to by both ends of the arrow may be a pixel combination used in the Laplacian filter.

[0114] For each kernel, the input image data (IDATA) may be different. Therefore, even when a Laplacian filter obtained using the same pixel combination is used, the result of applying the Laplacian filter to each kernel may be different. In this case, the target pixel DP may correspond to a blue pixel or a red pixel.

[0115] Fig. 11BAn example of 8 pixel combinations corresponding to each angular pattern used in the Laplacian filter is shown. At this time, the target pixel DP may correspond to a green pixel.

[0116] The Laplacian filter may represent a result indicating a difference between a value obtained by doubling pixel data of one pixel among three pixels combined for each pixel corresponding to the angular pattern and a value corresponding to the sum of pixel data values ​​of the remaining two pixels except the one pixel.

[0117] In addition, the Laplace filter can represent one of two results "large" and "small" obtained by comparing a predetermined threshold with the difference between a value obtained by doubling the pixel data of one pixel among three pixels combined for each pixel corresponding to the angular pattern and a value corresponding to the sum of the pixel data values ​​of the remaining two pixels except the one pixel.

[0118] The pixel combination used in the Laplacian filter may mean grouping pixels of the same color together, and the pixel combination used in the Laplacian filter corresponding to each pattern may be predetermined for each pattern.

[0119] For example, a combination of the eleventh pixel P11, the third pixel P3 and the fifteenth pixel P15 and a combination of the third pixel P3, the fifteenth pixel P15 and the twenty-third pixel P23 can be determined as pixel combinations corresponding to the first angular pattern, and a combination of the eleventh pixel P11, the twenty-third pixel P23 and the fifteenth pixel P15 and a combination of the twenty-third pixel P23, the fifteenth pixel P15 and the third pixel P3 can be determined as pixel combinations corresponding to the second angular pattern.

[0120] At this time, in the case where the first angle pattern and the second angle pattern are candidate patterns, when the Laplace filter corresponding to the first angle pattern is calculated for an arbitrary kernel, a result of comparing each of "abs(P3*2-P11-P15)" and "abs(P15*2-P15-P23)" with a predetermined threshold value can be obtained based on the calculated Laplace filter. In addition, when the Laplace filter corresponding to the second angle pattern is calculated, a result of comparing each of "abs(P23*2-P11-P15)" and "abs(P15*2-P23-P15)" with a predetermined threshold value based on the calculated Laplace filter can be obtained.

[0121] In addition, the third determiner 340 can calculate the Laplacian filter, and can determine the angular pattern corresponding to the target kernel based on the results listed according to the pixel combination. Therefore, in the case where the first angular pattern and the second angular pattern are candidate patterns, i) when the result corresponding to the first angular pattern is expressed as "(small, small)" and the result corresponding to the second angular pattern is expressed as "(small, large)", and ii) when the result of calculating the complex gradient combination corresponding to the first angular pattern relative to the target kernel is expressed as "(small, small)", and the result of calculating the complex gradient combination corresponding to the second angular pattern relative to the target kernel is expressed as "(small, small)", the first angular pattern can be determined as the angular pattern corresponding to the target kernel.

[0122] Return to reference Fig.11A , for example, in the case where the angle pattern determined by the second determiner 330 is Figure 3 In the case represented by PT_D and PT_H, the combination of the twelfth pixel P12, the eighth pixel P8 and the fourth pixel P4 and the combination of the sixteenth pixel P16, the twelfth pixel P12 and the eighth pixel P8 can be determined as the pixel combination corresponding to the corner pattern PT_D, and the combination of the eighteenth pixel P18, the fourteenth pixel P14 and the tenth pixel P10 and the combination of the twenty-second pixel P22, the eighteenth pixel P18 and the fourteenth pixel P14 can be determined as the pixel combination corresponding to PT_H.

[0123] When the Laplace filter corresponding to PT_D is used, the result of comparing each of "abs(P8*2-P12-P4)" and "abs(P12*2-P16-P8)" with a predetermined threshold can be obtained, and when the Laplace filter corresponding to PT_H is used, the result of comparing each of "abs(P14*2-P18-P10)" and "abs(P18*2-P22-P14)" with a predetermined threshold can be obtained.

[0124] At this time, when all the results obtained by comparing the result values ​​of each pixel combination with the threshold value using the Laplace filter are expressed as "small", the third determiner 340 can determine the angular pattern of the corresponding pixel combination as the angular pattern corresponding to the target kernel. Therefore, in the case where the angular patterns PT_D and PT_H are candidate patterns, i) when the result obtained by applying the Laplace filter corresponding to PT_D to the target kernel is expressed as "(small, small)", and ii) when the result obtained by applying the Laplace filter corresponding to PT_H to the target kernel is expressed as "(small, large)", the angular pattern PT_D can be determined as the angular pattern corresponding to the target kernel.

[0125] Return to reference Fig. 11B, for example, in the case where the angle pattern determined by the second determiner 330 is Figure 3 In the case represented by PT_D and PT_H, the combination of the eleventh pixel P11, the seventh pixel P7 and the third pixel P3 and the combination of the seventeenth pixel P17, the seventh pixel P7 and the ninth pixel P9 can be determined as the pixel combination corresponding to the corner pattern PT_D, and the combination of the seventeenth pixel P17, the nineteenth pixel P19 and the ninth pixel P9 and the combination of the twenty-third pixel P23, the nineteenth pixel P19 and the fifteenth pixel P15 can be determined as the pixel combination corresponding to PT_H.

[0126] When the Laplace filter corresponding to PT_D is used, the result of comparing each of "abs(P7*2-P11-P3)" and "abs(P7*2-P17-P9)" with a predetermined threshold can be obtained, and when the Laplace filter corresponding to PT_H is used, the result of comparing each of "abs(P19*2-P17-P9)" and "abs(P19*2-P23-P15)" with a predetermined threshold can be obtained.

[0127] At this time, in the case where the result value of each pixel combination is compared with the threshold value using the Laplacian filter, when the result of the pixel combination corresponding to each solid arrow is represented as "small" and the result of the pixel combination corresponding to each dotted arrow is represented as "large", the angular pattern of the corresponding pixel combination can be determined as the angular pattern corresponding to the target kernel. Therefore, in the case where PT_D and PT_H are candidate patterns, according to Fig. 11B The order of pixel combinations shown, the result of applying the Laplacian filter corresponding to PT_D to the target kernel is represented as "(small, large)", and according to Fig. 11B In the order of pixel combination shown, the result of applying the Laplacian filter corresponding to PT_H to the target kernel is expressed as "(small, small)", and the angle pattern PT_D can be determined as the angle pattern corresponding to the target kernel.

[0128] At this time, the third determiner 340 may determine the angle pattern PT_D as an angle pattern corresponding to the target kernel according to a result of applying the Laplacian filter to the target kernel, and may transmit information about the angle pattern PT_D to the pixel interpolator 350 .

[0129] Fig.12 Some implementations based on the disclosed technology are shown Figure 4 FIG. 1 is a schematic diagram of an example of a detailed operation of operation S140 . Fig.13 Some implementations based on the disclosed technology are shown Figure 4 FIG. 1 is a schematic diagram of an example of a detailed operation of operation S130 .

[0130] Reference Fig.12 , the pixel interpolator 350 can interpolate the target pixel DP based on the value obtained by applying the weight to the pixel data of some pixels, which belong to the same area as the corner pattern area including the target pixel DP and correspond to the same color as the target pixel DP. This is because when determining an corner pattern, the target kernel is similar to the corner pattern rather than the plane pattern, so it can be considered that the method of interpolating the corner pattern is more appropriate.

[0131] For example, when weights are not used and the angular pattern corresponding to the target kernel is determined to be PT_A, the pixel interpolator 350 can calculate the average value of pixel data of some pixels (i.e., the first pixel P1, the fifteenth pixel P15, the twenty-third pixel P23, and the twenty-fifth pixel P25) belonging to the same area as the angular pattern area including the target pixel DP and having the same color as the target pixel DP, and the calculated average value can be determined as the pixel data of the target pixel DP.

[0132] In addition, when weights are used, the pixel interpolator 350 can calculate a weighted average of a first value obtained when '3' is multiplied by pixel data of the fifteenth pixel P15 and the twenty-third pixel P23 in the pixels (P1, P15, P23, P25) and a second value obtained when '1' is multiplied by pixel data of the first pixel P1 and the twenty-fifth pixel P25 in the pixels (P1, P15, P23, P25), and can determine the calculated average as the pixel data of the target pixel DP. Here, the pixels P15 and P23 used to calculate the first value can be located closer to the target pixel DP, and the other pixels P1 and P25 used to calculate the second value can be located farther away from the target pixel DP.

[0133] Reference Fig.13 , the target pixel DP can be interpolated based on the value obtained by applying weights to the pixel data of the pixels having the same color as the target pixel DP. This is because when a plurality of angular patterns are determined, the target kernel is positioned similar to a plane pattern rather than an angular pattern, so that a method of interpolating a plane pattern can be considered more appropriate.

[0134] For example, when weights are not used and the angular pattern corresponding to the target kernel is determined as PT_A and PT_B, the pixel interpolator 350 can determine the average value of pixel data of some pixels (i.e., P1, P3, P5, P11, P15, P21, P23, P25) having the same color as the target pixel DP as the pixel data of the target pixel DP.

[0135] In addition, when using weights, the pixel interpolator 350 can calculate a weighted average of a first value obtained when '3' is multiplied by pixel data of some pixels (P3, P11, P15, P23) among the pixels (P1, P3, P5, P11, P15, P21, P23, P25) positioned closer to the target pixel DP and a second value obtained when '1' is multiplied by pixel data of some pixels (P1, P5, P21, P25) among the pixels (P1, P3, P5, P11, P15, P21, P23, P25) positioned farther away from the target pixel DP, and can determine the calculated average value as the pixel data of the target pixel DP.

[0136] Fig.14 is shown with Figure 1 A block diagram of an example of a computing device 1000 corresponding to an image signal processor.

[0137] Reference Fig.14 , the computing device 1400 may represent a computer program product for executing Figure 1 An embodiment of a hardware configuration of an operation of the image signal processor 100 is shown.

[0138] The computing device 1400 may be mounted on a chip separate from the chip on which the image sensing device is mounted. According to one embodiment, a chip on which the image sensing device is mounted and a chip on which the computing device 1400 is mounted may be implemented in one package (e.g., a multi-chip package (MCP)), but the scope of the disclosed technology is not limited thereto.

[0139] also, Figure 1 The internal configuration or arrangement of the image sensing device and the image signal processor 100 described in the foregoing may vary according to the implementation. For example, at least a portion of the image sensing device may be included in the image signal processor 100. Alternatively, at least a portion of the computing device 1400 may be included in the image sensing device. In this case, at least a portion of the computing device 1400 may be mounted together on a chip on which the image sensing device is mounted.

[0140] The computing device 1400 may include a processor 1410 , a memory 1420 , an input / output interface 1430 , and a communication interface 1440 .

[0141] Processor 1410 can process execution Figure 1 The processor 1410 may be data and / or instructions required for the operation of the components (100, 200) of the image signal processor 100 described in the embodiment of the present invention. That is, the processor 1410 may refer to the image signal processor 100, but the scope of the disclosed technology is not limited thereto.

[0142] The memory 1420 may store data and / or instructions required to perform operations of the components (100, 200) of the image signal processor 100, and may be accessed by the processor 1410. For example, the memory 1420 may be a volatile memory (e.g., a dynamic random access memory (DRAM), a static random access memory (SRAM), etc.) or a non-volatile memory (e.g., a programmable read-only memory (PROM), an erasable PROM (EPROM), etc.), an EEPROM (electrically erasable PROM), a flash memory, etc.).

[0143] That is, a computer program for executing the operation of the image signal processor 100 disclosed in this document is recorded in the memory 1420 , and is executed and processed by the processor 1410 , thereby realizing the operation of the image signal processor 100 .

[0144] The input / output interface 1430 is an interface that connects an external input device (eg, keyboard, mouse, touch panel, etc.) and / or an external output device (eg, display) to the processor 1410 to allow transmission and reception of data.

[0145] The communication interface 1440 is a component that can transmit and receive various data with an external device (eg, an application processor, an external memory, etc.), and may be a device that supports wired communication or wireless communication.

[0146] It is obvious from the above description that the image signal processor and the image signal processing method based on some implementations of the disclosed technology can interpolate the target pixel by determining the angular pattern corresponding to the target kernel, and even when the target kernel corresponds to the angular pattern, the correction accuracy of the defective pixel can be improved.

[0147] The embodiments of the disclosed technology can provide various effects that can be directly or indirectly recognized through the above-mentioned patent documents.

[0148] Although a number of exemplary embodiments have been described, it should be appreciated that variations and enhancements to the disclosed embodiments and other embodiments may be devised based on what is described and / or illustrated in this patent document.

[0149] CROSS-REFERENCE TO RELATED APPLICATIONS

[0150] This patent document claims priority to and the benefit of Korean Patent Application No. 10-2023-0092057, filed on Jul. 14, 2023, the disclosure of which is incorporated herein by reference in its entirety as a part of the disclosure of this patent document.

Claims

1. An image signal processor, the image signal processor comprising: a first determiner that compares a target gradient combination including a target kernel of a target pixel with a reference gradient combination of each of a plurality of angular patterns to calculate a similarity score, and determines at least one angular pattern based on the calculated similarity score; as well as A pixel interpolator interpolates the target pixel using pixel data of an interpolation pixel determined based on the angular pattern.

2. The image signal processor according to claim 1, wherein: The reference gradient combination includes a reference gradient obtained by comparing a threshold value with a difference between pixel data of each of a plurality of pixel pairs.

3. The image signal processor according to claim 2, wherein: The reference gradient combination is configured for the plurality of pixel pairs to be used in the reference gradient combination that vary according to a color of a defective pixel.

4. The image signal processor according to claim 2, wherein: The pixel pair is two pixels having the same color but located at different positions.

5. The image signal processor according to claim 1, wherein: The similarity score represents the number of pixel pairs where the target gradient is identical to the reference gradient when the target gradient combination is compared with the reference gradient combination based on each of the pixel pairs.

6. The image signal processor according to claim 1, wherein: The first determiner: The angular pattern having the highest similarity score is determined as the angular pattern corresponding to the target kernel.

7. The image signal processor according to claim 1, further comprising: a second determiner that determines at least one angular pattern of the target kernel based on a complex gradient combination when the first determiner determines a plurality of angular patterns; wherein the complex gradient combination is determined by the plurality of angular patterns determined by the first determiner with respect to the target kernel, and The complex gradient combination means a result obtained by comparing average values ​​of difference values ​​between pixel data of two pixels in each of a plurality of pixel combinations with each other.

8. The image signal processor according to claim 7, wherein: Each of the pixel combinations varies according to the color of the defective pixel.

9. The image signal processor according to claim 7, wherein: The complex gradient combination represents a pixel combination corresponding to a smaller average value when average values ​​of difference values ​​between pixel data of two pixels in each of the plurality of pixel combinations are compared with each other.

10. The image signal processor according to claim 7, wherein: The two pixels are a pair of pixels having the same color.

11. The image signal processor according to claim 7, further comprising: a third determiner that, when a plurality of angular patterns are determined by the second determiner, determines an angular pattern of the target kernel based on a result of applying a Laplacian filter corresponding to the angular pattern determined by the second determiner, Here, the Laplacian filter represents a difference between a value obtained by doubling pixel data of one pixel among three pixels of each pixel combination and a value corresponding to a sum of pixel data values ​​of two pixels other than the one pixel.

12. The image signal processor according to claim 11, wherein: The plurality of pixel combinations to be used in the Laplacian filter vary according to the color of the defective pixel.

13. The image signal processor according to claim 11, wherein: The Laplace filter: comparing a predetermined threshold with a difference between a value obtained by doubling pixel data of one pixel of three pixels of each pixel combination and a value corresponding to a sum of pixel data values ​​of the remaining two pixels; and The result of the comparison is enabled to be expressed by the expression "greater than or equal to" or "less than" based on three pixels.

14. The image signal processor according to claim 11, wherein: The three pixels are pixels where pixels having the same color are grouped together.

15. The image signal processor according to claim 11, wherein: The pixel interpolator: When the number of the angle patterns determined by the third determiner is a plural number, the target pixel is interpolated using pixel data of pixels having the same color as the target pixel.

16. The image signal processor according to claim 1, wherein: The corner pattern is a pattern filled with textured areas and non-textured areas, in, The texture area and the non-texture area are distinguished from each other by using a horizontal line passing through one side of the target pixel and a vertical line passing through the other side of the target pixel as boundary lines.

17. A method for processing an image signal, the method comprising the following steps: comparing a target gradient combination including a target kernel of the target pixel to a reference gradient combination for each of the plurality of angular patterns to calculate a similarity score; determining an angular pattern corresponding to the target kernel based on the similarity score; as well as The target pixel is interpolated using pixel data of an interpolation pixel determined based on the angle pattern.

18. The image signal processing method according to claim 17, wherein: The step of determining the angular pattern corresponding to the target kernel comprises the following steps: When the number of angle patterns determined based on the similarity score is plural, An angular pattern corresponding to the target kernel is determined based on a result of calculating a complex gradient combination with respect to the target kernel, the complex gradient combination corresponding to each of the angular patterns determined based on the similarity scores.

19. The image signal processing method according to claim 18, wherein: The step of determining the angular pattern corresponding to the target kernel comprises the following steps: When the number of the angle patterns determined based on the complex gradient combination is a complex number, An angular pattern corresponding to the target kernel is determined based on a result of applying a Laplacian filter to the target kernel, the Laplacian filter corresponding to each of the angular patterns determined based on the complex gradient combination.

20. The image signal processing method according to claim 17, wherein: The step of interpolating the target pixel comprises the following steps: The target pixel is interpolated using pixel data of pixels having the same color as the target pixel.

Images