Image processing device configured to perform edge-preserving smoothing and image processing method thereof

Abstract

An image processing device configured to perform edge-preserving smoothing and an image processing method thereof are provided. The image processing device includes: a low-resolution image generation circuit configured to generate a low-resolution image including a second pixel corresponding to the first pixel based on an input image including the first pixel; and an edge-preserving smoothing circuit configured to: generate the reliability of the second pixel based on the characteristics of the value of the first pixel, and perform edge-preserving smoothing on the input image using the value of the second pixel with the reflection rate adjusted based on the reliability of the second pixel.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2018-0155470 filed on December 5, 2018, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0002] The inventive concept relates to an image processing apparatus, and more particularly, to an image processing apparatus configured to perform edge preserving smoothing using a low-resolution image and an image processing method of the image processing apparatus. Background Art

[0003] When edge-preserving smoothing is performed using an edge-preserving smoothing filter with a wide tap, a low-resolution image can be used as a guide image to perform a high-speed image processing operation. In this case, since the edge-preserving property is not taken into account in the downsampling operation for generating the low-resolution image, the edge-preserving performance is degraded during edge-preserving smoothing using the low-resolution image. Summary of the invention

[0004] The inventive concept provides an image processing apparatus configured to enhance edge preserving performance when edge preserving smoothing is performed using a low-resolution image, and an image processing method of the image processing apparatus.

[0005] According to some example embodiments of the present invention, there is provided an image processing device comprising: a processor configured to: generate a low-resolution image including a second pixel corresponding to the first pixel from an input image including a first pixel, generate the reliability of the second pixel based on the characteristics of the value of the first pixel, and based on the reliability of the second pixel, perform edge-preserving smoothing on the input image using the value of the second pixel with adjusted reflection rate.

[0006] According to some example embodiments of the present inventive concept, there is provided an image processing method for performing edge-preserving smoothing, the method comprising: generating a low-resolution image including second pixels based on an input image including first pixels, generating the reliability of each of the second pixels based on the characteristics of the values ​​of the first pixels respectively corresponding to the second pixels, and performing edge-preserving smoothing on the input image using the values ​​of the second pixels whose reflection rates are adjusted based on the reliability of each of the second pixels.

[0007] According to some example embodiments of the present inventive concept, there is provided a device comprising at least one processor and a memory configured to store instructions. When the instructions are executed by the at least one processor, the at least one processor is prompted to perform the following operations: generating a low-resolution image including second pixels based on an input image including first pixels, generating a reliability indicating the degree of edge smoothing of each of the second pixels based on the degree of change of the values ​​of the first pixels respectively corresponding to the second pixels, and performing edge-preserving smoothing on the input image by limitedly using the values ​​of the second pixels based on the reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Example embodiments of the inventive concept will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0009] Figure 1 is a block diagram of an image processing apparatus according to some example embodiments;

[0010] Figure 2 yes Figure 1 A flowchart of an image processing method of an image processing device;

[0011] Figure 3A is a diagram of the edge of the image;

[0012] Figure 3B is a graph used to explain the input image;

[0013] Figure 3C and Figure 3D It is a graph used to explain the basic image;

[0014] Figure 4 is a block diagram of a reliability map generation module according to some example embodiments;

[0015] Figure 5 is a flow chart of an edge-preserving smoothing method of an edge-preserving smoothing circuit according to some example embodiments;

[0016] Fig. 6A and Figure 6B is a graph for explaining a method of generating reliability of a second pixel of a low-resolution image in an edge-preserving smoothing circuit according to some example embodiments;

[0017] Figure 7 is a reliability graph of reliability of a second pixel for generating a low-resolution image according to some example embodiments;

[0018] Figure 8 is a diagram illustrating an edge-preserving smoothing method of an edge-preserving smoothing circuit according to some example embodiments;

[0019] Fig. 9is a block diagram of a reliability map generation module according to some example embodiments;

[0020] Fig.10 is a flow chart of an edge-preserving smoothing method of an edge-preserving smoothing circuit according to some example embodiments;

[0021] Fig.11 It is used to explain Fig. 9 A graph of a second low resolution image shown in FIG.

[0022] Fig.12 and Fig.13 are respectively block diagrams of image processing devices configured to perform edge-preserving smoothing according to some example embodiments;

[0023] Fig.14 is a block diagram of a reliability map generation module configured to set a reliability function according to some example embodiments;

[0024] FIG. 15A to FIG. 15C is a graph showing various forms of reliability functions;

[0025] Fig.16 is a block diagram of an example of implementing an image processing method according to some example embodiments by using software; and

[0026] Fig.17 is a block diagram of a computing system including an image processing apparatus according to some example embodiments. DETAILED DESCRIPTION

[0027] Figure 1 is a block diagram of an image processing apparatus 10 according to some example embodiments.

[0028] Reference Figure 1 The image processing device 10 may include a low-resolution image generation circuit 20, an edge-preserving smoothing circuit 100, an image subtractor 25 and / or a post-processing circuit 30. The image processing device 10 may also include Figure 1 Components not shown in, and not limited to Figure 1. In addition, the low-resolution image generation circuit 20, the edge-preserving smoothing circuit (or edge-preserving smoothing filter) 100 and / or the post-processing circuit 30 can be implemented in various forms (such as hardware, software or hardware / software integrated forms). In addition, the edge-preserving smoothing circuit 100, the low-resolution generation circuit 20, the image subtractor 25 and / or the post-processing circuit 30 can be implemented as part of a processor, which is configured to perform a specific operation based on an input, or to run instructions in a code and perform a specific operation. The processor may include: a processing circuit, such as hardware including a logic circuit; a hardware / software combination, such as a processor running software; or a combination of a processing circuit and a hardware / software combination. For example, the processing circuit may more specifically include, but is not limited to: a central processing unit (CPU), an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a system on a chip (SoC), a programmable logic unit, a microprocessor, an application-specific integrated circuit (ASIC), etc. In the following, it is assumed that the edge in the image processing technology represents a portion of the image where the brightness suddenly changes. A sudden change in the brightness of an image can generally indicate the occurrence of a significant change in an object.

[0029] The low-resolution image generation circuit 20 may receive an input image IMG in , for the input image IMG in Perform a downsampling operation (or subsampling operation) and generate a low-resolution image IMG low In the following, the input image IMG in The pixels included in the low-resolution image IMG will be referred to as the first pixels. low The pixel included in is referred to as a second pixel. In some example embodiments, the low-resolution image generation circuit 20 may generate an image based on an input image IMG including a first pixel. in Generate a low-resolution image IMG including a second pixel corresponding to the first pixel low That is, the second pixel can be obtained by in The sample points are generated by downsampling the area corresponding to the coordinates of the first pixel. Therefore, the input image IMG in The area corresponding to the coordinates of the first pixel may correspond to the low-resolution image IMG low The area corresponding to the coordinates of the second pixel. Low-resolution image IMG low may include a plurality of second pixels, and the input image IMG in A plurality of first pixels respectively corresponding to the plurality of second pixels may be included.

[0030] The edge-preserving smoothing circuit 100 may include a reliability map generation module 110. In some example embodiments, the reliability map generation module 110 may calculate the low-resolution image IMGlow The reliability of the second pixel of the image IMG is determined according to the edge smoothing degree of the second pixel and a reliability map is generated. The reliability may be a parameter that may be determined according to the edge smoothing degree of the second pixel and may be used to adjust the reflection rate of the value of the second pixel when the edge-preserving smoothing circuit 100 performs edge-preserving smoothing. When the reflection rate of the value of the second pixel is not adjusted by using the value of the second pixel with a high edge smoothing degree, the low-resolution image IMG is used. low When performing edge-preserving smoothing, edge-preserving performance may degrade. Low-resolution image IMG low The degree of edge smoothing of the second pixel can be different and can be based on the input image IMG in The characteristic of the value of the first pixel corresponding to the second pixel is determined. Therefore, the reliability map generation module 110 can generate the reliability of each second pixel based on the characteristic of the value of the first pixel corresponding to the second pixel. Specifically, the reliability map generation module 110 can generate the reliability of the second pixel based on the degree of change of the value of the first pixel corresponding to the second pixel. When the degree of change of the value of the first pixel corresponding to the second pixel exceeds the first reference degree, the reliability map generation module 110 can determine the edge smoothing degree to be very high and generate the reliability of the second pixel with a very low value (for example, a value close to the value "0"). When the degree of change of the value of the first pixel corresponding to the second pixel is lower than the second reference degree less than the first reference degree, the reliability map generation module 110 can determine the edge smoothing degree to be very low and generate the reliability of the second pixel with a very high value (for example, a value close to the value "1"). That is, it can be inferred that when the degree of change of the value of the first pixel is high, the second pixel may have a high degree of edge smoothing proportional to the degree of change.

[0031] In some example embodiments, the edge-preserving smoothing circuit 100 may adjust the low-resolution image IMG used for edge-preserving smoothing by using the reliability map in The reflection rate of the second pixel value is adjusted, and the input image IMG is adjusted using the second pixel value with the adjusted reflection rate. in Perform edge-preserving smoothing. For example, as shown in Equation 1 and Equation 2, the edge-preserving smoothing circuit 100 can perform edge-preserving smoothing using the reliability map:

[0032] as well as

[0033]

[0034] Where x represents the target pixel (e.g., the pixel to be smoothed with edge preservation), and θ represents the filtering range. C is the normalized value of the filtering result of the target pixel x, that is, the low-resolution image IMG with the filtering range θ. low The second pixel y in lowThe coefficients of the first function f and the second function g are the sum of the first function f and the second function g of the target pixel x (or reference pixel). The first function f can be used to obtain the coefficients of the first function f and the second function g that depend on the target pixel x and the reference pixel y. low The coefficient of the distance between the pixels and the second function g can be used to obtain the pixel value (or pixel size) I that depends on the target pixel x. x With the second pixel y low Pixel value The coefficient of the difference between . Reliability function f adj Can be used to pass the input image IMG in The second pixel y low The corresponding characteristic value δ(y low ) is applied as a variable to obtain the second pixel y low The reliability obtained is f adj (δ(y low )) can be used to adjust the second pixel y for edge-preserving smoothing low Pixel value The reflection rate.

[0035] In some example embodiments, the reliability map generation module 110 may calculate the input image IMG in The standard deviation or variance of the value of the first pixel as the input image IMG in The second pixel y low The corresponding characteristic value δ(y low In some example embodiments, the reliability map generation module 110 may calculate the input image IMG in The difference between the maximum and minimum values ​​of the first pixel is taken as the input image IMG in The second pixel y low The corresponding characteristic value δ(y low ). In some example embodiments, the reliability map generation module 110 may generate a high-pass filter or a band-pass filter based on the frequency component characteristics by applying a high-pass filter or a band-pass filter to the input image IMG in Calculate the input image IMG in The value of the first pixel is taken as the input image IMG in The second pixel y low The corresponding characteristic value δ(y low ).

[0036] The reliability map generation module 110 can generate the reliability map by converting the input image IMG in The second pixel y low The corresponding characteristic value δ(y low ) is applied to the reliability function f adj To generate the second pixel y low Reliability. Figure 7 Given the reliability function f adj Detailed description.

[0037] The edge-preserving smoothing circuit 100 according to some example embodiments may be configured to perform edge-preserving smoothing by using a low-resolution image IMG low The reliability map of the second pixel is used to adjust the reflection rate of the value of the second pixel used for edge-preserving smoothing. As a result, when the edge-preserving smoothing circuit 100 uses the low-resolution image IMG low When edge-preserving smoothing is performed, during edge-preserving smoothing, the value of a pixel with a high degree of edge smoothing that may degrade edge-preserving performance may be limitedly reflected based on reliability, so that the edge-preserving performance may be enhanced.

[0038] As a result of edge-preserving smoothing, the edge-preserving smoothing circuit 100 may be based on the input image IMG in Generates the input image IMG in The edge of the base image IMG base The edge-preserving and smoothing unit 100 can reduce the number of pixels used to extract the basic image IMG by considering the characteristics of edge pixels. base The amount of calculation is reduced, while improving the basic image IMG base The quality of the base image IMG base Can be referred to by the input image IMG in The part with a sharp change in brightness is left and the input image IMG is in The image subtractor 25 can remove the portion with slight brightness change from the input image IMG. in Remove the base image IMG base And generate detail image IMG detail .

[0039] The post-processing circuit 30 may receive the basic image IMG base and the detail image IMG detail And for the base image IMG base and the detail image IMG detail For example, the post-processing circuit 30 may perform image processing operations by using the basic image IMG base and the detail image IMG detail The method further comprises performing image processing operations such as high dynamic range (HDR) compression, tone mapping, detail enhancement and / or image haze removal on at least one of the images.

[0040] Figure 2 yes Figure 1 Flow chart of the image processing method of the image processing device 10. Figure 1 right Figure 2 The flowchart is described.

[0041] Reference Figure 2 The low-resolution image generation circuit 20 can generate an input image IMG in Perform downsampling operation and generate low-resolution image IMG low (S10). The reliability map generation module 110 may generate a low-resolution image IMG low Specifically, the reliability map generation module 110 may generate a reliability map corresponding to the second pixel of the input image IMG based on the input image IMG. in IMG with low resolution images low The edge-preserving smoothing circuit 100 can use the low-resolution image IMG to generate the reliability of each second pixel. low Specifically, when the edge-preserving smoothing circuit 100 performs edge-preserving smoothing, in the low-resolution image IMG low The reliability map is reflected in the value of the second pixel, and the reflection rate of the value of the second pixel whose edge smoothing degree exceeds the reference degree can be adjusted to a small value. The edge-preserving smoothing circuit 100 can generate a basic image IMG base As a result of edge-preserving smoothing, and using an image subtractor 25, a detail image IMG is generated. detail The post-processing circuit 30 can process the basic image IMG base and the detail image IMG detail The image processing operation is performed (S40). In the following, the post-processing circuit 30 may process the processed basic image IMG base with processed details image img detail Synthesis (S50).

[0042] Figure 3A is a diagram of the edge of image 1, Figure 3B is a graph used to explain the input image, Figure 3C and Figure 3D It is a graph used to explain the basic image.

[0043] Reference Figure 3A , the image 1 may include a first region 2 having high brightness and a second region 3 having low brightness. Figure 3B , the pixel size on the dotted line of image 1 can be shown as curve 4. Figure 3B In FIG. 4 , the abscissa represents the position of the pixel on the dotted line of the image 1 in the horizontal direction, and the ordinate represents the pixel size (or pixel value) of each pixel. Figure 3A The first component 4A corresponding to the first region 2 and Figure 3A The second region 3 corresponds to the third component 4C and Figure 3AThe second component 4B corresponds to the boundary between the first area 2 and the second area 3. In the second component 4B, the pixel size of the image 1 may change drastically, and in the first component 4A and the third component 4C, the pixel size of the image 1 may change slightly due to the noise component.

[0044] When the edge-preserving smoothing circuit (referring to Figure 1 100) pairs of images (refer to Figure 3A 1) When performing edge-preserving smoothing, you can retain Figure 3B The second component 4B indicating the change in brightness can be removed by simply Figure 3B As a result, the edge-preserving smoothing circuit (referring to Figure 1 100) based on Figure 3B The corresponding image of curve 4 is generated with Figure 3C However, when edge-preserving smoothing is performed using a low-resolution image by completely reflecting the value of the second pixel of the low-resolution image, a basic image with reduced edge-preserving performance may be generated to correspond to Figure 3D The edge-preserving smoothing circuit (referring to Figure 1 100) in may adjust the reflection rate of each of the values ​​of the second pixel used for edge-preserving smoothing based on the reliability map to enhance the edge-preserving performance.

[0045] Figure 4 is a block diagram of the reliability map generation module 110 according to some example embodiments.

[0046] Reference Figure 4, the reliability map generation module 110 may include a pixel value characteristic generation module 112 and / or a characteristic-based reliability generation module 114. The pixel value characteristic generation module 112 may obtain a value of a first pixel of an input image corresponding to a second pixel of a low-resolution image, and generate a characteristic of the value of the first pixel. In some example embodiments, the pixel value characteristic generation module 112 may generate a predetermined (or alternatively, a given) value indicating a degree of change in the value of the first pixel, and the generated value may correspond to a variable used to generate the reliability of the second pixel. In an example, the pixel value characteristic generation module 112 may perform a calculation for obtaining a standard distribution or variance of a value of a first pixel of an input image corresponding to the second pixel of the low-resolution image, and the generated standard distribution or variance may be used to generate the reliability of the second pixel. In another example, the pixel value characteristic generation module 112 may perform a calculation for obtaining a difference between a maximum value and a minimum value in a value of a first pixel of an input image corresponding to the second pixel of the low-resolution image, and the obtained difference may be used to generate the reliability of the second pixel. However, the pixel value characteristic generation module 112 according to the above-described example embodiment is merely an example, and the inventive concept is not limited thereto. The pixel value characteristic generation module 112 may generate various characteristics of the value of the first pixel of the input image, wherein the edge smoothness of the second pixel of the low-resolution image may be determined by the various characteristics. In the above manner, the pixel value characteristic generation module 112 may generate characteristics corresponding to the second pixels of the low-resolution image, respectively. In some example embodiments, the reliability map generation module 110 may not include the pixel value characteristic generation module 112, but may directly receive information required for generating the reliability of the second pixel of the low-resolution image from another logical source.

[0047] The characteristic-based reliability generation module 114 may generate the reliability of each second pixel based on the characteristics respectively corresponding to the second pixels of the low-resolution image generated by the pixel value characteristic generation module 112. In some example embodiments, the characteristic-based reliability generation module 114 may determine the edge smoothness of each second pixel based on the characteristics respectively corresponding to the second pixels of the low-resolution image, and generate the reliability of each second pixel according to the edge smoothness. The characteristic-based reliability generation module 114 may generate the reliability of the second pixels of the low-resolution image based on the reliability function. That is, the characteristic-based reliability generation module 114 may apply the characteristics (or characteristic values) respectively corresponding to the second pixels received from the pixel value characteristic generation module 112 to the reliability function and generate the reliability of the second pixels. The following will refer to Figure 7 A detailed description of the reliability function is described.

[0048] Figure 5is a flow chart of a margin-preserving balancing method of a margin-preserving balancing circuit according to some example embodiments.

[0049] Reference Figure 5 , the edge-preserving smoothing circuit may obtain the value of the first pixel of the input image corresponding to the second pixel of the low-resolution image (S100). Hereinafter, the edge-preserving smoothing circuit may calculate the standard deviation or variance of the value of the first pixel (S110). The edge-preserving smoothing circuit may apply the calculation result to the reliability function. The edge-preserving smoothing circuit may apply the calculation result to the reliability function and obtain the reliability of the second pixel (S120). The edge-preserving smoothing circuit may perform edge-preserving smoothing on the input image using the value of the second pixel whose reflection rate is adjusted based on the reliability of the second pixel (S130). In the above manner, the edge-preserving smoothing circuit may adjust the reflection rate of each of the values ​​of the second pixel of the low-resolution image used for edge-preserving smoothing based on the reliabilities of all second pixels of the low-resolution image. As a result, during edge-preserving smoothing, the influence of pixels with a high degree of edge smoothing may be minimized.

[0050] Fig. 6A and Figure 6B is a graph for explaining a method of generating reliability of a second pixel of a low-resolution image in an edge-preserving smoothing circuit according to some example embodiments. Fig. 6A Curve 4 shows Figure 3A The dotted line portion of image 1. The pixel "PX1" of the low-resolution image can be generated by down-sampling the first component 4A. It is assumed that the pixel of the input image corresponding to the pixel "PX1" is the pixel "SG_1". The pixel "SG_1" can also be generated by sampling the first component 4A. In addition, the pixel "PX2" of the low-resolution image can be generated by down-sampling the second component 4B. It is assumed that the pixel of the input image corresponding to the pixel "PX2" is the pixel "SG_2". The pixel "SG_2" can also be generated by sampling the second component 4B.

[0051] Reference Fig. 6A , the degree of change of the pixel "SG_1" corresponding to the pixel "PX1" may be small, and therefore, the pixel "PX1" may have a small degree of edge smoothing. On the contrary, the degree of change of the pixel "SG_2" corresponding to the pixel "PX2" may be very large, and therefore, the pixel "PX2" may have a very large degree of edge smoothing. Therefore, the value of the pixel "PX1" may be fully used for edge-preserving smoothing, however, because the value of the pixel "PX2" may reduce the edge-preserving performance, it may be necessary to adjust the reflection rate of the value of the pixel "PX2" during edge-preserving smoothing.

[0052] In order to generate the reliability of the second pixel of the low-resolution image including the pixels "PX1" and "PX2", the edge-preserving smoothing circuit according to some example embodiments may calculate the standard deviation (or variance) of the values ​​of the first pixels of the input image respectively corresponding to the second pixels of the low-resolution image. Figure 6B The calculated standard deviation is graphically represented in .

[0053] Reference Figure 6B , the edge-preserving smoothing circuit can generate the coordinate "X" corresponding to the pixel "PX1" 1 " and generates the coordinate "X" corresponding to the pixel "PX2". 2 " The standard distribution value STD. The standard distribution value STD can be used to generate the reliability of the second pixel of the low-resolution image including the pixel "PX1" and the pixel "PX2". However, the edge-preserving smoothing circuit is only an example, and the inventive concept is not limited to this. As described above, the edge-preserving smoothing circuit can calculate the difference between the maximum value and the minimum value of the values ​​of the first pixels of the input image respectively corresponding to the second pixels of the low-resolution image to generate the reliability of the second pixel of the low-resolution image including the pixel "PX1" and the pixel "PX2".

[0054] Figure 7 is a reliability graph of the reliability of a second pixel for generating a low-resolution image according to some example embodiments. Figure 4 right Figure 7 Give a description.

[0055] Reference Figure 7 The characteristic value generated by the pixel value characteristic generation module 112 can be used as a variable X to be applied to the reliability function f adj , and the characteristic-based reliability generation module 114 can manage the reliability function f adj When the variable X exceeds the first reference value V 1 When the reliability function f adj A reliability close to the value "0" can be generated when the variable X is lower than the second reference value V 2 When the reliability function f adj A reliability close to the value "1" can be generated, and when the variable X ranges from the first reference value V 1 To the second reference value V 2 When the reliability function f adj It can produce a reliability that is inversely proportional to the variable X. Reliability function f adjIt can be divided into a first region R1 to a third region R3. The first region R1 may be a region configured to generate the reliability of the second pixel with a high degree of edge smoothness. The third region R3 may be a region configured to generate the reliability of the second pixel with a low degree of edge smoothness. The second region R2 may be a region configured to generate the reliability of the second pixel with a medium degree of edge smoothness. However, the reliability function f adj This is only an example, and the present invention is not limited thereto. The reliability function may have various forms so that the values ​​of pixels with a high degree of edge smoothing among the second pixels of the low-resolution image can be limitedly reflected during edge-preserving smoothing. In addition, depending on the type of variable X applied, the reliability function f adj For example, when the variable X is the inverse of the standard deviation of the value of the first pixel of the input image corresponding to each of the second pixels of the low-resolution image, the reliability function f adj Can be used with Figure 7 The reliability functions shown in are different.

[0056] Figure 8 is a diagram illustrating an edge-preserving smoothing method of an edge-preserving smoothing circuit according to some example embodiments.

[0057] Reference Figure 8 , the edge-preserving smoothing circuit may obtain the value of the first pixel of the input image corresponding to the second pixel of the low-resolution image (S200). Hereinafter, the edge-preserving smoothing circuit may calculate the difference between the maximum value and the minimum value among the values ​​of the first pixel (S210). The edge-preserving smoothing circuit may apply the calculation result to the reliability function. The edge-preserving smoothing circuit may apply the calculation result to the reliability function and obtain the reliability of the second pixel (S220). The edge-preserving smoothing circuit may perform edge-preserving smoothing on the input image using the value of the second pixel whose reflection rate is adjusted based on the reliability of the second pixel (S230).

[0058] Fig. 9 is a block diagram of a reliability map generation module 110 ′ according to some example embodiments.

[0059] Reference Fig. 9, the reliability map generating module 110' may include a pixel value characteristic generating module 112' and a characteristic-based reliability generating module 114'. The pixel value characteristic generating module 112' may include a high-pass filter 115' and a second low-resolution image generating module 116'. The high-pass filter 115' may receive an input image and pass pixels having frequency components having a predetermined (or optionally, given) frequency band or a higher frequency band. However, instead of the high-pass filter 115', the pixel value characteristic generating module 112' may include a band-pass filter configured to pass only a predetermined (or optionally, given) frequency band. In this case, the band-pass filter may be applied to the input image. Hereinafter, the case where the pixel value characteristic generating module 112' includes the high-pass filter 115' will be mainly described. The second low-resolution image generating module 116' may downsample the input image filtered by the high-pass filter 115' and generate a second low-resolution image. Each of the pixels of the second low-resolution image may have a value corresponding to the frequency component. For the sake of brevity, reference will also be made to Figure 3B right Fig. 9 Described. Pixels corresponding to the first component 4A and the third component 4C of the curve 4 may have frequency components included in the low frequency band. Pixels corresponding to the second component 4B may have frequency components included in the high frequency band. Therefore, the second low-resolution image may include only pixels corresponding to the second component 4B. In the example, when the degree of change in the value of the first pixel of the input image corresponding to the pixel of the second low-resolution image becomes larger, the pixel of the second low-resolution image may have a higher frequency component, and thus the pixel value of the second low-resolution image may be higher.

[0060] The characteristic-based reliability generation module 114' may generate respective reliabilities of second pixels of the first low-resolution image based on the values ​​of pixels of the second low-resolution image. Specifically, the characteristic-based reliability generation module 114' may apply the values ​​of pixels of the second low-resolution image to the reliability function and match the generated reliabilities with the second pixels of the first low-resolution image, respectively. For example, a pixel "PX_a" of the first low-resolution image may correspond to a pixel "PX_b" of the second low-resolution image, and the characteristic-based reliability generation module 114' may apply the value of the pixel "PX_b" to the reliability function and match the generated reliability with the pixel "PX_a". Therefore, during the subsequent edge-preserving smoothing, the reflection rate of the value of the pixel "PX_a" may be adjusted by the matched reliability.

[0061] Fig.10 is a flow chart of an edge-preserving smoothing method of an edge-preserving smoothing circuit according to some example embodiments.

[0062] Reference Fig.10, the edge-preserving smoothing circuit may receive an input image (S300). The edge-preserving smoothing circuit may pass the input image through a high-pass filter (S310). The edge-preserving smoothing circuit may generate a first low-resolution image based on the input image and generate a second low-resolution image based on the input image filtered by the high-pass filter (S320). The edge-preserving smoothing circuit may apply the pixel value of the second low-resolution image to the reliability function and generate the reliability of the second pixel of the first low-resolution image (S330). The edge-preserving smoothing circuit may perform edge-preserving smoothing on the input image using the value of the second pixel whose reflection rate is adjusted based on the reliability of the second pixel (S340). In the above manner, the edge-preserving smoothing circuit may adjust the reflection rate of each of the values ​​of the second pixels used for edge-preserving smoothing based on the reliability of the values ​​of all second pixels of the first low-resolution image.

[0063] Fig.11 It is used for explanation reference Fig. 9 The second low resolution image is described in detail. Fig. 9 right Fig.11 The curve diagram is described.

[0064] Reference Fig.11 , the input image may include a first pixel group PIXG_1 containing a frequency component having a reference frequency fc or lower and a second pixel group PXG_2 containing a frequency component exceeding the reference frequency fc. When the high pass filter HPF or 115' is applied to the input image, only the second pixel group PIXG_2 may pass through the high pass filter HPF. The second low resolution image generation module 116' may perform a down sampling operation on the second pixel group PIXG_2 and generate a second low resolution image. The value of the pixel corresponding to the frequency component of the second low resolution image may be determined according to the degree of change of the value of the first pixel corresponding to the pixel of the second low resolution image of the input image. For example, the value of the pixel of the second low resolution image may be proportional to the degree of change of the value of the first pixel corresponding to the pixel of the second low resolution image of the input image. The characteristic-based reliability generation module 114' may apply the pixel value of the generated second low resolution image to the reliability function and generate the reliability of the second pixel of the first low resolution image.

[0065] Fig.12 and Fig.13 1 and 2 are block diagrams of an image processing apparatus 10a and an image processing apparatus 10b, respectively, configured to perform edge-preserving smoothing according to some example embodiments.

[0066] Reference Fig.12 The image processing device 10a may include a low-resolution image generation circuit 20a, a reliability map generation module 110a and / or an edge-preserving smoothing circuit 100. The low-resolution generation circuit 20a may be based on the input image IMG inGenerate a low-resolution image IMG low And the low-resolution image IMG low Provided to the edge-preserving smoothing circuit 100. In addition, the low-resolution image generating circuit 20a can generate a low-resolution image IMG low The required intermediate result MR is provided to the reliability map generation module 110a. The reliability map generation module 110a can use the intermediate result MR to generate a low-resolution image IMG low In some example embodiments, the intermediate result MR may include a calculation result, such as the input image IMG in At the time point when the intermediate result MR is received from the low-resolution image generation circuit 20a, the reliability map generation module 110a may sequentially start the operation of generating the reliability map RM. The edge-preserving smoothing circuit 100 may use the reliability map RM, the low-resolution image IMG low and / or input image IMG in Perform edge-preserving smoothing.

[0067] Reference Fig.13 The image processing device 10b may include a low-resolution image generating circuit 20b, a reliability map generating module 110b and / or an edge-preserving smoothing circuit 100. The low-resolution image generating circuit 20b may be based on the input image IMG in Generate a low-resolution image IMG low And the low-resolution image IMG low Provided to the edge-preserving smoothing circuit 100. The reliability map generation module 110b can directly receive the input image IMG in parallel with the low-resolution image generation circuit 20b. in , and can generate a low-resolution image IMG low According to the above-described exemplary embodiments, the reliability map generation module 110b may perform various operations, such as calculating the reliability map RM of the second pixel of the input image IMG. in an operation of calculating the standard deviation or distribution of the first pixel of the input image IMG, an operation of calculating the difference between the maximum value and the minimum value, and / or an operation of making the input image IMG in Through the operation of the high-pass filter and / or the band-pass filter, the reliability map generation module 110b can generate a low-resolution image IMG low The edge-preserving and smoothing circuit 100 may use the reliability map RM, the low-resolution image IMG low and input image IMG in Perform edge-preserving smoothing.

[0068] Fig.14 is a block diagram of a reliability map generation module 110 ″ configured to set a reliability function according to some example embodiments. FIG. 15A to FIG. 15C is a reliability function f showing various forms adj1 、f adj2 、f adj3 Graph of the curve.

[0069] Reference Fig.14 The reliability map generation module 110" may include a pixel value characteristic generation module 112, a characteristic-based reliability generation module 114 and / or a reliability function setting module 116. Figure 4 The pixel value characteristic generation module 112 and the characteristic-based reliability generation module 114 are described in detail, so the reliability function setting module 116 will be mainly described.

[0070] The reliability function setting module 116 may determine the form of the reliability function differently in consideration of the sensing environment of the input image. The sensing environment of the input image may include a brightness condition when the image of the object is captured, a degree of movement of the object, and / or a noise condition. By changing the form of the reliability function in consideration of the sensing environment by using the reliability function setting module 116, the reflection rate of the pixel value that degrades the edge preserving performance may be adjusted to different degrees during edge preserving smoothing.

[0071] Reference Fig.15A The reliability function setting module 116 can set the reliability function f adj1 The second region R2' is Figure 7 The reliability function f adj The second region R2 in the embodiment is narrower. Fig. 15B The reliability function setting module 116 can set the reliability function f adj2 The second region R2' is Figure 7 The reliability function f adj The second region R2 in the embodiment is wider. Fig. 15C The reliability function setting module 116 can set the reliability function f adj3 Set to Figure 7 The reliability function f adj The comparison is inverted based on the abscissa. FIG. 15A to FIG. 15C The reliability function f adj1 、f adj2 and f adj3 This may be only an example, and the present inventive concept is not limited thereto. The reliability function setting module 116 may change the reliability function f in various forms in various sensing environments. adj1 、f adj2 and f adj3 This allows the reflection rate of the value of the second pixel of the low-resolution image having a high degree of edge smoothing to be adjusted to a small value during edge-preserving smoothing.

[0072] Fig.16 is a block diagram of an example of implementing an image processing method according to some example embodiments by using software.

[0073] Reference Fig.16 , the image processing device 500 may include a processor 510 and a working memory 520. The working memory 520 may be a non-transitory computer-readable recording medium. The processor 510 may include: a processing circuit, such as hardware including a logic circuit; a hardware / software combination, such as a processor that runs software; or a combination of a processing circuit and a hardware / software combination. For example, the processing circuit may more specifically include, but is not limited to, a central processing unit (CPU), an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a system on a chip (SoC), a programmable logic unit, a microprocessor, an application-specific integrated circuit (ASIC), etc. The processor 510 may run a computer program stored in the working memory 520. The working memory 520 may store a computer program configured to perform at least some of the various functions of edge-preserving smoothing according to the above-described example embodiments by using software processing, and the computer program may include an edge-preserving smoothing filter 525 according to the function.

[0074] As an example, the processor 510 may perform edge-preserving smoothing according to some example embodiments. For example, the processor 510 may run the edge-preserving smoothing filter 525 and generate a reliability map indicating the degree of edge smoothing of each pixel in the second pixels of the low-resolution image. Hereinafter, the processor 510 may run the edge-preserving smoothing filter 525 and adjust the reflection rate of the value of the second pixel with a high degree of edge smoothing among the second pixels of the low-resolution image by using the reliability map during edge-preserving smoothing. As a result, edge-preserving smoothing can be performed on a predetermined (or optionally, a given) input image by using the pixel value whose reflection rate is adjusted.

[0075] Fig.17 is a block diagram of a computing system 600 including an image processing apparatus according to the above-described exemplary embodiment.

[0076] Reference Fig.17 , the computing system 600 may include an application processor (AP) 610, a memory device 620, a storage device 630, an input / output (I / O) device 640, a power supply 650, and / or an image sensor 660. Fig.17 Not shown, but computing system 600 may also include ports capable of communicating with a video card, a sound card, a memory card, and / or a Universal Serial Bus (USB) device for communicating with other electronic devices.

[0077] The application processor 610 may include: a processing circuit, such as hardware including a logic circuit; a hardware / software combination, such as a processor running software; or a combination of a processing circuit and a hardware / software combination. For example, the processing circuit may more specifically include, but is not limited to, a central processing unit (CPU), an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a system on a chip (SoC), a programmable logic unit, a microprocessor, an application specific integrated circuit (ASIC), etc. The application processor 610 may perform specific calculations or tasks. According to some example embodiments, the application processor 610 may include a microprocessor and / or a central processing unit (CPU). The application processor 610 may communicate with a memory device 620, a storage device 630, and / or an I / O device 640 through an address bus, a control bus, and / or a data bus. The memory device 620 may store data required for the operation of the computing system 600. For example, the memory device 620 may be implemented as a dynamic random access memory (DRAM), a mobile DRAM, a static RAM (SRAM), a flash memory, a phase change RAM (PRAM), a ferroelectric RAM (FRAM), a resistive RAM (RRAM), and / or a magnetic RAM (MRAM). The storage device 630 may include a solid state drive (SSD), a hard disk drive (HDD), and / or a compact disk read only memory (CD-ROM). The I / O device 640 may include an input unit (e.g., a keyboard, a keypad, and a mouse) and / or an output unit (e.g., a printer and a display). The power supply 650 may supply an operating voltage required for the operation of the computing system 600.

[0078] According to some example embodiments, the application processor 610 may include a codec module 611 configured to perform an image processing operation. The codec module 611 may include an edge-preserving smoothing filter 611_1. The edge-preserving smoothing filter 611_1 may generate a reliability map indicating the degree of edge smoothing of each pixel in the second pixel of the low-resolution image. Hereinafter, during edge-preserving smoothing, the edge-preserving smoothing filter 611_1 may adjust the reflection rate of the pixel value having a high degree of edge smoothing among the second pixel of the low-resolution image by using the reliability map. As a result, edge-preserving smoothing may be performed on a predetermined (or optionally, a given) input image by using the pixel value whose reflection rate is adjusted.

[0079] Typical example embodiments of the inventive concept are disclosed in the above description and accompanying drawings. Although specific terms are used, they are used in a general and descriptive sense only and not for limiting purposes. It will be understood by those skilled in the art that various changes in form and / or details may be made to the disclosed example embodiments without departing from the spirit and / or scope of the inventive concept as defined by the claims.

Claims

1. An image processing device, comprising: The processor is configured as: generating, from an input image including the first pixel, a low-resolution image including a second pixel corresponding to the first pixel, generating a reliability of the second pixel based on a characteristic of the value of the first pixel, and performing edge-preserving smoothing on the input image using the value of the second pixel whose reflection rate is adjusted based on the reliability of the second pixel, so that during edge-preserving smoothing, the influence of the second pixel having a high degree of edge smoothing is minimized, wherein the characteristic of the value of the first pixel is a degree of change in the value of the first pixel, and The reliability of the second pixel indicates the edge smoothness of the second pixel.

2. The image processing apparatus according to claim 1, wherein: The processor is further configured to: The low-resolution image is generated by performing a down-sampling operation on the input image.

3. The image processing apparatus according to claim 1, wherein: The processor is further configured to generate the reliability of the second pixel based on a standard distribution or variance of values ​​of the first pixel.

4. The image processing apparatus according to claim 1, wherein: The processor is further configured to generate the reliability of the second pixel based on a difference between a maximum value and a minimum value among the values ​​of the first pixel.

5. The image processing apparatus according to claim 1, wherein: The processor is further configured to: applying a high pass filter or a band pass filter to the input image, and The reliability of the second pixel is generated based on a frequency component characteristic of the value of the first pixel.

6. The image processing apparatus according to claim 1, wherein: The processor is further configured to: applying a variable corresponding to the characteristic of the value of the first pixel to a reliability function, and The reliability of the second pixel is generated based on the reliability function.

7. The image processing apparatus according to claim 6, wherein: The processor is further configured to determine the reliability function based on image sensing environment information corresponding to the input image.

8. The image processing apparatus according to claim 6, wherein: The processor is further configured to set the reliability function to perform the following operations: When said variable exceeds a first reference value, said reliability is close to a value of 0, and The reliability is close to a value of 1 when the variable is below a second reference value which is smaller than the first reference value.

9. The image processing apparatus according to claim 1, wherein: The processor is further configured to: When the variation degree of the value of the first pixel exceeds a first reference value, the reliability having a low value is generated so that the value of the second pixel is limitedly reflected during the execution of edge-preserving smoothing.

10. The image processing apparatus according to claim 1, wherein: The processor is further configured to: When the degree of variation of the value of the first pixel is lower than a second reference value, the reliability having a high value is generated so that the value of the second pixel is sufficiently reflected during the execution of edge-preserving smoothing.

11. The image processing apparatus according to claim 1, wherein: The processor is further configured to: A base image indicating edges of the input image based on the input image due to edge-preserving smoothing is generated.

12. The image processing apparatus according to claim 11, wherein: The processor is further configured to: generating a detail image by removing the base image from the input image, and An image processing operation is performed on at least one of the base image and the detail image.

13. The image processing apparatus according to claim 1, wherein: The processor is further configured to: receiving information indicative of said characteristic of the value of the first pixel, and The reliability of the second pixel is generated using the information.

14. The image processing apparatus according to claim 1, wherein: The processor is further configured to: receiving the input image, generating information indicative of the characteristic of the value of the first pixel, and The reliability of the second pixel is generated using the information.

15. An image processing method for performing edge-preserving smoothing, the method comprising: generating a low-resolution image including second pixels based on an input image including first pixels; generating the reliability of each of the second pixels based on characteristics of the values ​​of the first pixels respectively corresponding to the second pixels; and performing edge-preserving smoothing on the input image using values ​​of second pixels whose reflection rates are adjusted based on the reliability of each of the second pixels, so that during edge-preserving smoothing, influence of second pixels having a high degree of edge smoothing is minimized, wherein the characteristic of the value of the first pixel is a degree of change in the value of the first pixel, and The reliability of each of the second pixels indicates a degree of edge smoothness of each of the second pixels.

16. The method of claim 15, wherein: The second pixel includes a third pixel, and the first pixel includes a fourth pixel corresponding to the third pixel, The step of generating the reliability of each of the second pixels further includes generating the reliability of the third pixel based on the degree of change of the value of the fourth pixel.

17. The method of claim 16, wherein: The step of generating the reliability of the third pixel further comprises generating the reliability of the third pixel based on a standard distribution or variance of values ​​of the fourth pixel.

18. The method of claim 16, wherein: The step of generating the reliability of the third pixel includes generating the reliability of the third pixel based on a difference between a maximum value and a minimum value of the value of the fourth pixel.

19. The method of claim 15, wherein: The second pixel includes a third pixel, and the first pixel includes a fourth pixel corresponding to the third pixel, The step of generating the reliability of each of the second pixels includes applying a high pass filter or a band pass filter to the input image, and generating the reliability of the third pixel based on the frequency component characteristics of the value of the fourth pixel.

20. The method of claim 15, wherein: The step of generating said reliability of each of the second pixels comprises: determining a degree of edge smoothness for each of the second pixels based on the characteristics of the values ​​of the first pixels respectively corresponding to the second pixels; and The reliability of each of the second pixels is generated based on the determination result.

21. The method of claim 20, wherein: The step of generating the reliability of each of the second pixels based on the determination result includes generating a lower reliability when the degree of edge smoothness of each of the second pixels is determined to be greater.

22. The method of claim 20, wherein: The step of performing edge-preserving smoothing on the input image using the value of the second pixel with adjusted reflection rate comprises: adjusting the values ​​of the second pixels so that the values ​​of the second pixels whose edge smoothing degree exceeds the first reference degree among the second pixels are limitedly reflected during edge-preserving smoothing; and The values ​​of the second pixels are adjusted so that the values ​​of the second pixels having an edge smoothing degree among the second pixels lower than a second reference degree smaller than the first reference degree are sufficiently reflected during edge preserving smoothing.

23. A device comprising: at least one processor; as well as a memory configured to store instructions, When the instruction is executed by the at least one processor, the at least one processor is prompted to perform the following operations: generating a low-resolution image including second pixels based on an input image including first pixels, generating reliability indicating the degree of edge smoothness of each of the second pixels based on the degree of change in the values ​​of the first pixels respectively corresponding to the second pixels, and By performing edge-preserving smoothing on the input image using the value of the second pixel whose reflection rate is adjusted based on the reliability, the influence of the second pixel having a high degree of edge smoothing is minimized during the edge-preserving smoothing.

Images