Method and apparatus for image processing and electronic device

Abstract

A method and apparatus for image processing, as well as electronic devices, are provided. The image processing method includes: setting an offset window for an offset pattern for kernel offset and an offset parameter for the application intensity of kernel offset; determining an output kernel by applying kernel offset to an input kernel based on the offset window and the offset parameter; and using the output kernel to adjust the contrast of a degraded image.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2021-0121837, filed September 13, 2021, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety by reference for all purposes. TECHNICAL FIELD

[0002] The following description relates to a method and apparatus having contrast adjustment. BACKGROUND

[0003] In a process of acquiring an image, the quality of the image can be degraded. Image restoration can restore an original image by removing a degradation element from a degraded image. The degradation between the original image and the degraded image can be modeled by a degradation kernel, and the image restoration can restore the degraded image to the original image using the degradation kernel. The image restoration can include a method using prior information and a method of learning a relationship between the original image and the degraded image. In the image restoration, the performance of the kernel can directly affect the restoration performance. SUMMARY

[0004] This summary is provided to introduce a selection of concepts, in a simplified form, that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to determine the scope of the claimed subject matter.

[0005] In one general aspect, a method having image processing includes setting an offset window of an offset pattern for kernel offset and an offset parameter of an application strength for kernel offset, determining an output kernel by applying the kernel offset to an input kernel based on the offset window and the offset parameter, and adjusting contrast of a degraded image using the output kernel.

[0006] The step of setting the offset window and the offset parameter can include setting a distribution type of offset values in the offset window, and setting a distribution parameter specifying a detailed distribution of the distribution type.

[0007] The distribution type can include a Gaussian distribution, and the distribution parameter can include a standard deviation of the Gaussian distribution.

[0008] The step of setting the offset window and the offset parameter can include setting a contrast adjustment parameter for a degree of contrast adjustment, and setting the offset parameter by normalizing the contrast adjustment parameter based on the offset values of the offset window.

[0009] The step of setting the contrast adjustment parameter can include setting the contrast adjustment parameter based on an image characteristic of the degraded image.

[0010] The size of the offset window can correspond to the size of the input kernel, and the strength of the contrast adjustment parameter can be adjusted according to the normalization based on the size of the offset window.

[0011] The method can include adjusting the size of the input kernel.

[0012] The step of setting the offset window and the offset parameter can include setting the size of the offset window and the offset parameter based on the adjusted size of the input kernel.

[0013] The method can include detecting a first region and a second region having different image characteristics in the degraded image, and the step of setting the offset window and the offset parameter can include setting the offset window and the offset parameter individually for each of the first region and the second region.

[0014] The first region and the second region can correspond to a flat region and an edge region, respectively, and the step of setting the offset window and the offset parameter can include setting the size of a first offset window for the first region to be greater than the size of a second offset window for the second region, and setting the strength of a first offset parameter for the first region to be weaker than the strength of a second offset parameter for the second region.

[0015] The step of determining the output kernel can include determining a new input kernel by applying the kernel offset to the input kernel, and determining the output kernel by normalizing the new input kernel based on kernel values of the new input kernel.

[0016] In another general aspect, one or more embodiments include a non-transitory computer-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform any one, any combination, or all of the operations and methods described herein.

[0017] In another general aspect, an apparatus having image processing includes a processor configured to set an offset window of an offset pattern for a kernel offset and an offset parameter for an application strength of the kernel offset, determine an output kernel by applying the kernel offset to an input kernel based on the offset window and the offset parameter, and adjust a contrast of a degraded image using the output kernel.

[0018] To set the offset window and the offset parameter, the processor can be configured to set a distribution type of offset values in the offset window, and set a distribution parameter specifying a detailed distribution of the distribution type.

[0019] To set the offset window and the offset parameter, the processor can be configured to set a contrast adjustment parameter for a degree of contrast adjustment, and set the offset parameter by normalizing the contrast adjustment parameter based on the offset values of the offset window.

[0020] The processor can be configured to detect a first region and a second region having different image characteristics in the degraded image, and to set the offset window and the offset parameter individually for each of the first region and the second region for setting the offset window and the offset parameter.

[0021] The first region and the second region can correspond to a flat region and an edge region, respectively, and to set the offset window and the offset parameter, the processor can be configured to set a size of a first offset window for the first region to be greater than a size of a second offset window for the second region, and to set an intensity of a first offset parameter for the first region to be weaker than an intensity of a second offset parameter for the second region.

[0022] To determine the output kernel, the processor can be configured to determine a new input kernel by applying the kernel offset to the input kernel, and to determine the output kernel by normalizing the new input kernel based on kernel values of the new input kernel.

[0023] The device can include a memory storing instructions that, when executed by a processor, configure the processor to perform the process of setting the offset window and the offset parameter, the process of determining the output kernel, and the process of adjusting the contrast.

[0024] In another general aspect, an electronic device includes a processor configured to set a distribution type of offset values in an offset window and a distribution parameter specifying a detailed distribution of the distribution type, set the offset window based on the distribution type and the distribution parameter, set a contrast adjustment parameter for a degree of contrast adjustment, set an offset parameter by normalizing the contrast adjustment parameter based on the offset values of the offset window, determine an output kernel by applying a kernel offset to an input kernel based on the offset window and the offset parameter, and adjust a contrast of a degraded image using the output kernel.

[0025] The processor can be configured to determine a new input kernel by applying the kernel offset to the input kernel, and to determine the output kernel by normalizing the new input kernel based on kernel values of the new input kernel.

[0026] In another general aspect, an electronic device includes one or more sensors configured to obtain an input image, and one or more processors configured to determine a kernel offset based on a distribution parameter and a contrast adjustment parameter, determine an output kernel by applying the kernel offset to an input kernel, and determine an output image by applying the output kernel to the input image.

[0027] To determine the kernel offset, the one or more processors can be configured to determine an offset window based on the distribution parameter, determine an offset parameter based on the contrast adjustment parameter, and determine the kernel offset based on the offset window and the offset parameter.

[0028] The kernel values of the offset window can increase toward a center portion of the offset window and decrease toward a peripheral portion of the offset window.

[0029] The one or more sensors can include an under-display camera (UDC), and to obtain the input image, the UDC can be configured to obtain a degraded image.

[0030] Other features and aspects will be apparent from the following detailed description and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 An example of an operation of an image processing device is shown.

[0032] Figure 2 is a flowchart showing an example of a kernel modification operation using kernel offset.

[0033] Figure 3 An example of a kernel offset setting operation is shown.

[0034] Figure 4 An example of changing the processing of a kernel is shown.

[0035] Figure 5 An example of using an output kernel to enhance image quality is shown.

[0036] Figure 6 An example of the effect of a contrast adjustment parameter on a result image is shown.

[0037] Figure 7 An example of the effect of an offset window on a result image is shown.

[0038] Figure 8 An example of mitigating glare by kernel modification is shown.

[0039] Figure 9 An example of kernel setting that is adaptive to image characteristics is shown.

[0040] Figure 10 An example of the effect of a difference in kernel setting is shown.

[0041] Figure 11 is a flowchart showing an example of an image processing method.

[0042] Figure 12 is a block diagram showing an example of an image processing device.

[0043] Figure 13 is a block diagram showing an example of an electronic device.

[0044] Throughout the drawings and detailed description, unless otherwise described or provided, like reference numerals will be understood to refer to like elements, features, and structures. The drawings can not be to scale and the dimensions, proportions, and shapes of the elements in the drawings can have been exaggerated for clarity, illustration, and convenience. DETAILED DESCRIPTION

[0045] The following detailed description is provided to help the reader obtain a thorough understanding of the methods, devices, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, devices, and / or systems described herein will be clear to those skilled in the art after understanding the present disclosure. For example, the order of the operations described herein is merely an example and is not limited to those set forth herein, but can be changed as will be apparent after understanding the present disclosure, except for operations that must occur in a particular order. Also, descriptions of known features can be omitted in order to more clearly and concisely describe embodiments of the present disclosure.

[0046] Although terms such as "first", "second", and "third" can be used herein to describe various elements, components, regions, layers or sections, these elements, components, regions, layers or sections should not be limited by these terms. Instead, these terms are only used to distinguish one element, component, region, layer or section from another element, component, region, layer or section. Thus, the first element, the first component, the first region, the first layer or the first section referred to in the examples described herein can also be referred to as the second element, the second component, the second region, the second layer or the second section without departing from the teachings of the examples.

[0047] Throughout the specification, when an element (such as a layer, region, or substrate) is referred to as being "on" another element, "connected to" or "coupled to" another element, it can be directly on, directly connected to, or directly coupled to the other element, or one or more other elements can be interposed therebetween. In contrast, when an element is referred to as being "directly on", "directly connected to", or "directly coupled to" another element, there are no other elements interposed therebetween. Likewise, each of the expressions "between... and" and "adjacent to" should also be interpreted in the same manner, respectively. As used herein, the term "and / or" includes any one of the associated listed items, as well as any combination of any two or more of the associated listed items.

[0048] The terminology used herein is for the purpose of describing various examples only and is not intended to be limiting of the disclosure. As used herein, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. The terms "comprises," "comprising," "includes," "including" and "has," "having" as used herein, specify the presence of stated features, numbers, operations, members, elements and / or a combination thereof, but do not preclude the presence or addition of one or more other features, numbers, operations, members, elements and / or a combination thereof.

[0049] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs and based on the understanding of the disclosure of the present application. Unless specifically defined herein, terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the disclosure of the application, and are not to be interpreted in an idealized or overly formal sense.

[0050] Hereinafter, examples will be described in detail with reference to the accompanying drawings. When the examples are described with reference to the drawings, the same reference numbers are used throughout the drawings and redundant descriptions will be omitted.

[0051] Figure 1 An example of an operation of an image processing device is shown. Referring to Figure 1 , the image processing device 100 can output a result image 102 by performing image restoration and / or image enhancement on a degraded image (also referred to as a degraded quality image) 101. In a process of generating the degraded image 101, the degraded image 101 can be accompanied by various degradation elements. For example, the degraded image 101 can be generated by an under display camera (UDC), and can be accompanied by degradation elements such as reduced sharpness, a haze effect, reduced contrast, a ghost effect, and / or increased noise. For example, the degradation can be modeled as Equation 1 below.

[0052] Equation 1:

[0053]

[0054] In Equation 1, y denotes measured data, A denotes a degradation kernel, x denotes a ground truth (GT) image, n denotes noise, represents a convolution operation. The degraded image 101 can correspond to measured data, and the result image 102 can correspond to a GT image. The GT image can also be referred to as an original image, and the degradation kernel can also be referred to simply as a kernel, a filter, or a point spread function (PSF). The convolution operation can be replaced by other operations.

[0055] The image processing device 100 can restore the degraded image 101 to the result image 102 using the provided kernel. For example, the image processing device 100 can perform image restoration by using a method of prior information and / or a method of learning a relationship between a measured image and a GT image. In both methods, the performance of the kernel can directly affect the restoration performance.

[0056] Unpredictable noise, manufacturing tolerances, and intervention of an image signal processing (ISP) pipeline can hinder a typical image processing device from obtaining a kernel for perfect degradation modeling. For example, even when a typical image processing device obtains the input kernel 110 by using optical modeling of a design of the UDC, the input kernel 110 can lack contrast restoration capability due to incomplete modeling. In addition, the typical image processing device can need to supplement contrast functionality (e.g., contrast adjustment functionality, contrast enhancement functionality, etc.) due to insufficient prior information or errors in the prior information.

[0057] In contrast, when the input kernel 110 is provided, the image processing device 100 of one or more embodiments can generate the output kernel 130 by applying the kernel offset 120 to the input kernel 110. The output kernel 130 can have improved contrast adjustment functionality than the input kernel 110 used by a typical image processing device. The image processing device 100 of one or more embodiments can perform image restoration using the output kernel 130, and thus, can obtain the result image 102 with enhanced contrast in the result image 102 compared to when the input kernel 110 is used by a typical image processing device (e.g., without using the kernel offset 120 and / or the output kernel 130). When the input kernel 110 is an existing kernel with predetermined functionality, the contrast adjustment functionality can be supplemented to the input kernel 110 by the kernel offset 120. When the input kernel 110 is a new kernel without functionality, the contrast enhancement functionality can be added to the input kernel 110 by the kernel offset 120.

[0058] The image processing device 100 of one or more embodiments can enhance contrast of the result image 102 by adjusting at least one of a kernel size, an offset pattern, and an application strength of the offset. When the kernel size and the offset pattern are determined, the image processing device 100 can finely adjust the contrast by tuning parameters (e.g., distribution parameters and contrast adjustment parameters).

[0059] The kernel offset 120 can be defined by an offset pattern related to an offset window and an offset parameter related to an application strength of the offset. The kernel offset 120 can be applied to the input kernel 110 to generate the output kernel 130 by summation (e.g., element-wise addition of pixels) of kernel values of the input kernel 110 and offset values of the offset window (or kernel offset), where the offset pattern can indicate a distribution of the offset values. For example, the offset values can follow a Gaussian distribution. In this case, in response to applying the kernel offset 120 to the input kernel 110, kernel values at a central portion of the input kernel 110 can be increased more than kernel values at a peripheral portion of the input kernel 110. The image processing device 100 can apply an offset pattern suitable for a degradation in the degraded image 101 or predetermined for the degradation in the degraded image 101 to the kernel offset 120. Different offset patterns can be applied according to the degradation. The offset parameter can determine an application strength of the offset. As a value of the offset parameter increases, kernel values can be increased more.

[0060] The kernel size can not only affect a computational amount of image restoration, but also affect a restoration result. The image processing device 100 can appropriately adjust the kernel size in consideration of the computational amount and / or the restoration result. Since the restoration result can also be affected by the offset parameter, the image processing device 100 can eliminate an influence of the kernel size on the offset parameter by normalizing the offset parameter based on the kernel size. Thus, the kernel size and the offset application strength can be operated separately.

[0061] Figure 2 is a flowchart illustrating an example of a kernel modification operation using a kernel offset. The operations 210 to 250 of Figure 2 may be performed sequentially or non-sequentially. For example, an order of the operations 210 to 250 can be changed, and / or at least two of the operations 210 to 250 can be performed in parallel.

[0062] Referring to Figure 2 , in operation 210, the image processing device can receive an input kernel. The input kernel can be represented as A in = A The input kernel can be an existing kernel or a new kernel. When the input kernel is a new kernel, the input kernel can be an identity kernel in which only a kernel value at a center is "1" and other kernel values are "0".

[0063] In operation 220, the image processing device can adjust a kernel size (e.g., a height and / or a width of a channel of the kernel). The input kernel having the adjusted size can be represented as A 1The kernel size can be reduced by cropping and increased by zero padding. When the kernel size is reduced, the computational burden can be reduced, but information at the outer part of the kernel can be lost. In contrast, when the kernel size is increased, information can not be lost, but the computational burden can increase. Also, according to the kernel size, artifacts can occur in the restored result image. Accordingly, the image processing device can adjust the kernel size in consideration of the computational burden and / or the degradation characteristic. In some cases, operation 220 can be omitted.

[0064] In operation 230, the image processing device can set a kernel offset. The kernel offset can be expressed as αW Applying the kernel offset to the input kernel can be expressed as A 1 + αW The application result can be expressed as A 2 .

[0065] W The offset window can be expressed as. The offset window can have a size and a dimension corresponding to the input kernel. In other words, the kernel values of the input kernel and the offset values of the offset window can have the same dimension and be provided in the same number. When the size of the input kernel is adjusted, the size of the offset window can be set based on and / or corresponding to the adjusted size of the input kernel. The offset window can determine an offset pattern of the kernel offset. The offset pattern can indicate a distribution of the offset values applied to the input kernel. The kernel offset can be applied to the input kernel by element-wise adding the kernel values of the input kernel and the offset values of the offset window (or the kernel offset).

[0066] α denotes an offset parameter and can determine an application strength of the offset values of the offset window. The offset values of the offset window can be scaled by the offset parameter. As the value of the offset parameter increases, the offset values can be scaled larger and applied to the input kernel.

[0067] In operation 240, the image processing device can normalize the kernel. When the kernel offset is applied to the input kernel, the kernel intensity can be different from the kernel intensity before the kernel offset is applied. The change in the kernel intensity can cause an unexpected effect in addition to the improvement effect expected by the kernel. Accordingly, the image processing device can bring the kernel intensity after the kernel offset is applied back to the kernel intensity before the kernel offset is applied through kernel normalization. When the input kernel after the kernel offset is applied is expressed as A 2 , the image processing device can normalize the kernel based on A 2 A 2 ​Normalization is performed. For example, the sum of the kernel values of the input kernel (K) can be normalized by dividing the sum of the kernel values of the input kernel (K) by the sum of the kernel values of the kernel (K') of the predetermined distribution. A 2 The sum of the kernel values of the input kernel (K) can be normalized by dividing the sum of the kernel values of the input kernel (K) by the sum of the kernel values of the kernel (K') of the predetermined distribution. A 2 The sum of the kernel values of the input kernel (K) can be normalized by dividing the sum of the kernel values of the input kernel (K) by the sum of the kernel values of the kernel (K') of the predetermined distribution. ∑A 2 The sum of the kernel values of the input kernel (K) can be normalized by dividing the sum of the kernel values of the input kernel (K) by the sum of the kernel values of the kernel (K') of the predetermined distribution. A 2 The sum of the kernel values of the input kernel (K) can be normalized by dividing the sum of the kernel values of the input kernel (K) by the sum of the kernel values of the kernel (K') of the predetermined distribution. A 3 .

[0068] In operation 250, the image processing device can determine an output kernel. The image processing device can determine the normalized result A 3 as the output kernel. The output kernel can be expressed as A out . The image processing device can perform image restoration using the output kernel.

[0069] Figure 3 An example of a kernel offset setting operation is illustrated. Referring to Figure 3 , the image processing device can determine an offset window based on a distribution parameter. The distribution parameter can be expressed as λ , and the offset window can be expressed as W . The offset window can include offset values of a predetermined distribution. The distribution of the offset values can be referred to as an offset pattern. Various types of distributions can be used according to characteristics of the degradation and / or a contrast adjustment direction. The image processing device can selectively use an offset pattern suitable for the characteristics of the degradation in the degraded image and / or determined according to the contrast adjustment direction of the characteristics of the degradation.

[0070] For example, the offset pattern can include a probability distribution such as a Gaussian distribution, a perturbation distribution to which modeling related to deep learning is applied, and / or an optical distribution reflecting optical characteristics of an optical device such as a UDC. The probability distribution and the perturbation distribution can alleviate possible artifacts caused by contrast enhancement. The probability distribution and the perturbation distribution can be used in a common case (e.g., to enhance contrast of a typical camera). For example, the probability distribution can alleviate blocking artifacts or shading, and the perturbation distribution can alleviate overfit in a deep learning-based restoration method. More specifically, when kernel offsets following a Gaussian distribution are used, kernel values at a central portion of the input kernel can increase more than kernel values at a peripheral portion of the input kernel (e.g., the kernel values can increase toward the central portion, and the kernel values can decrease toward the peripheral portion). Thus, blocking artifacts can be alleviated. The optical distribution can alleviate degradation of a UDC image. The optical distribution can be used in a special case (e.g., to enhance contrast of a special camera such as a UDC).

[0071] When the distribution type of the offset values is determined, a detailed distribution of the distribution type can be specified or determined by a distribution parameter. Each distribution type can define a parameter for specifying the detailed distribution, and the parameter can be set by the distribution parameter. For example, the distribution parameter for a Gaussian distribution can include a standard deviation. When the offset values are distributed in two dimensions in the offset window, the distribution parameter can include a first standard deviation of a distribution along a first axis of the two dimensions and a second standard deviation of a distribution along a second axis of the two dimensions.

[0072] The image processing device can determine the offset parameter based on a contrast adjustment parameter. The contrast adjustment parameter can be represented as μ , and the offset parameter can be represented as α The contrast adjustment parameter can determine a degree of contrast adjustment. For example, when a value of the contrast adjustment parameter increases, the kernel offset can increase and / or be more strongly applied. The image processing device can set the contrast adjustment parameter based on an image characteristic of the degraded image. For example, a small value of the contrast adjustment parameter can be applied to a flat characteristic for weak offset, and a large value of the contrast adjustment parameter can be applied to an edge characteristic or a texture characteristic for strong offset.

[0073] The contrast adjustment parameter can be normalized based on the offset values of the offset window. For example, the contrast adjustment parameter can be normalized by dividing the contrast adjustment parameter by a sum of the offset values of the offset window ∑W . The offset parameter can correspond to the normalized result. Accordingly, an effect of the kernel size on the offset parameter can be eliminated by the normalization. When the size of the input kernel is adjusted, the size of the offset window can be set based on the size of the adjusted input kernel, and the offset values according to the window size can be reflected in the offset parameter. The size of the offset window can correspond to the kernel size, and the strength of the contrast adjustment parameter can be determined according to the normalization based on the size of the offset window, or adjusted to be suitable for the size of the offset window according to the normalization.

[0074] The image processing device can set the kernel offset based on the offset window and the offset parameter. For example, the image processing device can set a multiplication result of the offset window and the offset parameter as the kernel offset. The offset parameter can scale the offset values of the offset window by the multiplication.

[0075] Figure 4 An example of changing the processing of the kernel is illustrated. Referring to Figure 4 , an input kernel 410 (e.g., Figure 2 in A in or A ) is received. The input kernel 410 can be an existing kernel designed to improve the image quality of the UDC image. The input kernel 420 (e.g., Figure 2 ) can be determined by adjusting the size of the input kernel 410.A 1 ). In Figure 4 example, the input kernel 420 can be determined by cropping the input kernel 410. The input kernel 420 can have a smaller size than the input kernel 410.

[0076] The output kernel 440 can be determined by applying a kernel offset to the input kernel 420 based on the offset window 430 and the offset parameter. The offset window 430 can be determined based on the size of the input kernel 420. The offset window 430 can have a size corresponding to the size of the input kernel 420. In Figure 4 example, the offset values of the offset window 430 can be distributed in a Gaussian pattern, and the kernel values at a central portion of the input kernel 420 can increase more than the kernel values at a peripheral portion of the input kernel 420 (e.g., the kernel values can increase toward the central portion, and the kernel values can decrease toward the peripheral portion). The offset values at the central portion of the offset window can have a maximum value, and the offset values can gradually decrease toward the peripheral portion of the offset window.

[0077] Comparing the input kernel 420 and the output kernel 440 in Figure 4 example, according to the distribution of the offset values of the offset window 430, the kernel values at the central portion of the input kernel 420 can increase more than the kernel values at the peripheral portion of the input kernel 420. The output kernel 440 can exhibit more excellent contrast enhancement performance than the input kernel 410 and the input kernel 420.

[0078] Figure 5 An example of enhancing image quality using the output kernel is illustrated. Referring to Figure 5 , the image processing device can output a result image by performing image quality improvement 510 based on a degraded image and the output kernel. The degraded image can be represented as y , the output kernel can be represented as A out , and the result image can be represented as x' The image processing device can improve image quality by using a method using prior information and / or a method of learning a relationship between the degraded image and the result image. For example, the method using prior information can include a direct filtering method using the output kernel and / or a repetition method. The relationship learning method can include a method of generating training data pairs each including a sample degraded image and a sample result image using the output kernel and learning a relationship between the training data pairs through deep learning.

[0079] Figure 6 An example of the effect of the contrast adjustment parameter on the result image is illustrated. Referring to Figure 6, showing a degraded image 610, a result image 620 according to an existing kernel, and result images 630 and 640 according to modified kernels. For example, the degraded image 610 can be acquired through UDC, and the existing kernel can be acquired or determined based on a photographed pattern. With reference to the result image 620, the result image 620 determined by a typical image processing device using the existing kernel (rather than the modified kernel) can achieve sharpness enhancement and ghost removal of the degraded image 610, but can have limitations in terms of haze removal and contrast enhancement. The image processing device of one or more embodiments can modify the existing kernel through kernel offset. With reference to the result image 630 and the result image 640, in addition to sharpness enhancement and ghost removal, the result images 630 and 640 determined by the image processing device of one or more embodiments using the modified kernel can achieve haze removal and contrast enhancement of the degraded image 610, the modified kernel can complement the existing kernel. The result image 630 can be a result according to an offset having a weak intensity, and the result image 640 can be a result according to an offset having a strong intensity. The offset intensity can be adjusted according to a purpose determined or predetermined according to image quality improvement or a preference of a user or a developer.

[0080] Figure 7 An example showing the effect of an offset window on a result image is shown. With reference to Figure 7 , a result image 710 can be obtained through an offset window without a pattern, and a result image 720 can be obtained through an offset window having a predetermined pattern (e.g., a Gaussian distribution). When kernel offset is applied, artifacts can be caused by strong contrast enhancement, and the offset window can mitigate such artifacts. Comparing the result image 710 and the result image 720, it can be learned that the result image 720 has less artifacts than the result image 710. Various offset patterns can be used according to the characteristics of degradation in a degraded image and / or according to the contrast adjustment direction of the characteristics of degradation.

[0081] Figure 8 An example showing mitigation of glare through kernel modification is shown. With reference to Figure 8 , showing a degraded image 810, a result image 820 according to an existing kernel, and a result image 830 determined by the image processing device of one or more embodiments according to a modified kernel. For example, the degraded image 810 can be acquired through UDC, and the existing kernel can be acquired based on a photographed pattern. Since glare is accompanied by saturation, different kernels can be used according to the intensity of a light source causing glare. Accordingly, the image processing device of one or more embodiments can process glare through different methods. Kernel modification by the image processing device of one or more embodiments can be effective in terms of glare mitigation. With reference to the result image 830, the modified kernel can achieve contrast enhancement even as a single kernel and further mitigate glare.

[0082] Figure 9 An example of kernel setting adaptive to image characteristics is shown. Referring to Figure 9 , the image processing device can detect a first region 910 and a second region 920 having different image characteristics in the degraded image 900. For example, the first region 910 can be a flat area, and the second region 920 can be an edge area. The flat area can be a region including few edges or no edges, and the edge area can represent a region including many edges. The edge area can correspond to a texture region. The image processing device can detect edges in the degraded image 900 using various edge detection techniques, such as a Laplacian filter, and extract the first region 910 and the second region 920 from the degraded image 900 based on the detected edges.

[0083] Since the flat area and the edge area have different image characteristics, different kernel offsets can be set individually. For example, different window sizes and / or offset strengths can be applied for the respective regions. For example, a large kernel size and an offset with weak strength can be suitable for the flat area, and a small kernel size and an offset with strong strength can be suitable for the edge area. The image processing device can set a window size of a first kernel offset of the first region 910 to be larger than a window size of a second kernel offset of the second region 920, and set a strength of an offset parameter of the first kernel offset to be weaker than a strength of an offset parameter of the second kernel offset. A value of a contrast adjustment parameter can be set to be large for a strong offset, and to be small for a weak offset.

[0084] Figure 10 An example of the effect of the difference of kernel setting is shown. Referring to Figure 10 , a degraded image 1010, a result image 1020 according to a uniform kernel, and a result image 1030 according to adaptive kernel are shown. For example, the degraded image 1010 can be acquired by UDC. Comparing the result image 1020 and the result image 1030, it can be learned that the result image 1030 determined by the image processing device of one or more embodiments achieves greater sharpness enhancement and artifact suppression in the edge area, and greater noise reduction in the flat area.

[0085] Figure 11 is a flowchart showing an example of an image processing method. The operations 1110-1130 of Figure 11 may be performed sequentially or non-sequentially. For example, the order of the operations 1110-1130 can be changed, and / or at least two of the operations 1110-1130 can be performed in parallel.

[0086] Referring to Figure 11In operation 1110, the image processing device can set an offset window of an offset pattern for kernel offset and an offset parameter of an application strength for kernel offset. Operation 1110 can include setting a distribution type of offset values in the offset window, and setting a distribution parameter specifying a detailed distribution of the distribution type. The distribution type can include a Gaussian distribution, and the distribution parameter can include a standard deviation of the Gaussian distribution.

[0087] Operation 1110 can include setting a contrast adjustment parameter for a degree of contrast adjustment, and setting the offset parameter by normalizing the contrast adjustment parameter based on the offset values of the offset window. The contrast adjustment parameter can be set based on the image characteristics of the degraded image. The size of the offset window can correspond to the size of the input kernel, and the strength of the contrast adjustment parameter can be adjusted to be suitable for the size of the offset window according to the normalization.

[0088] The image processing device can adjust the size of the input kernel. In this case, operation 1110 can include setting the size of the offset window and the offset parameter based on the size of the adjusted input kernel.

[0089] The image processing device can detect a first region and a second region having different image characteristics in the degraded image. In this case, operation 1110 can include setting the offset window and the offset parameter individually for each of the first region and the second region. The first region can correspond to a flat region, and the second region can correspond to an edge region. Operation 1110 can include setting the size of a first offset window for the first region to be greater than the size of a second offset window for the second region, and setting the strength of a first offset parameter for the first region to be weaker than the strength of a second offset parameter for the second region.

[0090] In operation 1120, the image processing device can determine an output kernel by applying kernel offset to the input kernel based on the offset window and the offset parameter. Operation 1120 can include determining a new input kernel by applying kernel offset to the input kernel, and determining the output kernel by normalizing the new input kernel based on kernel values of the new input kernel.

[0091] In operation 1130, the image processing device can adjust the contrast of the degraded image using the output kernel.

[0092] In addition, the descriptions provided with reference to Figures 1 to 10 , Figure 12 and Figure 13 can be applied to the image processing method.

[0093] Figure 12 is a block diagram illustrating an example of an image processing device. With reference to Figure 12The image processing device 1200 includes a processor 1210 (e.g., one or more processors) and a memory 1220 (e.g., one or more memories). The memory 1220 can be connected to the processor 1210, and can store instructions executable by the processor 1210, data to be operated on by the processor 1210, or data processed by the processor 1210. The memory 1220 can include a non-transitory computer readable medium (e.g., a high-speed random access memory) and / or a non-volatile computer readable medium (e.g., at least one disk storage device, a flash device, or another non-volatile solid state memory device).

[0094] The processor 1210 can execute instructions to perform any one or more or all of the operations and methods described above with reference to Figures 1 to 11 and below with reference to Figure 13 For example, the processor 1210 can set an offset window of an offset pattern for a kernel offset and an offset parameter of an application strength for the kernel offset, determine an output kernel by applying the kernel offset to an input kernel based on the offset window and the offset parameter, and adjust a contrast of a degraded image using the output kernel.

[0095] In addition, the descriptions provided with reference to Figures 1 to 11 and Figure 13 may be applicable to the image processing device 1200.

[0096] Figure 13 is a block diagram illustrating an example of an electronic device. Referring to Figure 13 , the electronic device 1300 can include a processor 1310 (e.g., one or more processors), a memory 1320 (e.g., one or more memories), a camera 1330, a storage 1340, an input device 1350, an output device 1360, and a network interface 1370, which can communicate with each other by a communication bus 1380. For example, the electronic device 1300 can be implemented as at least a part of a mobile device (such as a mobile phone, a smart phone, a PDA, a netbook, a tablet computer, or a laptop computer), a wearable device (such as a smart watch, a smart band, or smart glasses), a computing device (such as a desktop computer or a server), a home appliance (such as a television, a smart television, or a refrigerator), a security device (such as a door lock), or a vehicle (such as an autonomous vehicle or a smart vehicle). The electronic device 1300 can structurally and / or functionally include the image processing device 100 of Figure 1 and / or the image processing device 1200 of Figure 12 .

[0097] The processor 1310 executes functions and instructions for performing in the electronic device 1300. For example, the processor 1310 can process instructions stored in the memory 1320 or the storage 1340. The processor 1310 can perform any one or more or all of the operations and methods described above with reference to FIGS. 1 through 12. The memory 1320 can include a computer readable storage medium or a computer readable storage device. The memory 1320 can store instructions to be executed by the processor 1310, and can store information about software and / or applications as they are executed by the electronic device 1300. Figures 1 to 12

[0098] The camera 1330 can take photos and / or videos (e.g., the degraded image 101). The storage 1340 includes a computer readable storage medium or a computer readable storage device. The storage 1340 can store a larger amount of information than the memory 1320 for a long time. For example, the storage 1340 can include a magnetic hard disk, an optical disk, a flash memory, a floppy disk, or other types of non-volatile memory known in the art.

[0099] The input device 1350 can receive input from a user in a conventional input manner through a keyboard and a mouse, and in a new input manner such as touch input, voice input, and image input. For example, the input device 1350 can include a keyboard, a mouse, a touch screen, a microphone, or any other device that detects input from a user and transmits the detected input to the electronic device 1300. The output device 1360 can provide output of the electronic device 1300 to a user through a visual channel, an auditory channel, or a tactile channel. The output device 1360 can include, for example, a display, a touch screen, a speaker, a vibration generator, or any other device that provides output to a user. The network interface 1370 can communicate with external devices through a wired network or a wireless network.

[0100] Hereinabove Figures 1 to 13 ​The described image processing devices, processors, memories, cameras, storage devices, input devices, output devices, network interfaces, communication buses, image processing device 100, image processing device 1200, processor 1210, memory 1220, processor 1310, memory 1320, camera 1330, storage device 1340, input device 1350, output device 1360, network interface 1370, communication bus 1380, and other devices, apparatuses, units, modules, and components are implemented by or represent hardware components. Examples of hardware components that can be used to perform the operations described in this application include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components performing operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer can be implemented by one or more processing elements, such as an array of logic gates, a controller and arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices configured to respond to and process instructions in a defined manner to achieve desired results. In one example, a processor or computer includes or is connected to one or more memories that store instructions or software for execution by the processor or computer. The hardware components implemented by the processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described in this application. The hardware components can also access, manipulate, process, create, and store data in response to execution of the instructions or software. For simplicity, the singular term "processor" or "computer" can be used in the description of the examples described in this application, but in other examples, multiple processors or computers can be used, or a processor or computer can include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components can be implemented by one or more processors, or a processor and a controller, and one or more other hardware components can be implemented by one or more other processors, or additional processors and additional controllers. The one or more processors, or a processor and a controller, can implement a single hardware component or two or more hardware components.The hardware components can have any one or more of a variety of different processing configurations, examples of which include a single processor, multiple processors, parallel processors, single-instruction single-data (SISD) multiprocessing, single-instruction multiple-data (SIMD) multiprocessing, multiple-instruction single-data (MISD) multiprocessing, and multiple-instruction multiple-data (MIMD) multiprocessing.

[0101] The methods illustrated in the figures in the present application are performed by computing hardware (e.g., by one or more processors or computers) implemented to execute instructions or software as described above to perform the operations described in the present application performed by the methods. For example, a single operation or two or more operations can be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations can be performed by one or more processors, or a processor and a controller, and one or more other operations can be performed by one or more other processors, or further processors and further controllers. The one or more processors, or a processor and a controller, can perform a single operation or two or more operations. Figures 1 to 13 The methods illustrated in the figures in the present application are performed by computing hardware (e.g., by one or more processors or computers) implemented to execute instructions or software as described above to perform the operations described in the present application performed by the methods. For example, a single operation or two or more operations can be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations can be performed by one or more processors, or a processor and a controller, and one or more other operations can be performed by one or more other processors, or further processors and further controllers. The one or more processors, or a processor and a controller, can perform a single operation or two or more operations.

[0102] The instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement the hardware components and perform the methods as described above can be written as a computer program, a code segment, instructions, or any combination thereof, to individually or collectively instruct or configure one or more processors or computers to operate as a machine or special purpose computer to perform the operations performed by the hardware components and methods as described above. In one example, the instructions or software include machine code (such as produced by a compiler) directly executable by the one or more processors or computers. In another example, the instructions or software include high-level code to be executed by the one or more processors or computers using an interpreter. The instructions or software can be written in any programming language based on the block diagrams and flowcharts illustrated in the figures and corresponding descriptions in the specification, which disclose algorithms for performing the operations performed by the hardware components and methods as described above.

[0103] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement the hardware components and perform the methods as described above, as well as any associated data, data files, and data structures, can be recorded, stored, or fixed in one or more non-transitory computer-readable storage media or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include read-only memory (ROM), programmable read-only memory (PROM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random- access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, a hard disk drive (HDD), a solid-state drive (SSD), a card-type memory such as a multimedia card or a micro card (e.g., a secure digital (SD) or extreme digital (XD)), a magnetic tape, a floppy disk, a magneto-optical data storage device, an optical data storage device, a hard disk, a solid state disk, and any other device configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and provide the instructions or software and any associated data, data files, and data structures to one or more processors or computers so that the one or more processors and computers can execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed over a networked computer system so that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed by one or more processors or computers in a distributed manner.

[0104] While the present disclosure includes certain examples, it will be clear to those skilled in the art that various changes can be made to the form and details of these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered in a descriptive sense only and not for purposes of limitation. Descriptions of features or aspects in each example should be considered as being applicable to similar features or aspects in other examples. Suitable results can be achieved if the described techniques are performed in a different order, and / or if components in the described systems, architectures, devices, or circuits are combined in a different manner, and / or replaced or supplemented by other components or their equivalents.

Claims

1. A method of image processing, comprising: setting an offset window of an offset pattern for kernel offset and an offset parameter of an application strength for kernel offset; determining an output kernel by applying the kernel offset to an input kernel based on the offset window and the offset parameter; and adjusting contrast of a degraded image using the output kernel, and wherein the method further comprises adjusting a size of the input kernel, and wherein the step of setting the offset window and the offset parameter comprises setting a size of the offset window and the offset parameter based on the adjusted size of the input kernel. The step of setting the offset window and the offset parameter comprises:

2. The method of claim 1, wherein, setting a distribution type of offset values in the offset window; setting a distribution parameter specifying a detailed distribution of the distribution type; and determining the offset window based on the distribution parameter. 3.The method of claim 2, wherein the distribution type comprises a Gaussian distribution, and the distribution parameter comprises a standard deviation of the Gaussian distribution. The step of setting the offset window and the offset parameter comprises:

4. The method of claim 1, wherein, setting a contrast adjustment parameter of a degree of contrast adjustment; and setting the offset parameter by normalizing the contrast adjustment parameter based on the offset values of the offset window. The step of setting the contrast adjustment parameter comprises setting the contrast adjustment parameter based on an image characteristic of the degraded image.

5. The method of claim 4, wherein, 6.The method of claim 4, wherein a size of the offset window corresponds to a size of the input kernel, and a strength of the contrast adjustment parameter is adjusted according to the normalization based on the size of the offset window. 7.The method of claim 1, further comprising: detecting a first region and a second region having different image characteristics in the degraded image, wherein the step of setting the offset window and the offset parameter comprises setting the offset window and the offset parameter separately for each of the first region and the second region. 8.The method of claim 7, wherein the first region and the second region respectively correspond to a flat region and an edge region, and the step of setting the offset window and the offset parameter comprises: setting a size of a first offset window for the first region to be larger than a size of a second offset window for the second region; and setting a strength of a first offset parameter for the first region to be weaker than a strength of a second offset parameter for the second region. The step of determining the output kernel comprises:

9. The method of any one of claims 1 to 8, wherein, determining a new input kernel by applying the kernel offset to the input kernel; and determining the output kernel by normalizing the new input kernel based on kernel values of the new input kernel. 10.A non-transitory computer-readable storage medium storing instructions which, when executed by a processor, configure the processor to perform the method of any one of claims 1 to 9. 11.An apparatus of image processing, comprising: a processor configured to: set an offset window of an offset pattern for kernel offset and an offset parameter of an application strength for kernel offset, determine an output kernel by applying the kernel offset to an input kernel based on the offset window and the offset parameter, and adjust contrast of a degraded image using the output kernel, and wherein the processor is further configured to adjust a size of the input kernel, and wherein the processor is further configured to set a size of the offset window and the offset parameter based on the adjusted size of the input kernel. wherein the processor is further configured to set a size of the offset window and the offset parameter based on a size of the adjusted input kernel.

12. The apparatus of claim 11, wherein, To set the offset window and the offset parameter, the processor is further configured to: set a distribution type of offset values in the offset window; set a distribution parameter of a detailed distribution of the specified distribution type; and determine the offset window based on the distribution parameter.

13. The apparatus of claim 11, wherein, To set the offset window and the offset parameter, the processor is further configured to: set a contrast adjustment parameter for a degree of contrast adjustment; and set the offset parameter by normalizing the contrast adjustment parameter based on the offset values of the offset window.

14. The apparatus of claim 11, wherein, The processor is further configured to: detect a first region and a second region having different image characteristics in the degraded image; and To set the offset window and the offset parameter, set the offset window and the offset parameter individually for each of the first region and the second region.

15. The device of claim 14, wherein, the first region and the second region correspond to a flat region and an edge region, respectively, and To set the offset window and the offset parameter, the processor is further configured to: set a size of a first offset window for the first region to be larger than a size of a second offset window for the second region; and set a strength of a first offset parameter for the first region to be weaker than a strength of a second offset parameter for the second region. To determine the output kernel, the processor is further configured to:

16. The apparatus of claim 11, wherein, determine a new input kernel by applying the kernel offset to the input kernel; and determine the output kernel by normalizing the new input kernel based on kernel values of the new input kernel. a memory storing instructions that, when executed by the processor, configure the processor to perform the process of setting the offset window and the offset parameter, the process of determining the output kernel, and the process of adjusting the contrast.

17. The apparatus of any one of claims 11-16, further comprising:

18. An electronic device, comprising: a processor configured to: set a distribution type of offset values in the offset window and a distribution parameter of a detailed distribution of the specified distribution type; set the offset window based on the distribution type and the distribution parameter; set a contrast adjustment parameter for a degree of contrast adjustment; set the offset parameter by normalizing the contrast adjustment parameter based on the offset values of the offset window; determine an output kernel by applying a kernel offset to an input kernel based on the offset window and the offset parameter; and adjust a contrast of a degraded image using the output kernel, and wherein the processor is further configured to adjust a size of the input kernel, and wherein the processor is further configured to set a size of the offset window and the offset parameter based on a size of the adjusted input kernel. The processor is further configured to: 19.The electronic device of claim 18, wherein, determine a new input kernel by applying the kernel offset to the input kernel; and determine the output kernel by normalizing the new input kernel based on kernel values of the new input kernel.

20. An electronic device, comprising: one or more sensors configured to obtain an input image; and one or more processors configured to: determine a kernel offset based on the distribution parameter and the contrast adjustment parameter, determine an output kernel by applying the kernel offset to the input kernel, and ​ determining an output image by applying an output kernel to the input image, and wherein the one or more processors are further configured to adjust a size of the input kernel, and wherein the one or more processors are further configured to set a distribution parameter and a contrast adjustment parameter based on the adjusted size of the input kernel. 21.The electronic device of claim 20, wherein, To determine the kernel offset, the one or more processors are configured to: determine an offset window based on the distribution parameter; determine an offset parameter based on the contrast adjustment parameter; and determine the kernel offset based on the offset window and the offset parameter. 22.The electronic device of claim 21, wherein, The kernel values of the offset window increase toward a center portion of the offset window and decrease toward a peripheral portion of the offset window. 23.The electronic device of any one of claims 20-22, wherein, the one or more sensors include an under-display camera, and to obtain the input image, the under-display camera is configured to obtain a degraded image.

Images