Image system, operation method of image signal processor and contrast enhancement method

Abstract

An image system, an operating method of an image signal processor, and a contrast enhancement method are provided. The image signal processor (ISP) is configured to perform signal processing on an original image received from an image device. The operating method includes: generating a plurality of multi-scale images based on an input image, the plurality of multi-scale images having different resolutions from each other; iteratively performing an operation based on fast global weighted least squares (FGWLS) on each of the plurality of multi-scale images to generate a final illumination map; and outputting an enhanced image based on the final illumination map and the input image.

Description

[0001] This application claims the priority benefit of Korean Patent Application No. 10-2020-0038285 filed on March 30, 2020, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0002] Example embodiments of the present disclosure relate to an image device, and more particularly, to an image system including an image signal processor and an operating method of the image signal processor. Background Art

[0003] An image sensor included in a smart phone, a tablet personal computer (PC), or a digital camera obtains image information about an external object by converting light reflected from the external object into an electrical signal. Various image signal processing operations are performed to convert the electrical signal obtained from the image sensor into image information that a person can actually perceive or to improve image quality. Summary of the invention

[0004] One or more example embodiments provide an image system including an image signal processing unit and an operating method of an image signal processor.

[0005] According to an aspect of an example embodiment, there is provided an operating method of an image signal processor (ISP), the ISP being configured to perform signal processing on an original image received from an image device, the operating method comprising: generating a plurality of multi-scale images based on an input image, the plurality of multi-scale images having resolutions different from each other; iteratively performing an operation based on a fast global weighted least squares (FGWLS) on each of the plurality of multi-scale images to generate a final illumination map; and outputting an enhanced image based on the final illumination map and the input image.

[0006] According to an aspect of an example embodiment, an image system is provided, comprising: an image sensor configured to output an original image based on light reflected from an external object; and an image signal processor configured to: perform preprocessing on the original image to generate an input image, extract a final illumination map based on the input image through a multi-scale-based extraction operation, and generate an enhanced image based on the final illumination map and the input image, wherein the multi-scale-based extraction operation comprises: generating a plurality of multi-scale images with different resolutions based on the input image; iteratively performing an illumination map extraction operation on the plurality of multi-scale images; and outputting a final illumination map as a result of iteratively performing the illumination map extraction operation.

[0007] In one exemplary embodiment, the illumination map extraction operation is performed based on:

[0008]

[0009] Wherein, (x, y) indicates a pixel position, FGWLS indicates a function based on the operation of Fast Global Weighted Least Squares (FGWLS), and L i Indicates the calculation result of FGWLS, ↑ n indicates a function that amplifies the input by a factor of n. Indicates that the initial illumination map is reduced by n l The illumination diagram obtained by multiplying i Indicates the guidance image, λ i Indicates the level of flattening used in the function based on FGWLS operation, T i The number of times of row-based calculation and column-based calculation used in a function for an FGWLS-based operation is indicated, and an illuminance map extraction operation is performed based on “i” changing stepwise from “1” to “0”.

[0010] According to another aspect of an example embodiment, there is provided an image contrast enhancement method for an image signal processor, the method comprising: generating an initial illumination map based on an input image; iteratively reducing the initial illumination map by a factor of n to generate a plurality of multi-scale images, wherein n is a positive integer greater than 1; performing a first calculation based on a first image having the lowest resolution among the plurality of multi-scale images; performing a second calculation based on a result of the first calculation and a second image having the second lowest resolution among the plurality of multi-scale images; performing a third calculation based on a result of the second calculation and a third image having the highest resolution among the plurality of multi-scale images; generating a result of the third calculation as a final illumination map; and generating an enhanced image based on the final illumination map and the input image, wherein the first calculation, the second calculation, and the third calculation are based on fast global weighted least squares (FGWLS). BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and / or other aspects will become apparent by describing in detail example embodiments with reference to the accompanying drawings, in which:

[0012] Figure 1 is a block diagram illustrating an image system according to an example embodiment;

[0013] Figure 2 It is shown Figure 1 A block diagram of an image signal processing unit;

[0014] Figure 3 It is shown Figure 2 A flowchart of the operation of the enhancement module;

[0015] Figure 4 It is shown Figure 2 Block diagram of the enhancement module;

[0016] Figure 5 It is shown Figure 4 A flowchart of the operation of an illumination map extraction unit;

[0017] Figure 6 It is shown Figure 4 A block diagram of an illumination map extraction unit;

[0018] Figure 7 It is shown Figure 5 Flow chart of operation S123;

[0019] Fig. 8A and Figure 8B It is shown Figure 6 Block diagram of the multi-scale FGWLS;

[0020] Fig. 9 Is used to describe Figures 6 to 8B A diagram showing an example effect of the operation of an illumination map extraction unit;

[0021] Fig.10 It is shown Figure 4 Block diagram of the enhancement unit;

[0022] Fig.11 It is shown Figure 4 A block diagram of a color restoration unit;

[0023] Fig. 12A and Fig. 12B is a diagram for describing an example effect according to an example embodiment;

[0024] Fig.13 is a block diagram illustrating an image sensor device according to example embodiments; and

[0025] Fig.14 is a block diagram illustrating an electronic device according to an example embodiment. DETAILED DESCRIPTION

[0026] Hereinafter, example embodiments may be described in detail and clearly to such an extent that a person having ordinary skill in the art can implement the example embodiments.

[0027] Hereinafter, components described with reference to the terms "component", "unit", "module", "layer", etc. in the detailed description and functional blocks shown in the drawings may be implemented in the form of software, hardware, or a combination thereof. For example, software may be machine code, firmware, embedded code, and application software. For example, hardware may include circuits, electronic circuits, processors, computers, integrated circuits, integrated circuit cores, pressure sensors, inertial sensors, micro-electromechanical systems (MEMS), passive components, or a combination thereof.

[0028] Figure 1is a block diagram illustrating an image system according to an example embodiment. Figure 1 The image system 10 may include an image sensor (or an image sensor device) 11 and an image signal processor (or an image signal processing unit) 12. The image system 10 may be at least one of various electronic systems configured to obtain a video or image associated with an external object (such as, for example, a smart phone, a tablet computer, a black box, and an electronic device for a vehicle), or the image system 10 may be included in at least one of various electronic systems configured to obtain a video or image associated with an external object (such as, for example, a smart phone, a tablet computer, a black box, and an electronic device for a vehicle).

[0029] The image sensor 11 may receive light reflected from an external object, and may obtain an image, image information, or image data associated with the external object based on the received light. For example, the image sensor 11 may include at least one of various image sensors such as a complementary metal oxide semiconductor (CMOS) image sensor, a charge coupled device (CCD) image sensor, and a dynamic vision sensor (DVS) image sensor. The image sensor 11 may output an image generated based on light reflected from an external object as a raw image RAW. In an example embodiment, the raw image RAW may indicate image data obtained by the image sensor 11. That is, the raw image RAW may be an original of image data that has not yet undergone separate image processing. Alternatively, the raw image RAW may also be an image that has undergone image processing (e.g., a high dynamic range (HDR) or wide dynamic range (WDR) performed in the image sensor 11).

[0030] The image signal processor 12 may receive the raw image RAW from the image sensor 11, and may perform image signal processing on the received raw image RAW. The image signal processor 12 may output an output image IMG_out as a result of the image signal processing. In an example embodiment, the output image IMG_out may have enhanced image quality compared to the raw image RAW. The output image IMG_out may be provided to an external device (e.g., an application processor (AP), a graphics processor (GPU), or a display device).

[0031] Figure 2 It is shown Figure 1 The image signal processor block diagram is shown in Figure 1. Figure 1 and Figure 2The image signal processor 12 may perform various image signal processing on the raw image RAW received from the image sensor 11, and may generate or output an output image IMG_out with enhanced quality. For example, the image signal processor 12 may include a noise reduction module 12a, a white balance module 12b, a color filter array (CFA) interpolation module 12c, a correction module 12d, an enhancement module 100, and a memory device 12e.

[0032] The noise reduction module 12a may perform image signal processing such as noise reduction for reducing noise of a raw image RAW received from the image sensor 11. The white balance module 12b may perform image signal processing such as white balancing the output of the noise reduction module 12a. The color filter array interpolation module 12c may perform image signal processing such as color interpolation based on the color filter array of the image sensor 11 on the output of the white balance module 12b. The correction module 12d may perform image signal processing such as correction based on various factors such as color and gamma characteristics on the output of the color filter array interpolation module 12c.

[0033] The enhancement module 100 may perform image signal processing such as an enhancement operation for enhancing image quality (such as edges, contrast, and colors) on the output of the correction module 12d. The output of the enhancement module 100 may be output as an output image IMG_out. The memory device 12e may be configured to store outputs or intermediate results of the noise reduction module 12a, the white balance module 12b, the CFA interpolation module 12c, the correction module 12d, and the enhancement module 100 included in the image signal processor 12.

[0034] As described above, the image signal processor 12 may generate an output image IMG_out having enhanced quality by performing various image signal processing on the raw image RAW from the image sensor 11. In an exemplary embodiment, Figure 2 The configuration of the image signal processor 12 or the image signal processing operation shown in FIG. 1 is an example, and the embodiment is not limited thereto. For example, the image signal processor 12 may be based on Figure 2 A configuration different from the configuration shown in performs image signal processing on a raw image RAW.

[0035] In an example embodiment, the enhancement module 100 of the image signal processor 12 may be configured to perform contrast enhancement on an image. The image signal processing (in detail, image signal processing for contrast enhancement) of the enhancement module 100 in the image signal processor 12 will be described more fully below. For example, an example embodiment will be described as the image signal processor 12 or the enhancement module 100 performing image signal processing for contrast enhancement on an input image, but the embodiment is not limited thereto.

[0036] In the case where an image is obtained in a backlit environment or a low-illuminance environment, it may be necessary to perform image signal processing for contrast enhancement on the input image. For example, in the case where the image sensor 11 obtains an image in a shooting environment such as backlight or low illumination, because the brightness of a partial area of ​​the obtained image is relatively low, the object included in the partial area may not be recognized. In order to recognize the object in the partial area, the brightness of the entire image may be increased. However, since the brightness of different areas of the image (for example, relatively bright areas) is increased together, the object included in the different areas may not be recognized.

[0037] Therefore, in order to enhance the quality of images obtained in shooting environments such as backlight or low illumination, it is important to minimize artifacts (such as halo or fading) and effectively improve the local contrast of the image. Schemes for improving the quality (e.g., contrast) of images may include histogram-based schemes, defogging-based schemes, filtering-based schemes, retinex-based schemes, etc. Among the above schemes, the retinex-based scheme considers that the input image is defined by illuminance and reflectance, as shown in the following equation 1.

[0038] [Equation 1]

[0039] F(x, y) = R(x, y) L(x, y)

[0040] Referring to the above equation 1, F indicates an input image, R indicates reflectance information about the input image, L indicates illuminance information about the input image, and (x, y) indicates coordinates in the input image. As understood from the above equation 1, the input image F can be represented based on L as an illuminance component and R as a reflectance component.

[0041] In order to enhance the contrast of the input image, it is necessary to extract the illumination information or illumination map associated with the input image. The illumination of the input image may be determined based on the overall distribution of brightness in the input image, and the reflectivity may be determined by the unique characteristics of the object in the input image. For example, the illumination component of the input image may mainly include low-frequency components, and the reflectivity component of the input image may mainly include high-frequency components. Therefore, as shown in the following equation 2, the illumination information or illumination map may be extracted by applying a low-pass filter to the input image, and as shown in the following equation 3, the reflectivity may be extracted by removing information about the illumination map from the input image.

[0042] [Equation 2]

[0043] L(x, y) = LPF(F(x, y))

[0044] [Equation 3]

[0045] lnR(x,y)=lnF(x,y)-lmL(x,y)

[0046] In Equation 2 and Equation 3, LPF indicates a function corresponding to low pass filtering, and the remaining reference symbols are described with reference to the above Equation 1. In the case of extracting illuminance information based on Equation 2 and Equation 3, since the high frequency component is filtered, an edge (e.g., a high frequency component) is not retained in the extracted illuminance information or illuminance map. When the edge is not retained in the illuminance information, artifacts (such as a halo phenomenon) may occur in the final image.

[0047] In order to solve the above problems, a scheme for extracting an illumination map based on weighted least squares (WLS) can be used. The extraction scheme based on WLS indicates a scheme for extracting an illumination map by obtaining an optimal solution of a matrix composed of data items and constraint items. The extraction scheme based on WLS can strongly flatten similar brightness areas while retaining the edges of the input image. In this case, the extraction scheme based on WLS can obtain the optimal solution of the matrix by calculating the inverse matrix of a sparse Laplacian matrix. In this case, the elements of the sparse Laplacian matrix can indicate the similarity between pixels. The advantage of the extraction scheme based on WLS is that the flattening strength is large while retaining the edges. However, because the extraction scheme based on WLS needs to calculate a large-size matrix, a large amount of hardware resources are used, and the calculation speed is very slow, and it may be difficult to apply the extraction scheme based on WLS to mobile applications with limited hardware resources.

[0048] A fast global weighted least squares (FGWLS) extraction scheme may be used as an enhanced version of a WLS-based illumination map extraction scheme. An FGWLS-based extraction scheme or operation may reduce the amount of hardware resources used and the operation time by simplifying a WLS-based optimal solution. For example, an FGWLS-based extraction scheme may reduce the hardware resources and operation time required for calculation by iteratively performing row-based and column-based calculations instead of performing calculations on the entire matrix. However, in the case where the size or resolution of the input image is large, block artifacts may occur in the illumination map extracted by the FGWLS-based extraction scheme, and the block artifacts may cause a reduction in the quality of the output image IMG_out.

[0049] According to an example embodiment, the image signal processor 12 or the enhancement module 100 can prevent block artifacts from occurring in the extracted illumination map by generating a multiscale image for the input image and iteratively performing an FGWLS-based extraction operation on the generated multiscale image by performing a multiscale-based extraction operation. According to an example embodiment, how the image signal processor 12 or the enhancement module 100 processes an image signal to enhance contrast will be described in detail below.

[0050] Figure 3 It is shown Figure 2 Flowchart of the operation of the enhancement module 100. Performing fast global weighted least squares (FGWLS) may mean performing an FGWLS-based extraction or calculation on a specific image once or performing an FGWLS-based extraction or calculation on a specific image during a cycle. In this case, the multi-scale-based illumination map extraction operation according to the example embodiment may be performed by iteratively performing FGWLS on the multi-scale images. For example, performing FGWLS may include performing an FGWLS-based extraction operation on a specific image or one of the multi-scale images, and performing the multi-scale-based illumination map extraction operation may include iteratively performing an FGWLS-based extraction operation on multiple images. However, the embodiments are not limited thereto.

[0051] Reference Figure 2 and Figure 3 In operation S110, the enhancement module 100 may receive an input image IMG_in. The image of the enhancement module 100 input to the image signal processor 12 is referred to as an input image IMG_in. Figure 2 In the exemplary embodiment of , the input image IMG_in may be an image output from the correction module 12d. The input image IMG_in may be an image output from a module located before the enhancement module 100. The input image IMG_in may be an image (e.g., a color-enhanced image or a gamma-enhanced image) generated by processing the input image IMG_in at the enhancement module 100. For example, the input image IMG_in may be an image that has been subjected to image signal processing for contrast enhancement or a pre-processed image.

[0052] In operation S120, the enhancement module 100 may perform a multi-scale-based extraction operation on the input image IMG_in to generate a final illumination map LM_fin. For example, the enhancement module 100 may generate a multi-scale image based on the input image IMG_in. The multi-scale image may include a plurality of images, and the plurality of images may have different resolutions. The enhancement module 100 may perform a first FGWLS on a first image of the plurality of images included in the multi-scale image, and may perform a second FGWLS based on the result of the first FGWLS and the second image of the plurality of images. That is, the enhancement module 100 may perform a multi-scale-based extraction operation by iteratively performing FGWLS while transmitting the result of the current FGWLS to the next FGWLS. The enhancement module 100 may output the result of the multi-scale-based extraction operation (the result of iteratively performing FGWLS) as the final illumination map LM_fin. Operation S120 will be described more fully below with reference to the accompanying drawings.

[0053] In operation S130, the enhancement module 100 may generate a grayscale image GR based on the input image IMG_in. For example, the enhancement module 100 may generate a grayscale image GR associated with the input image IMG_in through a grayscale filter.

[0054] In operation S140 , the enhancement module 100 may perform contrast enhancement based on the grayscale image GR and the final illumination map LM_fin, and may generate an enhanced grayscale image GR_enh.

[0055] In operation S150, the enhancing module 100 may output an enhanced image IMG_enh based on the enhanced grayscale image GR_enh and the input image IMG_in.

[0056] Figure 4 It is shown Figure 2 The block diagram of the enhancement module 100 is shown in FIG. Figure 4 Description based on Figure 3 The operation of the flowchart (ie, image signal processing for contrast enhancement) is shown in FIG. Figures 2 to 4 The enhancement module 100 may include an illumination map extraction unit 110 , a grayscale image generation unit (or, a grayscale image generator) 120 , an enhancement unit 130 , and a color restoration unit 140 .

[0057] The illuminance map extraction unit 110 may generate a final illuminance map LM_fin based on the input image IMG_in. For example, the illuminance map extraction unit 110 may generate an initial illuminance map associated with the input image IMG_in. The illuminance map extraction unit 110 may generate a multi-scale image based on the initial illuminance map. The multi-scale image may include a plurality of images, and the plurality of images may have different resolutions.

[0058] The illumination map extraction unit 110 may perform a first FGWLS on a first image having a minimum resolution among a plurality of images included in the multi-scale image. The illumination map extraction unit 110 may enlarge a result of the first FGWLS, and may perform a second FGWLS on the enlarged result. In this case, the illumination map extraction unit 110 may use a second image having a next minimum resolution (e.g., a second minimum resolution) among the plurality of images as a guide image for the second FGWLS. In an exemplary embodiment, the guide image may be used as an image or information for preserving edges when FGWLS is performed.

[0059] As described above, the illuminance map extraction unit 110 may output a final illuminance map LM_fin as a result of iteratively performing FGWLS on a plurality of images, and the final illuminance map LM_fin may correspond to the initial illuminance map. The illuminance map extraction unit 110 may perform a multi-scale based extraction operation. Figures 5 to 9The configuration of the illumination map extraction unit 110 and the multi-scale based extraction operation are described more fully.

[0060] The grayscale image generation unit 120 may generate a grayscale image GR associated with the input image IMG_in. For example, the grayscale image generation unit 120 may generate the grayscale image GR associated with the input image IMG_in by using a grayscale filter.

[0061] The enhancement unit 130 may generate an enhanced grayscale image GR_enh based on the final illumination map LM_fin and the grayscale image GR. For example, the enhancement unit 130 may extract the reflectivity based on the grayscale image GR and the final illumination map LM_fin. The enhancement unit 130 may perform a calculation for enhancing the global contrast of the image based on the final illumination map LM_fin, and may perform a calculation for enhancing the local contrast of the image based on the reflectivity. The enhancement unit 130 may output the enhanced grayscale image GR_enh based on the calculation result. Fig.10 The configuration and operation of the enhancement unit 130 are described more fully.

[0062] The color restoration unit 140 may output an enhanced image IMG_enh by restoring the color of the enhanced grayscale image GR_enh based on the enhanced grayscale image GR_enh, the input image IMG_in, and the grayscale image GR. Fig.11 The configuration and operation of the color restoration unit 140 are described more fully.

[0063] As described above, according to an exemplary embodiment, the image signal processor 12 or the enhancement module 100 of the image signal processor 12 may extract an illumination map or a final illumination map of the input image IMG_in according to a multi-scale-based extraction operation. In this case, since FGWLS is performed on each multi-scale image generated from the input image and the result of the current FGWLS is transmitted as an input of the next FGWLS, even if the flattening level used in the FGWLS is reduced, a final illumination map LM_fin having an expected flattening level may be extracted. In addition, since the flattening level of the FGWLS is relatively low, block artifacts that occur in a conventional FGWLS (FGWLS for a single image) may be prevented. Therefore, the quality of the final output image may be enhanced.

[0064] Figure 5 It is shown Figure 4 Flowchart of the operation of the illumination map extraction unit. Figure 6 It is shown Figure 4 Block diagram of the illumination map extraction unit. Figures 4 to 6 In operation S121, the illumination map extraction unit 110 may generate an initial illumination map LM_int based on the input image IMG_in. Figure 6As shown in , the initial illumination map generator 111 of the illumination map extraction unit 110 may generate an initial illumination map LM_int based on the following Equation 4.

[0065] [Equation 4]

[0066]

[0067] Reference Figure 4 , Indicates an initial illuminance map LM_int, r(x, y), g(x, y), and b(x, y) respectively indicate the values ​​of a red pixel, a green pixel, and a blue pixel located at coordinates (x, y) in the input image IMG_in. In an exemplary embodiment, it is assumed that the above equation 4 indicates a value of a hue, saturation, and value (HSV) color space, and the illuminance is equal to or greater than the maximum value of the red value, the green value, and the blue value. In an exemplary embodiment, the initial illuminance map LM_int may be generated based on at least one of a scheme different from that of equation 4, for example, various illuminance map extraction schemes such as a low-pass filter, a bilateral filter, a Gaussian filter, and a WLS.

[0068] In operation S122, the illumination map extraction unit 110 may generate a multi-scale image img_ms based on the initial illumination map LM_int. Figure 6 As shown in FIG. 1 , the multi-scale image generator 112 may generate a multi-scale image img_ms based on the initial illumination map LM_int. The multi-scale image img_ms may include a plurality of images img_0 to img_1.

[0069] Multiple images img_0 to img_1 may have different resolutions. For example, the zeroth image img_0 may be the same image as the initial illumination map LM_int. The resolution of the zeroth image img_0 may be the same as the resolution of the initial illumination map LM_int or the resolution of the input image IMG_in. The first image img_1 may be an image obtained by reducing the zeroth image img_0 by n times or downsampling n times (n is a positive integer greater than 1 or a positive real number greater than 1). The first image img_1 may be smaller in size than the zeroth image img_0. The resolution of the first image img_1 may be lower than the resolution of the zeroth image img_0. The lth image img_1 may be an image obtained by reducing the zeroth image img_0 by n times or downsampling n times (n is a positive integer greater than 1 or a positive real number greater than 1). l times or downsampling by n l In one example embodiment, the lth image img_1 of the multiscale image img_ms may be an image with the lowest resolution or the smallest size. For example, the multiscale image generator 112 may generate the multiscale image img_ms by iteratively reducing the initial illumination map LM_int by n times.

[0070] In operation S123, the illumination map extraction unit 110 may iteratively perform FGWLS based on the multi-scale image img_ms. The illumination map extraction unit 110 may perform an illumination map extraction operation based on a multi-scale. Figure 6 As shown in FIG. 1 , the multi-scale extractor 113 may iteratively perform FGWLS on a plurality of images img_0 to img_1 included in the multi-scale image img_ms. Figures 7 to 8B Operation S123 is described in more detail.

[0071] In operation S124 , the illuminance map extraction unit 110 may output a final illuminance map LM_fin as a result of iteratively performing FGWLS (a result of a multi-scale based illuminance map extraction operation).

[0072] Conventional FGWLS can generate a final illuminance map by processing a single image with a single resolution. In this case, when the flattening level used for calculation exceeds a certain value or the size of the single image is large, block artifacts may occur in the final illuminance map.

[0073] In contrast, as described above, the image signal processor 12 or the enhancement module 100 according to the exemplary embodiment can generate a final illuminance map LM_fin by iteratively performing FGWLS on a plurality of images (multi-scale images img_ms) having different resolutions. In this case, the enhancement module 100 can prevent block artifacts from occurring in the final illuminance map LM_fin by using a relatively low flattening level. In addition, since the result of performing FGWLS on an image having a low resolution is used as an input of the next FGWLS (FGWLS for an image of the next resolution), even if a relatively low flattening level is used, the expected flattening strength can be obtained from the final illuminance map LM_fin.

[0074] Figure 7 It is shown Figure 5 Flow chart of operation S123. Figures 5 to 7 , operation S123 may include operations S123a to S123g.

[0075] In operation S123a, the variable i is set to 1, and the variable j is set to 1. In an exemplary embodiment, the variable i is only used to describe the reference numerals corresponding to the multiple images img_0 to img_1 in the multi-scale image img_ms, and the variable j is only used to describe the iterative operation of the multi-scale FGWLS, and the embodiment is not limited thereto. Here, l may be a reference numeral corresponding to an image with the smallest resolution among the multiple images img_0 to img_1 included in the multi-scale image img_ms. In an exemplary embodiment, l may be an integer greater than 2. For example, l may be "3".

[0076] In operation S123b, the multiscale extractor 113 may perform a j-th FGWLS based on the i-th multiscale image. The multiscale extractor 113 may perform a first FGWLS based on an l-th image img_l having the smallest resolution among a plurality of images img_0 to img_l included in the multiscale image img_ms. In an example embodiment, in the j-th FGWLS operation, the i-th image img_i may be used as a guide image. In the first FGWLS operation (j=1), the l-th image img_l (i=l) may be used as an input image and a guide image.

[0077] In operation S123c, the multiscale extractor 113 may amplify or expand the result of operation 123b (the result of the i-th FGWLS). In this case, the multiscale extractor 113 may amplify the result of operation 123b (the result of the i-th FGWLS) by as much as n times. In an example embodiment, n may correspond to a reduction rate for generating the multiscale image img_ms.

[0078] In operation S123d, the variable i is set to i-1, and the variable j is set to j+1.

[0079] In operation S123e, the multiscale extractor 113 may perform the jth FGWLS based on the result of the enlargement (the result of operation S123c) and the i-th multiscale image img_i. For example, the multiscale extractor 113 may perform the j-th FGWLS by using the result of the enlargement (the result of operation S123c) as an input and using the i-th multiscale image img_i as a guide image.

[0080] In operation S123f, it is determined whether the variable i is less than 0. When it is determined that the variable i is not less than 0, the multi-scale extractor 113 performs operation S123c again. That is, the multi-scale extractor 113 may perform the FGWLS operation on an image having a next resolution (a resolution higher than the resolution of the (l-1)th image img_l-1). In other words, the result of the FGWLS associated with the current resolution may be transmitted or used as an input of the FGWLS associated with the next resolution as a higher resolution.

[0081] When it is determined that the variable i is less than 0, the multi-scale extractor 113 may output a final illumination map LM_fin as a result of iteratively performing FGWLS in operation S123g.

[0082] As described above, the illuminance map extracting unit 110 according to example embodiments may extract the final illuminance map LM_fin by performing the FGWLS operation on each of the multi-scale images having different resolutions.

[0083] In one example embodiment, weighted least squares (WLS) may be performed on a specific image based on Equation 5 below.

[0084] [Equation 5]

[0085]

[0086] Referring to Equation 5 above, p indicates a pixel of a particular image, w indicates a set of pixels adjacent to the p pixel, and q indicates pixels adjacent to the p pixel. p,q (G) is a function used to calculate the similarity weight between two pixels p and q from the guidance image. a is to be applied to L p The weight of the pixel similarity condition. J(L) is the result of performing WLS. is the information about the initial illumination of pixel p, L p is the information about the illumination of pixel p, L q is information about the illuminance of pixel q. As described above, the fast global weighted least squares (FGWLS) may indicate a way of calculating by simplifying the optimal solution of WLS based on the above equation 5, and may flatten a specific image by iteratively performing calculations on the sparse Laplacian matrix in row units and in column units. The method according to the inventive concept may be performed based on the following equation 6: Figure 7 Flowchart of the multi-scale FGWLS operation.

[0087] [Equation 6]

[0088]

[0089] Referring to the above equation 6, FGWLS indicates an operation function for simplifying and calculating the optimal solution of WLS based on the above equation 5, ↑ n Indicates a function that magnifies or enlarges the input image by n times, L i Indicates the result of the i-th FGWLS, G i indicates the guidance image used in the i-th FGWLS, λ i indicates the flattening level used in the i-th FGWLS, T i Indicates the number of times the calculations for each row and for each column used in the i-th FGWLS are repeated. The remaining reference symbols are described above.

[0090] Guide image G i may correspond to one of the multiple images included in the multi-scale image img_ms. That is, the zeroth guide image G 0 It may correspond to the zeroth image img_0 or the initial illumination map LM_int. The first guide image G1 may correspond to the first image img_1 obtained by reducing the zeroth image img_0 by n times. That is, the initial illumination map LM_int (ie, ) or the zeroth image img_0 is reduced or decreased by n i times to determine the i-th guide image G i and the input image

[0091] like Figure 7 As shown in , the multi-scale FGWLS operation can be performed sequentially from the lth image img_1 having the lowest resolution to the zeroth image img_0 having the highest resolution. The result of the current FGWLS can be used or transmitted as the input of the next FGWLS. Therefore, even if the flattening level λ having a relatively small value is used i , the expected result can also be obtained from the final illumination map LM_fin. In addition, because the flattening level λ with a relatively small value is used i , so it is possible to prevent block artifacts from occurring in the final illuminance map LM_fin. In addition, since the results of processing images of various resolutions are combined, there may be advantages in terms of dynamic range compression (DRC) and tone reproduction.

[0092] In an example embodiment, in order to perform a multi-scale based extraction operation according to an example embodiment, it may be necessary to convert w p,q is defined as a function for calculating the similarity weight between two pixels. In the multi-scale based extraction operation according to the example embodiment, w p,q It can be defined based on the gradient of the guidance image, as shown in Equation 7 below.

[0093] [Equation 7]

[0094] w p，q (G)=(|G p -G q | α +β) -1

[0095] Referring to the above equation 7, α and β are variables used to adjust the magnitude of the similarity according to the gradient of the guidance image, G p is the value of pixel p in the guidance image, G q is the value of pixel q in the guide image. The remaining reference symbols are described above. In an example embodiment, α and β may be set to 1.32 and 0.001, respectively. However, the embodiment is not limited thereto. For example, α and β may be experimentally determined variables and may be set to various values ​​according to the manner in which the image signal processor is implemented and the image quality.

[0096] Fig. 8A and Figure 8B It is shown Figure 6 Block diagram of the multi-scale FGWLS. Figure 6 and Fig. 8A The multi-scale extractor 113 may include a plurality of FGWLS engines FGWLS_0 to FGWLS_1, a plurality of upscaling engines SU_1 to SU_1, and a plurality of variable engines VAR_0 to VAR_1.

[0097] The first FGWLS engine FGWLS_1 among the plurality of FGWLS engines FGWLS_0 to FGWLS_1 may receive the first image img_1 (e.g., the image with the smallest resolution among the multi-scale images img_ms) as input INPUT and the guide image GUID, and may receive the corresponding variable from the first variable engine VAR_1. The first FGWLS engine FGWLS_1 may perform the FGWLS operation based on the above equation 6. In this case, since the variable i is 1, and G i Each of can be the l-th image img_l, and λ i and T i The result (output OUT) of the first FGWLS engine FGWLS_1 is provided to the first amplification engine SU_1.

[0098] The 1st upscaling engine SU_1 may enlarge or amplify the output OUT of the 1st FGWLS engine FGWLS_1 by n times (in the figure, represented by Xn). In this case, n may correspond to the reduction rate of the multi-scale image img_ms. The output (the result of the upscaling) of the 1st upscaling engine SU_1 is provided to the (1-1)th FGWLS engine FGWLS_1-1.

[0099] The (l-1)th FGWLS engine FGWLS_l-1 may receive the output (magnification result) of the l-th amplification engine SU_l as input INPUT, and may receive the (l-1)th image img_l-1 of the multi-scale image img_ms as a guide image GUID. In this case, the resolution of the output (magnification result) of the l-th amplification engine SU_l may be the same as the resolution of the (l-1)th image img_l-1. The (l-1)th FGWLS engine FGWLS_l-1 may receive the corresponding variable from the (l-1)th variable engine VAR_l-1.

[0100] The (l-1)th FGWLS engine FGWLS_l-1 may perform FGWLS operations based on the above equation 6. In this case, since the variable i is l-1, L i+1It can be the output OUT of the first FGWLS engine FGWLS_1. That is, ↑ n (L i+1 ) may be the output of the first amplification engine SU_1. i can be the (l-1)th image img_l-1, and λ i and T i It can be set by the (l-1)th variable engine VAR_l-1. The result (output OUT) of the (l-1)th FGWLS engine FGWLS_l-1 is provided to the (l-1)th amplification engine SU_l-1.

[0101] The (l-1)th amplification engine SU_l-1 may expand or amplify the output OUT of the (l-1)th FGWLS engine FGWLS_l-1 by as much as n times, and the amplified result may be provided to the next FGWLS engine (eg, the (l-2)th FGWLS engine).

[0102] Each of the plurality of FGWLS engines FGWLS_0 to FGWLS_1 may perform the above FGWLS operation, and each of the plurality of amplification engines SU_1 to SU_1 may perform the above amplification operation. The zeroth FGWLS engine FGWLS_0 of the plurality of FGWLS engines FGWLS_0 to FGWLS_1 may receive the amplified result as input INPUT from the first amplification engine SU_1, may receive the zeroth image img_0 in the multi-scale image img_ms as the guide image GUID, and may receive variable information from the zeroth variable engine VAR_0. As described above, the zeroth FGWLS engine FGWLS_0 may perform the FGWLS operation based on the above equation 6, and the result of the zeroth FGWLS engine FGWLS_0 may be output as the final illumination map LM_fin.

[0103] Reference Figure 6 and Figure 8B , the multi-scale extractor 113 may include a FGWLS engine, a variable engine VAR, and an amplification engine SU. The FGWLS engine may perform a multi-scale-based extraction operation based on the above equation 6. For example, as described above, the FGWLS engine may receive the l-th image img_1 as an input INPUT, may receive a corresponding image (the l-th image img_1) in the multi-scale image img_ms as a guide image GUID, and may receive a corresponding variable from the variable engine VAR. The FGWLS engine may perform the FGWLS operation based on the above equation 6 by using the received information. The result of the FGWLS engine may be provided to the amplification engine SU, and the amplification engine SU may amplify the input result and provide the amplified result as the input INPUT of the FGWLS engine.

[0104] Thereafter, the FGWLS engine may perform an FGWLS operation on the up-scaled result and the multi-scale image img_ms based on the above Equation 6. A result of performing the FGWLS operation on the zeroth image img_0 having the highest resolution among the multi-scale images img_ms may be output as a final illumination map LM_fin.

[0105] As reference Fig. 8A As described above, the multi-scale extractor 113 according to the exemplary embodiment may have a pipeline structure in which a plurality of FGWLS operations are sequentially performed. Figure 8B As described above, the multi-scale extractor 113 according to the exemplary embodiment may have a feedback structure in which the result of the FGWLS operation is fed back as an input. However, the embodiment is not limited thereto. For example, the multi-scale FGWLS according to the embodiment may be variously changed or modified.

[0106] As described above, compared with the conventional FGWLS scheme associated with a single resolution, the illumination map extraction scheme using multi-scale FGWLS according to the exemplary embodiment can prevent block artifacts. In addition, since FGWLS is performed through multiple steps and the current FGWLS result is transmitted to the next FGWLS as input, there may be advantages in dynamic range compression (DRC) and tone reproduction.

[0107] Fig. 9 Is used to describe Figures 6 to 8B FIG. 1 is a diagram showing an example effect of the operation of the illumination map extraction unit. Fig. 9 , the first illuminance map LM_a, the second illuminance map LM_b, and the third illuminance map LM_c may be illuminance maps extracted by conventional FGWLS for a single image, and the final illuminance map LM_fin (e.g., LM_d) may be an illuminance map extracted by the illuminance map extraction unit 110 according to example embodiments.

[0108] The first illuminance map LM_a may be an illuminance map extracted by a conventional FGWLS with the flattening level λ set to a, the second illuminance map LM_b may be an illuminance map extracted by a conventional FGWLS with the flattening level λ set to b (b is greater than a), and the third illuminance map LM_c may be an illuminance map extracted by a conventional FGWLS with the flattening level λ set to c (c is greater than b). Fig. 9 As shown in , according to the conventional FGWLS, as the flattening level λ increases, the flattening intensity in the illuminance map may become stronger. However, like the first illuminance map LM_a, when the flattening level λ is relatively small, the expected flattening intensity may not appear in the extracted illuminance map. In addition, like the third illuminance map LM_c, when the flattening level λ is relatively large, block artifacts may occur in a specific area.

[0109] In contrast, even if the flattening level λ of each FGWLS i Being set to a relatively small value of d (eg, d=a), an expected flattening intensity may also appear in the final illuminance map LM_fin extracted by the multi-scale based extraction operation according to example embodiments. In addition, because the flattening level λ i It is relatively small, so it can prevent blocking artifacts from occurring in the final illumination map LM_fin.

[0110] Fig.10 It is shown Figure 4 Block diagram of the enhancement unit. Figure 4 and Fig.10 The enhancement unit 130 may perform signal processing based on the grayscale image GR and the final illuminance map LM_fin, and may generate an enhanced grayscale image GR_enh. For example, the enhancement unit 130 may include a reflectance extractor 131, an illuminance enhancer 132, a reflectance enhancer 133, and a combiner 134.

[0111] The reflectivity extractor 131 may extract the reflectivity RFL from the grayscale image GR based on the final illuminance map LM_fin. For example, as described above with reference to Equation 1, the specific image may be represented by reflectivity and illuminance. In the case where information such as about the grayscale image GR of the specific image and information such as about the final illuminance map LM_fin of illuminance are obtained, the reflectivity RFL may be extracted based on the above Equation 1. The final illuminance map LM_fin may be provided to the illuminance enhancer 132, and the reflectivity RFL may be provided to the reflectivity enhancer 133.

[0112] The luminance enhancer 132 may enhance the contrast of the final luminance map LM_fin. For example, the luminance enhancer 132 may enhance the contrast of the final luminance map LM_fin based on a gamma function. The gamma function may be represented by the following equation 8.

[0113] [Equation 8]

[0114] L enh =L γ Referring to Equation 8 above, L enh γ may indicate an enhanced illuminance map LM_enh which is information obtained by enhancing brightness and contrast of the final illuminance map LM_fin, and γ may indicate a gamma value. The brightness of a dark portion of the final illuminance map LM_fin may be adjusted according to γ.

[0115] The illuminance enhancer 132 may enhance the contrast of the final illuminance map LM_fin based on a windowed inverse sigmoid (WIS) function. The WIS function may be a function for ensuring a dynamic range of a result value by increasing the brightness of a dark portion of the final illuminance map LM_fin and simultaneously reducing the brightness of a bright portion of the final illuminance map LM_fin. The WIS function may be represented by the following equation 9.

[0116] [Equation 9]

[0117]

[0118]

[0119] L′ n =L n [f(v max )-f(v min )]+f(v min )

[0120]

[0121]

[0122] Referring to the above equation 9, f may indicate the S-shaped function, L may indicate the final illumination map LM_fin, v, v max and v min v may be a variable for determining the shape of the WIS. The increase in brightness of the dark portion of the enhanced illuminance map LM_enh and the decrease in brightness of the bright portion of the enhanced illuminance map LM_enh may be based on v max and v min is regulated.

[0123] In an exemplary embodiment, how the luminance enhancer 132 enhances the contrast of the luminance map is described with reference to Equation 8 or Equation 9 above, but the embodiment is not limited thereto. For example, for the purpose of enhancing the global contrast of the input image, the luminance enhancer 132 may use various function models or various image signal processing techniques.

[0124] The reflectivity enhancer 133 may output an enhanced reflectivity RFL_enh by enhancing the contrast (e.g., local contrast) of the reflectivity RFL. The final illuminance map LM_fin may mainly include low-frequency components, and may include only a portion of the high-frequency components corresponding to the reflectivity RFL. In this case (when the calculation for contrast enhancement is performed on the final illuminance map LM_fin), the value of the high-frequency component or reflectivity component included in the final illuminance map LM_fin may be changed by a nonlinear conversion, thereby reducing the overall clarity of the image. In order to prevent this problem, the reflectivity enhancer 133 may prevent the clarity of the final image from being reduced by enhancing the local contrast of the reflectivity RFL. For example, the reflectivity enhancer 133 may enhance the contrast of the reflectivity RFL based on the following equation 10.

[0125] [Equation 10]

[0126]

[0127]

[0128]

[0129] Referring to the above equation 10, R may indicate the extracted reflectance RFL, F may indicate the final illuminance map LM_fin, and R enh RFL_enh may indicate enhanced reflectivity, and P may be a variable for enhancing local contrast. Here, it is assumed that P is a number greater than 1. In the case where F(x, y) as the value of the central pixel is greater than L(x, y) as the average value of the values ​​of the surrounding pixels, R enh can become larger than R if F(x,y), which is the value of the center pixel, is not larger than L(x,y), which is the average value of the values ​​of the surrounding pixels. enh may become smaller than R. That is, the local contrast may be enhanced by making the value of the central pixel brighter when the central pixel is relatively bright and darker when the central pixel is relatively dark. In an example embodiment, L in the above equation 10 may be replaced with an enhanced luminance map LM_enh.

[0130] The combiner 134 may output an enhanced grayscale image GR_enh based on the enhanced illuminance map LM_enh and the enhanced reflectance RFL_enh. For example, as described above with reference to Equation 1, a specific image may be represented by information about illuminance and information about reflectance. That is, the combiner 134 may generate an enhanced grayscale image GR_enh by combining the enhanced illuminance map LM_enh and the enhanced reflectance RFL_enh based on Equation 1 above.

[0131] In an example embodiment, the combiner 134 may be configured to reduce the noise of the enhanced grayscale image GR_enh. For example, in the case where the brightness or contrast of a dark area in an image is enhanced, the noise present in the dark area may be amplified together. In order to reduce the above noise, the combiner 134 may combine the enhanced illuminance map LM_enh and the enhanced reflectivity RFL_enh based on the following equation 11.

[0132] [Equation 11]

[0133]

[0134] Referring to Equation 11 above, R enh The enhanced reflectivity RFL_enh can be indicated, and L n enh It can be a [0,1] value pair as the L of the enhanced illumination map LM_enh enh The normalized result. d Can instruct R through FGWLS enh In one example embodiment, the flattened result can be obtained based on the reference Figures 6 to 9 The multi-scale FGWLS described here is used to perform enh The FGWLS performed by R can be performed based on the conventional FGWLS. enh Executed FGWLS.

[0135] As shown in Equation 11, reference L n enh , according to R d and R enh The suppression or reduction of noise in dark areas is performed by a linear combination of R. The reason is that the size of the noise is not uniform in the entire area of ​​the image and is processed adaptively according to the brightness value. In the case of bright areas of the image, the pixel value is based on R enh is determined, and in the case of dark areas of the image, the pixel values ​​are based on d As described above, an enhanced grayscale image GR_enh in which noise amplified in a specific area (eg, a dark portion of an image) is suppressed by the above Equation 11 may be output.

[0136] Fig.11 It is shown Figure 4 Block diagram of the color restoration unit. Fig.11The color restoration unit 140 may generate an enhanced image IMG_enh based on the input image IMG_in and the enhanced grayscale image GR_enh. For example, the color restoration unit 140 may include first to third color channel engines 141a to 141c and an enhanced image generator engine 142.

[0137] The first color channel engine 141a, the second color channel engine 141b, and the third color channel engine 141c may generate enhanced color information r_enh, g_enh, and b_enh, respectively, based on corresponding color information r, g, and b from the input image IMG_in and corresponding information of the enhanced grayscale image GR_enh. For example, the first color channel engine 141a, the second color channel engine 141b, and the third color channel engine 141c may generate enhanced color information r_enh, g_enh, and b_enh, respectively, based on the following equation 12.

[0138] [Equation 12]

[0139]

[0140]

[0141]

[0142] Referring to the above equation 12, r, g, and b indicate color information obtained from the input image IMG_in, F enh indicates information corresponding to the enhanced grayscale image GR_enh, and F indicates information corresponding to the enhanced grayscale image GR_enh. Figure 4 The information corresponding to the grayscale image GR (the image before contrast enhancement) generated by the grayscale image generator 120 shown in FIG. enh , g enh and b enh Indicates enhanced color information about red, green, and blue, respectively.

[0143] The enhanced image generator engine 142 may combine the enhanced color information r_enh, g_enh, and b_enh generated based on the above Equation 12, and may generate an enhanced image IMG_enh.

[0144] As described above, according to example embodiments, the image signal processor 12 or the enhancement module 100 of the image signal processor 12 may generate a multiscale image associated with an input image, and may perform a multiscale FGWLS operation on the multiscale image. In this case, block artifacts occurring in conventional FGWLS may be prevented. Therefore, image quality may be enhanced. In this way, an image signal processor 12 having enhanced reliability and enhanced performance may be provided.

[0145] Fig. 12A and Fig. 12B is a diagram for describing an example effect according to an example embodiment. Fig. 12A The illumination map and output image obtained by performing conventional FGWLS for a single image on the input image INPUT are described, and reference will be made to Fig. 12B An illumination map and an output image obtained by performing MSFGWLS (FGWLS for multi-scale images) according to an exemplary embodiment on an input image INPUT are described.

[0146] exist Fig. 12A The illuminance map LM_FGWLS obtained by performing conventional FGWLS on the input image INPUT is shown in FIG. Fig. 12A As can be seen from the illumination map LM_FGWLS, blocking artifacts may occur in some areas. Fig. 12B , an illumination map LM_MSFGWLS obtained by performing MSFGWLS according to an exemplary embodiment on an input image INPUT is shown in FIG. Fig. 12A Block artifacts occurring in the illumination map LM_FGWLS can be reduced.

[0147] For example, referring to the illumination map LM_FGWLS subjected to conventional FGWLS, block artifacts may occur in a part of the region of the input image INPUT. In contrast, referring to the illumination map LM_MSFGWLS subjected to MSFGWLS according to example embodiments, block artifacts may not occur in the same region or may be reduced.

[0148] Results, such as Fig. 12A and Fig. 12B As shown in , an output image OUTPUT_MSFGWLS generated based on an illumination map LM_MSFGWLS subjected to MSFGWLS according to example embodiments may have better image quality than a final image OUTPUT_FGWLS generated based on an illumination map LM_FGWLS according to a conventional FGWLS.

[0149] Fig.13 is a block diagram illustrating an image sensor according to example embodiments. Fig.13 , the image sensor 1000 may include a pixel array 1100 , a peripheral circuit 1200 , and a processor including an enhancement module 1300 .

[0150] The pixel array 1100 may include a plurality of pixels. The peripheral circuit 1200 may be configured to process information obtained from the plurality of pixels of the pixel array 1100. In an example embodiment, the peripheral circuit 1200 may include various components necessary for generating image data at the image sensor 1000 (such as a row driver, an analog / digital converter (ADC), a memory, and a ramp generator).

[0151] The enhancement module 1300 included in the processor may perform image signal processing on the image data obtained by the peripheral circuit 1200, and may output an enhanced image IMG_enh. For example, an example embodiment in which an image signal processor is implemented independently of an image sensor is described above, but the embodiment is not limited thereto. Fig.13 As shown in , all or at least a portion of the components of the image signal processor may be included in the image sensor 1000 .

[0152] Fig.14 is a block diagram illustrating an electronic device according to an example embodiment. Fig.14 , the electronic device 2000 may include a main processor 2100, a touch panel 2200, a touch driver integrated circuit (TDI) 2202, a display panel 2300, a display driver integrated circuit (DDI) 2302, a system memory 2400, a storage device 2500, an audio processor 2600, a communication block 2700, and an image processor 2800. In an example embodiment, the electronic device 2000 may be one of various electronic devices such as a portable communication terminal, a personal digital assistant (PDA), a portable media player (PMP), a digital camera, a smart phone, a tablet computer, a laptop computer, and a wearable device.

[0153] The main processor 2100 may control the overall operation of the electronic device 2000. The main processor 2100 may control / manage operations of components of the electronic device 2000. The main processor 2100 may process various operations for operating the electronic device 2000.

[0154] The touch panel 2200 may be configured to sense a touch input from a user under the control of the touch driver integrated circuit 2202. The display panel 2300 may be configured to display image information under the control of the display driver integrated circuit 2302.

[0155] The system memory 2400 may store data used for the operation of the electronic device 2000. For example, the system memory 2400 may include a volatile memory such as a static random access memory (SRAM), a dynamic RAM (DRAM), or a synchronous DRAM (SDRAM) and / or a nonvolatile memory such as a phase change RAM (PRAM), a magnetoresistive RAM (MRAM), a resistive RAM (ReRAM), or a ferroelectric RAM (FRAM).

[0156] The storage device 2500 may store data regardless of whether power is supplied. For example, the storage device 2500 may include at least one of various non-volatile memories such as flash memory, PRAM, MRAM, ReRAM, and FRAM. For example, the storage device 2500 may include an embedded memory and / or a removable memory of the electronic device 2000.

[0157] The audio processor 2600 may process an audio signal by using the audio signal processor 2610. The audio processor 2600 may receive an audio input through a microphone 2620, or may provide an audio output through a speaker 2630.

[0158] The communication block 2700 may exchange signals with an external device / system via an antenna 2710. The transceiver 2720 and the modulator / demodulator (MODEM) 2730 of the communication block 2700 may process signals exchanged with an external device / system based on at least one of the following various wireless communication protocols: Long Term Evolution (LTE), Worldwide Interoperability for Microwave Access (WiMax), Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Bluetooth, Near Field Communication (NFC), Wireless Fidelity (Wi-Fi), and Radio Frequency Identification (RFID).

[0159] The image processor 2800 may receive light through the lens 2810. The image device 2820 and the image signal processor 2830 included in the image processor 2800 may generate image information about an external object based on the received light. In an example embodiment, the image signal processor 2830 may implement a reference Figures 1 to 13 For example, the image signal processor 2830 may be configured to Figures 1 to 13 The described multi-scale based extraction operation extracts an illumination map associated with image data from the image device 2820. In an example embodiment, the image signal processor 2830 may be included in the main processor 2100, and the image signal processor 2830 may use the system memory 2400 as a memory for image signal processing.

[0160] According to an example embodiment, an image signal processor may extract an illumination map associated with an input image by iteratively performing an operation based on a fast global weighted least squares (FGWLS) on images of different resolutions included in a multi-scale image. In this way, block artifacts of the extracted illumination map may be prevented. Therefore, the quality of a final image generated by the image signal processor is enhanced.

[0161] Although example embodiments have been described with reference to the accompanying drawings, it will be apparent to those of ordinary skill in the art that various changes and modifications may be made thereto without departing from the spirit and scope of the inventive concept as set forth in the claims.

Claims

1. An operating method of an image signal processor, the image signal processor being configured to perform signal processing on an original image received from an image device, the operating method comprising: generating a multi-scale image based on an input image, the multi-scale image including a plurality of images having different resolutions from each other; Iteratively performing a Fast Global Weighted Least Squares (FGWLS) based operation on the multi-scale image to generate a final illumination map; as well as Output enhanced image based on final illumination map and input image, Therein, during the iterative execution of the FGWLS-based operation, the result of the current FGWLS-based operation is transmitted to the input of the next FGWLS-based operation.

2. The operating method according to claim 1, wherein: The steps to generate multi-scale images include: generating an initial illumination map based on the input image; and The size of the initial illumination map is iteratively reduced by a factor of n to generate multi-scale images, where n is a positive real number.

3. The operating method according to claim 2, wherein: The steps of iteratively performing FGWLS-based operations on multi-scale images to generate the final illumination map include: performing a first FGWLS-based operation by using a first image having a minimum resolution among the multi-scale images as an input image and a guide image; amplifying a result of the first FGWLS-based operation by as much as n times to generate a first amplified result; performing a second FGWLS-based operation by using the first enlarged result as an input image and using a second image having the same resolution as that of the first enlarged result among the multi-scale images as a guide image; amplifying a result of the second FGWLS-based operation by as much as n times to generate a second amplified result; performing a third FGWLS-based operation by using the second enlarged result as an input image and using a third image having the same resolution as that of the second enlarged result among the multi-scale images as a guide image; and The result of the third FGWLS-based operation is output as a final illumination map.

4. The operating method according to claim 3, wherein: The third image corresponds to the initial illumination map.

5. The operating method according to any one of claims 1 to 4, wherein: The steps of outputting an enhanced image based on the final illumination map and the input image include: Generate a grayscale image based on an input image; extracting reflectance from the grayscale image based on the final illumination map; enhancing the global contrast of the final illumination map to generate an enhanced illumination map; enhancing the local contrast of reflectivity to generate enhanced reflectivity; and The enhanced illumination map and the enhanced reflectance are combined to output an enhanced image.

6. The operating method according to claim 5, wherein: The step of enhancing the global contrast of the final illumination map is performed based on a windowed inverse sigmoid function.

7. The operating method according to claim 5, wherein: The steps of combining the enhanced illumination map and the enhanced reflectance to output an enhanced image include: planarizing the enhanced reflectivity to generate a planarized reflectivity; normalizing the enhanced illumination map to generate a normalized illumination map; and Based on the normalized illumination map, an enhanced grayscale image is generated by a linear combination of the enhanced reflectance and the flattened reflectance.

8. The operating method according to claim 7, wherein: The step of flattening the enhanced reflectivity includes performing a FGWLS-based operation on the enhanced reflectivity.

9. The operating method according to claim 7, wherein: The step of outputting the enhanced image also includes: Based on the enhanced grayscale image, the input image, and the grayscale image, an enhanced image is output.

10. The operating method according to any one of claims 1 to 4, further comprising: receiving a raw image from an image device; as well as Preprocessing is performed on the original image to generate the input image.

11. The operating method according to claim 10, wherein: The steps to perform preprocessing include: Reduce the noise of the original image; performing white balancing on a result of reducing the noise; performing color filter array interpolation on the white balanced result; and Correction is performed on the result of the color filter array interpolation to generate the input image.

12. An image system comprising: an image sensor configured to output a raw image based on light reflected from an external object; as well as An image signal processor configured to: Perform preprocessing on the original image to generate the input image, Extracting the final illumination map from the input image via a multi-scale based extraction operation, and Generate an enhanced image based on the final illumination map and the input image, Among them, the multi-scale extraction operations include: generating a multi-scale image including a plurality of images having different resolutions based on an input image; Iteratively performing an illumination map extraction operation on the multi-scale image; and outputting a final illuminance map as a result of iteratively performing the illuminance map extraction operation, The step of iteratively performing an illumination map extraction operation on the multi-scale image comprises: iteratively performing an operation based on a fast global weighted least squares (FGWLS) on the multi-scale image to generate a final illumination map, Therein, during the iterative execution of the FGWLS-based operation, the result of the current FGWLS-based operation is transmitted to the input of the next FGWLS-based operation.

13. The imaging system according to claim 12, wherein: The multi-scale image is generated by iteratively downscaling an initial illumination map generated based on an input image by a factor of n to include a plurality of images having different resolutions, where n is a positive integer greater than 1.

14. The imaging system according to claim 13, wherein: The illumination map extraction operation is performed based on: where (x, y) is the pixel location, FGWLS is a function based on the operation of Fast Global Weighted Least Squares (FGWLS), and L i is the calculation result of FGWLS,↑ n is a function that amplifies the input by n times. is obtained by reducing the initial illumination map by n i The illumination diagram obtained by multiplying i is the guidance image, λ i is the flattening level used in the function based on FGWLS operation, T i is the number of row-based calculations and column-based calculations used in the function for the FGWLS-based operation, and The illumination map extraction operation is based on l The stepwise change of i to 0 is executed. l is an integer greater than 2.

15. The imaging system according to claim 14, wherein: The guidance image is the corresponding image in the multi-scale image.

16. An image contrast enhancement method for an image signal processor, the image contrast enhancement method comprising: generating an initial illumination map based on the input image; Iteratively downscaling the initial illumination map by a factor of n to generate a multi-scale image comprising a plurality of images, wherein n is a positive integer greater than 1; performing a first calculation based on a first image having a lowest resolution among the multi-scale images; performing a second calculation based on a result of the first calculation and a second image having a second lowest resolution among the multi-scale images; performing a third calculation based on a result of the second calculation and a third image having the highest resolution among the multi-scale images; generating a result of the third calculation as a final illumination map; and Generate an enhanced image based on the final illumination map and the input image, The first calculation, the second calculation and the third calculation are based on fast global weighted least squares FGWLS.

17. The image contrast enhancement method according to claim 16, wherein: A first computation is performed based on the first input, the first guide image, the first flattening level, and the first iteration count, wherein a second computation is performed based on a second input, a second guide image, a second flattening level, and a second iteration count, wherein a third calculation is performed based on a third input, a third guide image, a third flattening level, and a third iteration count, Among them, the first input is the first image, Wherein, the second input is an image obtained by enlarging the result of the first calculation by n times, Wherein, the third input is an image obtained by enlarging the result of the second calculation by n times, wherein the first guide image is the first image, wherein the second guide image is the second image, and Among them, the third guide image is the third image.

18. The image contrast enhancement method according to claim 16, wherein: The steps of generating an enhanced image based on the final illumination map and the input image include: extracting reflectance from a grayscale image corresponding to the input image based on the final illumination map; enhancing the global contrast of the final illumination map to generate an enhanced illumination map; enhancing the local contrast of reflectivity to generate enhanced reflectivity; and The enhanced illumination map and the enhanced reflectance are combined to output an enhanced grayscale image.

19. The image contrast enhancement method according to claim 18, wherein: The step of generating an enhanced image based on the final illumination map and the input image further includes: The colors of the enhanced grayscale image are restored based on the input image to generate an enhanced image.

Images