Image processing apparatus and method for improving picture quality of image

Abstract

Disclosed according to an embodiment is an image processing apparatus comprising: a memory for storing instructions; and a processor operating in accordance with instructions wherein the processor applies an input image to the neural network in order to obtain a parameter value of a tone curve and a first differential image having a higher resolution than a resolution of the input image, obtains a plurality of filtered images of different frequency bands by filtering the first differential image, and outputs the plurality of filtered images to the neural network. Applying a gain value corresponding to a sample value of an edge map of the first image to the plurality of filtered images, combining the plurality of filtered images to obtain a second differential image, combining the second differential image with the first image to obtain a second image, applying a gain value corresponding to an average sample value of the edge map to a parameter value, and outputting the second image. Determining a hue curve according to a parameter value to which the gain value is applied, and changing a sample value of the second image according to the hue curve in order to acquire an output image.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and more particularly, to an image processing device and method for improving image quality. Background Art

[0002] Recently, various image processing algorithms using neural networks have been developed. Deep learning-based image processing methods using training data can solve various problems that traditional methods cannot solve.

[0003] Recent research is moving towards changing the algorithms used to improve image quality from traditional linear systems to learnable systems based on neural networks.

[0004] For example, image super-resolution technology using neural networks is a technology that learns the difference between low-resolution images and high-resolution images and obtains a clearer and more detailed high-resolution image based on the low-resolution image. Since this super-resolution technology must restore high-frequency components that do not exist in the low-resolution image, it can usually be designed as a multi-convolutional layer structure.

[0005] In addition, as an example, a contrast expansion technique using a neural network is a technique of analyzing an image through a neural network, inferring a pixel-to-pixel mapping function (or mapping curve), and obtaining a three-dimensional image by applying the inferred mapping function to the image.

[0006] In order to install highly complex neural network-based algorithms on image processing products, it is very important to make the neural network lightweight. However, in traditional image processing, it is not easy to achieve lightweight neural networks.

[0007] Figure 1 is a diagram illustrating image processing applicable to an image processing product such as a television (TV).

[0008] like Figure 1 As shown, an output image can be obtained by sequentially applying a noise removal algorithm 1, an upscaling algorithm 2, a sharpness enhancement algorithm 3, a contrast enhancement algorithm 4 and a color expansion algorithm 5 to an input image.

[0009] Although the performance of the algorithm used in image processing can be improved by implementing the algorithm as a neural network-based algorithm, there is a problem in that hardware complexity is significantly increased due to the neural network used for the algorithm.

[0010] Furthermore, there is a problem in that it is difficult to consider the features of the image input to the first algorithm because algorithms subsequent to the first algorithm receive images output from the previous algorithm.

[0011] Therefore, a method is needed that can significantly improve image quality while implementing a lightweight neural network. Summary of the invention

[0012] The image processing apparatus 200 according to the embodiment may include: a memory configured to store at least one instruction; and at least one processor configured to operate according to the at least one instruction.

[0013] In an embodiment, the at least one processor may be configured to obtain a first difference image having a resolution greater than a resolution of the input image and a parameter value of a tone curve by applying the input image to the neural network 210 .

[0014] In an embodiment, the at least one processor may be configured to obtain a plurality of filtered images for different frequency bands by filtering the first difference image.

[0015] In an embodiment, the at least one processor may be configured to apply a gain value corresponding to a sample value of an edge map of the first image upscaled from the input image to each of the plurality of filtered images.

[0016] In an embodiment, the at least one processor may be configured to obtain the second difference image by combining a plurality of filtered images to which the gain values ​​are applied.

[0017] In an embodiment, the at least one processor may be configured to obtain the second image by combining the second difference image with the first image.

[0018] In an embodiment, the at least one processor may be configured to apply a gain value corresponding to the average sample value of the edge map to the parameter value.

[0019] In an embodiment, the at least one processor may be configured to determine a tone curve as a function of the parameter value to which the gain value is applied.

[0020] In an embodiment, the at least one processor may be configured to obtain the output image by changing sample values ​​of the second image according to the determined tone curve.

[0021] The image processing method according to the embodiment may include obtaining a first difference image having a resolution greater than that of the input image and a parameter value of the tone curve by applying the input image to the neural network 210 .

[0022] In an embodiment, the image processing method may include obtaining a plurality of filtered images of different frequency bands by filtering the first difference image.

[0023] In an embodiment, the image processing method may include applying a gain value corresponding to a sample value of an edge map of a first image upscaled from an input image to each of the plurality of filtered images.

[0024] In an embodiment, the image processing method may include obtaining a second difference image by combining a plurality of filtered images to which gain values ​​are applied.

[0025] In an embodiment, the image processing method may include obtaining a second image by combining the second difference image with the first image.

[0026] In an embodiment, the image processing method may include applying a gain value corresponding to an average sample value of the edge map to the parameter value.

[0027] In an embodiment, the image processing method may comprise determining a tone curve from a parameter value to which the gain value is applied.

[0028] In an embodiment, the image processing method may comprise obtaining an output image by changing sample values ​​of the second image according to the determined tone curve. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a diagram showing an example of image processing applicable to an image processing product.

[0030] Figure 2 is a block diagram showing the configuration of an image processing apparatus according to the embodiment.

[0031] Figure 3 is a diagram for describing a neural network according to an embodiment.

[0032] Figure 4 is a diagram for describing a tone curve according to the embodiment.

[0033] Figure 5 is a diagram for describing a method of generating a plurality of filtered images from a first differential image according to an embodiment.

[0034] Figure 6a is a diagram for describing a process of processing a plurality of filtered images according to an embodiment.

[0035] Figure 6b is a diagram for describing a relationship between sample values ​​of an edge map and gain values ​​applied to a plurality of filtered images according to an embodiment.

[0036] Figure 7 is a diagram for describing a relationship between a brightness value of a first image and a gain value applied to a second difference image according to an embodiment.

[0037] Figure 8is a diagram for describing a method of generating a block-wise tone curve from a parameter map output from a neural network according to an embodiment.

[0038] Fig. 9 is a diagram for describing a process of changing a value of a current sample of a second image according to a tone curve according to an embodiment.

[0039] Fig.10 is a flowchart for describing an image processing method according to an embodiment.

[0040] Fig.11 is a diagram for describing a training process of a first sub-neural network and a second sub-neural network according to an embodiment.

[0041] Fig.12 is a diagram for describing a training process of a third sub-neural network according to an embodiment.

[0042] Fig.13 is a diagram for describing the training process of the first sub-neural network, the second sub-neural network, and the third sub-neural network according to an embodiment.

[0043] Fig.14 is a block diagram showing a configuration of an image display apparatus including an image processing apparatus according to an embodiment. DETAILED DESCRIPTION

[0044] Since the present disclosure allows various changes and many embodiments, certain embodiments will be shown in the drawings and described in detail in the specific embodiments. However, this is not intended to limit the embodiments of the present disclosure, and the present disclosure includes all modifications, equivalents and substitutes that fall within the spirit and technical scope of the various embodiments.

[0045] When describing an embodiment, when the detailed description of related known technology is determined to be unnecessary to obscure the subject matter of the present disclosure, its detailed description is omitted herein. In addition, the numerals (e.g., first, second, etc.) used in the description of the embodiment correspond to identification symbols used to distinguish one element from another element.

[0046] Throughout this disclosure, the expression "at least one of a, b, or c" refers to only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0047] When one element is referred to as being “connected” or “coupled” to another element, the one element may be directly connected or coupled to the other element, but unless otherwise stated, the elements may be connected or coupled to each other via intervening elements therebetween.

[0048] In addition, the elements represented by "unit", "module" and the like in the present disclosure may be one element in which two or more elements are combined, or may be divided into two or more elements for each more subdivided function. In addition, in addition to its main function, each of the elements to be described below may also perform some or all of the functions that other elements are responsible for, and some of the main functions that the corresponding elements are responsible for may be dedicated by other elements.

[0049] In the present disclosure, an “image” may mean a still image (or frame), a moving image composed of a plurality of consecutive still images, or a video.

[0050] In the present disclosure, "neural network" is a representative example of an artificial neural network model that simulates brain nerves, and is not limited to an artificial neural network model that uses a specific algorithm. A neural network may also be referred to as a deep neural network.

[0051] In the present disclosure, "neural network parameters" are values ​​used in the calculation process of each layer constituting a neural network, and may include, for example, weights and / or biases used when applying input values ​​to a specific calculation formula. Neural network parameters are values ​​set as a result of training, and may be updated by separate training data as needed.

[0052] In the present disclosure, an "input image" refers to an image input to an image processing device as an image to be processed, and a "first image" refers to an image enlarged from the input image. In addition, a "second image" refers to an image obtained by sharpness enhancement processing, and a "third image" refers to an image obtained by contrast enhancement processing. In addition, an "output image" refers to an image output from the image processing device. According to an embodiment, the third image may correspond to the output image.

[0053] In the present disclosure, "sample" refers to data to be processed as data assigned to sampling positions of feature maps and images. For example, a pixel in a frame of a spatial domain may correspond to a sample. A unit including a plurality of samples may be defined as a block.

[0054] In the present disclosure, "tone curve" refers to information indicating a mapping relationship between input sample values ​​and output sample values. For example, a sample value of an image can be converted into one of a plurality of output sample values ​​according to a mapping relationship represented by the tone curve.

[0055] In the following, reference Figures 2 to 14 An image processing apparatus and an image processing method according to the embodiment are described.

[0056] Figure 2 is a block diagram showing the configuration of the image processing apparatus 200 according to the embodiment.

[0057] Reference Figure 2 The image processing device 200 may also include a neural network 210, a scaler 220, a filtering unit 230, an image analysis unit 240, a sharpness adjustment unit 250, a tone curve generation unit 260 and a tone curve application unit 270.

[0058] The scaler 220 , the filtering unit 230 , the image analyzing unit 240 , the sharpness adjusting unit 250 , the tone curve generating unit 260 , and the tone curve applying unit 270 may be implemented as one or more processors.

[0059] The scaler 220, the filtering unit 230, the image analyzing unit 240, the sharpness adjusting unit 250, the tone curve generating unit 260, and the tone curve applying unit 270 may operate according to instructions stored in the memory.

[0060] In an embodiment, the neural network 210 may be stored in a memory. In an embodiment, the neural network 210 may be implemented as at least one dedicated processor for artificial intelligence (AI).

[0061] The image processing apparatus 200 may process an input image and output an output image as a result of the processing. As described below, the output image may have a resolution greater than that of the input image and may have improved sharpness and contrast compared to the input image.

[0062] The output image may be reproduced by the image processing device 200 or may be sent to a separate display device for reproduction.

[0063] Regarding the operation of the image processing apparatus 200 , an input image may be processed by the neural network 210 , and the neural network 210 may output a first difference image having a resolution greater than that of the input image and a parameter value for a tone curve.

[0064] As reference below Fig.11 As described above, the neural network 210 may be trained to output an image similar to a difference image between a ground truth (GT) image having a resolution greater than that of an input image and an image enlarged from the input image.

[0065] The image processing apparatus 200 according to the embodiment can obtain an output image with improved sharpness by processing the first differential image using the method described below and then combining the first differential image with the first image, instead of combining the first differential image output from the neural network 210 with an image enlarged from the input image ( Figure 2 The first image shown) combination.

[0066] In an embodiment, the neural network 210 may include a first sub-neural network that processes an input image and outputs a feature map, a second sub-neural network that processes the feature map output from the first sub-neural network and outputs a first difference image, and a third sub-neural network that processes the feature map output from the first sub-neural network and outputs a parameter value.

[0067] Reference Figure 3 An example of the structure of the neural network 210 is described.

[0068] refer to Figure 3 , the first sub-neural network 310, the second sub-neural network 320, and the third sub-neural network 330 may include one or more convolutional layers. In an embodiment, at least one of the first sub-neural network 310, the second sub-neural network 320, and the third sub-neural network 330 may be implemented as a fully connected layer (FCL) or a recurrent neural network (RNN).

[0069] The first sub-neural network 310 may process the input image 305 through a convolutional layer and output a feature map 315 corresponding to the input image 305 .

[0070] The feature map 315 may represent a unique characteristic of the input image 305. For example, the feature map 315 may represent a vertical direction characteristic, a horizontal direction characteristic, or an edge characteristic of the input image 305.

[0071] Figure 3 It is shown that four feature maps 315 are output from the first sub-neural network 310, but this is only an example, and the number of feature maps output from the first sub-neural network 310 may vary according to the number of filter kernels of the last convolutional layer of the first sub-neural network 310.

[0072] The feature map 315 output from the first sub-neural network 310 may be input to the second sub-neural network 320 and the third sub-neural network 330 .

[0073] The second sub-neural network 320 may process the input feature map 315 through a convolutional layer and output a first difference image 325 having a resolution greater than that of the input image 305 .

[0074] The third sub-neural network 330 may process the input feature map 315 through a convolutional layer and output one or more parameter maps 335 including parameter values ​​for a tone curve.

[0075] Figure 3 It is shown that three parameter maps 335a, 335b, and 335c are output from the third sub-neural network 330, and each of the parameter maps 335a, 335b, and 335c may include parameter values ​​necessary for inferring a tone curve.

[0076] For example, the first parameter graph 335a may include a first parameter value indicating the location of a downward bump of the tone curve, the second parameter graph 335b may include a second parameter value indicating the location of an inflection point of the tone curve, and the third parameter graph 335c may include a third parameter value indicating the location of an upward bump of the tone curve.

[0077] In an embodiment, the tone curve generation unit 260 may determine a first average parameter value, a second average parameter value, and a third average parameter value by averaging the values ​​included in each of the first parameter map 335a, the second parameter map 335b, and the third parameter map 335c. A tone curve may be generated based on the first average parameter value, the second average parameter value, and the third average parameter value.

[0078] In addition, in an embodiment, the tone curve generation unit 260 may determine a first average parameter value for a block, a second average parameter value for a block, and a third average parameter value for a block by averaging the values ​​included in each of the first parameter map 335a, the second parameter map 335b, and the third parameter map 335c for each block. A tone curve for a block may be generated based on the first average parameter value for the block, the second average parameter value for the block, and the third average parameter value for the block.

[0079] Figure 4 is a diagram illustrating a tone curve 400 generated according to parameter values ​​according to an embodiment.

[0080] refer to Figure 4 Based on the first parameter value Lower_position, the second parameter value Cross_position, and the third parameter value Upper_position, the tone curve 400 may be determined by a curve fitting algorithm.

[0081] exist Figure 4 In the tone curve 400 shown, the horizontal axis may represent input sample values, and the vertical axis may represent output sample values.

[0082] In an embodiment, the first parameter value Lower_position may represent an intermediate value between 0 and the second parameter value Cross_position, and the third parameter value Upper_position may represent an intermediate value between n and the second parameter value Cross_position. For example, when the second parameter value Cross_position is m, the first parameter value Lower_position may be m / 2 and the third parameter value Upper_position may be (nm) / 2.

[0083] When the tone curve 400 is applied to an image, sample values ​​of the image may be changed to output sample values ​​that map to the sample values.

[0084] As described below, the image processing device 200 can modify the first parameter value (or the first average parameter value), the second parameter value (or the second average parameter value) and the third parameter value (or the third average parameter value) based on the characteristics of the image, and can generate a tone curve according to the modified parameter values, thereby further improving the contrast of the output image.

[0085] In an embodiment, the image processing device 200 may include at least one arithmetic logic unit (ALU) for the above-mentioned convolution processing. The ALU may be implemented as one or more processors. For the convolution processing, the ALU may include a multiplier and an adder, the multiplier performing a product operation between the sample value of the input image or feature map and the sample value of the filter kernel, and the adder adding the result value of the product.

[0086] Reference again Figure 2 , the input image may be scaled by the scaler 220 to obtain a first image having a resolution greater than that of the input image, separately from the input image processed by the neural network 210. The resolution of the first image may be equal to the resolution of the first difference image.

[0087] In an embodiment, the sealer 220 may apply conventional scaling (or rule-based scaling) to the input image. Conventional scaling may include, for example, at least one of bilinear scaling, bicubic scaling, Lanczos scaling, or step scaling.

[0088] The first differential image output by the neural network 210 may be input to the filtering unit 230. The filtering unit 230 may also obtain a plurality of filtered images of different frequency bands by filtering the first differential image.

[0089] Among the filtered images, the first filtered image can be used for the first frequency band, and the second filtered image can be used for the second frequency band. Here, the second frequency band can be larger than the first frequency band. That is, the first filtered image can be used for the low frequency band, and the second filtered image can be used for the high frequency band.

[0090] In an embodiment, when the number of filtered images obtained by the filtering unit 230 is three, the first filtered image may be used for the first frequency band, the second filtered image may be used for the second frequency band, and the third filtered image may be used for the third frequency band. Here, the first frequency band may be smaller than the second frequency band, and the second frequency band may be smaller than the third frequency band.

[0091] The filtering unit 230 may obtain a plurality of filtered images by using one or more filters. Figure 5 An example of a method of obtaining a plurality of filtered images is described.

[0092] refer to Figure 5 (a), the filtering unit 230 may obtain a first filtered image 515 for a low frequency band by processing the first difference image 325 with a low pass filter 510 .

[0093] refer to Figure 5 (b), the filtering unit 230 may obtain a third filtered image 535 for a high frequency band by processing the first difference image 325 through the high pass filter 520 .

[0094] At least one of the low pass filter 510 and the high pass filter 520 may be implemented as a Gaussian filter. A first filtered image 515 including a low frequency band component of the first difference image 325 and a third filtered image 535 including a high frequency band component thereof may be obtained by adjusting the Gaussian filter.

[0095] Next, refer to Figure 5 (c), the filtering unit 230 may obtain the second filtered image 525 for the middle frequency band by subtracting the first filtered image 515 and the third filtered image 535 from the first differential image 325 .

[0096] In one embodiment, the filtering unit 230 may obtain the second filtered image 525 for the middle frequency band by applying a middle pass filter (or a band pass filter) to the first difference image 325. In this case, the middle pass filter may be implemented as a Gaussian filter.

[0097] Reference again Figure 2 , the first image obtained by the scaler 220 is transmitted to the image analyzing unit 240 , and the image analyzing unit 240 obtains feature information of the first image and transmits the feature information of the first image to the sharpness adjusting unit 250 .

[0098] The image analysis unit 240 may provide the edge map of the first image as a feature of the first image to the sharpness adjustment unit 250 .

[0099] In an embodiment, the image analysis unit 240 may generate an edge map indicating samples corresponding to edges among samples of the first image. Sample values ​​in the edge map may represent edge strengths of samples of the first image. For example, when the sample value of a certain position in the edge map is large, it may indicate that the edge strength of the sample at the same position in the first image is large. Here, a large edge strength may mean that the sample value at a specific position in the first image has a large amount of change from its adjacent sample values.

[0100] In an embodiment, the image analysis unit 240 may provide brightness information of the first image as a feature of the first image to the sharpness adjustment unit 250 .

[0101] In an embodiment, the brightness information of the first image may include brightness values ​​of samples of the first image.In an embodiment, the brightness information of the first image may include an average brightness value of samples included in each of blocks of a specific size within the first image.

[0102] The sharpness adjustment unit 250 may modify the plurality of filtered images according to the feature information of the first image provided from the image analysis unit 240 .

[0103] Referring to FIG. 6A, FIG. 6B and Figure 7 A process of processing multiple filtered images according to an embodiment is described.

[0104] FIG. 6A is a diagram for describing a process of processing a plurality of filtered images according to an embodiment.

[0105] The sharpness adjustment unit 250 may determine a first gain value g1, a second gain value g2, and a third gain value g3 to be applied to the first filtered image 515, the second filtered image 525, and the third filtered image 535 according to the sample values ​​of the edge map 610. The first gain value g1, the second gain value g2, and the third gain value g3 may be multiplied by the sample values ​​of the first filtered image 515, the second filtered image 525, and the third filtered image 535, respectively.

[0106] In an embodiment, the first gain value g1, the second gain value g2, and the third gain value g3 may be determined based on samples. That is, the sharpness adjustment unit 250 may determine a sample-wise first gain value g1, a sample-wise second gain value g2, and a sample-wise third gain value g3 according to the sample values ​​of the edge map 610. In this case, the sample-wise first gain value g1 may be applied to each sample of the first filtered image 515, the sample-wise second gain value g2 may be applied to each sample of the second filtered image 525, and the sample-wise third gain value g3 may be applied to each sample of the third filtered image 535.

[0107] In an embodiment, the first gain value g1, the second gain value g2, and the third gain value g3 may be determined on a block basis. The sharpness adjustment unit 250 may determine the first gain value g1 for the block, the second gain value g2 for the block, and the third gain value g3 for the block, respectively corresponding to the average sample value of the block of the edge map 610. In this case, the first gain value g1 for the block may be applied to each block of the first filtered image 515, the second gain value g2 for the block may be applied to each block of the second filtered image 525, and the third gain value g3 for the block may be applied to each block of the third filtered image 535.

[0108] The first gain value g1 , the second gain value g2 , and the third gain value g3 respectively applied to the first filtered image 515 , the second filtered image 525 , and the third filtered image 535 may be determined to emphasize the sharpness of the second image 650 .

[0109] The third filtered image 535 includes high frequency components in the first differential image, and the high frequency components represent a portion having a large amount of variation in the first differential image, and therefore, a large third gain value may be determined according to a large sample value in the edge map 610. Therefore, the high frequency components in the second image 650 may be emphasized.

[0110] For example, when the sample value in the edge map 610 is greater than a predetermined value, the third gain value may be determined to be greater than the second gain value based on the sample value, and the second gain value may be determined to be greater than the first gain value. Therefore, for samples with high edge strength, the sample value in the third filtered image 535 may be significantly changed. In addition, for example, when the sample value in the edge map 610 is less than a predetermined value, the first gain value may be determined to be greater than the second gain value based on the sample value, and the second gain value may be determined to be greater than the third gain value. Therefore, for samples with low edge strength, the sample value in the first filtered image 515 may be significantly changed.

[0111] As another example, when the sample value in the edge map 610 is greater than a predetermined value, it can be determined based on the sample value that the first gain value is greater than the second gain value, and it can be determined that the second gain value is greater than the third gain value. In addition, when the sample value in the edge map 610 is less than a predetermined value, it can be determined based on the sample value that the third gain value is greater than the second gain value, and it can be determined that the second gain value is greater than the first gain value.

[0112] FIG. 6B is a diagram for describing a relationship between sample values ​​of an edge map 610 and gain values ​​g1 , g2 , and g3 applied to filtered images 515 , 525 , and 535 , according to an embodiment.

[0113] In FIG. 6B , the values ​​of the edge axis may correspond to sample values ​​of the edge map 610 normalized to 1. In FIG.

[0114] 6B, for example, when the edge value is less than 0.3, the third gain value g3 may be greater than the second gain value g2 and the second gain value g2 may be greater than the first gain value g1. In addition, for example, when the edge value is greater than or equal to 0.6 and less than 0.8, the first gain value g1 may be greater than the second gain value g2, and the second gain value g2 may be greater than the third gain value g3.

[0115] 6A , the first filtered image 515, the second filtered image 525, and the third filtered image 535 to which the first gain value g1, the second gain value g2, and the third gain value g3 are respectively applied may be combined to obtain a second difference image 620. Here, combining may mean adding sample values ​​at the same position in the image.

[0116] In an embodiment, the second differential image 620 may be modified based on the brightness information 630 of the first image 640. For example, the sharpness adjustment unit 250 may determine a fourth gain value g4 according to the brightness values ​​of the samples of the first image 640, and apply the fourth gain value g4 to each of the samples of the second differential image 620. In this case, the number of the fourth gain values ​​g4 may be equal to the number of samples of the second differential image 620.

[0117] In addition, for example, the sharpness adjustment unit 250 may determine a fourth gain value g4 according to the average brightness value of the blocks of the first image 640, and apply the fourth gain value g4 to each of the blocks of the second differential image 620. In this case, the number of the fourth gain values ​​g4 may be equal to the number of the blocks of the second differential image 620.

[0118] The fourth gain value g4 is intended to improve the sharpness of the second image 650. The fourth gain value g4 may be determined to be large for a bright portion in the first image 640, and may be determined to be small for a dark portion in the first image 640. When the user tends to focus on a bright portion rather than a dark portion in an image, a sample value of an area in the second differential image 620 corresponding to a bright area in the first image 640 increases.

[0119] Figure 7 6 is a diagram for describing a relationship between a brightness value of a first image 640 and a gain value applied to a second differential image 620 according to an embodiment.

[0120] Reference Figure 7 , when the brightness value is in the range of 0 to a, the fourth gain value may be determined as p, when the brightness value is in the range of b to c, the fourth gain value may be determined as r, and when the brightness value is greater than or equal to d, the fourth gain value may be determined as q.

[0121] In addition, when the brightness value is within the range of a to b, the fourth gain value may linearly increase as the brightness value increases, and when the brightness value is within the range of c to d, the fourth gain value may linearly decrease as the brightness value increases.

[0122] exist Figure 7 In the example, when the brightness value is in the range of b to c, the fourth gain value is the largest. Figure 7 In the embodiments disclosed in , it can be understood that the sharpness of the area with intermediate brightness is increased.

[0123] According to an embodiment, when the fourth gain value is adaptively determined according to the magnitude of the brightness value of the first image 640, the second image 650 having improved sharpness may be obtained.

[0124] After applying the fourth gain value g4 to the second differential image 620, the second differential image 620 may be combined with the first image 640. As a result of the combination, a second image 650 having improved sharpness may be obtained.

[0125] In an embodiment, the first filtered image 515, the second filtered image 525, and the third filtered image 535 to which the first gain value g1, the second gain value g2, and the third gain value g3 are respectively applied may be combined and then combined with the first image 640 to obtain the second image 650. That is, the process of applying the fourth gain value g4 to the second differential image 620 based on the brightness information may be omitted.

[0126] Reference again Figure 2 , each of the second differential image and the second image output from the sharpness adjustment unit 250 may be provided to the image analysis unit 240 and the tone curve application unit 270 .

[0127] The image analysis unit 240 may provide at least one of the edge map of the first image and the distribution information of the sample values ​​of the second difference image to the tone curve generation unit 260 .

[0128] In an embodiment, the distribution information of the sample values ​​of the second differential image may include histogram information of the second differential image. Here, the histogram information may indicate the number of samples of the second differential image having a predetermined value or a predetermined range of values.

[0129] In an embodiment, the distribution information of the sample values ​​of the second differential image may include histogram information of the blocks of the second differential image. Here, the histogram information of the blocks may represent the number of samples in each block of the second differential image having a predetermined value or a predetermined range of values.

[0130] The tone curve generation unit 260 may change the parameter value (or the average parameter value) obtained from the neural network 210 according to the sample value of the edge map and may generate the tone curve by using the changed parameter value.

[0131] The neural network 210 may output a first parameter map, a second parameter map, and a third parameter map. In an embodiment, the tone curve generation unit 260 may obtain a first average parameter value by averaging the sample values ​​of the first parameter map, and may obtain a second average parameter value by averaging the sample values ​​of the second parameter map. In addition, a third average parameter value may be obtained by averaging the sample values ​​of the third parameter map. In the case where the first average parameter value, the second average parameter value, and the third average parameter value are obtained one by one, a tone curve for the second image may be generated.

[0132] In another embodiment, the tone curve generation unit 260 may obtain a first average parameter value for each block by averaging the sample values ​​in each block of the first parameter map, and may obtain a second average parameter value for each block by averaging the sample values ​​in each block of the second parameter map. In addition, a third average parameter value for each block may be obtained by averaging the sample values ​​in each block of the third parameter map. In the case where the first average parameter value, the second average parameter value, and the third average parameter value are obtained for each block, a plurality of tone curves for each block of the second image may be generated. Figure 8 Describe this.

[0133] Figure 8 335a, 335b, and 335c output from the neural network 210 according to an embodiment of the present invention.

[0134] The neural network 210 may output a first parameter map 335a including the first parameter value as a sample value, a second parameter map 335b including the second parameter value as a sample value, and a third parameter map 335c including the third parameter value as a sample value.

[0135] As described above, in the case where the tone curve generation unit 260 generates a tone curve, the first average parameter value, the second average parameter value, and the third average parameter value may also be obtained by averaging the sample values ​​of the first parameter map 335a, the sample values ​​of the second parameter map 335b, and the sample values ​​of the third parameter map 335c. The tone curve generation unit 260 may modify at least one of the first average parameter value, the second average parameter value, and the third average parameter value by considering the edge map of the first image, and may generate a tone curve by using the modified average parameter value.

[0136] When generating the tone curve 800 for the block, the tone curve generation unit 260 may also obtain a first average parameter value for the block, a second average parameter value for the block, and a third average parameter value for the block by averaging the sample values ​​in each of the blocks included in the first parameter map 335a, the second parameter map 335b, and the third parameter map 335c.

[0137] like Figure 8 As shown in , when the first parameter map 335a, the second parameter map 335b and the third parameter map 335c are divided into nine blocks, the tone curve generation unit 260 can obtain the first average parameter value of the first block (upper left block) 811a of the first parameter map 335a by averaging the sample values ​​in the first block 811a of the first parameter map 335a, can obtain the second average parameter value of the first block 811b of the second parameter map 335b by averaging the sample values ​​in the first block 811b of the second parameter map 335b, and can obtain the third average parameter value of the first block 811c of the third parameter map 335c by averaging the sample values ​​in the first block 811c of the third parameter map 335c.

[0138] Similarly, the tone curve generation unit 260 can obtain a first average parameter value of the second block (the block to the right of the upper left block) 813a of the first parameter map 335a by averaging the sample values ​​in the second block 813a of the first parameter map 335a, can obtain a second average parameter value of the second block 813b of the second parameter map 335b by averaging the sample values ​​in the second block 813b of the second parameter map 335b, and can obtain a third average parameter value of the second block 813c of the third parameter map 335c by averaging the sample values ​​in the second block 813c of the third parameter map 335c.

[0139] The tone curve generation unit 260 may modify at least one of the first average parameter value of the block, the second average parameter value of the block, and the third average parameter value of the block by considering the edge map of the first image, and may generate a tone curve 800 for the block of the second image by using the modified average parameter value of the block.

[0140] At least one of the first average parameter value for the block, the second average parameter value for the block, and the third average parameter value for the block can be modified, and samples of the edge map of the first image can also be considered for each block. In other words, the average sample value of the block of the edge map of the first image can be used to modify at least one of the first average parameter value for the block, the second average parameter value for the block, and the third average parameter value for the block.

[0141] Reference Figure 8 , the tone curve generation unit 260 may generate a tone curve 831 of the first block of the second image by using the first average parameter value of the first block 811a of the first parameter map 335a, the second average parameter value of the first block 811b of the second parameter map 335b, and the third average parameter value of the first block 811c of the third parameter map 335c.

[0142] In addition, the tone curve generation unit 260 can generate a tone curve 833 of the second block of the second image by using the first average parameter value of the second block 813a of the first parameter map 335a, the second average parameter value of the second block 813b of the second parameter map 335b, and the third average parameter value of the second block 813c of the third parameter map 335c.

[0143] Hereinafter, a method of modifying at least one of the first parameter value (or the first average parameter value), the second parameter value (or the second average parameter value), and the third parameter value (or the third average parameter value) performed by the tone curve generation unit 260 is described.

[0144] The tone curve generation unit 260 may determine a fifth gain value corresponding to the sample value of the edge map, and may apply the fifth gain value to at least one of the first parameter value, the second parameter value, and the third parameter value. For example, the tone curve generation unit 260 may also multiply the fifth gain value by the first parameter value and the third parameter value.

[0145] In an embodiment, when the sample value of the edge map is large, the fifth gain value may be determined to be large, and when the sample value of the edge map is small, the fifth gain value may be determined to be small. For example, the sample value of the edge map and the fifth gain value may be proportional to each other.

[0146] In an embodiment, the tone curve generation unit 260 may determine an average sample value by averaging sample values ​​of the edge map, and may apply a fifth gain value corresponding to the determined average sample value to at least one of the first average parameter value, the second average parameter value, and the third average parameter value.

[0147] In an embodiment, the tone curve generation unit 260 may obtain an average sample value of the block by averaging the sample values ​​of the edge map for each block, and may determine a fifth gain value of the block corresponding to the average sample value of the block. In an embodiment, the fifth gain value of the block may be applied to (e.g., multiplied by) at least one of the first average parameter value of the block, the second average parameter value of the block, and the third average parameter value of the block.

[0148] A large average sample value of a block in the edge map means a large amount of variation in sample values ​​in a corresponding block in the first image. Therefore, the tone curve application unit 270 may adjust the tone curve for the corresponding block to increase contrast.

[0149] In an embodiment, when generating a tone curve for the second image, the tone curve generation unit 260 may modify the tone curve by considering the distribution information of the sample values ​​of the second difference image. For example, the tone curve generation unit 260 may determine a sixth gain value according to the distribution information of the sample values ​​of the second difference image, and may adjust the tone curve by applying the sixth gain value to the output sample values ​​mapped to the input sample values ​​of the tone curve.

[0150] In an embodiment, the distribution information of the sample values ​​of the second differential image may include an average value of squares of the sample values ​​of the second differential image. In the case where a tone curve is generated for each block of the second image, the distribution information of the sample values ​​of the second differential image may include an average value of squares of the sample values ​​in each block included in the second differential image.

[0151] Since the second difference image is an image in which high-frequency components are emphasized by the sharpness adjustment unit 250, a large sample value may mean that the contrast of the area including the corresponding sample needs to be improved. That is, the tone curve generation unit 260 may also identify the portion in which sharpness is emphasized in the second difference image, and adjust the tone curve so that the contrast of the identified portion is improved.

[0152] In an embodiment, the tone curve generation unit 260 may determine the sixth gain value according to the average value of the square of the sample values ​​of the second differential image. For example, the average value of the square of the sample values ​​of the second differential image and the sixth gain value may be proportional to each other. Therefore, when the average value of the square of the sample values ​​of the second differential image is large, the sixth gain value may be determined to be large, and when the average value of the square of the sample values ​​of the second differential image is small, the sixth gain value may be determined to be small.

[0153] The tone curve generation unit 260 may provide the tone curve applied unit 270 with a modified tone curve according to the distribution information of the sample values ​​of the second difference image.

[0154] The tone curve application unit 270 may also change the sample values ​​of the second image by using the tone curve received from the tone curve generation unit 260. For example, the tone curve application unit 270 may identify output sample values ​​mapped to sample values ​​of the second image from the tone curve, and may obtain an output image including the identified output sample values.

[0155] In case the tone curve for blocks is received from the tone curve generation unit 260 , the tone curve application unit 270 may also change the sample values ​​of the second image for each block.

[0156] refer to Fig. 9 A method of applying a tiled tone curve to the second image 650 is described.

[0157] like Fig. 9 As shown, when the second image 650 is divided into nine blocks, the tone curve application unit 270 may apply each of the nine tone curves received from the tone curve generation unit 260 to each block of the second image 650 .

[0158] For example, the tone curve for the first block (the upper left block) 910 of the second image 650 can be applied to samples of the first block 910 of the second image 650, and the tone curve for the second block (the block to the right of the upper left block) 920 of the second image 650 can be applied to samples of the second block 920 of the second image 650.

[0159] In an embodiment, blocking artifacts may appear in the output image when a tone curve is applied to each block of the second image 650. Thus, when a tone curve is applied to samples within one block of the second image 650, the tone curve application unit 270 may also use tone curves of neighboring blocks.

[0160] Reference Fig. 9 , when changing the current sample curr in the first block 910 of the second image 650, the tone curve application unit 270 can use not only the tone curve for the first block 910, but also the tone curves for the adjacent blocks, i.e., the tone curve for the second block 920, the tone curve for the fourth block 930, and the tone curve for the fifth block 940.

[0161] In an embodiment, the tone curve application unit 270 may obtain a first output sample corresponding to the current sample curr according to the tone curve used for the first block 910, may obtain a second output sample corresponding to the current sample curr according to the tone curve used for the second block 920, may obtain a third output sample corresponding to the current sample curr according to the tone curve used for the fourth block 930, and may obtain a fourth output sample corresponding to the current sample curr according to the tone curve used for the fifth block 940.

[0162] The tone curve application unit 270 can obtain a final output sample corresponding to the current sample curr by performing weighted sum or weighted average on the first output sample, the second output sample, the third output sample, and the fourth output sample according to the distance between the current sample curr and the center sample c1 in the first block 910, the distance between the current sample curr and the center sample c2 in the second block 920, the distance between the current sample curr and the center sample c3 in the fourth block 930, and the distance between the current sample curr and the center sample c4 in the fifth block 940.

[0163] The weights by which the first, second, third, and fourth output samples are multiplied may be inversely proportional to the distances between the current sample curr and the center samples c1, c2, c3, and c4 of the corresponding blocks 1010, 1020, 1030, and 1040. For example, as the distance between the current sample curr and any one of the center samples increases, a small weight may be determined for an output sample obtained according to a tone curve for a block including the corresponding center sample. In addition, as the distance between the current sample curr and any one of the center samples decreases, a large weight may be determined for an output sample obtained according to a tone curve for a block including the corresponding center sample.

[0164] In an embodiment, when samples within any one block are changed according to a tone curve, the tone curves of adjacent blocks are also used, and thus, continuity between blocks within an output image may be improved.

[0165] Fig.10 is a flowchart for describing an image processing method according to an embodiment.

[0166] Reference Fig.10 In operation S1010, the image processing apparatus 200 obtains a first difference image having a resolution greater than a resolution of the input image and a parameter value for a tone curve by applying the input image to the neural network 210.

[0167] In operation S1020, the image processing apparatus 200 obtains a plurality of filtered images by filtering the first differential image. The filtered images may include a filtered image for a low frequency band and a filtered image for a high frequency band.

[0168] In an embodiment, the filtered image may include a filtered image for a low frequency band, a filtered image for a middle frequency band, and a filtered image for a high frequency band.

[0169] In operation S1030, the image processing apparatus 200 applies a gain value corresponding to the sample value of the edge map of the first image to the filtered image. For example, the image processing apparatus 200 may multiply the gain value corresponding to the sample value of the edge map of the first image by the sample value of the filtered image.

[0170] In operation S1040, the image processing apparatus 200 obtains a second differential image by combining the filtered images to which the gain values ​​are applied.

[0171] In an embodiment, the image processing apparatus 200 may obtain the second difference image by combining sample values ​​of the filtered images to which the gain values ​​are applied.

[0172] In operation S1050, the image processing apparatus 200 obtains a second image by combining the second differential image with the first image. For example, the image processing apparatus 200 may obtain the second image by summing a sample value of the second differential image and a sample value of the first image.

[0173] In an embodiment, the second difference image may be combined with the first image after being modified based on brightness information of the first image.

[0174] In operation S1060, the image processing apparatus 200 applies a gain value corresponding to an average sample value of the edge map to a parameter value obtained from the neural network 210. The average sample value of the edge map may be an average of sample values ​​within the edge map or an average of sample values ​​of each block within the edge map.

[0175] In operation S1070, the image processing apparatus 200 determines a tone curve according to the parameter value to which the gain value is applied. In an embodiment, the image processing apparatus 200 may determine the tone curve by using a curve fitting algorithm based on the parameter value.

[0176] In operation S1080, the image processing apparatus 200 obtains an output image by changing the second image according to the tone curve.

[0177] In an embodiment, the image processing apparatus 200 may modify the tone curve by considering distribution information of sample values ​​of the second difference image, and may change the second image according to the modified tone curve.

[0178] In the following, reference Figures 11 to 13 The training process of neural network 210 is described.

[0179] Fig.11 is a diagram for describing the training process of the first sub-neural network 310 and the second sub-neural network 320 according to an embodiment, and Fig.12 is a diagram for describing a training process of the third sub-neural network 330 according to an embodiment.

[0180] As described above, the neural network 210 may include a first sub-neural network 310, a second sub-neural network 320, and a third sub-neural network 330. After the training of the first sub-neural network 310 and the second sub-neural network 320 is completed, the third sub-neural network 330 may be additionally trained.

[0181] First, refer to Fig.11 , the training input image 1101 may be input to the first sub-neural network 310. The training input image 1101 may correspond to the input image input to the image processing device 200. The training input image 1101 may be obtained by downscaling the GT image 1104.

[0182] The first sub-neural network 310 may output a feature map to the second sub-neural network 320 by processing the training input image 1101, and the second sub-neural network 320 may output a training first difference image 1102 by processing the feature map.

[0183] The first sub-neural network 310 and the second sub-neural network 320 can process the training input image 1101 and the feature map according to preset neural network parameters.

[0184] Separate from inputting the training input image 1101 to the first sub-neural network 310 , the scaler 1110 may upscale the training input image 1101 and output an upscaled image 1103 having a resolution greater than that of the training input image 1101 .

[0185] A difference image 1105 corresponding to the difference between the enlarged image 1103 and the GT image 1104 may be compared with the training first difference image 1102 , and loss information 1106 may be obtained as a result of the comparison.

[0186] The loss information 1106 corresponding to the result of comparing the difference image 1105 and the training first difference image 1102 may include at least one of an L1 norm value, an L2 norm value, a structural similarity (SSIM) value, a Peak Signal-To-Noise Ratio-Human Vision System (PSNR-HVS) value, a multi-scale SSIM (MS-SSIM) value, a Variance Inflation Factor (VIF) value, and a Video Multimethod Assessment Fusion (VMAF) value of the difference between the difference image 1105 and the training first difference image 1102.

[0187] The first sub-neural network 310 and the second sub-neural network 320 may update the neural network parameters so that the loss information 1106 is reduced or minimized.

[0188] Next, the third sub-neural network 330 may be trained through a cooperative operation with the trained first sub-neural network 310.

[0189] Reference Fig.12 , the training input image 1201 can be input to the first sub-neural network 310, and the first sub-neural network 310 can output a feature map by processing the training input image 1201 according to the neural network parameters set through the training process.

[0190] The third sub-neural network 330 may output a parameter value (or a parameter map) of a tone curve by processing the feature map according to preset neural network parameters.

[0191] The training input image 1201 may be changed into a tone-mapped image 1202 through a tone curve application process 1210 using a tone curve determined according to parameter values.

[0192] Loss information 1204 corresponding to a result of comparing the tone-mapped image 1202 with the GT image 1203 may be obtained, and the third sub-neural network 330 may update the neural network parameters so that the loss information 1204 is reduced or minimized.

[0193] In an embodiment, the first sub-neural network 310, the second sub-neural network 320, and the third sub-neural network 330 included in the neural network 210 may be trained together. Fig.13 Describe this.

[0194] Fig.13 3 is a diagram for describing the training process of the first sub-neural network 310, the second sub-neural network 320, and the third sub-neural network 330 according to an embodiment.

[0195] Reference Fig.13 , the training input image 1301 may be input to a training device 1300 including a neural network 210. The training device 1300 may correspond to the above-mentioned image processing device 200 or a separate server.

[0196] As Figure 2 The image processing device 200 shown processes an input image and outputs an output image, and the training device 1300 can process a training input image 1301 and output a training output image 1302 .

[0197] Loss information 1304 corresponding to the result of comparing the training output image 1302 with the GT image 1303 may be obtained. Neural network parameters of the first sub-neural network 310, the second sub-neural network 320, and the third sub-neural network 330 included in the neural network 210 may be updated so that the loss information 1304 is reduced or minimized.

[0198] On the other hand, regarding Figure 2 , the tone curve application unit 270 has been described as applying the tone curve to the second image, but in an embodiment, the tone curve application unit 270 may also apply the tone curve to the first image. In this case, instead of the second differential image, the image analysis unit 240 may provide the tone curve generation unit 260 with distribution information of sample values ​​of the first differential image obtained from the neural network 210.

[0199] The tone curve generation unit 260 may modify the parameter value obtained from the neural network 210 by considering the distribution information of the sample value of the first differential image (or the second differential image), and may generate a tone curve of the first image by using the modified parameter value. The tone curve application unit 270 may obtain a third image by applying the tone curve to the first image. An output image may be obtained by combining the second image with the third image.

[0200] Fig.14 is a block diagram showing a configuration of an image display apparatus 1400 including the image processing apparatus 200 according to an embodiment.

[0201] Fig.14 The image display device 1400 shown may include Figure 2 An image processing device 200 is shown.

[0202] Reference Fig.14 According to an embodiment, the image display device 1400 may include a tuner unit 1440, a processor 1410, a display unit 1420, a communication unit 1450, a sensing unit 1430, an input / output unit 1470, a video processing unit 1480, an audio processing unit 1485, an audio output unit 1460, a memory 1490 and a power supply unit 1495.

[0203] The tuner unit 1440 according to an embodiment can tune and select only the frequency of a channel to be received among a plurality of radio wave components by amplifying, mixing, resonating, etc. a broadcast signal received in a wired or wireless manner. The broadcast signal includes audio, video, and additional information (e.g., Electronic Program Guide (EPG)).

[0204] The tuner unit 1440 may receive broadcast signals from various sources, such as terrestrial broadcasting, cable broadcasting, satellite broadcasting, and Internet broadcasting. The tuner unit 1440 may also receive broadcast signals from sources such as analog broadcasting or digital broadcasting.

[0205] The sensing unit 1430 may sense a user's voice, a user's image, or a user's interaction, and may include a microphone 1431 , a camera unit 1432 , and an optical receiving unit 1433 .

[0206] The microphone 1431 receives a voice uttered by the user. The microphone 1431 may convert the received voice into an electric signal and output the electric signal to the processor 1410. The user's voice may include, for example, a voice corresponding to a menu or function of the image display device 1400.

[0207] The camera unit 1432 may receive images (eg, successive frames).

[0208] The optical receiving unit 1433 receives an optical signal (including a control signal) from an external control device through an optical window (not shown) of a bezel of the display unit 1420. The optical receiving unit 1433 may receive an optical signal corresponding to a user input (e.g., touch, press, touch gesture, voice, or motion) from the control device. The control signal may be extracted from the received optical signal under the control of the processor 1410.

[0209] The input / output unit 1470 may receive video (e.g., moving images, etc.), audio (e.g., voice, music, etc.), and additional information (e.g., EPG, etc.) from the outside of the image display device 1400 under the control of the processor 1410. The input / output interface may include any one of a high-definition multimedia interface (HDMI), a mobile high-definition link (MHL), a universal serial bus (USB), a display port (DP), thunderbolt, a video graphics array (VGA) port, an RGB port, a D-subminiature (D-SUB), a digital visual interface (DVI), a component jack, and a PC port.

[0210] The processor 1410 controls the overall operation of the image display device 1400 and the signal flow between the internal components of the image display device 1400, and performs data processing functions. When there is a user input or a preset and stored condition is met, the processor 1410 can execute an operating system (OS) and various applications stored in the memory 1490.

[0211] The processor 1410 may include a random access memory (RAM) that stores signals or data input from outside the image display device 1400 or is used as a storage area corresponding to various tasks performed in the image display device 1400 (e.g., an image noise elimination task), a read-only memory (ROM) that stores a control program for controlling the image display device 1400, and a processor.

[0212] The video processing unit 1480 performs processing on the video data received by the image display device 1400. The video processing unit 1480 may perform various image processing such as decoding, scaling, noise reduction, frame rate conversion, or resolution conversion on the video data.

[0213] The audio processing unit 1485 may perform processing on the audio data. The audio processing unit 1485 may perform various processing on the audio data, such as decoding, amplification, or noise reduction. On the other hand, the audio processing unit 1485 may include a plurality of audio processing modules configured to process audio corresponding to a plurality of contents.

[0214] The audio output unit 1460 outputs audio included in a broadcast signal received through the tuner unit 1440 under the control of the processor 1410. The audio output unit 1460 may output audio (e.g., voice, sound, etc.) input through the communication unit 1450 or the input / output unit 1470. In addition, the audio output unit 1460 may output audio stored in the memory 1490 under the control of the processor 1410. The audio output unit 1460 may include at least one of a speaker, a headphone output terminal, or a Sony / Philips Digital Interface (S / PDIF) output terminal.

[0215] The power supply unit 1495 supplies power input from an external power source to components inside the image display device 1400 under the control of the processor 1410. In addition, the power supply unit 1495 can supply power output from one or two batteries (not shown) located inside the image display device 1400 to the internal components under the control of the processor 1410.

[0216] The memory 1490 may store various data, programs or applications that drive and control the image display device 1400 under the control of the processor 1410. The memory 1490 may include a broadcast receiving module (not shown), a channel control module (not shown), a volume control module (not shown), a communication control module (not shown), a voice recognition module (not shown), a motion recognition module (not shown), an optical receiving module (not shown), a display control module (not shown), an audio control module (not shown), an external input control module (not shown), a power control module (not shown), a power control module (not shown) of an external device connected wirelessly (e.g., via Bluetooth), a voice database (DB) (not shown) or a motion DB (not shown). The modules (not shown) and DB (not shown) of the memory 1490 may be implemented in the form of software to perform a broadcast receiving control function, a channel control function, a volume control function, a communication control function, a voice recognition function, a motion recognition function, an optical receiving control function, a display control function, an audio control function, an external input control function, a power control function or a power control function of an external device connected wirelessly (e.g., via Bluetooth) in the image display device 1400. The processor 1410 may perform each function by using software stored in the memory 1490 .

[0217] on the other hand, Fig.14The block diagram of the image display device 1400 shown is a block diagram for an embodiment. Each element of the block diagram can be integrated, added or omitted according to the specification of the image display device 1400 actually implemented. That is to say, when necessary, two or more elements can be integrated into one element, or one element can be subdivided into two or more elements. In addition, the function performed by each block is to describe an embodiment, and its specific operation or equipment are not intended to limit the scope of the present disclosure.

[0218] The image processing apparatus 200 and method according to the embodiment are intended to manufacture a lightweight neural network for image processing.

[0219] In addition, the image processing apparatus 200 and method according to the embodiment are intended to further improve the image quality of an image obtained through image processing.

[0220] The image processing apparatus 200 according to the embodiment may include: a memory configured to store at least one instruction; and at least one processor configured to operate according to the at least one instruction.

[0221] In an embodiment, the at least one processor may obtain a first difference image having a resolution greater than a resolution of the input image and a parameter value of the tone curve by applying the input image to the neural network 210 .

[0222] In an embodiment, the at least one processor may obtain a plurality of filtered images of different frequency bands by filtering the first difference image.

[0223] In an embodiment, the at least one processor may apply a gain value corresponding to a sample value of an edge map of the first image upscaled from the input image to each of the plurality of filtered images.

[0224] In an embodiment, the at least one processor may obtain the second difference image by combining a plurality of filtered images to which the gain values ​​are applied.

[0225] In an embodiment, the at least one processor may obtain the second image by combining the second difference image with the first image.

[0226] In an embodiment, the at least one processor may apply a gain value corresponding to an average sample value of the edge map to the parameter value.

[0227] In an embodiment, the at least one processor may determine a tone curve based on parameter values ​​to which the gain values ​​are applied.

[0228] In an embodiment, the at least one processor may obtain the output image by changing sample values ​​of the second image according to the determined tone curve.

[0229] In an embodiment, the neural network 210 may include: a first sub-neural network 310, configured to output a feature map by processing an input image; a second sub-neural network 320, configured to output a first difference image by processing the feature map output from the first sub-neural network 310; and a third sub-neural network 330, configured to output a parameter value by processing the feature map output from the first sub-neural network 310.

[0230] In an embodiment, the gain values ​​corresponding to the sample values ​​of the edge map may include a first gain value applied to the filtered image for a first frequency band and a second gain value applied to the filtered image for a second frequency band, the first frequency band may be larger than the second frequency band, and when the sample value of the edge map is greater than a preset value, it can be determined that the first gain value is greater than the second gain value.

[0231] In an embodiment, the at least one processor may be further configured to change the second difference image according to brightness information of the first image, and obtain the second image by combining the changed second difference image with the first image.

[0232] In an embodiment, the brightness information may include a brightness value of the first image.

[0233] In an embodiment, the at least one processor may be further configured to apply a gain value corresponding to the brightness value to the samples of the second difference image.

[0234] In an embodiment, the at least one processor may be further configured to: adjust the determined tone curve according to distribution information of sample values ​​of the second difference image, and obtain the output image by changing the sample values ​​of the second image according to the adjusted tone curve.

[0235] In an embodiment, the distribution information of the sample values ​​may include an average value of squares of the sample values ​​of the second image.

[0236] In an embodiment, the at least one processor may be further configured to adjust the tone curve by applying a gain value corresponding to the average value to the output sample values ​​mapped to the input sample values ​​of the tone curve.

[0237] In an embodiment, the tone curve may include tone curves respectively corresponding to blocks of the second image.

[0238] In an embodiment, at least one processor may further be configured to obtain a first output sample corresponding to the current sample based on a tone curve corresponding to a current block including the current sample of a second image, obtain a second output sample corresponding to the current sample based on a tone curve corresponding to an adjacent block adjacent to the current block, and obtain a final output sample of the output image corresponding to the current sample by performing a weighted sum of the first output sample and the second output sample based on a distance between the current sample and a center sample of the current block and a distance between the current sample and a center sample of the adjacent block.

[0239] The image processing method according to the embodiment may include obtaining a first difference image having a resolution greater than that of the input image and a parameter value of the tone curve by applying the input image to the neural network 210 .

[0240] In an embodiment, the image processing method may include obtaining a plurality of filtered images of different frequency bands by filtering the first difference image.

[0241] In an embodiment, the image processing method may include applying a gain value corresponding to a sample value of an edge map of a first image upscaled from an input image to each of the plurality of filtered images.

[0242] In an embodiment, the image processing method may include obtaining a second difference image by combining a plurality of filtered images to which gain values ​​are applied.

[0243] In an embodiment, the image processing method may include obtaining a second image by combining the second difference image with the first image.

[0244] In an embodiment, the image processing method may include applying a gain value corresponding to an average sample value of the edge map to the parameter value.

[0245] In an embodiment, the image processing method may comprise determining a tone curve from a parameter value to which the gain value is applied.

[0246] In an embodiment, the image processing method may comprise obtaining an output image by changing sample values ​​of the second image according to the determined tone curve.

[0247] In an embodiment, the gain values ​​corresponding to the sample values ​​of the edge map may include a first gain value applied to the filtered image for a first frequency band and a second gain value applied to the filtered image for a second frequency band, the first frequency band may be larger than the second frequency band, and when the sample value of the edge map is greater than a preset value, it can be determined that the first gain value is greater than the second gain value.

[0248] In an embodiment, obtaining the second image may include: changing the second differential image according to brightness information of the first image; and obtaining the second image by combining the changed second differential image with the first image.

[0249] In an embodiment, obtaining the output image may include: adjusting the determined tone curve according to distribution information of sample values ​​of the second difference image; and obtaining the output image by changing the sample values ​​of the second image according to the adjusted tone curve.

[0250] Since the image processing apparatus 200 and method according to the embodiment can improve the resolution and contrast of an image using a single neural network, the neural network used for image processing can be lightweight.

[0251] In addition, the image processing apparatus 200 and method according to the embodiment may further improve image quality by sharing data representing characteristics of an image with different algorithms used for image processing.

[0252] In addition, the image processing apparatus 200 and method according to the embodiment may improve image sharpness and contrast together in units of micro-blocks by combining existing pixel-centric sharpness improvement and block-centric contrast improvement.

[0253] On the other hand, the embodiments of the present disclosure may be written as a program executable on a computer, and the written program may be stored in a machine-readable storage medium.

[0254] The machine-readable storage medium may be provided in the form of a non-transitory storage medium. A "non-transitory storage medium" is a tangible device and only means that signals (e.g., electromagnetic waves) are not included. The term does not distinguish between a case where data is semi-permanently stored in a storage medium and a case where data is temporarily stored in a storage medium. For example, a non-transitory storage medium may include a buffer that temporarily stores data.

[0255] According to an embodiment, the method according to various embodiments of the present disclosure may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a commodity. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) is at least temporarily stored on a machine-readable storage medium, such as a memory of a manufacturer's server, an application store's server, or a relay server, or may be temporarily generated.

[0256] The technical concept of the present disclosure has been described in detail with reference to the preferred embodiments, but the technical concept of the present disclosure is not limited to the above embodiments, and a person of ordinary skill in the art can make various modifications and changes thereto within the scope of the technical concept of the present disclosure.

Claims

1. An image processing device (200), comprising: a memory configured to store at least one instruction; as well as at least one processor configured to operate according to the at least one instruction to: Obtaining a first difference image and parameter values ​​of a tone curve by applying an input image to a neural network (210), the first difference image having a resolution greater than a resolution of the input image; Obtaining a plurality of filtered images of different frequency bands by filtering the first differential image; applying a gain value corresponding to a sample value of an edge map of the first image upscaled from the input image to each of the plurality of filtered images; obtaining a second difference image by combining the plurality of filtered images to which the gain values ​​are applied; obtaining a second image by combining the second difference image with the first image; applying a gain value corresponding to the average sample value of the edge map to the parameter value; determining a tone curve based on the parameter value to which the gain value is applied; as well as An output image is obtained by changing the sample values ​​of the second image according to the determined tone curve.

2. The image processing apparatus according to claim 1, wherein: The neural network (210) comprises: A first sub-neural network (310) is configured to output a feature map by processing an input image; a second sub-neural network (320) configured to output a first difference image by processing the feature map output from the first sub-neural network (310); and The third sub-neural network (330) is configured to output a parameter value by processing the feature map output from the first sub-neural network (310).

3. The image processing device according to claim 1 or 2, wherein: The gain values ​​corresponding to the sample values ​​of the edge map include a first gain value applied to the filtered image of the first frequency band and a second gain value applied to the filtered image of the second frequency band, The first frequency band is greater than the second frequency band, and When the sample value of the edge map is greater than a preset value, the first gain value is determined to be greater than the second gain value.

4. The image processing apparatus according to any one of claims 1 to 3, wherein: The at least one processor is further configured to: changing the second difference image according to the brightness information of the first image; and A second image is obtained by combining the changed second difference image with the first image.

5. The image processing apparatus according to any one of claims 1 to 4, wherein: The brightness information includes the brightness value of the first image, and The at least one processor is further configured to apply gain values ​​corresponding to the luminance values ​​to samples of the second difference image.

6. The image processing apparatus according to any one of claims 1 to 5, wherein: The at least one processor is further configured to: adjusting the determined tone curve according to distribution information of sample values ​​of the second difference image; and An output image is obtained by changing the sample values ​​of the second image according to the adjusted tone curve.

7. The image processing apparatus according to any one of claims 1 to 6, wherein: The distribution information of the sample values ​​includes an average value of squares of the sample values ​​of the second image, and The at least one processor is further configured to adjust the tone curve by applying a gain value corresponding to the average value to output sample values ​​mapped to input sample values ​​of the tone curve.

8. The image processing apparatus according to any one of claims 1 to 7, wherein: The tone curves include tone curves respectively corresponding to blocks of the second image, and The at least one processor is further configured to: obtaining a first output sample corresponding to the current sample according to a tone curve corresponding to a current block including the current sample of the second image; obtaining a second output sample corresponding to the current sample according to a tone curve corresponding to a neighboring block adjacent to the current block; as well as A final output sample of an output image corresponding to a current sample is obtained by performing weighted summation on the first output sample and the second output sample according to a distance between the current sample and a central sample of a current block and a distance between the current sample and a central sample of an adjacent block.

9. An image processing method performed by an image processing device (200), the image processing method comprising: Obtaining a first difference image and parameter values ​​of a tone curve by applying an input image to a neural network (210), the first difference image having a resolution greater than a resolution of the input image; Obtaining a plurality of filtered images of different frequency bands by filtering the first differential image; applying a gain value corresponding to a sample value of an edge map of the first image upscaled from the input image to each of the plurality of filtered images; obtaining a second difference image by combining the plurality of filtered images to which the gain values ​​are applied; obtaining a second image by combining the second difference image with the first image; applying a gain value corresponding to the average sample value of the edge map to the parameter value; determining a tone curve based on the parameter value to which the gain value is applied; as well as An output image is obtained by changing the sample values ​​of the second image according to the determined tone curve.

10. The image processing method according to claim 9, wherein: The gain values ​​corresponding to the sample values ​​of the edge map include a first gain value applied to the filtered image of the first frequency band and a second gain value applied to the filtered image of the second frequency band, The first frequency band is greater than the second frequency band, and When the sample value of the edge map is greater than a preset value, the first gain value is determined to be greater than the second gain value.

11. The image processing method according to claim 9 or 10, wherein: Obtaining the second image includes: changing the second difference image according to the brightness information of the first image; and A second image is obtained by combining the changed second difference image with the first image.

12. The image processing method according to any one of claims 9 to 11, wherein: Obtaining the output image includes: adjusting the determined tone curve according to distribution information of sample values ​​of the second difference image; and An output image is obtained by changing the sample values ​​of the second image according to the adjusted tone curve. 13 . A computer-readable recording medium having recorded thereon a program for causing a computer to execute the image processing method according to claim 9 .

Images