Image processing device and image processing method thereof

Abstract

An image processing device filters an image and obtains a signal within a threshold range as a shadow. The image processing device obtains boundary information by applying boundary detection filters associated with different directions. Based on the boundary information, the shadow is applied to a portion of the input image to provide an output image with improved sharpness.

Description

Technical Field

[0001] The present disclosure relates to enhancement processing of input images. Background Art

[0002] The development of electronic technology has led to the development and distribution of various types of electronic devices. In particular, in recent years, display devices used in various places such as homes, offices, and public places have continued to develop.

[0003] Recently, the demand for high-resolution image services has been increasing. This demand has led to the development of technologies for improving image detail as the resolution of display devices increases. Specifically, there is a need to develop detail enhancement technologies that, when upscaling images, sharpen details lost during compression or blurring. Summary of the Invention

[0004] Embodiments may overcome the above disadvantages and other disadvantages not described above. In addition, embodiments are not required to overcome the above disadvantages, and embodiments may not overcome any of the problems described above.

[0005] The present disclosure provides an image processing apparatus and an image processing method for accurately detecting boundaries of objects in an image and applying shadows to the detected boundaries to enhance details of the image.

[0006] According to an embodiment, the present disclosure relates to an image processing apparatus and an image processing method thereof for enhancing details of an image by accurately detecting boundaries of an object in an image and applying shadows to the detected boundaries.

[0007] An embodiment provides an image processing apparatus comprising: one or more processors; and one or more memories storing program code. Execution of the program code by the one or more processors is configured to cause the image processing apparatus to: obtain a shadow based on filtering an input image, where the shadow is a signal within a threshold range; obtain boundary information by applying multiple boundary detection filters to the input image, where each boundary detection filter is associated with a different direction; and obtain an output image by applying the shadow to a region identified based on the boundary information.

[0008] In some embodiments of the image processing apparatus, the one or more processors are further configured to obtain the output signal by filtering with a second-order differential filter, and obtain the shadow by limiting the positive signal or the negative signal from the output signal.

[0009] In some embodiments of the image processing device, the shadow corresponds to at least one of the following: an inner boundary of a first object in the input image, the inner boundary including a first pixel having a first pixel value less than a first threshold, or an outer boundary of a second object in the input image, the outer boundary including a second pixel having a second pixel value greater than or equal to a second threshold, wherein the second threshold is greater than the first threshold.

[0010] In some embodiments of the image processing device, the boundary information includes position information and amplitude information, the one or more processors are further configured to identify the boundary area based on the position information, and the one or more processors are further configured to obtain the output image by the following operations: applying the amplitude information corresponding to the boundary area to the shadow, and inserting the shadow to which the amplitude information is applied into the boundary area.

[0011] In some embodiments of the image processing apparatus, the one or more processors are further configured to perform filtering using a second-order differential filter, and a first size of each of the plurality of boundary detection filters is larger than a second size of the second-order differential filter.

[0012] In some embodiments of the image processing device, the input image includes a first pixel block, and the one or more processors are further configured to obtain boundary information by: obtaining a first plurality of filter output values ​​by applying a plurality of boundary detection filters to the first pixel block, and obtaining first boundary information associated with the first pixel block based on a first value of the first plurality of filter output values.

[0013] In some embodiments of the image processing device, the one or more processors are further configured to obtain boundary information associated with the first pixel block based on: the amplitude of the minimum negative value of the first plurality of filter output values, or the maximum absolute value of the first plurality of filter output values.

[0014] In some embodiments of the image processing apparatus, the one or more processors are further configured to obtain weights by performing normalization by applying a threshold to i) the magnitude of the minimum negative value or ii) the maximum absolute value, and obtain an output image by applying the weights to the obtained shadows.

[0015] In some embodiments of the image processing apparatus, the one or more processors are further configured to determine at least one of a number or a size of the plurality of boundary detection filters based on characteristics of the boundary region.

[0016] In some embodiments of the image processing device, the multiple boundary detection filters include: a first filter, the same filter coefficients overlap in units of rows in a first direction; a second filter, the same filter coefficients overlap in units of rows in a second direction, a third filter, the same filter coefficients overlap in units of rows in a third direction, and a fourth filter, the same filter coefficients overlap in units of rows in a fourth direction.

[0017] In some embodiments of the image processing apparatus, based on the number of the plurality of boundary detection filters being N, the direction of the nth boundary detection filter is calculated as follows: angle of the nth boundary detection filter=180*(n-1) / N.

[0018] In some embodiments of the image processing device, the image processing device includes a display, and the one or more processors are further configured to control the display to output the output image, and the output image is a 4K ultra high definition (UHD) image or an 8K UHD image.

[0019] Also provided herein is an image processing method, comprising: filtering an input image; obtaining a shadow based on the filtering of the input image, wherein the shadow is a signal within a threshold range; obtaining boundary information by applying a plurality of boundary detection filters to the input image, wherein each boundary detection filter in the plurality of boundary detection filters is associated with a different direction; identifying a boundary region based on the boundary information; and obtaining an output image by applying the shadow to the region identified based on the boundary information.

[0020] In some embodiments of the image processing method, obtaining the shadow includes obtaining an output signal by filtering with a second-order differential filter, and limiting a positive signal or a negative signal from the output signal.

[0021] In some embodiments of the image processing method, the shadow corresponds to at least one of the following: an inner boundary of a first object in the input image, the inner boundary including a first pixel having a first pixel value less than a first threshold, or an outer boundary of a second object in the input image, the outer boundary including a second pixel having a second pixel value greater than or equal to a second threshold, wherein the second threshold is greater than the first threshold.

[0022] In some embodiments of the image processing method, the boundary information includes position information and amplitude information, and obtaining the output image includes: identifying the boundary area based on the position information, applying the amplitude information corresponding to the boundary area to the shadow, and inserting the shadow to which the amplitude information is applied into the boundary area.

[0023] In some embodiments of the image processing method, obtaining the shadow includes filtering using a second-order differential filter, and a first size of each of the plurality of boundary detection filters is larger than a second size of the second-order differential filter.

[0024] In some embodiments of the image processing method, the input image includes a first pixel block, and obtaining boundary information includes: obtaining a first plurality of filter output values ​​by applying a plurality of boundary detection filters to the first pixel block, and obtaining first boundary information associated with the first pixel block based on a first value of the first plurality of filter output values.

[0025] In some embodiments of the image processing method, obtaining boundary information includes: obtaining boundary information associated with the first pixel block based on the magnitude of the smallest negative value of the first plurality of filter output values, or the largest absolute value of the first plurality of filter output values.

[0026] Also provided herein is a non-transitory computer-readable recording medium storing computer instructions, which, when one or more processors of the image processing device execute the computer instructions, enable the image processing device to perform operations, the operations including: obtaining shadows based on filtering of an input image, wherein the shadows are signals within a threshold range; obtaining boundary information by applying multiple boundary detection filters to the input image, wherein each boundary detection filter in the multiple boundary detection filters is associated with a different direction; and obtaining an output image by applying the shadows to an area identified based on the boundary information.

[0027] According to various embodiments, by accurately detecting the boundaries of an object in a high-resolution image to apply shading, the sharpness of the image can be enhanced by making the outline of the object clear while minimizing the amplification of noise of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above and / or other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings, in which:

[0029] Figure 1 is a diagram for describing implementation of an image processing apparatus according to an embodiment;

[0030] Figure 2 is a block diagram showing a configuration of an image processing apparatus according to an embodiment;

[0031] Figure 3A is a diagram used to describe the characteristics of a second-order differential signal to help understand the present disclosure;

[0032] Figure 3B are diagrams showing various examples of Laplacian filters according to the embodiment;

[0033] Figure 3C is a view for describing a method for applying a Laplacian filter according to an embodiment;

[0034] Figure 4 is a diagram for describing an output of a filter according to an embodiment;

[0035] Figure 5 is a diagram for describing a plurality of boundary detection filters according to an embodiment;

[0036] Figure 6A is a diagram for describing a plurality of boundary detection filters according to another embodiment;

[0037] Figure 6B is a diagram for describing a plurality of boundary detection filters according to another embodiment;

[0038] Figure 7A is a view for describing an example of an input image according to an embodiment;

[0039] Figure 7B is a view for describing a second-order differential image according to an embodiment;

[0040] Figure 7C is a view showing a shadow image according to an embodiment;

[0041] Figure 7D is a view for describing an output image according to an embodiment;

[0042] Figure 8A is a diagram for describing a boundary detection effect according to an embodiment;

[0043] Figure 8B is a diagram for describing a boundary detection effect according to an embodiment;

[0044] Figure 8C is a diagram for describing a boundary detection effect according to an embodiment;

[0045] Figure 9 is a view for describing an example of implementation of the image processing method according to the embodiment;

[0046] Figure 10 is a view showing implementation of an image processing apparatus according to another embodiment;

[0047] Figure 11 is a view for describing a shadow insertion effect according to an embodiment;

[0048] Figure 12A is a diagram for describing a method for determining whether to apply boundary enhancement as provided in various embodiments;

[0049] Figure 12B is a diagram for describing a method for determining whether to apply boundary enhancement as provided in various embodiments;

[0050] Figure 12C is a diagram for describing a method for determining whether to apply boundary enhancement as provided in various embodiments;

[0051] Figure 12D is a diagram for describing a method for determining whether to apply boundary enhancement as provided in various embodiments;

[0052] Figure 13 is a view for describing a method for determining whether to apply boundary enhancement as provided in various embodiments; and

[0053] Figure 14 is a flowchart provided for describing a method of image processing according to an embodiment. DETAILED DESCRIPTION

[0054] The present disclosure will be further described with reference to the accompanying drawings.

[0055] Terms used in this specification will be briefly described, and the present disclosure will be described in more detail.

[0056] Taking into account the functions in the present disclosure, general terms that are currently widely used are selected as the terms used in the embodiments of the present disclosure, but they may be changed according to the intentions of those skilled in the art or judicial precedents, the emergence of new technologies, etc. In addition, in specific cases, there may be terms arbitrarily selected by the applicant. In this case, the meanings of these terms will be mentioned in detail in the corresponding description sections of the present disclosure. Therefore, the terms used in the embodiments of the present disclosure should be defined based on the meaning of the terms and the content throughout the present disclosure rather than the simple names of the terms.

[0057] Terms such as "first", "second", etc. may be used to describe various elements, but these elements should not be limited by these terms. These terms are only used to distinguish one element from another.

[0058] Unless otherwise specified, singular expressions include plural expressions. It should be understood that terms such as "including" or "consisting of..." are used herein to indicate the presence of a feature, number, step, operation, element, component, or combination thereof, but do not exclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, elements, components, or combinations thereof.

[0059] It should be understood that at least one of A or B means one of "A," "B," or "A and B."

[0060] Terms such as "module", "unit", "part", etc. are used to indicate an element that performs at least one function or operation, and such an element can be implemented as hardware or software, or a combination of hardware and software. In addition, in addition to requiring each of multiple "modules", "units", "parts", etc. to be implemented in a single piece of hardware, these components can be integrated into at least one module or chip and can be implemented in at least one processor (not shown). In some embodiments, at least one processor is configured to execute program code obtained from at least one memory.

[0061] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement these embodiments. However, the present disclosure can be embodied in many different forms and is not limited to the embodiments described herein. For the purpose of clearly illustrating the disclosure in the accompanying drawings, parts not relevant to the description are omitted, and the same reference numerals have been assigned to similar parts throughout the specification.

[0062] Figure 1 is a view for describing an example of implementation of the image processing apparatus according to the embodiment.

[0063] like Figure 1 As shown, the image processing apparatus 100 may be implemented as a TV or a set-top box, but is not limited thereto, and may be applied to any device having image processing and / or display functions, such as a smart phone, a tablet computer, a laptop computer, a head-mounted display (HMD), a near-eye display (NED), a large format display (LFD), a digital signage, a digital information display (DID), a video wall, a projector display, a camera, a camcorder, a printer, and the like.

[0064] The image processing apparatus 100 can receive various compressed images or images of various resolutions. For example, the image processing apparatus 100 can receive images in compressed formats such as Moving Picture Experts Group (MPEG) (e.g., MP2, MP4, MP7, etc.), Joint Photographic Coding Experts Group (JPEG), Advanced Video Coding (AVC), H.264, H.265, High Efficiency Video Codec (HEVC), etc. The image processing apparatus 100 can receive any of standard definition (SD) images, high definition (HD) images, full HD images, or ultra HD images.

[0065] According to an embodiment, even if the image processing device 100 is implemented as a UHD TV, UHD content is not sufficient. Therefore, it often happens that an SD, HD, full HD, or other image is input (hereinafter referred to as a low-resolution image). In this case, the input low-resolution image can be converted to a UHD image (hereinafter referred to as a high-resolution image) and used. However, the texture or edges of the image are blurred during the image conversion process, resulting in a problem of degraded image details. In some cases, the viewer may not be able to clearly see the boundaries. The unclearness may be caused by noise.

[0066] According to another embodiment, even when inputting high-resolution images, detail loss can occur during image compression and decompression processes. As the number of pixels increases, digital images require more data, and when compressing large amounts of data, compression-induced detail degradation is difficult to avoid. This detail degradation can include increased image distortion after decompression relative to the original image processed during the image compression phase.

[0067] Images contain various types of edges. For example, edges can be categorized as complex edges with various orientations and straight edges with clear orientations. Straight edges with clear orientations, which primarily appear at object boundaries or within text, are the first elements to be recognized by image viewers. Therefore, processing specifically designed to enhance or render straight edges is an important technique for improving the appearance or quality of image details.

[0068] Therefore, various embodiments for enhancing image details by accurately detecting straight edges with clear directions and applying shadowing effects will be described below.

[0069] Figure 2 is a block diagram showing the configuration of an image processing apparatus according to an embodiment.

[0070] Reference Figure 2 , the image processing apparatus 100 includes an input device 110 and a processor 120 .

[0071] The input device 110 receives various types of content. For example, the input device 110 can receive an image signal from an external device (e.g., a source device), an external storage medium (e.g., a universal serial bus (USB) memory), an external server (e.g., a network hard drive), etc., in a streaming or downloading manner via a communication method (such as Wi-Fi (wireless local area network (LAN)) based on an access point (AP), Bluetooth, Zigbee, wired / wireless LAN, wide area network (WAN), Ethernet, IEEE 1394, High-Definition Multimedia Interface (HDMI), USB, Mobile High-Definition Link (MHL), Audio Engineering Society / European Broadcasting Union (AES / EBU), optical, coaxial, etc.). Here, the image signal can be a digital image signal of any one of SD, HD, full HD, or ultra HD images, but is not limited thereto.

[0072] The processor 120 may control the overall operation of the image processing apparatus 100 .

[0073] According to an embodiment, the processor 120 may be implemented with a digital signal processor (DSP), a microprocessor, an artificial intelligence (AI) processor, and a time controller (TCON) for processing a digital image signal, but is not limited thereto. The processor 120 may include one or more of a central processing unit (CPU), a microcontroller unit (MCU), a microprocessing unit (MPU), a controller, an application processor (AP), a communication processor (CP), and an advanced reduced instruction set computing (RISC) machine (ARM) processor, or may be defined as a corresponding term. The processor 120 may be implemented in a system on chip (SoC) type or a large-scale integration (LSI) type with a built-in processing algorithm, or a field programmable gate array (FPGA) type.

[0074] In some embodiments, processor 120 includes one or more CPUs. In some embodiments, processor 120 includes one or more processors. In some embodiments, memory 130 includes one or more memories. In some embodiments, the one or more processors are configured to read instructions and data from the one or more memories. In some embodiments, the one or more memories are configured to store program code including executable instructions. In some embodiments, the program code is configured to cause the one or more processors to execute Figure 14 logic.

[0075] The processor 120 performs image processing on the input image and obtains an output image. Here, the input image may be an SD image, an HD image, a full HD image, or the like.

[0076] Specifically, the processor 120 can detect the boundary area in the input image, insert the shadow into the detected boundary area (or perform shadow processing on the detected boundary area), and obtain an output image. Here, the output image can be an ultra-high-definition (UHD) image, in particular a 4K (3840×2160) UHD image or an 8K (7680×4320) UHD image, but is not limited thereto. According to the example, various preprocessings can be performed on the input image before shadow processing. In the following, for ease of description, the input image and the preprocessed image are not distinguished from each other and are referred to as the input image. In addition, the shadow processing according to the embodiment can be performed on an image pre-stored in a memory (not shown), but for ease of description, it is limited to the input image.

[0077] For ease of description, the method for obtaining (or generating) shadows will be described first, and then the method for obtaining information about the boundaries included in the input image (i.e., boundary information) will be described. However, depending on the embodiment, generating shadows and obtaining boundary information can be performed in parallel, or either can be performed before the other.

[0078] According to an embodiment, the processor 120 may obtain an output image by generating a shadow based on an input image and applying the generated shadow to a boundary area.

[0079] <Shadow Generation>

[0080] According to an embodiment, the processor 120 may filter the input image and obtain a signal of a threshold range from the filtered signal as a shadow. Here, the filtered signal may be an output signal of a filter applied to the input image and may be referred to as filtered data or a filtered value.

[0081] Typically, if a first-order or second-order edge detection filter is applied to an input image, a filtered signal including edge amplitude and edge direction information (perpendicular to the gradient) can be obtained. Here, a first-order edge detection filter refers to a filter that detects edges based on a first-order differential signal, and a second-order edge detection filter refers to a filter that detects edges based on a second-order differential signal. In some embodiments, the first-order edge detection filter includes one or more positive filter coefficients, one or more negative filter coefficients, and presents a zero crossing point between the positive and negative filter coefficients. For example, see 6B. In some embodiments, the second-order edge detection filter presents two zero crossing points. For example, see Figure 6A An edge can refer to an area where the values ​​of spatially adjacent pixels change dramatically. For example, an edge can be an area where the brightness of an image changes rapidly from a low value to a high value or from a high value to a low value.

[0082] like Figure 3AAs shown, the first-order differential signal has a negative or positive value at the edge, while the second-order differential signal has a zero-crossing characteristic where the sign changes at the edge. Specifically, when the downward trend of the input signal becomes stronger or the upward trend of the input signal becomes weaker at the edge, the second-order differential signal has a negative value. Conversely, when the downward trend of the input signal becomes weaker or the upward trend becomes stronger, the second-order differential signal has a positive value. In other words, the second-order differential signal has a positive value at the inner boundary and a negative value at the outer boundary for dark objects, and has a positive value at the outer boundary and a negative value at the inner boundary for bright objects.

[0083] According to an embodiment, the processor 120 may generate a shadow based on the characteristics of the second-order differential signal. For example, the processor 120 may generate a shadow based on a positive signal or a negative signal (i.e., a signal having a positive (+) value or a negative (-) value) in the output signal of the second-order differential filter.

[0084] According to one embodiment, the processor 120 may apply a second-order differential filter to the input image and generate a shadow based on a signal having a negative (-) value at the filter's output. For example, the second-order differential filter may be implemented as a Laplacian filter. The left and right portions (or left, right, top, and bottom) of the Laplacian filter are symmetrical about the filter's center, with the coefficients summing to zero. This value decreases as the distance from the center increases, potentially changing the sign from positive to negative. In other words, the center coefficient may be a non-negative number including zero. As described above, the output of a Laplacian filter with a positive center coefficient has an opposite sign to that of the second-order differential signal. Therefore, when the processor 120 uses the second-order differential signal, the sign of the positive signal included in the second-order differential signal may be inverted to generate the shadow. In some cases, the processor 120 may adjust the negative signal in the output of the Laplacian filter to generate the shadow. Here, the adjustment may be a clipping operation, applying a weight to the negative signal, or subtracting / adding a signal of a predetermined size.

[0085] Figure 3B 1 is a diagram illustrating various examples of a Laplacian filter according to an embodiment. That is, the Laplacian filter may be in a two-dimensional form of n*n and may be applied to a two-dimensional pixel area. Figure 3B A Laplacian filter of 3*3 format is shown. In this case, the filter used for shadow generation can be implemented to be smaller than the size of the filter used for boundary detection described later, but is not limited thereto. In some embodiments, the size of the filter refers to the number n of coefficients along one dimension in the filter.

[0086] Applying a Laplacian filter to an input image can mean convolving the Laplacian filter with the input image. Convolution is an image processing technique that uses weighted filters, which means that the pixel values ​​of the input image are multiplied by the corresponding weights (or coefficients) included in the filter, and then the sum is obtained. Therefore, convolution usually refers to the sum of products. The multiplicand in each product is taken from the input signal and the filter coefficients. In this context, the filter is called a mask, window, or kernel. That is, the number of pixels included in the filter is the sum of the weights (or coefficients) included in the filter. Figure 3B The values ​​in the Laplacian filter shown in may be weights (values ​​indicating how much of the corresponding pixel to use).

[0087] Figure 3C is a view for describing a method for applying a Laplacian filter according to an embodiment.

[0088] like Figure 3C As shown, when P7 is the pixel to be processed, the Laplacian filter can be applied with P7 as the center pixel. In this case, the Laplacian filter can be obtained by P1*0+P2*(-1)+P3*0+P6*(-1)+P7*4+P8*(-1)+P 11 *0+P 12 *(-1)+P 13 *0 is calculated to obtain the convolution P corresponding to P7 7C The convolution values ​​corresponding to the remaining pixel values ​​are obtained by the same calculation. However, the pixels located at the corners of the image (e.g., P1 to P5, P6, P 11 、P 16 、P 21 etc.) lack surrounding pixels for applying the filter. In this case, the filter may be applied after making the image larger by adding pixel values ​​(e.g., pixel value 0) to the outer region except the original image, or after adding pixels having the same color as the outer boundary of the original image. Figure 3C In the example of , there are, for example, 16 input samples of an input image and 9 output values. The filter has 9 values, 5 of which are non-zero. In some examples, the output values ​​are example filter outputs for obtaining shadows or identifying boundary regions.

[0089] Figure 4 is a view for describing an output of a filter according to an embodiment.

[0090] like Figure 3B As shown, the Laplacian filter can be a two-dimensional (2D) filter. However, in Figure 4 In , for the convenience of description, a one-dimensional filter is applied to a one-dimensional signal.

[0091] Figure 4 is shown when the first filter {-1, 2, -1} is applied to Figure 3BThe output value of the filter is a one-dimensional signal in the filter. Figure 4 As shown in FIG, the outputs 412 and 422 of the filters in Case #0 and Case #1 have positive values ​​when the downward trend of the input signals 411 and 421 becomes stronger or the upward trend becomes weaker, and conversely, when the downward trend of the input signals 411 and 421 becomes weaker or the upward trend becomes stronger, the outputs 412 and 422 have negative values. That is, if a Laplace filter with a positive central coefficient is used, an output with a sign opposite to that of the second-order differential signal can be obtained. In this case, the processor 120 can use a signal with a negative value at the output of the Laplace filter with a positive central coefficient to obtain a shadow.

[0092] According to one embodiment, the processor 120 may utilize a clipping function that generates an output signal by limiting the input signal to a given range in order to generate a shadow. For example, the processor 120 applies a Laplacian filter f to the input image I and clips only the negative portion to generate a shadow S. This can be represented by the following equation 1.

[0093] Equation 1

[0094] S=min{0,f*I}

[0095] In Equation 1, in some embodiments, the "*" operator refers to convolution. The result of the convolution is then compared to 0 term by term, and the minimum value is taken. Therefore, there are no negative values ​​in S.

[0096] exist Figure 4 In case #0 and case #1, when shadows 413 and 423 obtained based on negative values ​​among the outputs of the Laplace filter are applied to the input signals 411 and 421, the following can be obtained: Figure 4 Output signals 414 and 424 are shown. For example, negative values ​​of shadows 413 and 423 are added to the pixel values ​​(or coefficient values) of the input signals 411 and 421, and accordingly, the pixel values ​​(or coefficient values) of the pixel regions to which shadows 413 and 423 are added in the output signals 414 and 424 may have values ​​smaller than the pixel values ​​(or coefficients) of the input signals 411 and 421. However, this is described to facilitate understanding, and according to one embodiment, the generated shadows may be applied to the input image based on boundary information, as described below.

[0097] At the same time, according to another embodiment, the processor 120 can use an artificial neural network based on deep learning (or deep artificial neural network), that is, a learning network model, to generate shadows. The learning network model can be designed to obtain the output data desired by the user through continuous convolution operations on the input image, and can be implemented as a system for learning a large number of images. For example, at least one of the coefficients (or parameters) or sizes of the filter included in the edge detection filter can be learned to output an appropriate shadow image. As an example, when an image is input, the learning network model can be learned to output an image (hereinafter referred to as a shadow image) including shadows whose amplitude is adjusted based on the characteristics of the input image. As another example, if an input image and a shadow image (for example, a shadow image generated based on a second-order differential signal) are input, the learning network model can be learned to output a shadow image including shadows whose amplitude is adjusted based on the characteristics of the input image. Here, the characteristics of the input image can include various image characteristics related to the input image, such as resolution, type, edge area distribution, texture area distribution, color distribution, etc.

[0098] For example, the learning network model can be implemented as at least one deep neural network (DNN) model among a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory network (LSTM), a gated recurrent unit (GRU), or a generative adversarial network (GAN).

[0099] <Boundary Detection>

[0100] In addition to shadow generation, the processor 120 can obtain information about the boundaries included in the input image, i.e., boundary information. In the present disclosure, a boundary refers to an edge with a clear and straight direction and / or an edge with a clear direction and a thickness greater than or equal to a threshold, and can therefore be distinguished from a complex edge with a variety of directions. For example, a boundary can be a boundary of an object, text, etc. Straight edges can be identified by recognizing when the same boundary detection filter has a high value for two adjacent pixel blocks (consistency between neighbors about direction). Complex edges can correspond to two adjacent pixel blocks corresponding to the maximum boundary detection filter outputs of different boundary detection filters (inconsistency between neighbors about direction). In some embodiments, a single longer filter is used to achieve averaging between adjacent pixel blocks. More description of the boundary detection filter is provided below.

[0101] Shading is only applied to edges with clear, straight directions. Detection identifies these straight boundaries. In high-resolution images, such as 8K images, shading of edges with clear, straight directions (i.e., boundaries) improves image quality. However, applying shading to even complex edges with varying directions can increase noise, degrading image quality. It is important to determine which edges are suitable for adding or inserting shading.

[0102] The boundary information may be information including at least one of position information (or coordinate information), amplitude information, or direction information of pixels detected as boundaries. According to one example, the boundary information may be implemented in the form of at least one of a graph or a table. For example, the boundary information may have a structure in which the amplitude information corresponding to each pixel block is arranged in a matrix form of pixel block units. As used herein, the term "pixel block" refers to a set of adjacent pixels including at least one pixel, and "region" refers to a term for a portion of an image and may represent at least one pixel block or a set of pixel blocks.

[0103] Specifically, the processor 120 can obtain boundary information by applying multiple boundary detection filters with different directions to the input image. Here, the boundary detection filter can be implemented by an n-order differential filter (e.g., a first-order differential filter or a second-order differential filter). For example, the processor 120 can use a Laplacian filter as a second-order differential filter for boundary detection. However, the embodiment is not limited to this, and for some cases, at least one of a Roberts filter, a Sobel filter, a directional filter, a gradient filter, a difference filter, or a Prewitte filter can be used. An example of a Prewitte filter or kernel is a first-order differential filter for detecting horizontal edges, and has 9 filter coefficients arranged in a 3*3 matrix. The first row of the matrix has elements [1 1 1]. The second row has elements [0 00]. The third row has elements [-1 -1 -1]. However, for boundary detection, it may be desirable to use the same type of filter as used in shadow generation. For example, if a second-order differential filter is used for shadow generation and a first-order differential filter is used for boundary detection, the position of the boundary detected by the first-order differential filter may be different from the edge position detected by the second-order differential filter used for shadow generation. However, in some cases where a shadow is not applied to the boundary detected by the first-order differential filter, the boundary can also be detected by using a first-order differential filter that is different from the second-order differential filter used for shadow generation.

[0104] Hereinafter, for convenience of description, a case where a Laplacian filter is used to detect a boundary will be described.

[0105] In a high-resolution image with a large number of pixels, the number of pixels constituting the edge to be detected also increases. Therefore, the size of the filter used for edge detection also increases. For example, when a compressed low-resolution input image is enlarged and output on a high-resolution display, the edges included in the input image are also enlarged. In order to detect the edges in this enlarged image, a corresponding large-sized filter must be used. Therefore, the Laplacian filter used for boundary detection can be larger than the Laplacian filter used for the above-mentioned shadow generation. For example, the size can be 5*5 or larger (or 7*7 or larger), but is not limited to this. According to one embodiment, in the case of shadow generation, if the shadow is generated in a small edge unit, a more natural shadow generation is available. Therefore, when generating a shadow, a filter having a size smaller than the size of the filter used in the case of boundary detection can be used to generate the shadow.

[0106] If the number of boundary detection filters is N according to the example, the nth boundary detection filter may have a direction of 180*(n-1) / N. Here, the plurality of boundary detection filters may be four or more. For example, the plurality of boundary detection filters may include a first filter (in which the same filter coefficients overlap (or repeat) in units of rows in a first direction), a second filter (in which the same filter coefficients overlap (or repeat) in units of rows in a second direction), a third filter (in which the same filter coefficients overlap (or repeat) in units of rows in a third direction), and a fourth filter (in which the same filter coefficients overlap (or repeat) in units of rows in a fourth direction). However, as the size of the filter increases, the performance of distinguishing between complex edges and straight edges may decrease. Therefore, in the present disclosure, a plurality of filters with different directions may be added in proportion to the size of the filter to accurately detect straight edges, i.e., boundaries with clear directions.

[0107] Figure 5 is a diagram for describing a plurality of boundary detection filters according to an embodiment.

[0108] exist Figure 5 In the embodiment, the number of the plurality of boundary detection filters is four, and the size of the plurality of boundary detection filters is a size of 5*5, but the embodiment is not limited thereto. In addition, it is assumed that the plurality of boundary detection filters are symmetrical two-dimensional filters having a Laplace filter form in the corresponding directions, and the coefficients {-19, 6, 26, 6, -29} are repeated in the corresponding directions (see Figure 5 , G[1], G[2], G[3] and G[4]), but the embodiment is not limited thereto.

[0109] In this case, if the nth boundary detection filter is implemented to have a direction of angle 180*(n-1) / N, the first boundary detection filter G(1) has a direction of 0 degrees, and the second boundary detection filter G(2) has a direction of 180*(2-1) / 4=45 degrees. Similarly, the third boundary detection filter G(3) and the fourth boundary detection filter G(4) have directions of 90 degrees and 135 degrees, respectively. As described above, when four boundary detection filters are used, it is possible to detect a boundary having a symbol For example, edges in directions of 0, 45, 90, 135, 180, 225, 270, and 315 degrees can be detected.

[0110] In this case, the second to fourth boundary detection filters can be obtained by rotating the first boundary detection filter by a corresponding angle. Therefore, as long as the first boundary detection filter is stored in a memory (not shown), the second to fourth boundary detection filters can be obtained. According to an embodiment, the first to fourth boundary detection filters can all be stored in a memory (not shown).

[0111] Figure 6A and Figure 6B is a view provided for describing a plurality of boundary detection filters according to another embodiment.

[0112] Figure 6A It is shown that the plurality of boundary detection filters are eight second-order differential filters. The plurality of boundary detection filters may be implemented in a format having Laplacian filters in corresponding directions, but the embodiment is not limited thereto.

[0113] In this case, if the nth boundary detection filter is implemented to have a direction angle of 180*(n-1) / N, the first boundary detection filter G(1) has a direction of 0 degrees, and the second boundary detection filter G(2) has a direction of 180*(2-1) / 8=22.5 degrees. Similarly, the third to eighth boundary detection filters G(3), G(4), G(5), G(6), G(7) and G(8) have directions of 45 degrees, 67.5 degrees, 90 degrees, 112.5 degrees, 135 degrees and 157.5 degrees, respectively. That is, with Figure 5 Compared to the embodiment shown, filters with directions of 22.5 degrees, 67.5 degrees, 112.5 degrees and 157.5 degrees can be added. As described above, when eight boundary detection filters are used, it is possible to detect edges with symbols For example, edges in directions of 0, 22.5, 45, 67.5, 90, 112.5, 135, 157.5, 180, 202.5, 225, 247.5, 270, 292.5, 315, and 337.5 degrees can be detected.

[0114] In this case, the second to eighth boundary detection filters can be obtained by rotating the first boundary detection filter by a corresponding angle. Therefore, as long as the first boundary detection filter is stored in a memory (not shown), the second to eighth boundary detection filters can be obtained. At the same time, according to an embodiment, the first to eighth boundary detection filters can be stored in a memory (not shown).

[0115] At the same time, the center coefficient of the filter coefficients may be different depending on the thickness of the boundary area to be detected. For example, a larger center coefficient can detect a thinner edge. Therefore, the processor 120 can apply a filter with a corresponding coefficient based on the thickness of the boundary to be detected.

[0116] Figure 6B An embodiment in which the plurality of boundary detection filters are eight first-order differential filters is shown.

[0117] and Figure 6A Similarly, each of the eight filters G(1) to G(8) has directions of 0 degrees, 22.5 degrees, 45 degrees, 67.5 degrees, 90 degrees, 112.5 degrees, 135 degrees, and 157.5 degrees. The eight filters can be arranged in the same manner as Figure 6A The same way to detect the symbol For example, edges in directions of 0, 22.5, 45, 67.5, 90, 112.5, 135, 157.5, 180, 202.5, 225, 247.5, 270, 292.5, 315, and 337.5 degrees can be detected.

[0118] For example, Figure 6A and Figure 6B The size of the boundary detection filter shown in Figure 5 The size of the boundary detection filter is shown in . In this case, with Figure 5 Compared with the boundary detection filter shown in , it can detect thicker edges. However, as the filter size increases, the performance of distinguishing complex edges from straight edges may deteriorate. Therefore, as Figure 6A and Figure 6B As shown, you can also use Figure 5 Multiple filters in different directions.

[0119] According to an embodiment, the processor 120 may apply filters of different sizes according to characteristics of the boundary area to be detected.

[0120] For example, in the case of applying shading only to edges having a thickness greater than or equal to a first threshold, the processor 120 may use a first filter of a size corresponding to the thickness.

[0121] As another example, in the case of applying shading only to edges having a thickness less than a first threshold and greater than or equal to a second threshold, a second filter having a size corresponding to the thickness may be used.

[0122] As another example, when the processor 120 is configured to apply the shadow only to edges whose thickness is less than a second threshold value, a third filter having a size corresponding to the thickness can be used. The thickness to be detected can be indicated, for example, in the program code, in the input data provided to the processor, or by a learning process. Here, the size of the second filter can be smaller than the size of the first filter, and the size of the third filter can be smaller than the size of the second filter. In this case, the first filter can be a low-pass filter that filters relatively low-band signals (e.g., thick edges) in the frequency domain, the second filter can be a band-pass filter that filters mid-band signals (e.g., edges of medium thickness), and the third filter can be a high-frequency filter that filters high-band signals (e.g., fine edges).

[0123] At the same time, the processor 120 can obtain a plurality of filter output values ​​by applying each of the plurality of boundary detection filters to the pixel block, and obtain boundary information of the pixel block based on at least one of the plurality of filter output values. The method of applying the boundary detection filter to the pixel block is similar to Figure 3B Here, the plurality of filter output values ​​may include at least one of amplitude, position information, or direction information of a boundary included in the image.

[0124] Because the directions of the multiple boundary detection filters are different, the multiple filter output values ​​are different for each pixel block. For example, if processor 120 uses a second-order differential filter, the first pixel block includes the minimum negative value of the output value of the first boundary detection filter. In the case of the second pixel block, the minimum negative value of the output value of the second boundary detection filter may be included. In other words, the minimum negative value of the output value of the boundary detection filter having the direction most similar to the direction of the edge included in the corresponding pixel block may be included.

[0125] According to one embodiment, when processor 130 uses the second order differential filter (for example, Laplace filter) with positive center coefficient to detect boundary, boundary information can be obtained based on the amplitude of the maximum negative value among a plurality of filter output values. This is because the output value of the second order differential filter has a zero-crossing characteristic in which the sign changes in the boundary area, and the shadow according to the embodiment is generated based on the negative value in the output value of the second order differential filter. The boundary detected according to it can overlap with at least some of the generated shadow area. The reason for using the minimum negative value in a plurality of filter output values ​​is that the amplitude of the output value of each second order differential filter can be different according to the direction of the boundary included in each pixel block. That is, the output value of the second order differential filter with a direction most similar to the direction of the edge included in the corresponding pixel block can be the minimum negative value.

[0126] The above-described process can be expressed using a max function as shown in Equation 2 below.

[0127] Equation 2

[0128] B=max{0,-(G[n]*I)}

[0129] Here, B0 may be the magnitude of the smallest negative value among the plurality of filter output values, G[n] is the nth filter, I is the input image, and G[n]*I may be the output value of the nth filter determined by convolution. In terms of numerical value, for example, -10 is smaller than -9.

[0130] According to an embodiment, the processor 120 may normalize the amplitude B0, which is the minimum negative value among the plurality of filter output values, and obtain boundary information. For example, the processor 120 may obtain a boundary map (B) by converting B0 based on the following equation 3.

[0131] Equation 3

[0132]

[0133] As shown in Equation 3, when the min function is applied to 1 and the value obtained by subtracting the threshold value Th from B0, a boundary map (B) normalized to a value between 0 and 1 can be obtained. The numbers included in the obtained boundary map (B) can be used as weights for shading. That is, a boundary map (B) can be obtained in which, when the amplitude is low because it is less than or equal to the threshold value, a weight of 0 is applied, and when the amplitude increases, the weight increases in proportion to the increased amplitude.

[0134] In this case, the threshold value and the slope value can be predefined fixed values, or variable values ​​that vary according to at least one of the image characteristics, image type, boundary amplitude, shadow type, or shadow amplitude. For example, the threshold value can vary according to the amplitude of the boundary to be detected. For example, when a boundary with a strong amplitude is to be detected, the threshold value can be set to a relatively small value, and when a boundary with a weak amplitude is to be detected, the threshold value can be set to a relatively large value. The slope value can adjust the degree to which the size B0 of the minimum negative value is proportional to the boundary information B. Therefore, when the weight is to be adjusted to be relatively large in proportion to the amplitude, the slope value can be set to a larger value, and when the weight value is to be adjusted to be relatively small in proportion to the amplitude, the slope value can be set to a smaller value.

[0135] According to another embodiment, when the processor 120 uses a first-order differential filter to detect the boundary, the boundary information can be obtained based on the maximum absolute value among the multiple filter output values. The output value of the first-order differential filter may not distinguish between the inside and outside of the object, such as Figure 3A Therefore, unlike the output value of the second-order differential filter, the boundary area where the shadow is generated may not be accurately detected, but in the case of inserting the shadow by detecting a wide boundary area, the boundary area can be detected using the first-order differential filter.

[0136] The following Equation 4 expresses as an equation the process of acquiring boundary information based on the maximum absolute value among the output values ​​of a plurality of first-order differential filters.

[0137] Equation 4

[0138] Mag=max{|G[i]|}

[0139] According to an embodiment, the processor 120 may obtain boundary information by normalizing the maximum absolute value among the plurality of filter output values. For example, by applying a threshold to the maximum absolute value max{|G[i]|} among the plurality of filter output values ​​(G[i]), boundary information (B) in the range of 0 to 1 may be obtained.

[0140] The following equation 5 expresses the process of obtaining the maximum absolute value from a plurality of filter output values ​​as an equation.

[0141] Equation 5

[0142] Mag=max{[G[i]|}

[0143] Equation 6 below represents a process of obtaining boundary information by normalizing an output value having a maximum absolute value among a plurality of filter output values.

[0144] Equation 6

[0145]

[0146] In Equation 6, B0 of Equation 3 is replaced by Mag, and thus will not be described further.

[0147] In the above embodiment, the pixel block boundary information is obtained based on the minimum negative value or the maximum absolute value among the multiple filter output values, but it is not limited to this. For example, in the case of a second-order differential filter (Laplacian filter), the average amplitude of the negative values ​​among the multiple filter output values ​​can be used, and in the case of a first-order differential filter, the pixel block boundary information can be obtained based on the average value of the multiple filter values. Alternatively, the boundary information can be obtained by multiplying the magnitude of the minimum negative value (or positive value) or the maximum absolute value by an arbitrary weight.

[0148] The processor 120 may apply the shadow to the input image based on the boundary information and obtain an output image. Here, the boundary information may include location information of the boundary area. That is, the processor 120 may insert the shadow only into the boundary area identified by the boundary information among the many edge areas that generate the shadow.

[0149] In addition, the boundary information may include amplitude information of the boundary area. In this case, the processor 120 may identify the boundary area based on the position information, apply the amplitude information corresponding to the identified boundary area to the shadow corresponding to the identified boundary area, and insert the shadow to which the amplitude information is applied into the identified boundary area.

[0150] In addition, the boundary information may include at least one of direction information or size information of the boundary area. In this case, the processor 120 may apply different weights to the magnitude of the boundary area based on at least one of the direction information or size information.

[0151] According to an embodiment, the processor 120 may obtain an output image by multiplying the obtained shadow S by boundary information (ie, boundary map B) as a weight, and then add the weighted shadow to the input image I. This process may be expressed as Equation 7 below.

[0152] Equation 7

[0153] O=I+B*S

[0154] According to another embodiment, the processor 120 may additionally adjust at least one of the shading or the boundary magnitude based on characteristics of the boundary area.

[0155] According to an embodiment, the processor 120 may apply additional weight to at least one of the shadow or boundary magnitude based on at least one of the magnitude or direction of the boundary region. In this case, the processor 120 may apply different weights based on at least one of the magnitude or direction of the boundary region.

[0156] In another example, the processor 120 may apply an additional weight to at least one of the shadow or the boundary amplitude based on at least one of the distance (or density) between the boundary region and another boundary region or the characteristics of the region adjacent to the boundary region. In this case, the processor 120 may apply different weights based on at least one of the distance (or density) between the boundary region and another boundary region or the characteristics of the region adjacent to the boundary region.

[0157] As another example, processor 120 may apply additional weights to at least one of shading or border magnitude based on at least one of the size or type of the border region. For example, processor 120 may apply different weights to the borders of objects such as text and buildings. Alternatively, processor 120 may apply different weights to borders of absolute (e.g., based on a threshold) or relatively large size, or borders of other sizes.

[0158] As described above, the processor 120 may detect a boundary area for applying a shadow by adjusting at least one of a size of a filter, a threshold value (Equations 3 and 6), or a slope value (Equations 3 and 6).

[0159] According to another embodiment, a deep learning-based artificial neural network (or deep artificial neural network) may be used, i.e., by using a learning network model, to perform boundary region detection. For example, the learning network model may learn at least one of coefficients or sizes of multiple boundary detection filters to detect boundary regions in an input image suitable for shadow insertion.

[0160] According to another embodiment, a learning network model can be used to perform both the above-mentioned shadow generation and boundary detection. For example, the learning network model can learn at least one of the coefficients, sizes or amplitudes of the edge detection filter used to generate the shadow, and learn at least one of the coefficients, sizes and amplitudes of multiple boundary detection filters to detect boundary areas suitable for shadow insertion. In this case, the learning network model may include a first learning network model for shadow generation and a second learning network model for obtaining boundary information. The processor 120 can obtain an output image by multiplying the output of the first learning network model with the output of the second learning network model. However, according to another embodiment, the above-mentioned shadow generation, boundary detection and shadow insertion can all be performed using a learning network model. That is, when the input image is input to the learning network model, an output image in which the shadow is inserted can be output from the learning network model. In some cases, the first learning network model and the second learning network model can perform learning by exchanging parameters.

[0161] Although only shadow insertion processing according to the embodiment is described in the above embodiment, other image processing such as noise removal processing, sharpening, texture processing, scaling, etc. may be performed in addition. According to one example, the processor 120 may perform noise removal processing before edge detection. Generally, noise is generated during the process of compressing and transmitting images, etc. The reason for performing noise removal processing is that noise not only degrades image quality but also reduces the effects of other image processing, such as the above-mentioned boundary detection. According to one embodiment, the processor 120 may perform noise removal by using non-local filtering and self-similar methods, a smoothing filter using low-pass filtering, etc.

[0162] At the same time, depending on the embodiment, the shadow insertion process can be performed before or after scaling the image. For example, the above image processing can be performed after scaling to enlarge the low-resolution image to a high-resolution image, or the above image processing can be performed after scaling in the process of decoding the compressed image.

[0163] 7A to 7D are views provided for describing an image processing method according to an embodiment.

[0164] When a second-order differential filter (e.g., a Laplace filter with a positive central coefficient) is applied to Figure 7A When the input image is Figure 7B The filter output.

[0165] In this case, the filter output has negative values ​​711 at the inner boundary of the dark object "circle" and positive values ​​712 at the outer boundary, as shown in FIG. Figure 7B In addition, the filter output is The outer boundary of the dark object has a negative value 721, and the inner boundary of the bright object has a positive value 722. In some embodiments, the object text can be in a language other than English. As shown in Figure 3, the second-order differential value of the Laplacian filter has a negative value at the inner boundary of the dark object and a positive value at the outer boundary, and has a negative value at the outer boundary of the bright object and a positive value at the inner boundary.

[0166] Then, if Figure 7C As shown, you can use only Figure 7B The negative values ​​711 and 721 in the filter output in the image are used to generate shadows. For example, the processor 120 may use a clipping function to identify an area with negative values, which limits the input signal to a given range and generates an output signal. In this case, the area with negative values ​​may be at least one of the inner boundary of a dark object or the outer boundary of a bright object. Figure 7C As shown, it can be confirmed that the inner boundary of the circle (dark object) and the text are generated. Shadows where the outer edges (of bright objects) become darker.

[0167] By adding the generated shadow ( Figure 7C ) is added to the input image, we can obtain Figure 7D Output image. Figure 7D ,Compared with the input image, the outline of the text becomes clear and the sharpness is enhanced.

[0168] Figure 8A and Figure 8B : are views provided for describing the effects of boundary detection according to the embodiment.

[0169] Figure 8A It is shown that by Figure 6A or Figure 6B ) is applied to the Laplace filter as Figure 7A The results of detecting boundaries using the input image shown in the figure. As shown in the figure, it can be seen that filters with different orientations successfully detect boundaries in the corresponding directions, but fail to detect boundaries in other directions. In addition, since a Laplacian filter with a positive center coefficient is used, it can be confirmed that the inner and outer boundaries of the object have different signs (positive and negative).

[0170] Figure 8B FIG. 4 shows a result of estimating the same form of boundary in all directions based on the minimum negative value or the maximum positive value among a plurality of boundary detection values ​​obtained in each pixel area according to an embodiment. Figure 8A The smallest negative value in the output of each filter shown in , can detect the inner boundary of dark objects (circular areas) and bright objects (text ), and when the maximum positive value is used, the outer boundaries of dark objects (circular areas) and bright objects (text ). Therefore, we can obtain Figure 8B The boundary detection results are shown.

[0171] Figure 8C Only shows Figure 8B The negative area (dark area) extracted from the boundary detection result is used to insert the shadow effect. Here, the extracted negative area can be a bright object (text ) or the inner boundary of a dark object (circular area).

[0172] Figure 9 is a view for describing an example of implementation of the image processing method according to the embodiment.

[0173] exist Figure 9 , it is shown that the Laplacian filter is used to generate shadows and detect boundaries, but it is not limited thereto.

[0174] Reference Figure 9 , the processor 120 applies a first Laplacian filter f for generating shadows and a second Laplacian filter F[i] for detecting boundaries of the input image I in steps S911 and S921, respectively. Here, the first Laplacian filter f may be implemented in a smaller size than the second Laplacian filter F[i]. In addition, the second Laplacian filter F[i] may include a plurality of filters having different orientations.

[0175] The processor 120 can generate the shadow S by clipping the negative portion of the output of the first Laplacian filter f (i.e., the second-order differential signal 912). This process can be expressed as the above equation 1. In addition, the processor 120 can estimate the boundary based on the output F(i)*I of the second Laplacian filter (F(i)) 922. Specifically, the processor 120 can estimate the boundary based on the size B0 of the smallest negative value among multiple filter output values ​​with different directions. This process can be expressed as the above equation 2.

[0176] The processor 120 can obtain boundary information (B) (e.g., a boundary map), i.e., weights, by applying a threshold value Th to B0, which is the smallest negative value among the plurality of filter output values ​​923. By this processor, a weight of 0 is applied to amplitudes less than or equal to the threshold, and the weight can be increased in proportion to the increasing amplitude. This process can be expressed as Equation 3 shown above.

[0177] Thereafter, the processor 120 can obtain an output image by multiplying the generated shadow S with the obtained boundary information (B). For example, the processor 120 can obtain an output image by multiplying each pixel data included in the shadow map with each pixel data of the corresponding position in the boundary map. In this case, the shadow S is not applied to the area where the boundary information (B) is 0, and the shadow S is applied to the area where the boundary information (B) is 0 or greater. Therefore, according to an embodiment, the shadow is applied to the boundary area.

[0178] exist Figure 9 In

[0045] , the boundary B can be a mask for selecting the shadow (shown as a multiplication operator). The insertion of the shadow is shown by the addition operation. These are illustrative, non-limiting examples of operators.

[0179] Figure 10 is a diagram showing an implementation example of an image processing apparatus according to another embodiment.

[0180] Reference Figure 10 The image processing apparatus 100 ′ includes an input device 110 , a processor 120 , a memory 130 , a display 140 , an output device 150 , and a user interface 160 . Figure 10 In the configuration, Figure 2 Overlapping configurations will not be described further.

[0181] The memory 130 is electrically connected to the processor 120 and can store data required for various embodiments. In this case, depending on the purpose of data use, the memory 130 can be implemented as a memory embedded in the image processing device 100', or can be implemented as a removable memory in the image processing device 100'. For example, data used to drive the image processing device 100' can be stored in the memory embedded in the image processing device 100', and data for additional functions of the image processing device 100' can be stored in the removable memory of the image processing device 100'. The memory embedded in the image processing device 100' can be a volatile memory (such as dynamic random access memory (DRAM), static random access memory (SRAM), synchronous dynamic random access memory (SDRAM)) or a non-volatile memory (such as a one-time programmable ROM (OTPROM), a programmable ROM (PROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a mask ROM, a flash ROM, a flash memory (such as a NAND flash memory or a NOR flash memory), a hard disk drive, or a solid-state drive (SSD)). In the case of a memory that is detachably mounted to the image processing apparatus 100', the memory may be implemented as a memory card (e.g., Compact Flash (CF), Secure Digital (SD), Micro Secure Digital (micro-SD), Mini Secure Digital (mini-SD), Extreme Digital (xD), Multi Media Card (MMC)), etc.), an external memory connectable to a USB port (e.g., a USB memory), etc.

[0182] According to an embodiment, the memory 130 may store images received from an external device (eg, source device), an external storage medium (eg, US), an external server (eg, network hard disk), etc. Here, the image may be a digital moving image, but is not limited thereto.

[0183] According to an embodiment, the memory 130 may be implemented as a single memory for storing data generated from various operations of the present disclosure.

[0184] According to another embodiment, the memory 130 may be implemented to include first to third memories.

[0185] The first memory may store at least a portion of the image input through the inputter 110. Specifically, the first memory may store at least a portion of an area of ​​the input image frame. In this case, according to an embodiment, at least a portion of the area may be an area required for performing image processing. According to one embodiment, the first memory may be implemented as an N-line memory. For example, the N-line memory may be a memory having a capacity equal to 17 lines in the vertical direction, but is not limited thereto. For example, when a full HD image of 1080p (resolution of 1,920×1,080) is input, an image area of ​​only 17 lines in the full HD image is stored in the first memory. The reason for implementing the first memory as an N-line memory and storing only a portion of the input image frame for image processing is that the memory capacity of the first memory is limited due to hardware limitations.

[0186] The second memory is a memory for storing shadow images, boundary information, etc., and can be implemented with various sizes according to various embodiments. For example, when storing shadow images generated according to the embodiment, the memory can be implemented with a size suitable for storing shadow images.

[0187] The third memory is a memory for storing an output image that has undergone image processing (e.g., shadow insertion processing), and can be implemented with memories of various sizes according to various embodiments. The third memory can be implemented with a size equal to or larger than the size of the input image. According to another embodiment, when an image is output in units of an image corresponding to the size of the first memory, or when an image is output in units of pixel rows, it can be implemented with an appropriate size for storing the image. However, if the output image is overwritten in the first memory or the second memory, or the output image is displayed without being stored, the third memory may not be required.

[0188] The display 140 can be implemented as a display including self-luminous elements or a display including non-self-luminous elements and a backlight. For example, the display 140 can be implemented as various types of displays, such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a light emitting diode (LED), a plasma display panel (PDP), a quantum dot light emitting diode (QLED), etc. In the display 160, a backlight unit, a driving circuit that can be implemented as an a-si TFT, a low-temperature polycrystalline silicon (LTPS) TFT, an organic TFT (OTFT), etc. can also be included. At the same time, the display 140 can be implemented as a touch screen coupled to a touch sensor, a flexible display, a rollable display, a three-dimensional (3D) display, a display with multiple display modules physically connected, etc. The processor 120 can control the display 140 to output an output image processed according to various embodiments.

[0189] The output unit 150 outputs a sound signal. For example, the output unit 150 can convert a digital sound signal processed by the processor 120 into an analog sound signal, amplify and output the analog sound signal. For example, the output unit 150 may include at least one speaker unit capable of outputting at least one channel, a D / A converter, an audio amplifier, etc. According to an example, the output unit 150 can be implemented to output various multi-channel sound signals. In this case, the processor 120 can control the output unit 150 to process the input sound signal according to the enhanced processing of the input image. For example, the processor 120 can convert the input two-channel sound signal into a virtual multi-channel (e.g., 5.1-channel) sound signal, identify the position of the image processing device 100' to process the signal into a cubic sound signal optimized to the space, or provide an optimized sound signal according to the type of the input image (e.g., content genre).

[0190] The user interface 160 may be implemented as a device such as a button, a touchpad, a mouse, and a keyboard, or as a touch screen or a remote control transceiver capable of performing the aforementioned display and operation input functions. The remote control transceiver may receive a remote control signal from an external remote controller, or transmit a remote control signal, using at least one communication method such as infrared communication, Bluetooth communication, or Wi-Fi communication. Depending on the embodiment, the shadow insertion process according to the embodiment may be selected through a user setting menu, and in this case, a user command for inserting a shadow may be received through the remote control transceiver.

[0191] According to an embodiment, the image processing device 100 may further include a tuner and a demodulator. The tuner (not shown) may receive a radio frequency (RF) broadcast signal by tuning a channel selected by a user or all pre-stored channels in the RF broadcast signal received via an antenna. The demodulator (not shown) may receive and demodulate the digital intermediate frequency (IF) signal converted by the tuner and perform channel decoding, etc. The input image received via the tuner according to one embodiment may be processed by the demodulator (not shown) and then provided to the processor 120 for shading processing according to one embodiment.

[0192] Figure 11 is a view for describing a shadow insertion effect according to an embodiment. Figure 11 A previous method of processing an input image using a previous method and a proposed method of processing an input image according to an embodiment are respectively shown when an image of a wall composed of bricks is captured.

[0193] Reference Figure 11In addition to thick edges between bricks, the input image also contains thin edges on the brick surfaces. Conventional techniques, by simultaneously amplifying frequency signals corresponding to both thick and thin edges, can increase noise as sharpness increases. Furthermore, according to embodiments, by shading thick edges, it can be confirmed that noise is amplified less and detail enhancement is superior compared to conventional techniques.

[0194] 12A to 12D is a view provided for describing a method for determining whether to apply boundary enhancement as provided in various embodiments.

[0195] Figure 12A An example of an input image is shown. Figure 12A As shown, the input image may be a pattern image that changes from a thick circular pattern to a fine circular pattern from the center toward the edge. For example, the central circle may be a thick edge with a thickness of several tens of pixels, and the outermost circle may be a thin edge with a thickness of one pixel.

[0196] Figure 12B The image shows detecting an edge having a thickness less than a first threshold value from edges included in an input image. For example, the processor 120 may use a Laplacian filter having a first size to detect an edge having a thickness less than the first threshold value.

[0197] Figure 12C An image is shown for detecting edges having a thickness less than a second threshold value among edges included in an input image. Here, the second threshold value may be a value greater than the first threshold value. For example, processor 120 may use a Laplacian filter having a second size smaller than the first size to detect edges having a thickness less than the second threshold value.

[0198] Figure 12D An image is shown for detecting edges included in an input image having a thickness greater than or equal to a first threshold and less than a second threshold. For example, the processor 120 may use a Laplacian filter having a third size greater than the second size and smaller than the first size to detect edges having a thickness greater than or equal to the first threshold and less than the second threshold.

[0199] According to another embodiment, when using a fixed-size filter (e.g., Figure 6A ) After edge detection, the threshold of Equation 3 can be adjusted to detect edges of different amplitudes.

[0200] according to 12B to 12D In the image shown in , it is confirmed that discontinuous areas (dark areas) are formed in eight directions in the central area of ​​the image. This means that eight filters with different directions are used to detect edges (e.g., Figure 6A ). That is, when Figure 12A The output image of the input image shown has the following 12B to 12D From the shape shown, it can be seen that one of the described embodiments is used.

[0201] Figure 13 is a view for describing a method for determining whether to apply boundary enhancement as provided in various embodiments.

[0202] Figure 13 An input image 1310, an output image 1320, and a difference image 1330 are sequentially shown. A difference image is an image that represents the difference between pixel values ​​obtained by comparing the input image and the output image. Referring to difference image 1330, obtained by comparing input image 1310 and output image 1320, negative values ​​are included in dark edge locations. This means that shadows are inserted at the boundaries of objects (e.g., the inner boundaries of dark objects), and it can be seen that one of the embodiments described above is used.

[0203] Figure 14 is a flowchart provided for describing a method of image processing according to an embodiment.

[0204] according to Figure 14 In the image processing method, in step S1410, when an input image is received, the input image is filtered and a signal within a threshold range in the filtered signal is obtained as a shadow.

[0205] In step S1420 , boundary information is obtained by applying a plurality of boundary detection filters having different directions to the input image.

[0206] Then, in step S1430, an output image is obtained by applying shading to the region identified by the boundary information in the input image. Here, the output image may be a 4K UHD image or an 8K UHD image.

[0207] In step S1410 , in order to obtain the shadow, a second-order differential signal is obtained by filtering the input image using a second-order differential filter, and the shadow can be obtained by limiting the positive signal or the negative signal from the second-order differential signal.

[0208] In this case, the obtained shadow can correspond to at least one of the inner boundary of an object included in the input image having a pixel value less than the first threshold or the outer boundary of an object included in the input image having a pixel value greater than or equal to a second threshold (which is greater than the first threshold).

[0209] In addition, the boundary information may include position information and amplitude information of the boundary area. In step S1430 of acquiring the output image, the boundary area may be identified based on the position information, the amplitude information corresponding to the identified boundary area may be applied to the shadow corresponding to the identified boundary area, and the shadow to which the amplitude information is applied may be inserted into the identified boundary area.

[0210] In the step S1410 of obtaining the shadow, the shadow may be obtained by applying a second-order differential filter to the input image. In this case, in step S1420, in order to obtain boundary information, the boundary information may be obtained by applying a plurality of boundary detection filters having a size greater than the second-order differential filter to the input image.

[0211] Furthermore, in step S1420 of obtaining boundary information, a plurality of filter output values ​​may be obtained by applying a plurality of boundary detection filters to a pixel block included in the input image, and boundary information about the pixel block may be obtained based on one of the plurality of filter output values.

[0212] In step S1420 of obtaining boundary information, boundary information about the pixel block may be obtained based on the magnitude of the minimum negative value or the maximum absolute value among the plurality of filter output values.

[0213] In addition, in step S1430 of obtaining the output image, normalization can be performed by applying a threshold to at least one of the magnitude of the minimum negative value or the maximum absolute value among the multiple filter output values, and the output image can be obtained by applying the weight obtained by normalization to the shadow.

[0214] In the step S1420 of obtaining boundary information, at least one of the number or size of the plurality of boundary detection filters may be determined based on characteristics of the region to which shading is to be applied.

[0215] When the number of the plurality of boundary detection filters is N, the direction of the nth boundary detection filter may be calculated by Equation 180*(n-1) / N.

[0216] According to the various embodiments described above, the boundaries of objects in high-resolution images can be accurately detected to apply shading, thereby enhancing the sharpness of the image by making the outline of the object clear while avoiding noise amplification within the image.

[0217] Various embodiments can be applied not only to image processing apparatuses but also to other electronic devices such as image receiving devices (such as set-top boxes) and display devices (such as TVs).

[0218] The methods according to various embodiments can be implemented as applications that can be installed in traditional image processing devices. Alternatively, an artificial intelligence neural network (or deep artificial neural network) based on deep learning, ie a learning network model, can be used to perform the methods according to various embodiments.

[0219] In addition, the methods according to various embodiments may be implemented only through software upgrade or hardware upgrade for a conventional image processing apparatus.

[0220] Various embodiments may be performed by an embedded server provided in the image processing apparatus or a server external to the image processing apparatus.

[0221] According to an embodiment, the various embodiments described above can be implemented with software including instructions stored in a machine-readable storage medium that is readable by a machine (e.g., a computer). The device may be a device that can call instructions from a storage medium and operate according to the called instructions, and may include an electronic device according to the disclosed embodiment (e.g., an image processing device (A)). When the processor executes the instructions, the processor may use other components to perform the functions corresponding to the instructions directly or under the control of the processor. The instructions may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" means that the storage medium does not include a signal and is tangible, but does not distinguish whether the data is stored semi-permanently or temporarily on the storage medium.

[0222] According to an embodiment, the methods according to various embodiments disclosed herein may be provided in a computer program product. The computer program product may be exchanged as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)) or through an application store (e.g., PlayStore). TM ) Online distribution. In the case of online distribution, at least part of the computer program product may be temporarily stored or at least temporarily stored in a storage medium, such as a memory of a manufacturer's server, an application store's server or a relay server.

[0223] Each element (e.g., module or program) according to various embodiments can be composed of a single entity or multiple entities, and some sub-elements of the above-mentioned sub-elements can be omitted. These elements can also be included in various embodiments. Alternatively or additionally, some elements (e.g., modules or programs) can be integrated into one entity to perform the same or similar functions performed by each corresponding element before integration. According to various embodiments, the operations performed by modules, programs or other elements can be performed sequentially in a parallel, repeated or heuristic manner, or at least some operations can be performed in different orders.

[0224] Although preferred embodiments have been shown and described, the present disclosure is not limited to the specific embodiments and it should be understood that the present disclosure is not limited to the specific embodiments described above, and those skilled in the art will understand that various changes in form and details may be made therein without departing from the spirit and scope defined by the appended claims and their equivalents.

Claims

1. An image processing device, comprising: one or more processors; as well as One or more memories storing program codes, wherein execution of the program codes by the one or more processors is configured to cause the image processing apparatus to: obtaining a filtered input image based on filtering the input image by using a second-order differential filter, and obtaining the shadow by using only a positive signal of the filtered input image or only a negative signal of the filtered input image; Obtaining boundary information by applying a plurality of boundary detection filters to the input image, wherein each boundary detection filter of the plurality of boundary detection filters is associated with a different direction; and An output image is obtained by applying the shading to the regions identified based on the boundary information.

2. The image processing apparatus according to claim 1, wherein: The one or more processors are further configured to: The positive signal or the negative signal is obtained by clipping the filtered input image.

3. The image processing apparatus according to claim 1, wherein: The shading corresponds to at least one of the following: an inner boundary of a first object in the input image, wherein the inner boundary comprises a first pixel having a first pixel value less than a first threshold, or An outer boundary of a second object in the input image, wherein the outer boundary includes second pixels having a second pixel value greater than or equal to a second threshold, wherein the second threshold is greater than the first threshold.

4. The image processing device according to claim 1, in, The boundary information includes position information and amplitude information, wherein the one or more processors are further configured to identify a boundary area based on the location information, and The one or more processors are further configured to obtain the output image by performing the following operations: applying amplitude information corresponding to the boundary region to the shadow, and A shadow to which the amplitude information is applied is inserted into the boundary area.

5. The image processing device according to claim 1, in, A first size of each of the plurality of boundary detection filters is larger than a second size of the second-order differential filter.

6. The image processing device according to claim 1, in, The input image includes a first pixel block, and The one or more processors are further configured to obtain the boundary information by performing the following operations: obtaining a first plurality of filter output values ​​by applying the plurality of boundary detection filters to the first pixel block, and First boundary information associated with the first pixel block is obtained based on a first value of the first plurality of filter output values.

7. The image processing device according to claim 6, in, The one or more processors are further configured to obtain boundary information associated with the first pixel block based on: the magnitude of the smallest negative value of the first plurality of filter output values, or The maximum absolute value of the first plurality of filter output values.

8. The image processing device according to claim 7, in, The one or more processors are further configured to: The weights are obtained by performing normalization by applying a threshold to either i) the magnitude of the smallest negative value or ii) the largest absolute value, and The output image is obtained by applying the weights to the obtained shading.

9. The image processing device according to claim 1, in, The one or more processors are further configured to determine at least one of a number or a size of the plurality of boundary detection filters based on characteristics of the boundary region.

10. The image processing device according to claim 1, in, The plurality of boundary detection filters include: a first filter in which the same filter coefficients are repeated in units of rows in a first direction; a second filter in which the same filter coefficients are repeated in rows in a second direction; a third filter in which the same filter coefficients are repeated in units of rows in a third direction; and A fourth filter in which the same filter coefficients are repeated in units of rows in a fourth direction.

11. The image processing device according to claim 1, in, Based on the number of the plurality of boundary detection filters being N, the direction of the nth boundary detection filter is calculated by the following equation: The angle of the nth edge detection filter = 180 * (n-1) / N.

12. The image processing apparatus according to claim 1, further comprising: monitor, The one or more processors are further configured to control a display to output the output image, and The output image is a 4K ultra high definition (UHD) image or an 8K UHD image.

13. An image processing method for an image processing device, the image processing method comprising: obtaining a filtered input image based on filtering the input image by using a second-order differential filter, and obtaining the shadow by using only a positive signal of the filtered input image or only a negative signal of the filtered input image; obtaining boundary information by applying a plurality of boundary detection filters to the input image, wherein each boundary detection filter of the plurality of boundary detection filters is associated with a different direction; as well as An output image is obtained by applying the shading to the region identified based on the boundary information.

14. The image processing method according to claim 13, The filtered input image is clipped to obtain the positive signal or the negative signal.

15. The image processing method according to claim 13, wherein: The shading corresponds to at least one of the following: an inner boundary of a first object in the input image, wherein the inner boundary comprises a first pixel having a first pixel value less than a first threshold, or An outer boundary of a second object in the input image, wherein the outer boundary includes second pixels having a second pixel value greater than or equal to a second threshold, wherein the second threshold is greater than the first threshold.

16. A non-transitory computer-readable recording medium storing computer instructions that, when one or more processors of an image processing apparatus execute the computer instructions, enable the image processing apparatus to perform operations comprising: obtaining a filtered input image based on filtering the input image by using a second-order differential filter, and obtaining the shadow by using only a positive signal of the filtered input image or only a negative signal of the filtered input image; obtaining boundary information by applying a plurality of boundary detection filters to the input image, wherein each boundary detection filter of the plurality of boundary detection filters is associated with a different direction; as well as An output image is obtained by applying the shading to the region identified based on the boundary information.

Images