Image processing apparatus and image processing method thereof

Abstract

The image processing device applies the image to a first learning network model to optimize the edge of the image, applies the image to a second learning network model to optimize the texture of the image, and based on information about the edge area and texture area of the image, applies a first weight to the first image and applies a second weight to the second image to obtain an output image.

Description

Technical Field

[0001] The present disclosure relates to an image processing apparatus and an image processing method thereof, and more particularly, to an image processing apparatus and an image processing method thereof for enhancing image features by using a learning network model. Background Art

[0002] Driven by the development of electronic technology, various types of electronic devices have been developed and released. In particular, image processing devices have been deployed in various places such as homes, offices and public places, and have been continuously developing in recent years.

[0003] High-resolution display panels, such as 4K UHD TVs, have recently become available and widely distributed. However, the availability of high-resolution content for reproduction on these high-resolution display panels is somewhat limited. Consequently, various technologies are being developed to generate high-resolution content from lower-resolution content. In particular, there is a growing demand for efficient processing of the numerous operations required to generate high-resolution content within limited processing resources.

[0004] Furthermore, artificial intelligence systems that mimic human-level intelligence have recently been used in various fields. Unlike conventional rule-based intelligent systems, artificial intelligence systems are systems that learn, determine, and execute processes independently. As the system operates iteratively, the artificial intelligence system shows further improved recognition rates and, for example, becomes able to more accurately understand user preferences. As a result, conventional rule-based intelligent systems are increasingly being replaced by artificial intelligence systems based on deep learning.

[0005] Artificial intelligence technologies consist of machine learning (e.g., deep learning) and elemental technologies that leverage machine learning.

[0006] Machine learning refers to algorithmic technology that independently classifies and learns the characteristics of input data. Meanwhile, elemental technology refers to technology that simulates human brain functions such as cognition and determination by using machine learning algorithms such as deep learning, and includes technical fields such as language understanding, visual understanding, reasoning / prediction, knowledge representation, and operational control.

[0007] Attempts have been made to enhance image features using artificial intelligence techniques in conventional image processing devices. However, the performance of conventional image processing devices is limited by the amount of processing required to generate high-resolution images, and this process takes a significant amount of time. Therefore, there is a need for a technology that enables image processing devices to generate and provide high-resolution images by performing only a small number of operations. Summary of the Invention

[0008] The present disclosure addresses the above needs and provides an image processing apparatus and an image processing method thereof for obtaining high resolution with improved image features by using multiple learning network models.

[0009] An image processing apparatus according to an embodiment of the present disclosure for achieving the above-mentioned purpose includes: a memory storing computer-readable instructions; and a processor configured to execute the computer-readable instructions so as to: apply an input image as a first input to a first learning network model, obtain a first image including enhanced edges optimized based on the edges of the input image from the first learning network model, apply the input image as a second input to a second learning network model, and obtain a second image including enhanced texture optimized based on the texture of the input image from the second learning network model. The processor identifies edge regions and texture regions included in the image, applies a first weight to the first image and a second weight to the second image based on information about the edge region and the texture region, and obtains an output image optimized from the input image based on the first weight applied to the first image and the second weight applied to the second image.

[0010] Furthermore, the first type of the first learned network model is different from the second type of the second learned network model.

[0011] In addition, the first learning network model can be one of the following models: a deep learning model for optimizing the edges of the input image by using multiple layers, or a machine learning model trained to optimize the edges of the input image by using multiple pre-learned filters.

[0012] In addition, the second learning network model can be one of the following models: a deep learning model for optimizing the texture of the input image by using multiple layers, or a machine learning model for optimizing the texture of the input image by using multiple pre-learned filters.

[0013] At the same time, the processor may obtain a first weight corresponding to the edge area and a second weight corresponding to the texture area based on ratio information of the edge area and the texture area.

[0014] In addition, the processor may downscale the input image to obtain a downscaled image having a resolution less than that of the input image. In addition, the first learning network model may obtain a first image with enhanced edges from the first learning network model that upscales the downscaled image, and the second learning network model may obtain a second image with enhanced texture from the second learning network model that upscales the downscaled image.

[0015] In addition, the processor can obtain region detection information in which edge regions and texture regions have been identified based on the downsampled image, and provide the region detection information and the image to the first and second learning network models, respectively.

[0016] In addition, the first learning network model may acquire the first image by upscaling the edge region, and the second learning network model may acquire the second image by upscaling the texture region.

[0017] In addition, the first image and the second image may be a first residual image and a second residual image, respectively. Furthermore, the processor may apply a first weight to the first residual image based on an edge region and a second weight to the second residual image based on a texture region, and then mix the first residual image, the second residual image, and the input image to obtain an output image.

[0018] At the same time, the second learning network model can be a model that stores multiple filters corresponding to each of multiple image modes, classifies each image block included in the image into one of the multiple image modes, and applies at least one filter among the multiple filters corresponding to the classified image mode to the image block, and provides a second image.

[0019] Here, the processor may accumulate index information of the image pattern corresponding to each classified image block, identify the image as one of a natural image or a graphic image based on the index information, and adjust the first weight and the second weight based on the result of identifying the input image as one of a natural image or a graphic image.

[0020] Here, the processor may increase at least one of the first weight or the second weight based on the input image being recognized as a natural image, and decrease at least one of the first weight or the second weight based on the input image being recognized as a graphic image.

[0021] At the same time, the image processing method of the image processing device according to an embodiment of the present disclosure may include the following steps: applying the input image as a first input to a first learning network model, obtaining a first image including enhanced edges optimized based on the edges of the input image from the first learning network model, applying the input image as a second input to a second learning network model, obtaining a second image including enhanced texture optimized based on the texture of the input image from the second learning network model, identifying the edge area of the edge included in the input image, identifying the texture area of the texture included in the input image, applying a first weight to the first image based on the edge area, applying a second weight to the second image based on the texture area, and obtaining an output image optimized from the input image based on the first weight applied to the first image and the second weight applied to the second image.

[0022] Here, the first learning network model and the second learning network model may be different types of learning network models from each other.

[0023] In addition, the first learning network model can be one of the following models: a deep learning model for optimizing the edges of the input image by using multiple layers, or a machine learning model trained to optimize the edges of the input image by using multiple pre-learned filters.

[0024] In addition, the second learning network model can be one of the following models: a deep learning model for optimizing the texture of the input image by using multiple layers, or a machine learning model trained to optimize the texture of the input image by using multiple pre-learned filters.

[0025] Furthermore, the image processing method may include the following steps: acquiring the first weight and the second weight based on ratio information of an edge region in the input image and a texture region in the input image.

[0026] Furthermore, the image processing method may include the following steps: downscaling the input image to obtain a downscaled image having a resolution smaller than that of the input image. Simultaneously, the first learning network model may upscale the downscaled image to obtain the first image, and the second learning network model may upscale the downscaled image to obtain the second image.

[0027] Here, the image processing method may include the following steps: obtaining first region detection information for identifying an edge region of an input image and second region detection information for identifying a texture region of the input image, and providing the region detection information and the image to the first and second learning network models, respectively.

[0028] Meanwhile, the first learning network model may acquire the first image by upscaling the edge region, and the second learning network model may acquire the second image by upscaling the texture region.

[0029] In addition, the first image and the second image may be a first residual image and a second residual image, respectively.

[0030] According to the various embodiments of the present disclosure described above, high-resolution images are generated by applying learning network models that are different from each other to images, thereby reducing the amount of operations required to generate high-resolution images, thereby enabling high-resolution images to be generated within the limited resources of an image processing device, and enabling the images to be provided to users. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a diagram illustrating an implementation example of an image processing apparatus according to an embodiment of the present disclosure;

[0032] Figure 2is a block diagram showing a configuration of an image processing apparatus according to an embodiment of the present disclosure;

[0033] Figure 3 is a diagram illustrating first and second learning network models according to an embodiment of the present disclosure;

[0034] Figure 4 is a diagram illustrating frequency reduction according to an embodiment of the present disclosure;

[0035] Figure 5 is a diagram illustrating a deep learning model and a machine learning model according to an embodiment of the present disclosure;

[0036] Figure 6 is a diagram illustrating first and second learning network models according to another embodiment of the present disclosure;

[0037] Figure 7 is a diagram illustrating first and second learning network models according to another embodiment of the present disclosure;

[0038] Figure 8 is a diagram schematically illustrating the operation of a second learning network model according to an embodiment of the present disclosure;

[0039] Figure 9 is a diagram illustrating index information according to an embodiment of the present disclosure;

[0040] Figure 10 is a diagram illustrating a method of acquiring a final output image according to an embodiment of the present disclosure;

[0041] Figure 11 It shows Figure 2 A block diagram of a detailed configuration of an image processing apparatus shown in ;

[0042] Figure 12 is a block diagram illustrating a configuration of an image processing apparatus for learning and using a learning network model according to an embodiment of the present disclosure; and

[0043] Figure 13 is a flowchart illustrating an image processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0044] Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

[0045] As the term used in the embodiments of the present disclosure, taking into account the functions described in the present disclosure, conventionally widely used general terms are selected as much as possible. However, these terms can change according to the intention of those skilled in the relevant art or the emergence of new technologies. In addition, in certain cases, a term can be specified, and in this case, the meaning of the term will be described in detail in the relevant description of the present disclosure. Therefore, the term used in the present disclosure should be defined based on the meaning of the term and the overall content of the present disclosure rather than just based on the name of the term.

[0046] In this specification, expressions such as “having,” “may have,” “including,” and “may include” should be interpreted as indicating the presence of such features (e.g., elements such as values, functions, operations, and components), and these terms are not intended to exclude the existence of additional features.

[0047] In addition, the expression “at least one of A and / or B” should be understood to mean any one of “A” or “B” or “A and B”.

[0048] In addition, the expressions "first", "second", etc. used in this specification can be used to describe various elements, regardless of any order and / or degree of importance. In addition, such expressions are only used to distinguish one element from another element and are not intended to limit these elements.

[0049] At the same time, the description of an element (e.g., a first element) in the present disclosure being "(operationally or communicatively) coupled to" or "connected to" another element (e.g., a second element) should be understood to mean that the one element can be directly coupled to the other element, or the one element is coupled to the other element through another element (e.g., a third element).

[0050] Unless the context clearly defines otherwise, singular expressions include plural expressions. In addition, in the present disclosure, terms such as "including" and "having" should be interpreted as indicating the presence of such features, numbers, steps, operations, elements, components, or combinations thereof described in the specification, but do not exclude the presence or possibility of adding one or more other features, numbers, steps, operations, elements, components, or combinations thereof in advance.

[0051] In addition, in the present disclosure, a "module" or "unit" can perform at least one function or operation and can be implemented as hardware or software, or as a combination of hardware and software. In addition, in addition to "modules" or "units" that need to be implemented as specific hardware, multiple "modules" or multiple "units" can be integrated into at least one module and implemented as at least one processor.

[0052] Furthermore, in this specification, the term "user" may refer to a person or a device (eg, an artificial intelligence electronic device) that operates an electronic device.

[0053] Hereinafter, embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings.

[0054] Figure 1 is a diagram illustrating an implementation example of an image processing apparatus according to an embodiment of the present disclosure.

[0055] The image processing apparatus 100 may be implemented as follows Figure 1 However, the image processing apparatus 100 is not limited thereto, and the image processing apparatus 100 may be implemented as any device equipped with an image processing function and / or a display function, such as a smartphone, a tablet PC, a laptop PC, a head-mounted display (HMD), a near-eye display (NED), a large-format display (LFD), a digital signage, a digital information display (DID), a video wall, a projector display, a camera, a camcorder, a printer, etc., without limitation.

[0056] The image processing apparatus 100 can receive images of various resolutions or various compressed images. For example, the image processing apparatus 100 can receive images 10 formatted according to any one of standard definition (SD), high definition (HD), full HD, and ultra HD images. In addition, the image processing apparatus 100 can receive images 10 encoded in a format or compressed in a format such as MPEG (e.g., MP2, MP4, MP7, etc.), AVC, H.264, HEVC, etc.

[0057] Even if the image processing device 100 is implemented as a UHD TV according to an embodiment of the present disclosure, due to the limited availability of UHD content, there are many cases where only images such as standard definition (SD), high definition (HD), and full HD images (hereinafter referred to as low-resolution images) are available. In such cases, a method can be provided for upscaling a low-resolution input image to a UHD image (hereinafter referred to as a high-resolution image) and providing the resulting image. As an example, a low-resolution image can be applied as input to a learning network model so that the low-resolution image can be enlarged, and thereby a high-resolution image can be obtained as output for display on the image processing device 100.

[0058] However, in order to upscale a low-resolution image to a high-resolution image, a large number of complex processing operations are generally required to transform the image data. Therefore, an image processing device 100 with high performance and high complexity is required to perform such a transformation. As an example, in order to upscale a 60P image of SD level with a resolution of 820×480 to a high-resolution image, the image processing device 100 should perform operations on 820×480×60 pixels per second. Therefore, a processing unit with high performance is required, such as a central processing unit (CPU) or a graphics processing unit (GPU) or a combination thereof. As another example, the image processing device 100 should perform operations on 3840×2160×60 pixels per second to upscale a 60P image of UHD level with a resolution of 4K to an image with a resolution of 8K. Therefore, a processing unit is required that can process at least 24 times as many operations as the case of upscaling an SD-level image.

[0059] Therefore, hereinafter, various embodiments will be described that provide an image processing apparatus 100 that reduces the amount of operations required to upscale a lower resolution image to a higher resolution image, thereby maximizing limited resources of the image processing apparatus 100 .

[0060] In addition, various embodiments will be described in which the image processing apparatus 100 acquires an output image while emphasizing or enhancing at least one image feature among various features of an input image.

[0061] Figure 2 is a block diagram illustrating a configuration of an image processing apparatus according to an embodiment of the present disclosure.

[0062] according to Figure 2 , the image processing apparatus 100 includes a memory 110 and a processor 120 .

[0063] The memory 110 is electrically connected to the processor 120 and can store data required for executing various embodiments of the present disclosure. For example, the memory 110 can be implemented as an internal memory such as ROM (e.g., electrically erasable programmable read-only memory (EEPROM)), RAM, etc. included in the processor 120, or as a memory separate from the processor 120.

[0064] Depending on the use of the stored data, the memory 110 may be implemented as a memory embedded in the image processing apparatus 100, or may be implemented as a memory attached to or detached from the image processing apparatus 100. For example, in the case where data is used to operate the image processing apparatus 100, the data may be stored in a memory embedded in the image processing apparatus 100, and in the case where data is used for extended functions of the image processing apparatus 100, the data may be stored in a memory that can be attached to or detached from the image processing apparatus 100. In the case of being implemented as a memory embedded in the image processing device 100, the memory 110 can be at least one of the following items: a volatile memory (for example: dynamic RAM (DRAM), static RAM (SRAM) or synchronous dynamic RAM (SDRAM)), etc.) or a non-volatile memory (for example: one-time programmable ROM (OTPROM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (for example: NAND flash or NOR flash, etc.), hard disk drive or solid state drive (SSD)).

[0065] Meanwhile, in the case of being implemented as a memory that can be attached to or detached from the image processing apparatus 100, the memory 110 can be a memory card (e.g., Compact Flash (CF), Secure Digital (SD), Micro Secure Digital (Micro-SD), Mini Secure Digital (Mini-SD), Extreme Digital (xD), MultiMedia Card (MMC), etc.), a memory of an external memory that can be connected to a USB port (e.g., a USB memory), etc.

[0066] According to an embodiment of the present disclosure, the memory 110 may store at least one program for causing the processor 120 to execute instructions. Here, the instructions may be instructions for the processor 120 to obtain an output image by applying the image 10 to the learning network.

[0067] According to another embodiment of the present disclosure, the memory 110 may store a learning network model according to various embodiments of the present disclosure.

[0068] The learning network model according to an embodiment of the present disclosure is a determination model trained based on multiple images based on an artificial intelligence algorithm, and the learning network can be a model based on a neural network. The trained determination model can be designed to simulate human intelligence and decision-making on a computer, and can include multiple network nodes with weights that simulate neurons of a human neural network. Each of the multiple network nodes can form a connection relationship to simulate the synaptic activity of neurons that send and receive signals through synapses. In addition, the trained determination model can include, for example, a machine learning model, a neural network model, or a deep learning model developed from a neural network model. In a deep learning model, multiple network nodes can be located at different depths (or layers) from each other, and data can be sent and received according to a convolutional connection relationship.

[0069] As an example, the learning network model can be a convolutional neural network (CNN) model trained based on an image. CNN is a multi-layer neural network with a special connection structure designed for speech processing, image processing, etc. At the same time, the learning network model is not limited to CNN. For example, the learning network model can be implemented as at least one deep neural network (DNN) model among a recursive neural network (RNN) model, a long short-term memory network (LSTM) model, a gated recurrent unit (GRU) model, or a generative adversarial network (GAN) model.

[0070] For example, the learning network model can restore or convert a low-resolution image into a high-resolution image based on a super-resolution GAN (SRGAN). At the same time, the memory 110 according to an embodiment of the present disclosure can store multiple learning network models of the same type or different types. The number and type of learning network models are not limited. However, according to another embodiment of the present disclosure, at least one learning network model according to various embodiments of the present disclosure can be stored in at least one of an external device or an external server.

[0071] The processor 120 is electrically connected to the memory 110 and controls the overall operation of the image processing apparatus 100 .

[0072] According to an embodiment of the present disclosure, the processor 120 may be implemented as a digital signal processor (DSP), a microprocessor, an artificial intelligence (AI) processor, and a timing controller (T-CON) for processing digital signals. However, the processor 120 is not limited thereto, and the processor 120 may include one or more of a central processing unit (CPU), a microcontroller unit (MCU), a microprocessing unit (MPU), a controller, an application processor (AP) or a communication processor (CP), and an ARM processor, or may be defined by terms. In addition, the processor 120 may be implemented as a system on chip (SoC) or a large-scale integration (LSI) having a processing algorithm stored therein, or in the form of a field programmable gate array (FPGA).

[0073] Processor 120 may apply image 10 as input to a learning network model and obtain an image with improved, enhanced, optimized, or enhanced image features. Here, a feature of image 10 may refer to at least one of edge direction, edge strength, texture, grayscale value, brightness, contrast, or gamma value based on multiple pixels included in image 10. For example, processor 120 may apply the image to the learning network model and obtain an image with enhanced edges and texture. Here, an edge in an image may refer to an area where the values of spatially adjacent pixels change dramatically. For example, an edge may be an area where the brightness of an image changes dramatically from a low value to a high value or from a high value to a low value. The texture of an image may be a unique pattern or shape of areas in the image that are considered to have the same feature. The texture of an image may also be composed of fine edges, and thus processor 120 may obtain an image in which edge components equal to or greater than a first threshold strength (or threshold thickness) and edge components less than a second threshold strength (or threshold thickness) have been enhanced. Here, according to embodiments of the present disclosure, the first threshold strength may be a value used to demarcate edge components, and the second threshold strength may be a value used to demarcate texture components. Both may be predetermined values or values set based on image features. However, hereinafter, for convenience of explanation, the features described above are referred to as edges and textures.

[0074] Meanwhile, the image processing apparatus 100 according to an embodiment of the present disclosure may include a plurality of learning network models. Each of the plurality of learning network models may enhance different features of the image 10. Figure 3 This aspect will be described in detail.

[0075] Figure 3 is a diagram for illustrating first and second learning network models according to an embodiment of the present disclosure.

[0076] refer to Figure 3The processor 120 according to an embodiment of the present disclosure may apply the image 10 as input to the first learning network model and obtain as output a first image in which the edges of the image 10 are improved at operation S310. The image 10 may also be provided as input to the second learning network model and obtain as output a second image in which the texture of the image 10 is improved at operation S320.

[0077] At the same time, the image processing device 100 according to an embodiment of the present disclosure can use a first learning network model and a second learning network model based on different artificial intelligence algorithms in parallel. Alternatively, the image 10 can be processed serially by the first learning network model and the second learning network model. Here, the first learning network model can be a model trained by using larger resources other than those used to train the second learning network model. Here, the resources can be various data necessary for training and / or processing the learning network model, and can include, for example, whether real-time learning is performed, the amount of learning data, the number of convolutional layers included in the learning network model, the number of parameters, the capacity of the memory used in the learning network model, the extent to which the learning network uses a GPU, etc.

[0078] For example, the GPU provided in the image processing device 100 may include a texture unit, a special function unit (SFU), an arithmetic logic unit, and the like. Here, the texture unit is a resource for adding materials or textures to the image 10, and the special function unit is a resource for processing complex operations (such as square roots, reciprocals, and algebraic functions). Meanwhile, the integer arithmetic logic unit (ALU) is a resource for processing floating-point and integer operations, comparisons, and data movement. The geometry unit is a resource for calculating the position or viewpoint of an object, the direction of a light source, and the like. The grid unit is a resource for projecting three-dimensional data on a two-dimensional screen. In this case, the deep learning model can use more of the various resources included in the GPU for learning and operation than the machine learning model. Meanwhile, the resources of the image processing device 100 are not limited to the resources of the GPU, and the resources may be resources of various components included in the image processing device 100, such as the storage area of the memory 110, power, and the like.

[0079] The first learning network model and the second learning network model according to an embodiment of the present disclosure may be different types of learning network models.

[0080] As an example, the first learning network model may be one of the following models: a deep learning-based model that learns based on multiple images to improve the edges of the image 10, or a machine learning model that is trained to improve the edges of the image by using multiple pre-learned filters. The second learning network model may be a deep learning model that learns by using multiple layers to improve the texture of the image, or a machine learning-based model that is trained to improve the texture of the image by using multiple pre-learned filters based on multiple images and a pre-learned database (DB). Here, the pre-learned DB may be a plurality of filters corresponding to each of a plurality of image patterns, and the second learning network model may identify an image pattern corresponding to an image block included in the image 10, and optimize the texture of the image 10 by using a filter corresponding to the identified pattern among the plurality of filters. According to an embodiment of the present disclosure, the first learning network model may be a deep learning model, and the second learning network model may be a machine learning model.

[0081] The machine learning model includes a plurality of pre-learned filters that are pre-learned based on various information and data input methods such as supervised learning, unsupervised learning, and semi-supervised learning, and identifies a filter to be applied to the image 10 among the plurality of filters.

[0082] A deep learning model is a model that performs learning based on a large amount of data and includes multiple hidden layers between an input layer and an output layer. Therefore, a deep learning model may require more additional resources of the image processing apparatus 100 than a machine learning model for performing learning and operation.

[0083] As another example, the first learning network model and the second learning network model can be models based on the same artificial intelligence algorithm but with different sizes or configurations. For example, the second learning network model can be a low-complexity model that is smaller than the first learning network model. Here, the size and complexity of the learning network model can be proportional to the number of convolutional layers and the number of parameters that constitute the model. In addition, according to an embodiment of the present disclosure, the second learning network model can be a deep learning model, and the first learning network model can be a deep learning model that uses fewer convolutional layers than the second learning network model.

[0084] As another example, each of the first and second learning network models may be a machine learning model. For example, the second learning network model may be a low-complexity machine learning model that is smaller in size than the first learning network model.

[0085] Meanwhile, various embodiments of the present disclosure have been described based on the assumption that the first learning network model is a model trained using more resources than the second learning network model, but this is merely an example and the present disclosure is not limited thereto. For example, the first learning network model and the second learning network model may be models having the same or similar complexity, and the second learning network model may be a model trained using more resources than the first learning network model.

[0086] According to an embodiment of the present disclosure, the processor 120 may identify edge regions and texture regions included in the image 10 at operation S330. Then, at operation S340, the processor 120 may apply a first weight to the first image and a second weight to the second image based on information regarding the edge regions and texture regions. For example, the processor 120 may obtain a first weight corresponding to the edge regions and a second weight corresponding to the texture regions based on information regarding the ratio of the edge regions and texture regions included in the image 10. For example, if the ratio indicates that there are more edge regions than texture regions, the processor 120 may apply a greater weight to the first image, whose edge regions have been improved, than to the second image, whose texture has been improved. As another example, if the ratio indicates that there are more texture regions than edge regions, the processor 120 may apply a greater weight to the second image, whose texture has been improved, than to the first image, whose edge regions have been improved. The processor 120 may then obtain the output image 10 based on the first image to which the first weight has been applied and the second image to which the second weight has been applied.

[0087] As another example, the first and second images acquired from the first and second learning network models may be residual images. Here, the residual image may be an image that includes only residual information rather than the original image. For example, the first learning network model may identify edge regions in image 10, optimize the identified edge regions, and acquire the first image. The second learning network model may identify texture regions in image 10, optimize the identified texture regions, and acquire the second image.

[0088] Then, the processor 120 may blend the image 10 with the first image and the second image and acquire the output image 20. Here, blending may be a process of adding the corresponding pixel value of each of the first image and the second image to the value of each pixel included in the image 10. In this case, the output image 20 may be an image having enhanced edges and textures due to the first image and the second image.

[0089] The processor 120 according to an embodiment of the present disclosure may apply the first weight and the second weight to the first image and the second image, respectively, and then mix the images with the image 10 , thereby acquiring the output image 20 .

[0090] As another example, processor 120 may divide image 10 into multiple regions. Processor 120 may then identify the ratio of edge regions to texture regions in each of the multiple regions. For a first region having a high ratio of edge regions among the multiple regions, processor 120 may set a first weight greater than a second weight. Furthermore, for a second region having a high ratio of texture regions among the multiple regions, processor 120 may set a second weight greater than the first weight.

[0091] Then, at operation S340, the processor 120 may mix the first image and the second image to which the weights have been applied with the image 10 and obtain an output image. The image 10 and the output image 20 corresponding to the image 10 may be as shown in the following equation 1.

[0092] [Equation 1]

[0093] Y res =Y img +a*Network_Model1(Y img )+b*Network_Model2(Y img )

[0094] Here, Y_img represents the image 10, Network_Model1(Y_img) represents the first image, Network_Model2(Y_img) represents the second image, 'a' represents the first weight corresponding to the first image, and 'b' represents the second weight corresponding to the second image.

[0095] Meanwhile, as another example, the processor 120 may apply the image 10 as input to the third learning network model and obtain a first weight for applying to the first image and a second weight for applying to the second image. For example, the third learning network model may be trained to identify edge regions and texture regions included in the image 10, and output a first weight for emphasizing the edge region and a second weight for emphasizing the texture region based on the ratio of the identified edge region and texture region, features of the image 10, etc.

[0096] Figure 4 is a diagram illustrating frequency reduction according to an embodiment of the present disclosure.

[0097] refer to Figure 4At operation S410, the processor 120 may identify edge regions and texture regions in the input image 10' and obtain a first weight corresponding to the edge region and a second weight corresponding to the texture region. As an example, the processor 120 may apply a guided filter to the input image 10' and identify edge regions and texture regions. The guided filter may be a filter for dividing the image 10 into a base layer and a detail layer. The processor 120 may identify edge regions based on the base layer and identify texture regions based on the detail layer.

[0098] Then, at operation S420, the processor 120 may downsample the input image 10′ and obtain a resolution image 10 having a resolution smaller than that of the input image 10′. As an example, the processor 120 may apply downsampling to the input image 10′ and downsample the resolution of the input image 10′ to a target resolution. Here, the target resolution may be a low resolution lower than the resolution of the input image 10′. For example, the target resolution may be the resolution of the original image corresponding to the input image 10′. Here, the resolution of the original image may be estimated by a resolution estimation program, or the resolution of the original image may be identified based on additional information received together with the input image 10′, but the resolution and its identification are not limited thereto. At the same time, the processor 120 may apply various known downsampling methods other than downsampling to obtain an image 10 corresponding to the input image 10′.

[0099] For example, if the input image 10′ is a 4K UHD image, then in order to apply the input image 10′ as input to the first and second learning network models and obtain the output image 20, a line buffer memory at least 5.33 times (3840 / 820) is required compared to the case of applying an 820×480 SD image to the first and second learning network models. In addition, there is a problem that the space of the memory 110 used to store the intermediate operation results of each of the multiple hidden layers included in the first learning network model increases exponentially, and the required CPU / GPU performance increases exponentially according to the increase in the number of operations required for the first learning network model to obtain the first image.

[0100] Therefore, the processor 120 according to an embodiment of the present disclosure can apply the downscaled input image 10 to the first and second learning network models to reduce the amount of operations required in the first and second learning network models, the storage space of the memory 110, etc.

[0101] When the downscaled image 10 is input, at operation S430, the first learning network model according to an embodiment of the present disclosure may perform an upscaling operation to enhance the high-frequency components corresponding to the edges included in the input image 10, and obtain a first image with high resolution. At the same time, at operation S440, the second learning network model may perform an upscaling operation to enhance the high-frequency components corresponding to the texture included in the image 10, and obtain a second image with high resolution. Here, the resolutions of the first image and the second image may be the same as the resolution of the input image 10'. For example, if the input image 10 is a 4K high-resolution image and the downscaled image 10 is a 2K resolution image, the first and second learning network models may perform upscaling on the image 10 and obtain a 4K resolution image therefrom as output.

[0102] The processor 120 according to an embodiment of the present disclosure may mix the upscaled first and second images with the input image 10' at operation S450 and obtain a high-resolution output image 20 in which edges and textures in the input image 10' have been enhanced. Figure 4 In the illustrated embodiment, the process of obtaining the input image 10 ′ and the output image 20 corresponding to the input image 10 ′ may be as shown in the following equation 2.

[0103] Equation 2

[0104] Y res =Y org +a*Network_Model1(DownScaling(Y org ))+b*Network_Model2(DownScaling(Y org ))

[0105] Here, Y_org represents the input image 10', DownScaling(Y_org) represents the image 10, Network_Modell(DownScaling(Y_org)) represents the first image, Network_Model2(DownScaling(Y_org)) represents the second image, 'a' represents the first weight corresponding to the first image, and 'b' represents the second weight corresponding to the second image.

[0106] Figure 5 is a diagram illustrating a deep learning model and a machine learning model according to an embodiment of the present disclosure.

[0107] refer to Figure 5As described above, the first learning network model may be a deep learning model that learns to enhance the edges of the image 10 by using a plurality of layers, and the second learning network model may be a machine learning model that is trained to enhance the texture of the image 10 by using a plurality of pre-learned filters.

[0108] According to an embodiment of the present disclosure, a deep learning model can be modeled with a deep structure comprising a total of ten or more layers in a configuration of repeating two convolutional layers and one pooling layer. In addition, the deep learning model can perform operations by using various types of activation functions (e.g., Identity function, Logistic Sigmoid function, Hyperbolic Tangent (tanh) function, ReLU function, LeakyReLU function, etc.). In addition, the deep learning model can be resized differently by performing padding, stride, etc. during the process of performing convolution. Here, padding refers to filling a specific value (e.g., pixel value) as much as a predetermined size around the received input value. Stride refers to the shift interval of the weight matrix when performing convolution. For example, if stride = 3, the learning network model can perform convolution on the input value while shifting the weight matrix by up to three spaces at a time.

[0109] According to an embodiment of the present disclosure, a deep learning model can learn to optimize one feature of the various features of image 10 to which the user is highly sensitive, and a machine learning model can optimize at least one of the remaining features of image 10 by using a plurality of pre-learned filters. For example, it can be assumed that there is a close relationship between the clarity of the edge region (e.g., edge direction, edge strength) and the clarity perceived by the user regarding image 10. The image processing device 100 can enhance the edges of image 10 by using a deep learning model, and as an example of the remaining features, the image processing device 100 can enhance the texture by using a machine learning model. Since the deep learning model learns based on a larger amount of data than the machine learning model and performs iterative operations, it is assumed that the processing results of the deep learning model are better than the processing results of the machine learning model. However, the present disclosure is not necessarily limited to this, and both the first and second learning network models can be implemented as models based on deep learning or models based on machine learning. As another example, the first learning network model can be implemented as a model based on machine learning, and the second learning network model can be implemented as a model based on deep learning.

[0110] In addition, although the various embodiments of the present disclosure are explained based on the assumption that the first learning network model enhances edges and the second learning network model enhances textures, the specific operations of the learning network models are not limited thereto. For example, it can be assumed that there is a case where the closest relationship exists between the degree of noise of the processed image 10 and the clarity felt by the user regarding the image 10. In this case, the image processing device 100 can perform image processing on the noise of the image 10 by using a deep learning model, and as an example of the remaining image features, the image processing device 100 can enhance the texture by using a machine learning model. As another example, if the closest relationship exists between the degree of brightness of the processed image 10 and the clarity felt by the user regarding the image 10, the image processing device 100 can perform image processing on the brightness of the image 10 by using a deep learning model, and as an example of the remaining image features, the image processing device 100 can filter the noise by using a machine learning model.

[0111] Figure 6 is a diagram illustrating first and second learning network models according to an embodiment of the present disclosure.

[0112] The processor 120 according to an embodiment of the present disclosure may downsample the input image 10' and obtain an image 10 with a relatively low resolution at operation S610, and obtain region detection information of the identified edge region and texture region based on the downsampled image 10' at operation S620. Figure 5 In the illustrated embodiment, the processor 120 may identify edge regions and texture regions included in the input image 10' at the resolution of the original image. Figure 6 , the processor 120 may identify edge regions and texture regions included in the image 10 in which the resolution of the input image 10 ′ has been downscaled to the target resolution.

[0113] Then, the processor 120 according to an embodiment of the present disclosure may provide the region detection information and the image 10 to the first and second learning network models, respectively.

[0114] At operation S630, the first learning network model according to an embodiment of the present disclosure may perform an upscaling operation based on the region detection information to enhance only the edge region of the image 10. At operation S640, the second learning network model may perform an upscaling operation based on the region detection information to enhance only the texture region of the image 10.

[0115] As another example, processor 120 may provide an image including only some pixel information included in image 10 to the learning network model based on the region detection information. Since processor 120 provides only some information included in image 10 rather than image 10 to the learning network model, the amount of work performed by the learning network model may be reduced. For example, processor 120 may provide an image including only pixel information corresponding to an edge region to a first learning network model based on the region detection information, and provide an image including only pixel information corresponding to a texture region to a second learning network model.

[0116] Then, the first learning network model may upscale the edge region and acquire a first image, and the second learning network model may upscale the texture region and acquire a second image.

[0117] Next, the processor 120 may add the first image and the second image to the input image 10 ′ and obtain an output image 20 in operation S650 .

[0118] Figure 7 is a diagram illustrating first and second learning network models according to another embodiment of the present disclosure.

[0119] refer to Figure 7 At operation S710, the processor 120 according to an embodiment of the present disclosure may apply the input image 10' as input to the first learning network model and obtain a first image. As an example, at operation S710, when the first learning network model performs an upscaling operation to enhance high-frequency components corresponding to edges included in the input image 10', the processor 120 may obtain a high-resolution first image. Here, the first image may be a residual image. The residual image may be an image that includes only residual information, rather than the original image. The residual information may indicate the difference between each pixel or pixel group of the original image and the high-resolution image.

[0120] In addition, the processor 120 according to an embodiment of the present disclosure may apply the input image 10' as input to the second learning network model at operation S720 and obtain a second image. As an example, when the second learning network model performs an upscaling operation to enhance high-frequency components corresponding to texture included in the input image 10', the processor 120 may obtain a high-resolution second image. Here, the second image may be a residual image. The residual information may indicate the difference between each pixel or pixel group of the original image and the high-resolution image.

[0121] According to an embodiment of the present disclosure, the first and second learning network models each perform an upscaling operation that enhances at least one feature of the input image 10', and thus the first and second images have higher resolutions than the input image 10'. For example, if the resolution of the input image 10' is 2K, the resolutions of the first and second images may be 4K, and if the resolution of the input image 10' is 4K, the resolutions of the first and second images may be 8K.

[0122] According to an embodiment of the present disclosure, the processor 120 may upscale the input image 10' and obtain a third image at operation S730. According to an embodiment of the present disclosure, the image processing device 100 may include a separate processor for upscaling the input image 10', and the processor 120 may upscale the input image 10' and obtain a high-resolution third image. For example, the processor 120 may upscale the input image 10' using bilinear interpolation, bicubic interpolation, cubic spline interpolation, Lanczos interpolation, edge-directed interpolation (EDI), etc. At the same time, this is only an example, and the processor 120 may upscale the input image 10' based on various upscaling (or super-resolution) methods.

[0123] As another example, the processor 120 may apply the input image 10' as input to a third learning network model and obtain a high-resolution third image corresponding to the input image 10'. Here, the third learning network model may be a deep learning-based model or a machine learning-based model. According to an embodiment of the present disclosure, if the resolution of the input image 10' is 4K, the resolution of the third image may be 8K. In addition, according to an embodiment of the present disclosure, the resolutions of the first to third images may be the same.

[0124] Then, the processor 120 may mix the first to third images and obtain an output image 20 at operation S740 .

[0125] According to an embodiment of the present disclosure, the processor 120 may mix a first residual image, a second residual image, and a third residual image to obtain an output image, wherein the first residual image is obtained by upscaling the input image 10' by emphasizing edges in the input image 10', the second residual image is obtained by upscaling the input image 10' by emphasizing texture in the input image 10', and the third residual image is obtained by upscaling the input image 10'. Here, the processor 120 may identify edge regions in the input image 10' and apply the identified edge regions to the first learned network model, thereby emphasizing the edge regions and thus obtaining the upscaled first residual image. Furthermore, the processor 120 may identify texture regions in the input image 10' and apply the identified texture regions to the second learned network model, thereby emphasizing the texture regions and thus obtaining the upscaled second residual image. However, this is merely an example, and the configuration and operation are not limited thereto. For example, the processor 120 may apply the input image 10' to both the first and second learned network models. Then, the first learning network model can identify edge regions based on edge features among various image features of the input image 10' and enhance the identified edge regions, thereby obtaining a high-resolution upscaled first residual image. The second learning network model can identify texture regions based on texture features among various image features of the input image 10' and enhance the identified texture regions, thereby obtaining a high-resolution upscaled second residual image.

[0126] In addition, the processor 120 according to an embodiment of the present disclosure may upscale the input image 10' and obtain a third image with a high resolution. Here, the third image may be an image obtained by upscaling the original image (not the residual image).

[0127] According to an embodiment of the present disclosure, the processor 120 may blend the first to third images and obtain an output image 20 having a higher resolution than the input image 10'. Here, the output image 20 may be an upscaled image with enhanced edge and texture regions, rather than an image with only upscaled resolution. While this is merely an example, the processor 120 may also obtain multiple residual images with enhanced image features of the input image 10', blend the third image obtained by upscaling the input image 10' with the multiple residual images, and obtain the output image 20 based on the resulting blended images.

[0128] Figure 8 is a diagram schematically illustrating the operation of the second learning network model according to an embodiment of the present disclosure.

[0129] The processor 120 according to an embodiment of the present disclosure may apply the image 10 as an input to the second learning network model and acquire a second image with enhanced texture.

[0130] The second learning network model according to an embodiment of the present disclosure may store multiple filters corresponding to each of a plurality of image patterns. Here, the plurality of image patterns may be classified according to the characteristics of the image blocks. For example, the first image pattern may be an image pattern having a large number of lines in the horizontal direction, and the second image pattern may be an image pattern having a large number of lines in the rotational direction. The plurality of filters may be filters pre-learned by an artificial intelligence algorithm.

[0131] In addition, the second learning network model according to an embodiment of the present disclosure can read an image block of a predetermined size from the image 10. Here, the image block can be a group consisting of a plurality of pixels included in the image 10, the plurality of pixels including the object pixel and a plurality of surrounding pixels. As an example, the second learning network model can read a first image block of 3×3 pixels on the upper left end of the image 10 and perform image processing on the first image block. Then, the second learning network model can scan up to the unit pixel to the right from the upper left end of the image 10, read a second image block of 3×3 pixels, and perform image processing on the second image block. By scanning across pixel blocks, the second learning network model can perform image processing on the image 10. At the same time, the second learning network model can read the first to nth image blocks from the image 10 by itself, and the processor 120 can sequentially apply the first to nth image blocks as input to the second learning network model and perform image processing on the image 10.

[0132] To detect high-frequency components in the image block, the second learning network model may apply a filter of a predetermined size to the image block. As an example, the second learning network model may apply a 3×3 Laplacian filter 810 corresponding to the size of the image block to the image block, thereby eliminating low-frequency components in image 10 and detecting high-frequency components. As another example, the second learning network model may obtain high-frequency components of image 10 by applying various types of filters, such as Sobel, Prewitt, Robert, Canny, etc., to the image block.

[0133] The second learning network model can then calculate a gradient vector 820 based on the high-frequency components obtained from the image block. Specifically, the second learning network model can calculate horizontal and vertical gradients, and calculate a gradient vector based on the horizontal and vertical gradients. Here, the gradient vector can represent the amount of change in each pixel relative to a pixel located in a predetermined direction. In addition, the second learning network model can classify the image block into one of multiple image modes based on the directionality of the gradient vector.

[0134] Next, the second learning network model can search for a filter to be applied to the high-frequency component detected from the image 10 (perform a filter search) 830 by using the index matrix 850. Specifically, the second learning network model can identify index information indicating the pattern of the image block based on the index matrix, and search 830 for a filter corresponding to the index information. For example, if the index information corresponding to the image block is identified as 32 among index information 1 to 32 indicating the pattern of the image block, the second learning network model can obtain a filter mapped to index information 32 from among a plurality of filters. Meanwhile, the specific index values above are merely examples, and the index information may be reduced or increased depending on the number of filters. In addition, the index information may be represented in various ways other than integers.

[0135] The second learning network model may then retrieve at least one filter from a plurality of filters in a filter database (DB) 860 based on the search results, and apply the at least one filter to the image block 840 to obtain a second image. For example, the second learning network model may identify a filter corresponding to the pattern of the image block from among the plurality of filters based on the search results, and apply the identified filter to the image block to obtain a second image in which the texture region is upscaled.

[0136] Here, the filters included in the filter database 860 can be acquired based on the results of learning the relationship between the low-resolution image block and the high-resolution image block through the artificial intelligence algorithm. For example, the second learning network model can learn the relationship between the low-resolution first image block and the high-resolution second image block (wherein the texture area of the first image block has been upconverted by the artificial intelligence algorithm), identify the filter to be applied to the first image block, and store the identified filter in the filter database 860. However, this is merely an example, and the present disclosure is not limited thereto. For example, the second learning network model can identify a filter that enhances at least one of the various features of the image block by using the learning results of the artificial intelligence algorithm, and store the identified filter in the filter database 860.

[0137] Figure 9 is a diagram illustrating index information according to an embodiment of the present disclosure.

[0138] The processor 120 according to an embodiment of the present disclosure may accumulate index information of image patterns corresponding to each classified image block and obtain an accumulation result. Figure 9 , the processor 120 may obtain the index information corresponding to the image mode of the image block from the index information indicating the image mode. Then, the processor 120 may accumulate the index information of each of the plurality of image blocks included in the image 10, thereby obtaining an accumulation result, such as Figure 9 shown.

[0139] The processor 120 can analyze the accumulated results and identify the image 10 as one of a natural image or a graphic image. For example, if, based on the accumulated results, the number of image blocks that do not include a pattern (or do not show directionality) among the image blocks included in the image 10 is equal to or greater than a threshold, the processor 120 can identify the image 10 as a graphic image. As another example, if, based on the accumulated results, the number of image blocks that do not include a pattern among the image blocks included in the image 10 is less than a threshold, the processor 120 can identify the image 10 as a natural image. As yet another example, if, based on the accumulated results, the number of image blocks with a pattern in the vertical direction or a pattern in the horizontal direction is equal to or greater than a threshold, the processor 120 can identify the image 10 as a natural image. At the same time, the recognition and classification of images are merely exemplary, and a threshold value can be specified according to the manufacturer's purpose, the user's settings, etc.

[0140] As another example, the processor 120 may calculate the number and ratio of specific index information based on the accumulation result, and identify the type of the image 10 as a natural image or a graphic image based on the accumulation result. For example, the processor 120 may calculate at least three features based on the accumulation result.

[0141] If specific index information among the index information is information indicating an image block for which a pattern is not recognized (or directionality is not shown), the processor 120 may calculate the proportion of the index information based on the accumulated result. Hereinafter, an image block for which a pattern is not recognized is generally referred to as an image block including a flat area. The proportion of image blocks including a flat area among all image blocks may be calculated based on the following equation 3.

[0142] Equation 3

[0143]

[0144] Here, Histogram[i] represents the number of image blocks having index information "i" identified based on the accumulation result. In addition, based on the assumption that the index information indicating that the image blocks including flat areas are 32, Histogram

[32] represents the number of image blocks having index information 32, and P1 represents the ratio of image blocks including flat areas among all image blocks.

[0145] If an image block includes a pattern, processor 120 can determine whether the pattern is located in the central region within the image block based on the index information. For example, the pattern of image blocks with index information of 13 to 16 may be located in the central region within the block, compared to image blocks with index information of 1 to 12 and 17 to 31. Hereinafter, an image block with an image pattern located in the central region within the image block is generally referred to as a centrally distributed image block. Processor 120 can then calculate the proportion of centrally distributed image blocks based on the accumulated result and the following equation 4.

[0146] Equation 4

[0147]

[0148] Here, the processor 120 may calculate the number of cells having index information 1 to 31. The number of image blocks to identify the number of image blocks including the pattern (excluding image blocks including flat areas). In addition, the processor 120 can calculate the number of central distribution image blocks. Meanwhile, the image block having index information 13 to 15 is merely an example of a case where the pattern is located in the central area inside the image block, and the present disclosure is not necessarily limited thereto. As another example, P2 may be calculated based on the number of index information 11 to 17.

[0149] Then, the processor 120 may acquire average index information of the image 10 based on index information of each of the plurality of image blocks included in the image 10. According to an embodiment of the present disclosure, the processor 120 may calculate the average index information based on Equation 5 below.

[0150] Equation 5

[0151]

[0152] Here, “i” represents index information, Histogram[i] represents the number of image blocks corresponding to the index information i, and P3 represents average index information.

[0153] The processor 120 according to an embodiment of the present disclosure may calculate a 'Y' value based on Equation 6 below.

[0154] Equation 6

[0155] Y=W1*P1+W2*P2+W3*P3+Bias

[0156] Here, P1 represents the proportion of image blocks including flat areas, P2 represents the proportion of image blocks with central distribution, and P3 represents average index information. In addition, W1, W2, W3, and Bias represent parameters pre-learned by using an artificial intelligence algorithm model.

[0157] If the Y value exceeds 0, the processor 120 according to an embodiment of the present disclosure may recognize the image 10 as a graphic image, and if the 'Y' value is equal to or less than 0, the processor 120 may recognize the image 10 as a natural image.

[0158] The processor 120 may then adjust the first weight and the second weight corresponding to the first image and the second image, respectively, based on the recognition result. As an example, if the image 10 is recognized as a natural image, the processor 120 may increase at least one of the first weight corresponding to the first image or the second weight corresponding to the second image. In addition, the processor 120 may increase at least one of the parameters 'a' or 'b' in Equations 1 and 2. At the same time, if the image 10 is a natural image, since the first image with improved edges or the second image with improved texture is added to the image 10 or the input image 10', the processor 120 may obtain a high-resolution image with improved clarity, and therefore the processor 120 may increase at least one of the first weight or the second weight.

[0159] As another example, if image 10 is recognized as a graphic image, processor 120 may reduce at least one of the first weight corresponding to the first image or the second weight corresponding to the second image. Furthermore, processor 120 may reduce at least one of parameters 'a' or 'b' in Equations 1 and 2. Meanwhile, if image 10 is a graphic image, processor 120 may obtain a distorted image because the first image with enhanced edges or the second image with enhanced texture is added to image 10 or input image 1', and thus processor 120 may reduce at least one of the first weight or the second weight to minimize the occurrence of distortion.

[0160] Here, a graphic image may be an image that manipulates an image of the real world or an image newly created by using a computer, an imaging device, or the like. For example, a graphic image may include an illustrated image, a computer graphics (CG) image, an animated image generated using known software, or the like. A natural image may be an image other than a graphic image. For example, a natural image may include an image of the real world, a landscape image, a portrait image, or the like captured by a camera.

[0161] Figure 10 is a diagram illustrating a method of acquiring a final output image according to an embodiment of the present disclosure.

[0162] According to an embodiment of the present disclosure, if the final output image 30' (i.e., the displayed image) has a resolution greater than that of the output image 30, the processor 120 may upscale the output image 30 and obtain the final output image 30' at operation S350. For example, if the output image 30 is a 4K UHD image and the final output image is an 8K image, the processor 120 may upscale the output image 30 to an 8K UHD image and obtain the final output image 30'. Meanwhile, according to another embodiment of the present disclosure, a separate processor may be provided in the image processing device 100 to perform upscaling of the output image 30. For example, the image processing device 100 may include a first processor and a second processor, and may use the first processor to obtain the output image 30 in which edges and textures have been enhanced, and use the second processor to obtain the high-resolution final output image 30' in which the resolution of the output image 30 has been expanded.

[0163] At the same time, each of the first and second learning network models according to various embodiments of the present disclosure may be an on-device machine learning model in which the image processing apparatus 100 performs learning on its own without relying on an external device. Meanwhile, this is merely an example, and some learning network models may be implemented in the form of operations based on the device, while other learning network models may be implemented in the form of operations based on an external server.

[0164] Figure 11 It is shown in Figure 2 A block diagram of a detailed configuration of an image processing apparatus shown in FIG.

[0165] according to Figure 11 , the image processing apparatus 100 includes a memory 110, a processor 120, an input device 130, a display 140, an output device 150 and a user interface 160. Figure 11 When the components are shown, the Figure 2 Components shown are redundant descriptions of similar components.

[0166] According to an embodiment of the present disclosure, the memory 110 may be implemented as a single memory that stores data generated according to various operations of the present disclosure.

[0167] The memory 110 may be implemented to include first to third memories.

[0168] The first memory can store at least a portion of the image input through the inputter 130. In particular, the first memory can store at least some areas of the input image frame. In this configuration, according to an embodiment of the present disclosure, at least some areas may be areas necessary for performing image processing. At the same time, according to an embodiment of the present disclosure, the first memory can be implemented as an N-line memory. For example, the N-line memory may be a memory with a capacity of up to 17 lines in the horizontal direction, but is not limited thereto. For example, in the case of inputting a full HD image of 1080p (resolution of 1920 pixels), only image areas of 17 lines in the full HD image are stored in the first memory. The reason why the first memory as described above is implemented as an N-line memory and only stores some areas of the input image frame for image processing is that the storage capacity of the first memory is constrained according to hardware limitations. At the same time, the second memory may be a memory area allocated to the learning network model among the entire area of the memory 110.

[0169] The third memory is a memory for storing the first and second images and the output image, and according to various embodiments of the present disclosure, the third memory can be implemented as a memory of various sizes. According to an embodiment of the present disclosure, the processor 120 applies the image 10 obtained by downscaling the input image 10' to the first and second learning network models, and therefore the size of the third memory for storing the first and second images obtained from the first and second learning network models can be implemented to be the same as or similar to the size of the first memory.

[0170] The input device 130 may be a communication interface for receiving various types of content (e.g., image signals from an image source), such as a wired Ethernet interface or a wireless communication interface. For example, the input device 130 may receive image signals from an external device (e.g., a source device), an external storage medium (e.g., a USB), an external server (e.g., a Web hard drive), or the like via one or more networks such as the Internet through various communication methods such as Wi-Fi (wireless LAN network) based on an access point, Bluetooth, Zigbee, wired / wireless local area network (LAN), wide area network (WAN), Ethernet, LTE, fifth generation (5G), IEEE 1394, High-Definition Multimedia Interface (HDMI), Mobile High-Definition Link (MHL), Universal Serial Bus (USB), DisplayPort (DP), Thunderbolt, Video Graphics Array (VGA) port, RGB port, D-Sub (D-SUB), Digital Video Interface (DVI), and the like, through streaming or downloading. In particular, the 5G communication system is a communication using ultra-high frequency (mmWave) bands (for example, millimeter wave bands such as 26, 28, 38, and 60 GHz bands), and the image processing device 100 can send or receive 4K and 8K UHD images in a streaming environment.

[0171] Here, the image signal may be a digital signal, but the image signal is not limited thereto.

[0172] The display 140 can be implemented in various forms, such as a liquid crystal display (LCD), an organic light-emitting diode (OLED), a light-emitting diode (ED), a micro-LED, a quantum dot light-emitting diode (QLED), a liquid crystal on silicon (LCoS), a digital light processing (DLP), and a quantum dot (QD) display panel. In particular, the processor 120 according to an embodiment of the present disclosure can control the display 140 to display the output image 30 or the final output image 30'. Here, the final output image 30' can include a real-time UHD image of 4K or 8K, a streaming image, etc.

[0173] The outputter 150 outputs an acoustic signal.

[0174] For example, the outputter 150 may convert the digital acoustic signal processed by the processor 120 into an analog acoustic signal, amplify the signal, and output the signal. For example, the outputter 150 may include at least one speaker unit capable of outputting at least one channel, a D / A converter, an audio amplifier, and the like. According to an embodiment of the present disclosure, the outputter 150 may be implemented to output various multi-channel acoustic signals. In this case, the processor 120 may control the outputter 150 to perform enhancement processing on the input acoustic signal to correspond to the enhancement processing of the input image and output the signal. For example, the processor 120 may convert the input two-channel acoustic signal into a virtual channel (e.g., 5.1-channel) acoustic signal, or identify the location of the image processing device 100 within the environment of a room or building and process the signal into a stereo signal optimized for the space, or provide an acoustic signal optimized according to the type of the input image (e.g., the genre of the content). Meanwhile, the user interface 160 may be implemented as a device such as a button, a touchpad, a mouse, and a keyboard, or as a touch screen, a remote control receiver, and the like capable of receiving user input to perform the above-mentioned display functions and manipulation input functions. The remote control transceiver can receive a remote control signal from an external remote control device or send a remote control signal to an external remote control device through at least one communication method among infrared communication, Bluetooth communication or Wi-Fi communication. Figure 9 Although not shown in the figure, according to an embodiment of the present disclosure, free filtering for removing noise from the input image may be applied before image processing. For example, significant noise may be removed by applying a smoothing filter (e.g., a Gaussian filter) or a guided filter that filters the input image by comparing it with a predetermined guide.

[0175] Figure 12 is a block diagram illustrating a configuration of an image processing apparatus for learning and using a learning network model according to an embodiment of the present disclosure.

[0176] refer to Figure 12 , the processor 1200 may include at least one of a learning unit 1210 or a recognition unit 1220 . Figure 12 The processor 1200 in Figure 2 Corresponding to the processor 120 of the image processing device 100 or the processor of the data learning server.

[0177] The processor 1200 of the image processing apparatus 100 for learning and using the first and second learning network models may include at least one of a learning part 1210 or a recognition part 1220 .

[0178] The learning unit 1210 according to an embodiment of the present disclosure may acquire an image whose image features have been enhanced, and acquire an output image based on the image 10 and the image whose image features have been enhanced. The learning unit 1210 may then generate or train a recognition model having a standard for minimizing distortion of the image 10 and acquiring a high-resolution upscaled image corresponding to the image 10. Furthermore, the learning unit 1210 may generate a recognition model having a certain standard by using collected learning data.

[0179] As an example, the learning unit 1210 may generate, train, or update a learning network model so that at least one of an edge region or a texture region of the output image 30 is enhanced more than that of the input image 10 ′.

[0180] The recognition part 1220 may use predetermined data (for example, an input image) as input data of a trained recognition model, thereby estimating an object for recognition or a situation included in the predetermined data.

[0181] At least a portion of the learning unit 1210 and at least a portion of the recognition unit 1220 may be implemented as a software module or manufactured in the form of at least one hardware chip and installed on an image processing device. For example, at least one of the learning unit 1210 or the recognition unit 1220 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI) or as part of a conventional general-purpose processor (e.g., a CPU or application processor) or a graphics-specific processor (e.g., a GPU), and installed on the above-mentioned various types of image processing devices or object recognition devices. Here, the dedicated hardware chip for artificial intelligence is a dedicated processor dedicated to probabilistic operations and has higher performance than conventional general-purpose processors in parallel processing and is capable of quickly processing operations in artificial intelligence fields such as machine learning. In the case where the learning unit 1210 and the recognition unit 1220 are implemented as one or more software modules (or program modules including instructions), the software modules may be stored in a non-transitory computer-readable medium. In this case, the software module may be provided by an operating system (OS) or a specific application. Alternatively, a portion of the software module may be provided by an operating system (OS), and the other portion may be provided by a specific application.

[0182] In this case, the learning unit 1210 and the recognition unit 1220 may be installed in one image processing device, or may be installed in separate image processing devices respectively. For example, one of the learning unit 1210 and the recognition unit 1220 may be included in the image processing device, and the other may be included in an external server. In addition, the learning unit 1210 and the recognition unit 1220 may be connected by wire or wirelessly, or may be a separate software module of a larger software module or application. The model information constructed by the learning unit 1210 may be provided to the recognition unit 1220, and the data input to the recognition unit 1220 may be provided to the learning unit 1210 as additional learning data.

[0183] Figure 13 is a flowchart illustrating an image processing method according to an embodiment of the present disclosure.

[0184] according to Figure 13 In the image processing method shown, first, at operation S1310, an image is applied to a first learning network model, and a first image with enhanced edges of the image is obtained.

[0185] Then, the image is applied to a second learning network model at operation S1320, and a second image in which the texture of the image is enhanced is acquired.

[0186] Next, at operation S1330, an edge region and a texture region included in the image are identified, a first weight is applied to the first image and a second weight is applied to the second image based on information about the edge region and the texture region, and an output image is acquired.

[0187] Here, the first learning network model and the second learning network model may be different types of learning network models from each other.

[0188] The first learning network model according to an embodiment of the present disclosure may be one of the following models: a deep learning model that learns to enhance the edges of an image by using multiple layers, or a machine learning model that is trained to enhance the edges of an image by using multiple pre-learned filters.

[0189] The first learning network model according to an embodiment of the present disclosure may be one of the following models: a deep learning model that learns to optimize the texture of an image by using multiple layers, or a machine learning model that is trained to optimize the texture of an image by using multiple pre-learned filters.

[0190] Furthermore, the operation S1330 of acquiring the output image may include acquiring a first weight corresponding to the edge region and a second weight corresponding to the texture region based on ratio information of the edge region and the texture region.

[0191] In addition, the image processing method according to an embodiment of the present disclosure may include the following steps: downscaling the input image and acquiring an image having a resolution smaller than that of the input image. Simultaneously, the first learning network model may acquire the first image by performing an upscaling operation that emphasizes the edges of the image, and the second learning network model may acquire the second image by performing an upscaling operation that emphasizes the texture of the image.

[0192] In addition, the image processing method according to an embodiment of the present disclosure may include the following steps: obtaining region detection information of edge regions and texture regions that have been identified based on the down-converted image, and providing the region detection information and the image to the first and second learning network models respectively.

[0193] Here, the steps of providing the region detection information and the image to the first and second learning network models, respectively, may include providing the image including only pixel information corresponding to the edge region based on the region detection to the first learning network model, and providing the image including only pixel information corresponding to the texture region to the second learning network model. Simultaneously, the first learning network model may obtain the first image by upscaling the edge region, and the second learning network model may obtain the second image by upscaling the texture region.

[0194] In addition, the first image and the second image according to an embodiment of the present disclosure may be a first residual image and a second residual image, respectively. In addition, in operation S1330 of acquiring an output image, a first weight may be applied to the first residual image, and a second weight may be applied to the second residual image, and then the residual image may be mixed with the image to acquire the output image.

[0195] In addition, the second learning network model can be a model that stores multiple filters corresponding to each of a plurality of image modes, classifies each image block included in the image into one of the plurality of image modes, and applies at least one filter corresponding to the classified image mode among the plurality of filters to the image block, and provides a second image.

[0196] In addition, operation S1330 of acquiring an output image according to an embodiment of the present disclosure may include the following steps: accumulating index information of image patterns corresponding to each classified image block, identifying the type of the image as, for example, a natural image or a graphic image based on the accumulation result, and adjusting the weight based on the recognition result.

[0197] Here, the step of adjusting the weight may include the following steps: based on the image being identified as a natural image, increasing at least one of the first weight corresponding to the first image or the second weight corresponding to the second image, and based on the image being identified as a graphic image, reducing at least one of the first weight or the second weight.

[0198] The output image may be a 4K ultra high definition (UHD) image, and the image processing method according to an embodiment of the present disclosure may include a step of upscaling the output image to an 8K UHD image.

[0199] Meanwhile, various embodiments of the present disclosure may be applied to all electronic devices capable of performing image processing (such as an image receiving device and an image processing device, etc., such as a set-top box), as well as image processing devices.

[0200] In addition, the various embodiments described so far can be implemented in a recording medium that can be read by a computer or a computer-like device using software, hardware, or a combination thereof. In some cases, the embodiments described in this specification can be implemented as processor 120 itself. According to the implementation by software, embodiments such as the processing and functions described in this specification can be implemented as independent software modules. Each software module can perform one or more functions and operations described in this specification.

[0201] Meanwhile, computer instructions for executing the processing operations of the image processing apparatus 100 according to the various embodiments of the present disclosure may be stored in a non-transitory computer-readable medium. When the computer-readable instructions stored in such a non-transitory computer-readable medium are executed by a processor of a specific device, the processing operations of the image processing apparatus 100 according to the various embodiments are executed by the specific device.

[0202] Non-transitory computer-readable media refers to media that semi-permanently stores data and is machine-readable, rather than media that temporarily stores data (such as registers, caches, and memories). Specific examples of non-transitory computer-readable media include CDs, DVDs, hard disks, Blu-ray discs, USBs, memory cards, and ROMs.

[0203] Although the embodiments of the present disclosure have been illustrated and described, the present disclosure is not limited to the aforementioned specific embodiments, and it is apparent that various modifications may be made by those skilled in the art without departing from the spirit of the present disclosure as claimed in the appended claims. In addition, it is intended that such modifications should not be interpreted as being independent of the technical ideas or prospects of the present disclosure.

Claims

1. An image processing device, comprising: a memory storing computer-readable instructions; as well as A processor configured to execute the computer-readable instructions to: Provide the input image as the first input to the first neural model, obtaining a first image including enhanced edges optimized based on edges of the input image from the first neural model, wherein the first image is a first residual image, providing the input image as a second input to a second neural model, obtaining a second image including an enhanced texture optimized based on the texture of the input image from the second neural model, wherein the second image is a second residual image, identifying edge regions of edges included in the input image, identifying a texture region of texture included in the input image, obtaining a first weight corresponding to the edge area and a second weight corresponding to the texture area based on a ratio of the edge area and the texture area identified in the input image, applying the first weight to the first residual image based on the edge region, applying the second weight to the second residual image based on the texture region, and After applying the first weight and the second weight, mixing the first residual image, the second residual image, and the input image to obtain an output image, and The processor executing the computer-readable instructions is further configured to: Accumulating index information of image modes corresponding to each image block of the input image, identifying the input image as one of a natural image or a graphic image based on the index information, and The first weight and the second weight are adjusted based on a result of recognizing the input image as one of the natural image or the graphic image.

2. The image processing apparatus according to claim 1, wherein: A first type of the first neural model is different from a second type of the second neural model.

3. The image processing apparatus according to claim 1, wherein: The first neural model is one of: a deep learning model for optimizing the edge of the input image by using a plurality of layers, or a machine learning model trained to optimize the edge of the input image by using a plurality of pre-learned filters.

4. The image processing apparatus according to claim 1, wherein: The second neural model is one of: a deep learning model for optimizing the texture of the input image by using a plurality of layers, or a machine learning model trained to optimize the texture of the input image by using a plurality of pre-learned filters.

5. The image processing apparatus according to claim 1, wherein: The processor executing the computer readable instructions is further configured to: Downscaling the input image to obtain a downscaled image having a resolution smaller than that of the input image, providing the downscaled image as the first input to the first neural model, obtaining the first image with enhanced edges from the first neural model, the first neural model upscaling the downscaled image, providing the downscaled image as the second input to the second neural model, and The second image with enhanced texture is obtained from the second neural model, which upscales the downscaled image. The image processing apparatus according to claim 1 , wherein: The processor executing the computer readable instructions is further configured to: obtaining first region detection information for identifying the edge region of the input image and second region detection information for identifying the texture region of the input image, providing the input image and the first region detection information as the first input to the first neural model, and The input image and the second region detection information are provided as the second input to the second neural model.

7. The image processing apparatus according to claim 6, wherein: The first neural model obtains the first image by upscaling the edge region, and The second neural model obtains the second image by upscaling the texture region.

8. The image processing apparatus according to claim 1, wherein: The second neural model is a model that stores multiple filters corresponding to multiple image modes, classifies each of the image blocks included in the input image into an image mode among the multiple image modes, and provides at least one filter corresponding to the image mode among the multiple filters to the image block.

9. The image processing apparatus according to claim 1, wherein: The processor executing the computer readable instructions is further configured to: increasing at least one of the first weight or the second weight based on the input image being identified as the natural image, and Based on the input image being identified as the graphic image, at least one of the first weight or the second weight is reduced.

10. An image processing method for an image processing device, comprising: providing an input image as a first input to a first neural model; obtaining, from the first neural model, a first image including enhanced edges optimized based on edges of the input image, wherein the first image is a first residual image; providing the input image as a second input to a second neural model; obtaining, from the second neural model, a second image including an enhanced texture optimized based on the texture of the input image, wherein the second image is a second residual image; identifying edge regions of edges included in the input image; identifying a texture region of a texture included in the input image; obtaining a first weight corresponding to the edge region and a second weight corresponding to the texture region based on a ratio of the edge region identified in the input image to the texture region identified; applying the first weight to the first residual image based on the edge region, applying the second weight to the second residual image based on the texture region, and After applying the first weight and the second weight, mixing the first residual image, the second residual image, and the input image to obtain an output image, and The image processing method further includes: Accumulating index information of image modes corresponding to each image block of the input image, identifying the input image as one of a natural image or a graphic image based on the index information, and The first weight and the second weight are adjusted based on a result of recognizing the input image as one of the natural image or the graphic image.

11. The image processing method according to claim 10, wherein: The first neural model is one of: a deep learning model for optimizing the edge of the input image by using a plurality of layers, or a machine learning model trained to optimize the edge of the input image by using a plurality of pre-learned filters.

12. The image processing method according to claim 10, wherein: The second neural model is one of: a deep learning model for optimizing the texture of the input image by using a plurality of layers, or a machine learning model trained to optimize the texture of the input image by using a plurality of pre-learned filters.

Images