Image processing device and operation method thereof

Abstract

Provided are an image processing device and an operating method thereof. The image processing device comprises: a memory for storing one or more instructions; and at least one processor coupled to the memory and including a processing circuit. The one or more instructions, when executed individually or collectively by the at least one processor, instruct the image processing device to: store the cumulative quality of content including a plurality of input images on the basis of a viewing frequency of the content; determine a model storage condition on the basis of the viewing frequency and the cumulative quality; obtain a reference model corresponding to the model storage condition; store the reference model in the memory; and train the reference model stored in the memory by using training data corresponding to the quality of a first image, thereby generating a target model corresponding to the first image.

Description

Image processing device and its operating method

[0001] The disclosed embodiments relate to an image processing device and an operating method thereof. More specifically, the disclosed embodiments relate to an image processing device and an operating method thereof for processing a low-resolution image to output a high-resolution image.

[0002] With the advancement of deep learning technology, various learning-based upscaling methods are being developed. Learning-based upscaling methods exhibit excellent performance when the quality of the training images is similar to that of the input images being processed. However, if the characteristics of the image to be processed differ from the input quality assumed during training, the image quality deteriorates significantly, causing a problem known as the "domain gap."

[0003] To address these issues, research is being conducted on on-device learning, which processes and adapts AI models based on input data. On-device learning can refer to the operation of training a neural network model installed on a device in real time by collecting and computing information on the device itself without going through a cloud server. A device that performs on-device learning can be referred to as an edge device. In the field of on-device learning research, a paper on image processing and image quality improvement (ZSSR, CVPR 2018, Zero-Shot Super-Resolution using Deep Internal Learning, hereinafter referred to as Paper 1) was recently published.

[0004] Paper 1, called Zero-Shot Super Resolution (ZSSR), is a technology that constructs a database (DB) using input images tailored to the deterioration characteristics of the input image and then uses a model trained on this database to enlarge the image. However, ZSSR has the drawback of high learning complexity and difficulty in applying it to videos with significant image quality fluctuations, as it creates a new database from scratch for each input image and trains the model using this database.

[0005] To improve these problems, another paper (MetaSR, ECCV 2021, Fast Adaptation to Super-Resolution Networks via Meta-Learning, hereinafter referred to as Paper 2) was published. Paper 2 is a technology that learns the initial meta model from an external database and finds a model that fits the characteristics of the input image through transfer learning to reduce the learning computational complexity of ZSSR. However, since the technology of Paper 2 uses only one meta model, there is a performance limitation in including all the characteristics of various input images in one meta model. In addition, in environments that use low-capacity networks such as edge devices, this limitation of the meta model becomes a factor that limits the performance of on-device learning.

[0006] Additionally, there is a method of selecting a meta model that matches the characteristics of the input image by using multiple meta models that match the characteristics of the input image, but it is impossible to train the model by predicting all actual input images.

[0007] Accordingly, research is being conducted to reduce the amount of computation and resource usage required for training models with neural networks on edge devices with limited computational resources and internal memory, and to shorten the time required for training.

[0008] An image processing device according to one embodiment of the present disclosure includes a memory storing one or more instructions, and one or more processors coupled to the memory and including a processing circuit.

[0009] The image processing device stores the cumulative quality of content including a plurality of input images based on the viewing frequency of the content by individually or collectively executing one or more instructions by one or more processors according to one embodiment of the present disclosure.

[0010] An image processing device according to one embodiment of the present disclosure determines model storage conditions based on viewing frequency and cumulative quality.

[0011] An image processing device according to one embodiment of the present disclosure stores a reference model in a memory upon obtaining a reference model corresponding to a model storage condition.

[0012] A processor according to one embodiment of the present disclosure generates a target model corresponding to a first image by training a reference model stored in a memory using learning data corresponding to the quality of the first image.

[0013] An operating method of an image processing device according to one embodiment of the present disclosure includes a step of storing a cumulative quality of content including a plurality of input images based on a viewing frequency of the content, a step of determining a model storage condition based on the viewing frequency and the cumulative quality, a step of obtaining a reference model corresponding to the model storage condition and storing the reference model in a memory, and a step of generating a target model corresponding to the first image by training the reference model stored in the memory using learning data corresponding to the quality of the first image.

[0014] FIG. 1 is a drawing showing an image processing device according to one embodiment of the present disclosure.

[0015] Figure 2a is an example of a model learning process shown in a quality information graph of an image.

[0016] FIG. 2b is a graph showing real-time quality changes of multiple frame images included in video content according to one embodiment.

[0017] FIG. 3 is a flowchart illustrating an operation method of an image processing device according to one embodiment of the present disclosure.

[0018] Figure 4 is an example for explaining classification information according to one embodiment.

[0019] FIG. 5 is an example for explaining the cumulative quality of input images stored according to one embodiment.

[0020] FIG. 6 is an example for explaining an operation of an image processing device according to one embodiment of the present invention to store a second reference model.

[0021] FIG. 7 is an example of a first reference model and a second reference model stored in a memory of an image processing device according to one embodiment.

[0022] FIG. 8 is a diagram showing the configuration of an image processing device according to one embodiment of the present disclosure.

[0023] FIG. 9 is a drawing specifically showing a model learning unit of an image processing device according to one embodiment of the present disclosure.

[0024] FIG. 10 is a flowchart illustrating an operation method of an image processing device and a server according to one embodiment of the present disclosure.

[0025] FIG. 11 is a block diagram of an image processing device according to one embodiment of the present disclosure.

[0026] FIG. 12 is a detailed block diagram of an image processing device according to one embodiment of the present disclosure.

[0027] In this disclosure, the expression “at least one of a, b or c” may refer to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “all of a, b and c”, or variations thereof.

[0028] Below, embodiments of the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

[0029] The terms used in this disclosure are described as currently common terms, taking into account the functions mentioned herein. However, these terms may mean various other terms depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Therefore, the terms used in this disclosure should not be interpreted solely based on their names, but rather based on the meanings of the terms and the overall content of this disclosure.

[0030] Additionally, the terminology used in this disclosure is for the purpose of describing specific embodiments only and is not intended to limit the present disclosure.

[0031] Throughout the specification, when a part is said to be "connected" to another part, this includes not only the cases where it is "directly connected" but also the cases where it is "electrically connected" with another element in between.

[0032] As used herein, and particularly in the claims, the terms "above" and "above" and similar referents may refer to both the singular and the plural. Furthermore, unless the order of steps in a method according to the present disclosure is explicitly specified, the steps described may be performed in any appropriate order. The present disclosure is not limited by the order in which the steps are described.

[0033] The appearances of phrases such as “in some embodiments” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment.

[0034] Some embodiments of the present disclosure may be represented by functional block configurations and various processing steps. Some or all of these functional blocks may be implemented by various hardware and / or software configurations that perform specific functions. For example, the functional blocks of the present disclosure may be implemented by one or more microprocessors or by circuit configurations for a given function. Furthermore, for example, the functional blocks of the present disclosure may be implemented in various programming or scripting languages. The functional blocks may be implemented by algorithms that execute on one or more processors. Furthermore, the present disclosure may employ conventional techniques for electronic configuration, signal processing, and / or data processing. Terms such as “mechanism,” “element,” “means,” and “configuration” may be used broadly and are not limited to mechanical and physical configurations.

[0035] Additionally, the connecting lines or connecting members between components depicted in the drawings are merely exemplary representations of functional connections and / or physical or circuit connections. In an actual device, connections between components may be represented by various functional connections, physical connections, or circuit connections that may be replaced or added.

[0036] Additionally, terms such as “part”, “module”, etc. described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software, or a combination of hardware and software.

[0037] In the specification, a processor may include various processing circuits and / or multiple processors. For example, the term “processor” as used herein, including in the claims, may include various processing circuits, including at least one processor. At least one processor, one or more processors, may be configured to perform various functions described herein, individually and / or collectively, in a distributed fashion. As used herein, “processor,” “at least one processor,” and “one or more processors” may be configured to perform multiple functions. However, these terms encompass, without limitation, situations where one processor performs some of the functions and other processor(s) perform other parts of the functions, and situations where a single processor may perform all of the functions. Furthermore, the at least one processor may include a combination of processors that perform various of the disclosed functions in a distributed manner. At least one processor may execute program instructions to achieve or perform various functions.

[0038] Additionally, the term "user" in this specification refers to a person who utilizes the image processing device, and may include a consumer, evaluator, viewer, administrator, or installer. Furthermore, the term "manufacturer" or "provider" in this specification may refer to a manufacturer that manufactures the image processing device and / or components included in the image processing device.

[0039] In the present disclosure, 'image' may refer to a still image, a picture, a frame, a moving image composed of a plurality of consecutive still images, or a video.

[0040] In this disclosure, "neural network" refers to a representative example of an artificial neural network model that mimics brain neurons, and is not limited to an artificial neural network model using a specific algorithm. A neural network may also be referred to as a deep neural network.

[0041] FIG. 1 is a drawing showing an image processing device according to one embodiment of the present disclosure.

[0042] Referring to FIG. 1, the image processing device (100) may be an electronic device capable of processing and outputting an image. The image processing device (100) may be implemented in various forms including a display. For example, the image processing device (100) may be implemented in various electronic devices such as a TV, a mobile phone, a tablet PC, a digital camera, a camcorder, a laptop computer, a desktop, an e-book reader, a digital broadcasting terminal, a PDA (Personal Digital Assistant), a PMP (Portable Multimedia Player), a navigation device, an MP3 player, a wearable device, etc.

[0043] The image processing device (100) can receive and output various contents created by content providers through external devices. The contents may include video contents such as still images and moving images, audio contents, subtitle contents, and other additional information. The content provider may refer to a terrestrial broadcasting station, cable broadcasting station, satellite broadcasting station, IPTV (Internet Protocol Television) service provider, or OTT (Over the Top) service provider that provides various contents to consumers. The external device may be implemented as a variety of source devices such as a PC, a set-top box (e.g., terrestrial broadcasting set-top box, cable broadcasting set-top box, satellite broadcasting set-top box, internet broadcasting set-top box), Blu-ray disc player, mobile phone, game console, home theater, audio player, USB, etc. The external device is connected to the image processing device (100) through an input / output unit such as HDMI and can provide various contents to the image processing device (100). Alternatively, an external device can be connected to the image processing device (100) via a wired or wireless communication network such as Wi-Fi or WLAN to provide various contents to the image processing device (100).

[0044] In one example, an OTT may provide streaming content to a device connected to the Internet via the Internet. The OTT service may be provided by a third party other than the manufacturer of the video processing device (100) and the Internet Service Provider (ISP). An OTT may provide streaming content using a subscription-based Video on Demand (SVoD) service, also known as a “streaming platform,” which provides access to movies and TV content. In one example, the video processing device (100) may include an internal tuner or may be connected to an external tuner that provides live broadcast content. In one embodiment, the video processing device (100) may provide a user interface for selecting live broadcast content or streaming content. In one example, the video processing device (100) may provide a Picture-in-Picture (PIP) feature, in which one portion of the display screen displays content from a first source (e.g., streaming content from an OTT) and another portion of the display screen displays content from a second source (e.g., live broadcast content from a tuner).

[0045] The image processing device (100) can output video content. The video content may include multiple frame images. The video content is captured, compressed, transmitted, and then restored and output by the image processing device (100). Due to limitations in the physical characteristics of the device used to capture the video and limited bandwidth, information may be lost, resulting in image distortion. This image distortion may degrade the quality of the video content. The video content may have various qualities depending on the degree of image distortion.

[0046] An image processing device (100) according to one embodiment can process the image quality of an image. The image processing device (100) can process the image quality of video content whose quality has been reduced due to image distortion. For example, the image processing device (100) can obtain a second image (120, output image) by performing image quality processing on a first image (110, input image). For example, the image processing device (100) can obtain a high-resolution (or high-quality) output image by upscaling a low-resolution (or low-quality) input image using an image quality processing model. The image quality processing model according to one embodiment can include a neural network model trained using learning data corresponding to the first image (110) from a neural network model obtained based on the quality of the first image (110).

[0047] An image processing device (100) according to one embodiment can train an image quality processing model installed in the image processing device (100) in real time within the device itself. For example, the image processing device (100) can obtain quality information of a first image (110), obtain learning data corresponding to the quality of the first image (110), obtain an image quality processing model for processing the image quality of the first image (110), and update the image quality processing model using the learning data. An image processing device (100) according to one embodiment can obtain a second image (120) by performing image quality processing on the first image (110) using the updated image quality processing model.

[0048] The image quality processing model according to one embodiment may be referred to as a meta model. The meta model may refer to a neural network model that can quickly learn or generalize to new data. The meta model may be pre-trained using training data corresponding to various quality information stored in the cloud, and then used to process the image quality of actual images in the image processing device (100). For example, the meta model may be pre-trained using a corpus data set containing various image objects and image scenes. The image processing device (100) may reduce the domain gap problem by training the meta model using actual images as training data. The image processing device (100) may generate a meta model that is adaptive to the input image.

[0049] To train a neural network model, training data consisting of pairs of low-resolution images and high-resolution images can be used. For example, one pair of images can include a first version of a low-resolution image and a second version of a high-resolution image. The neural network model can be trained through a process in which the parameters of the neural network model are updated to reduce the difference (training error or loss) between the image output by processing the low-resolution image in the training data through the neural network model and the high-resolution image in the training data. The process of updating the parameters of the neural network model can involve multiple iterations. For example, the image processing device (100) can perform multiple iterations to update an image quality processing model previously stored in an internal memory to a model corresponding to a specific quality.

[0050] The training time of a neural network model can be determined by the computing power of the device (e.g., number of computing resources, bit width, memory capacity), the complexity of the algorithm used (e.g., accuracy of the quality analyzer, type of model learning algorithm), the size (or depth) of the neural network, the size of the training data, the type of the neural network model, etc.

[0051] Since the image processing device (100) trains a neural network model through limited computational resources (e.g., CPU, NPU, DSP, etc.) and internal memory compared to a server, the learning speed may be slower than that of the server. For example, the learning speed of the image processing device (100) may decrease as the number of iterations for updating the image quality processing model to a model corresponding to a specific quality increases. For example, the learning speed may decrease as the difference between the image quality processing model pre-stored in the image processing device (100) and the specific quality increases. For example, if the number of image quality processing models pre-stored in the image processing device (100) is small, the probability that there are few image quality processing models close to the specific quality is high, and thus the learning speed may decrease. The slower the learning speed of the meta-model for the input image, the more difficult it is to generate a meta-model that adapts to real-time quality changes of the input image, and thus the upscaling performance may decrease.

[0052] Below, a method for increasing the learning speed of a neural network model performed in an image processing device (100) is described.

[0053] Figure 2a is an example of a model learning process shown in a quality information graph of an image.

[0054] Figure 2a is a two-dimensional graph representing the results of analyzing the image quality of an input image and the model that processes the image quality on a quality plane. The graph illustrates the process of generating an updated model by training the model to process the image quality of the input image. The image quality of the input image can be analyzed using a quality analyzer.

[0055] Referring to FIG. 2a, Graph 1 (201) and Graph 2 (202) are quality information graphs that divide the quality of an image into two quality factors, where the horizontal axis represents quality factor A and the vertical axis represents quality factor B. For example, the quality factors may include Kernel Sigma, compression quality (Quality Factor, QF), etc. The Kernel Sigma value is a value that represents the blur quality of an image, and QF may be a value that represents the degree of deterioration due to compression.

[0056] The image processing device (100) can generate a model for processing the image quality of an input image in response to a quality value (210) of the input image. In graph 1 (201), an X represents a quality value (210) of the input image, Model 1, Model 2, Model 3, and Model 4 represent reference models (220), and a checkered circle symbol represents a model (230) updated from the reference model (220). An arrow represents an update direction of the reference model (220), and the end of the arrow points to the updated model (230). Model 1, Model 2, Model 3, and Model 4 may be image quality processing models that have been previously learned based on learning data having different quality values. The updated model (230) may be an image quality processing model having adjusted parameters by being learned in response to the quality value (210) of the input image from the reference model (220). The updated model (230) may be a target model corresponding to the quality value (210) of the input image or a meta-model adaptive to the input image. For example, the target model may be trained to improve the quality value (e.g., blur) of the input image.

[0057] The image processing device (100) can generate an updated model (230) by updating the parameters of model 3 in response to the quality value (210) of the input image based on model 3, which is the reference model closest to the quality value (210) of the input image among reference models (220) including model 1, model 2, model 3, and model 4. For example, the quality value (210) can have quality A of value X1 and quality B of value Y2. In addition, model 3 can output an image having quality A of value X0 and quality B of value Y0. The distance between the quality value (210) and the quality of the image output by model 3 is (e.g., Pythagorean theorem). Based on this calculation, it can be determined that Model 3 has a closer distance to the quality value (210) than Model 1, Model 2, and Model 4. The updated model (230) may have higher image quality processing performance for the input image. In the present disclosure, the reference model closest to the quality value of the image may mean a model having parameters closest to the quality value of the image. Meanwhile, the method for selecting the reference model is not limited to the method of selecting the reference model closest to the quality value (210) of the input image. For example, the reference model may be selected by considering the weights for each model.

[0058] In the present disclosure, the reference model (220) of graph 1 (201) may be referred to as the first reference model in graph 2 (202), and the updated model (230) of graph 1 (201) may be referred to as the second reference model in graph 2 (202).

[0059] In one embodiment, the first reference model may be a reference model pre-stored in the image processing device (100). The first reference model may be a model that has been learned in advance by a server or an external device and stored in the image processing device (100). The first reference model may be a meta-model that is adaptive to the input image as it is learned using the input image acquired by the image processing device (100). The first reference model may be stored in a non-volatile memory (e.g., NAND flash) and may not be reset even when the image processing device (100) is powered on / off.

[0060] In one embodiment, the second reference model may be an updated model from the first reference model. The second reference model may have parameters adjusted from the first reference model in response to the quality of the input image. For example, the adjusted parameters may improve the performance of the second reference model for image quality enhancement. The image processing device (100) may generate a second reference model with adjusted parameters from the first reference model for image quality processing of an input image input in real time. For example, the second reference model may be a target model for a quality value (e.g., 210) of a specific input image having a viewing history. The second reference model is stored in a volatile memory (e.g., DRAM) and, unlike the first reference model, may be reset when the image processing device (100) is powered on / off.

[0061] In one embodiment, the image processing device (100) may store some of the generated second reference models in non-volatile memory. The second reference models stored in the non-volatile memory may not be reset even when the image processing device (100) is powered on / off. The non-volatile memory may correspond to the internal memory of the image processing device (100) or a database of a server.

[0062] In one embodiment, the image processing device (100) may store a second reference model used to process the quality of frequently viewed videos by the user in non-volatile memory. The image processing device (100) may use the additionally stored second reference model to quickly generate a target model for the quality of frequently viewed videos. The additionally stored second reference model may be a quality processing model having parameters adjusted to process the quality of frequently viewed videos by the user. Accordingly, the number of iterations required to adjust the parameters of the quality processing model in response to the quality of frequently viewed videos may be reduced, and the learning time may be reduced. In one example, if a video is viewed a predetermined number of times or more within a predetermined interval, the video may be classified as frequently viewed.

[0063] In Graph 2 (202), the X represents the quality value (240) of the input image, Model 1, Model 2, Model 3, and Model 4 represent the first reference model (270), the checkered circle symbol represents the second reference model (260), and the hatched circle symbol represents the updated model (250) from the second reference model (260) (or the target model (250) for the quality value (240) of the input image). The arrow represents the update direction of the second reference model (260). The second reference model (260) may correspond to a target model (e.g., the updated model (230)) for the quality value (210) of the input image viewed in the past. The distance between the second reference model (260) and the target model (250) may be shorter than the distance between the first reference model (270) and the target model (250).

[0064] In one embodiment, since a second reference model (260) corresponding to the quality of frequently viewed images is additionally stored in the non-volatile memory of the image processing device (100), the image processing device (100) can use the second reference model (260). The second reference model (260) may be a reference model that outputs an image having a quality closest to the quality value (240) of the input image. Since the image processing device (100) generates a target model (250) using the second reference model (260), the number of iterations for learning may be reduced, the time required to generate the target model (250) may be reduced, and the learning speed of the neural network may be increased. Accordingly, the image processing device (100) can quickly generate a target model in response to real-time quality changes in the input image, and output an image with image quality processing for the user's preferred content. A method for determining the conditions of the second reference model stored in the non-volatile memory will be further described in FIG. 3.

[0065] Meanwhile, frequently viewed videos can be categorized by content, resolution, or object. Considering the cumulative frequency of quality changes in frequently viewed videos, a second reference model corresponding to the frequently accumulated quality can be stored in non-volatile memory. For example, the quality of frequently viewed videos can be accumulated by content, resolution, or object. Accordingly, the second reference model can be additionally stored by content, resolution, or object.

[0066] Meanwhile, the image processing device (100) can analyze the input image to obtain further quality factors in addition to Kernel Sigma and QF of each input image. For example, the image processing device (100) can analyze the input image to obtain further quality factors indicating the degree of noise included in the input image. In this case, the quality value of each input image obtained by the image processing device (100) can be expressed as a three-dimensional graph representing Kernel Sigma, QF, and the degree of noise along three axes.

[0067] When the number of quality factor axes increases to k, the target model must be generated from the k-dimensional quality information graph, so the amount of computation may increase by more than the k power (k is a natural number). For example, when generating a target model from a three-dimensional quality information graph, the reference model may be updated in response to not only quality factor A and quality factor B, but also quality factor C. Accordingly, the amount of computation may increase by more than 1 power compared to when generating a target model from a two-dimensional quality information graph. The image processing device (100) according to one embodiment of the present disclosure can quickly generate a target model even when the number of quality factor axes increases to k. FIG. 2B is a graph showing a real-time quality change of a plurality of frame images included in video content according to one embodiment.

[0068] Referring to Figure 2b, the horizontal axis of the graph represents the number of multiple frame images, and the vertical axis represents the quality level of the frame images. The graph shows changes in the blur quality and compression quality of the frame images.

[0069] In the graph, video content can vary in quality in real time, frame by frame. For example, the video content has approximately 3,500 frames, and quality can vary by 50% to 90% over time. For example, the video content is OTT content and has a bitrate of 4 Mbps.

[0070] In one embodiment, the image processing device (100) can generate a meta model that adapts to real-time changing qualities for each frame, for image quality processing of video content. The image processing device (100) needs to learn quickly to generate a meta model that adapts to each real-time changing quality. By generating a meta model that adapts to each real-time changing quality, the image processing device (100) can provide upscaled image quality for video content.

[0071] Meanwhile, an image processing device (100) according to one embodiment can store cumulative quality data that accumulates qualities that change in real time in response to video content. The cumulative quality graph is further described in FIG. 5.

[0072] FIG. 3 is a flowchart illustrating an operation method of an image processing device according to one embodiment of the present disclosure.

[0073] Referring to FIG. 3, in operation 310, the image processing device (100) according to one embodiment may store the cumulative quality of an input image based on the viewing frequency of each content. The viewing frequency of each content may correspond to the user's preference for each content. For example, the user's preference for each content may correspond to the number of times the content has been viewed within a set interval.

[0074] In one embodiment, the quality of the input image may refer to the image quality or degree of deterioration of the image. The quality of the input image may refer to at least one of compression quality, blur quality, and noise. In one embodiment, the cumulative quality of the input image may refer to information accumulated about the quality of each of the multiple frame images included in the video content.

[0075] In one embodiment, the image processing device (100) may store the cumulative quality of input images corresponding to frequently viewed content. The frequently viewed content may correspond to the user's preferred content. The image processing device (100) may obtain information about content with high viewing frequency to content with low viewing frequency based on the user's accumulated viewing history over a predetermined period of time. In one embodiment, the image processing device (100) may store the cumulative quality of input images corresponding to frequently viewed content and may not store the cumulative quality of input images corresponding to content with low viewing frequency. The cumulative quality may be stored in the memory only for frequently viewed content based on a medium-term or long-term viewing time (e.g., one month). In one embodiment, the image processing device (100) may store a reference model corresponding to frequently viewed content in a non-volatile memory according to operation 330 described below.

[0076] In one embodiment, the image processing device (100) may store the cumulative quality of the input image by type of classification information. For example, the classification information may include at least one of the type of content, the resolution of the content, and a combination of the type of content and the resolution of the content. For example, the cumulative quality of the input image may be accumulated by type of content, by resolution of the content, or by a combination of the type of content and the resolution of the content. The classification information will be further described with reference to FIG. 4.

[0077] In one embodiment, the type of content may be determined based on an external device connected to the image processing device (100). For example, the content may include OTT content received from an OTT content provider via short-range wireless communication such as Wi-Fi or WLAN, broadcast content such as real-time broadcast programs received from terrestrial broadcast stations or cable broadcast stations connected via a set-top box, game content provided by a game console device, or cloud game content provided by a cloud game provider via wireless communication. In addition, the content may be further classified by content provider. In the case of OTT content, it may differ by OTT content provider (e.g., App #1, App #2), in the case of broadcast content, it may differ by broadcast channel (e.g., #7, #231), and in the case of game content, it may differ by game content provider. In one embodiment, the image processing device (100) may store a reference model for each type of content in a non-volatile memory according to operation 330 described below.

[0078] In one embodiment, the resolution of the content may include SD (Standard Definition), HD (High Definition), FHD (Full High Definition), and UHD (Ultra High Definition). The resolution of the content may vary depending on at least one of the network transmission speed, the type of subscription product, the genre of the broadcast content, or the resolution of the original content, even for the same type of content. In one embodiment, the image processing device (100) may store a reference model for each resolution of the content in a non-volatile memory according to operation 330 described below.

[0079] In one embodiment, the image processing device (100) can store the cumulative quality of input images corresponding to frequently viewed content by type of classification information. The cumulative quality of input images corresponding to preferred content is stored for each content, and may be stored separately for each content resolution. The cumulative quality stored in the image processing device (100) will be further described with reference to FIG. 5.

[0080] In operation 320, the image processing device (100) according to one embodiment may determine model storage conditions based on the viewing frequency and cumulative quality of each content.

[0081] In one embodiment, the model storing condition may refer to the condition of a reference model to be stored in non-volatile memory. The model storing condition may refer to the condition of a quality processing model for processing quality corresponding to frequently viewed content. The reference model may be a quality processing model pre-trained using training data. As the parameters of the reference model are adjusted in response to the quality of the input image, a target model (or a meta-model adaptive to the input image) for quality processing of the input image may be generated.

[0082] In one embodiment, the reference model may include a first reference model pre-stored in the image processing device (100). The first reference model may be stored in a non-volatile memory. In one embodiment, the first reference model may include reference models for each type of content. For example, the first reference model may include a reference model for OTT content, a reference model for broadcast content, a reference model for game content, etc. In addition, in one embodiment, the first reference model may include reference models for each resolution. For example, the first reference model may include a reference model for SD resolution, a reference model for HD resolution, a reference model for FHD resolution, a reference model for UHD resolution, etc. In addition, in one embodiment, the first reference model may include reference models for each object, such as reference models for faces, letters, general areas, etc.

[0083] In one embodiment, the reference model may include a second reference model learned from the first reference model in response to the quality of the input image. The second reference model may be composed of a plurality of second reference models corresponding to a plurality of input images included in the video content. The second reference model may be stored in volatile memory. In one embodiment, the model storage condition may refer to a condition of a reference model to be stored in non-volatile memory among the plurality of second reference models updated from the first reference model in response to the plurality of input images. Since the description of the first reference model and the second reference model has been described with reference to FIG. 2A, redundant details will be omitted.

[0084] In one embodiment, the image processing device (100) may determine model storage conditions corresponding to a second reference model frequently used in the image processing device (100). In one embodiment, the second reference model to be frequently used may be determined based on the viewing frequency and cumulative quality of each content. For example, since frequently viewed content has similar quality depending on the type of content, the second reference model generated corresponding to the frequently viewed content may be frequently used. In addition, since frequently viewed content has similar quality depending on the resolution of the content, the second reference model generated corresponding to the resolution of the frequently viewed content may be frequently used. In addition, if the cumulative frequency of a specific quality value among the qualities of frequently viewed content is high, the second reference model generated corresponding to the quality value with a high cumulative frequency may be frequently used.

[0085] In one embodiment, the model storage condition may include at least one of information on frequently viewed content, information on the resolution of the content, high-frequency points of cumulative quality, or the number of model storages. For example, the model storage condition may be determined for each frequently viewed content and may be determined for each resolution of the content. The model storage condition may include quality value information corresponding to high-frequency points of cumulative quality. The model storage condition may include information regarding the number of model storages corresponding to the number of high-frequency points of cumulative quality. Examples of the model storage condition are described in detail in the graph (630) of FIG. 6 or FIG. 7.

[0086] In operation 330, the image processing device (100) according to one embodiment may store the reference model in memory as it obtains a reference model corresponding to the model storage condition.

[0087] In one embodiment, the image processing device (100) may store a second reference model corresponding to a model storage condition among a plurality of second reference models generated from a first reference model in non-volatile memory. The image processing device (100) may store a frequently used second reference model separately from the first reference model in non-volatile memory.

[0088] In one embodiment, the image processing device (100) can generate a plurality of second reference models by applying learning data corresponding to the quality of each input image included in the video content to the first reference model. The operation of generating the plurality of second reference models can be performed separately from the operation of determining the second reference model to be stored. The plurality of second reference models can be generated in real time according to real-time input images.

[0089] In one embodiment, the image processing device (100) may store the second reference model in non-volatile memory if the second reference model is a model learned in response to the quality of frequently viewed content. The image processing device (100) may additionally store a model learned using training data corresponding to the quality of frequently viewed content. The additionally stored second reference model may have parameters for processing the quality of frequently viewed content.

[0090] Additionally, in one embodiment, the image processing device (100) may store a second reference model in non-volatile memory if the quality of the current input image corresponds to a high-frequency point of accumulated quality. The image processing device (100) may additionally store a second reference model trained with learning data corresponding to a high-frequency accumulated quality value. The additionally stored second reference model may have parameters for processing the high-frequency point of accumulated quality.

[0091] In one embodiment, the image processing device (100) may store a second reference model learned in response to a high-frequency accumulated quality value among the accumulated qualities of each content for each content with a high viewing frequency in a non-volatile memory.

[0092] In one embodiment, the image processing device (100) may store a second reference model for each type of classification information. For example, the classification information may include at least one of the type of content, the resolution of the content, and a combination of the type of content and the resolution of the content. For example, the second reference model may be stored for each type of content, for each resolution of the content, or for each combination of the type of content and the resolution of the content. For example, the image processing device (100) may store a second reference model for a first content with a high viewing frequency separately from a second reference model for a second content with a high viewing frequency. An example of a second reference model stored for each type of classification information is described below in FIG. 7.

[0093] In one embodiment, frequently viewed content may correspond to a predetermined number of frequently viewed content, the top M% of frequently viewed content, or content with a viewing frequency greater than a predetermined value (where M is a positive number). For example, the image processing device (100) may store cumulative qualities corresponding to three frequently viewed content. For example, the image processing device (100) may store a second reference model corresponding to the top 30% of frequently viewed content.

[0094] In one embodiment, a high-frequency point of cumulative quality may correspond to a predetermined number of points with a high cumulative frequency, points in the top N% with a high cumulative frequency, or points with a cumulative frequency greater than or equal to a predetermined value (where N is a positive integer). For example, if a predetermined value specifies a signal-to-noise ratio (SNR) and a 100-frame video has 10 frames with a highest SNR of 10 dB, the high-frequency points in the video may include the 10 video frames with an SNR of 10 dB.

[0095] In operation 340, an image processing device (100) according to one embodiment may generate a target model corresponding to a first image by training a stored reference model using learning data corresponding to the quality of the first image. The first image may be an input image of the image processing device (100). The first image may be an image input to the image processing device (100) after the second reference model is stored.

[0096] In one embodiment, the image processing device (100) can obtain quality information of the first image through a quality analyzer. The quality analyzer can check the quality information of the image by analyzing the characteristics of the image, such as brightness, noise, etc. The image processing device (100) can obtain learning data corresponding to the quality of the first image. The learning data can include a high-resolution image (correct image, label) and a low-resolution image. The image processing device (100) can obtain a reference model corresponding to the quality of the first image. The image processing device (100) can generate a target model corresponding to the quality of the first image by applying the learning data to the reference model corresponding to the quality of the first image.

[0097] In one embodiment, the image processing device (100) can generate a low-resolution image that has been deteriorated from the first image by applying a deterioration process to the first image that is equivalent to the degree of deterioration of the image quality of the first image. The image processing device (100) can generate learning data that includes the first image used as the correct image and the low-resolution image that has been deteriorated from the first image.

[0098] In one embodiment, the image processing device (100) can obtain a model that outputs an image with a quality closer to that of the first image among the first reference model that has been previously stored or the second reference model that has been additionally stored.

[0099] Alternatively, in one embodiment, the image processing device (100) may interpolate a plurality of reference models to obtain a target model. For example, the image processing device (100) may interpolate a plurality of reference models by applying weights of each reference model to the parameters of the plurality of reference models. The method by which the image processing device (100) obtains the reference model is not limited to the above-described example.

[0100] In one embodiment, the image processing device (100) may transfer learn a reference model corresponding to the quality of the first image using learning data corresponding to the quality of the first image. Transfer learning may refer to a process of fine-tuning the parameters of a pre-trained neural network model. The image processing device (100) may generate a target model corresponding to the quality of the first image. The target model may have parameters adjusted for image quality processing of the first image.

[0101] In one embodiment, the image processing device (100) can obtain a second image whose quality has been processed from a first image based on a target model.

[0102] An image processing device (100) according to one embodiment can quickly generate a target model corresponding to the quality of an input image by pre-storing a reference model corresponding to the quality of content frequently viewed by a user in a non-volatile memory. The image processing device (100) can quickly provide an upscaled image for a user-preferred content by increasing the generation speed of a target model having parameters corresponding to real-time quality changes according to real-time quality changes. Fig. 4 is an example for explaining classification information according to one embodiment. In operation 310 of Fig. 3, the accumulated quality of the input image can be stored for each type of classification information.

[0103] In one embodiment of the present disclosure, the quality of an input image may be accumulated by type of classification information. The classification information may include first classification information (410) corresponding to the type of content and second classification information (420) corresponding to the resolution of the content. Each content included in the first classification information (410) may be classified into detailed classification information (415).

[0104] In one embodiment, the type of content may correspond to first classification information (410) for classifying and accumulating the input quality of input images. For example, in the case of input images, the accumulated quality may be stored for each content. For example, the content may include OTT content, broadcast content, game console content, cloud gaming, etc. For example, the quality of video frames included in OTT content, the quality of video frames included in broadcast content, the quality of video frames included in game console content, and the quality of video frames included in cloud gaming may be accumulated separately.

[0105] In one embodiment, for OTT content, input quality may be stored by OTT content provider. The type of OTT content provider may correspond to detailed classification information (415) for classifying and accumulating the input quality of input videos. For example, the quality of input videos provided by a first OTT content provider (App #1) may be accumulated separately from the quality of input videos provided by a second OTT content provider (App #2). However, this is not limited to this, and the quality of OTT content may not be subdivided by content provider.

[0106] In one embodiment, for broadcast content, input quality may be stored for each broadcast channel. The type of broadcast channel may correspond to detailed classification information (415) for classifying and accumulating the input quality of input images. For example, the quality of input images provided by channel 7 (#7) may be accumulated separately from the quality of input images provided by channel 231 (#231). For example, channel 7 (#7) may be received from a terrestrial broadcasting station server, and channel 231 (#231) may be received from a cable broadcasting station server. Terrestrial broadcasting station servers may primarily provide content in FHD resolution. Cable broadcasting stations may primarily provide content in SD resolution. However, this is not limited thereto, and the quality of broadcast content may not be sub-classified for each broadcasting channel.

[0107] In one embodiment, for a game console, input quality may be stored for each game content. The game content may refer to a type of game software. The type of game content may correspond to detailed classification information (415) for classifying and accumulating the input quality of input images. For example, the quality of the input image provided by the first game content (Game Content #1) may be accumulated separately from the quality of the input image provided by the second game content (Game Content #2). For example, the first game content (Game Content #1) may provide content in FHD resolution, and the second game content (Game Content #2) may provide content in UHD resolution.

[0108] In one embodiment, for cloud gaming, input quality may be stored for each cloud gaming provider. The type of cloud gaming provider may correspond to detailed classification information (415) for classifying and accumulating the input quality of input images. For example, the quality of input images provided by a first cloud gaming provider (Game App #1) may be accumulated separately from the quality of input images provided by other cloud gaming providers.

[0109] In one embodiment, the resolution of the content may correspond to second classification information (420) for classifying and accumulating the input quality of input images. For example, for input images, the input quality may be stored according to the resolution of the content. The resolution of the content may include SD, HD, FHD, and UHD.

[0110] In one embodiment, the type of subscription plan may be a factor in determining the content resolution. For example, OTT content providers may offer content in SD, HD, FHD, or UHD resolutions depending on the user's subscription plan. For example, a standard plan may offer low-resolution content, while a premium plan may offer high-resolution content.

[0111] Additionally, in one embodiment, network transmission speed may be one of the factors determining the resolution of content. For example, OTT content providers may provide content in SD, HD, FHD, or UHD resolutions depending on the network transmission speed. For example, cloud gaming providers may provide content in SD, HD, FHD, or UHD resolutions depending on the network transmission speed. Network transmission speeds may vary by region and may also vary depending on the network bandwidth supported by the device, such as 5G or 3G. For example, when the network bandwidth is wide, high-resolution content may be provided, while when the network bandwidth is limited, low-resolution content may be provided.

[0112] Additionally, in one embodiment, the genre of broadcast content may be a factor in determining its resolution. For example, a terrestrial broadcasting station server connected to a set-top box may provide content in UHD resolution for award ceremonies or soccer matches, and in FHD resolution for dramas or news. The genre of broadcast content can be identified through the program schedule provided by the broadcasting station server.

[0113] Additionally, in one embodiment, the resolution of the original content may be one of the factors determining the resolution of the content. For example, the original resolution of content produced in the past may be lower than that of recently produced content.

[0114] Meanwhile, in operation 330 of FIG. 3, the second reference model may also be stored by type of classification information. For example, the second reference model may be stored by type of content and by resolution of the content. FIG. 5 is an example of the cumulative quality of an input image according to an embodiment of the present disclosure. FIG. 5 illustrates the cumulative quality described in operation 310 of FIG. 3. Table (501) illustrated in FIG. 5 indicates items in which input quality is stored. In table (501), column items indicate types of content, and row items indicate types of resolution.

[0115] In one embodiment of the present disclosure, the accumulated quality of input images corresponding to preferred content may be stored in the image processing device (100). The accumulated quality of input images not corresponding to preferred content may not be stored in the image processing device (100). For example, in the table (501) of FIG. 5, if a user frequently views the first OTT app (OTT App #1), the seventh channel (#7), and the first game content (game), the image processing device (100) may accumulate the quality of input images corresponding to content with a high viewing frequency.

[0116] In one embodiment of the present disclosure, the accumulated quality of input images corresponding to preferred content may be stored by type of classification information. For content with a high viewing frequency, the image processing device (100) may accumulate the quality of input images by the same content and / or by the same resolution.

[0117] For example, the cumulative quality of input video can be stored by OTT content, broadcast content, and game content. If the input video is OTT content, the cumulative quality of the input video can be stored by OTT content provider. If the input video is broadcast content, the cumulative quality of the input video can be stored by broadcast channel. If the input video is game content, the cumulative quality of the input video can be stored by game content type.

[0118] For example, the cumulative quality of input images can be stored separately for each resolution of the same content. For example, the cumulative quality of input images provided by OTT content can be stored for each resolution of SD, HD, FHD, and UHD. For example, the cumulative quality of input images provided by broadcast content can be stored for each resolution of SD, HD, FHD, and UHD. For example, the quality of input images provided by game content can be stored for each resolution of SD, HD, FHD, and UHD.

[0119] Even if images have the same content and / or the same resolution, their quality values ​​may be distributed in various ways. For example, the quality values ​​of images may be distributed in various ways depending on degradation that occurs during the acquisition, transmission, and storage of images. For example, even if input images have the same content and the same resolution, the quality of the input images may vary depending on bitrate information such as 40 Mbps, 30 Mbps, and / or codec information such as H.264, HEVC, etc. For example, video content with SDR (Standard Dynamic Range) resolution may include a first image with an SNR of 5 dB and a second image with an SNR of 8 dB. In one embodiment, the quality of the input image may be accumulated based on bitrate information and / or codec information that change in real time. The bitrate information or the codec information may be included in the metadata of the input image.

[0120] Graphs 1 (502), 2 (503), and 3 (504) of FIG. 5 represent cumulative quality graphs of images corresponding to the same content and the same resolution. Graph 1 (502) is an example of a cumulative quality graph that accumulates quality values ​​of images corresponding to a first OTT app (OTT App #1) having HD resolution, graph 2 (503) is an example of a cumulative quality graph that accumulates quality values ​​of images corresponding to channel 7 (#7) having FHD resolution, and graph 3 (504) is an example of a cumulative quality graph that accumulates quality values ​​of images corresponding to a first game content (game) having UHD resolution. Cumulative quality graphs such as graph 1 (502), graph 2 (503), and graph 3 (504) may be stored for each content item and resolution item.

[0121] Graph 1 (502), Graph 2 (503), and Graph 3 (504) are each three-dimensional cumulative quality graphs, where the x-axis represents quality factor A, the y-axis represents quality factor B, and the z-axis represents cumulative frequency. For example, a point with a large value on the z-axis represents a high-frequency point of cumulative quality, and the quality values ​​corresponding to the x-axis and y-axis coordinate values ​​of a point with a large value on the z-axis represent the quality values ​​of the input image corresponding to the high-frequency point.

[0122] In one embodiment of the present disclosure, the image processing device (100) may store a second reference model learned with learning data corresponding to quality values ​​of input images corresponding to high-frequency points of accumulated quality. The image processing device (100) may store a second reference model learned with learning data corresponding to quality values ​​of input images accumulated for each content. The image processing device (100) may store a second reference model learned with learning data corresponding to quality values ​​of input images accumulated for each resolution of the content. This will be described in detail in FIG. 6.

[0123] FIG. 6 is an example for explaining an operation of an image processing device according to one embodiment of the present invention to store a second reference model.

[0124] Figure 6 shows graph 1 (610), graph 2 (620), graph 3 (630), and graph 4 (640).

[0125] Graph 1 (610) is a two-dimensional quality graph representing first reference models previously stored in an image processing device (100). The first reference models may be image quality processing models previously learned based on learning data having four different quality values.

[0126] Graph 2 (620) is a three-dimensional cumulative quality graph representing the cumulative quality of input video over a given period of time. The cumulative quality of the input video may correspond, for example, to a first OTT app (OTT App #1) with HD resolution. The cumulative quality of the input video may correspond to the user's preferred content.

[0127] Graph 3 (630) is a two-dimensional quality graph showing model storage conditions, which are conditions for second reference models to be additionally stored in the image processing device (100). The model storage conditions may include information about quality values ​​(e.g., x-axis values, y-axis values) corresponding to high-frequency points of cumulative quality (e.g., two points with the highest z-axis values) exemplified in Graph 2 (620). The model storage conditions may include information about two quality values ​​corresponding to two high-frequency points. In one embodiment, the model storage conditions may include information about frequently viewed content, resolution information of frequently viewed content, high-frequency points of cumulative quality, and / or the number of models to be stored. For example, the image processing device (100) may determine to additionally store two reference models corresponding to high-frequency points of cumulative quality of a first OTT app (OTT App #1) having an HD resolution.

[0128] Graph 4 (640) is a two-dimensional quality graph showing first reference models previously stored in the image processing device (100) and second reference models additionally stored. In Graph 4 (640), the second reference model is indicated by a checkered circle symbol. The second reference models may be reference models corresponding to model storage conditions. The image processing device (100) may store the second reference model when the second reference model corresponding to the model storage conditions is generated among the second reference models generated in real time. For example, the second reference models additionally stored in the image processing device (100) may be reference models having parameters corresponding to high-frequency points of the cumulative quality of the first OTT app (OTT App #1) having HD resolution. For example, the quality may refer to the SNR of the image (e.g., the higher the SNR, the lower the noise of the image), and the high-frequency points of Graph 2 (620) may be 5 dB and 8 dB. Therefore, the storage condition of the second reference model can be specified to store the trained first reference model that outputs an image with an SNR exceeding 5 dB.

[0129] FIG. 7 is an example of a first reference model and a second reference model stored in a memory of an image processing device according to one embodiment.

[0130] Table (701) illustrated in Fig. 7 represents model items in which the first reference model and the second reference model are stored. In table (701), column items represent content types, and row items represent resolution types. The first reference model and the second reference model are stored separately for each item.

[0131] In one embodiment, the reference model may include a reference model for each type of content (e.g., OTT content, broadcast content, game content) and / or a reference model for each resolution (e.g., SD, HD, FHD, UHD). In one embodiment, the reference models for each type of content may include an OTT content processing model, a broadcast content processing model, and a game content processing model. For example, the OTT content processing model may be a quality processing model trained with training data corresponding to various qualities of OTT content. In one embodiment, the reference models for each resolution may include an SD resolution processing model, an HD resolution processing model, an FHD resolution processing model, and a UHD resolution processing model. For example, the HD resolution processing model may be a quality processing model trained with training data corresponding to various qualities of an image having an HD resolution. Each reference model may be distinguished by identification information (e.g., S1 to S16, O1 to O12, G1 to G8, etc.). The identification information of the reference model may include quality processing content information, quality processing resolution information, and parameter information corresponding to a specific quality value.

[0132] In one embodiment, the image processing device (100) generates second reference models from a first reference model stored in advance, and stores a second reference model corresponding to a model storage condition among the second reference models for each item.

[0133] In one embodiment, the model storage condition may include at least one of information on content with a high viewing frequency, information on resolution of content with a high viewing frequency, high-frequency points of cumulative quality, or the number of model storages.

[0134] For example, if a user frequently watches channel 7 (#7), the first OTT app (OTT App #1), and the first game content, the image processing device (100) may determine model storage conditions to store a second reference model corresponding to the content with a high viewing frequency. The model storage conditions may include identification information of a model for processing channel 7 (#7), the first OTT app (OTT App #1), and the first game content.

[0135] Additionally, for example, if a user watches more images of channel 7 having HD resolution than images of channel 7 having SD resolution, the image processing device (100) may determine model storage conditions to store a larger number of second reference models corresponding to images of channel 7 having HD resolution than the second reference models corresponding to images of channel 7 having SD resolution.

[0136] Additionally, for example, the image processing device (100) may determine model storage conditions to store a second reference model learned as a high-frequency point of cumulative quality of an image. For example, for an item of channel 7 having HD resolution, the model storage conditions may be determined to store a second reference model (e.g., SA3, SA4, SA5) learned as a high-frequency point of cumulative quality.

[0137] In one embodiment, the second reference model may be stored for each piece of content. The second reference model may be stored for each type of content and / or each resolution of the content. For example, if a user frequently watches channel 7 (#7) and a first OTT app (OTT App #1), the second reference model (e.g., SA1 to SA8) generated for quality processing of videos from channel 7 may be stored as a separate item from the second reference model (e.g., OA1 to OA7) generated for quality processing videos from the first OTT app.

[0138] In one embodiment, the second reference model may be stored by content type and content resolution. The second reference model may be stored by resolution even for the same content. For example, if a user frequently watches channel 7 (#7), a second reference model (e.g., SA8) generated for processing the quality of images (e.g., soccer broadcasts) on channel 7 with UHD resolution may be stored as a separate item from the second reference models (e.g., SA6, SA7) generated for processing images (e.g., news broadcasts) on channel 7 with FHD resolution.

[0139] In one embodiment, the image processing device (100) may store a second reference model corresponding to the cumulative quality according to the type of content and / or the resolution of the content in a non-volatile memory. For example, in the graph (702), the image processing device (100) may store a second reference model (e.g., SA3, SA4, SA5) corresponding to the cumulative quality among a plurality of second reference models generated for processing an image of channel 7 (#7) having an HD resolution in a non-volatile memory. In this case, the cumulative quality of an image of channel 7 (#7) having an HD resolution may have three high-frequency points.

[0140] In one embodiment, the second reference model may be an updated model from the first reference model. For example, the second reference models (e.g., SA3, SA4, SA5) generated for processing images of channel 7 (#7) having HD resolution may be updated models from the first reference models (e.g., S5, S6, S7, S8) previously stored for processing images of channel 7 (#7) having HD resolution. For example, the second reference model may include a model updated from the first reference model (e.g., S7) that is closest to the high-frequency point value of the cumulative quality among the first reference models.

[0141] Meanwhile, in one embodiment, the reference model may further include reference models for each object type. For example, the reference models for each object type may include a face model, a character model, and a general region model excluding faces or characters. For example, a face model that smoothly processes the boundaries of eyes, nose, and mouth, a character model that focuses on character regions for quality processing, and a general region model that sharpens boundaries may each be distinguished. In one example, the general region model may perform quality processing for specific types of landscapes (e.g., oceans, forests, mountains, etc.). Furthermore, in one embodiment, the reference model may include a specific quality model that focuses on specific qualities, such as noise. In one embodiment, a second reference model may also be stored for each object type. For example, if a user frequently views video call content and document editing content, a second reference model generated for processing face objects corresponding to video call content may be stored as a separate item from a second reference model generated for processing character objects corresponding to document editing content.

[0142] FIG. 8 is a diagram showing the configuration of an image processing device according to one embodiment of the present disclosure.

[0143] Referring to FIG. 8, an image processing device (100) according to one embodiment may include a quality analysis unit (810), a model learning unit (820), an image quality processing unit (830), an accumulated quality analysis unit (840), and an additional model selection unit (850). The quality analysis unit (810), the model learning unit (820), the image quality processing unit (830), the accumulated quality analysis unit (840), and the additional model selection unit (850) may be implemented by at least one processor. The quality analysis unit (810), the model learning unit (820), the image quality processing unit (830), the accumulated quality analysis unit (840), and the additional model selection unit (850) may operate according to at least one instruction stored in a memory.

[0144] A quality analysis unit (810) according to one embodiment may analyze or evaluate the image quality or quality of a first image. The first image may be an input image of the image processing device (100). The image quality may indicate the degree of image degradation. The quality analysis unit (810) may evaluate or determine at least one of the compression degradation of the first image, the compression degree of the first image, the degree of blur, the degree of noise, and the resolution of the image.

[0145] According to one embodiment, a quality analysis unit (810) may analyze or evaluate the quality of a first image using a neural network trained to analyze or evaluate the quality of the first image. For example, the neural network may be a neural network trained to evaluate the quality of an image or video using an Image Quality Assessment (IQA) technology, a Video Quality Assessment (VQA) technology, or the like. For example, the neural network may be a neural network trained to input a first image and output a kernel sigma value representing a blur quality of the first image and a Quality Factor (QF) representing a compression quality of the image. The quality of the output first image may be expressed as a quality plane graph such as that illustrated in FIG. 2A.

[0146] According to one embodiment, a quality analysis unit (810) may provide the quality of the first image to a model learning unit (820). The quality analysis unit (810) may provide the quality of the first image to a cumulative quality analysis unit (840) and an additional model selection unit (850).

[0147] According to one embodiment, the model learning unit (820) can perform on-device learning operations. The on-device learning operations can include an operation of adaptively learning a quality processing model for processing the quality of an input image to the input image.

[0148] According to one embodiment, the model learning unit (820) can generate a meta model corresponding to the quality of the first image provided by the quality analysis unit (810). The model learning unit (820) can generate an updated model by learning (transfer learning) the generated meta model using learning data corresponding to the first image. According to one embodiment, the model learning unit (820) can provide the updated model to the quality processing unit (830). The model learning unit (820) can provide the updated model to the additional model selection unit (850). The updated model provided to the quality processing unit (830) can be referred to as a target model for the first image. The updated model provided to the additional model selection unit (850) can be referred to as a second reference model.

[0149] The operation of the model learning unit (820) according to one embodiment to generate a meta model and an updated model will be described in detail later with reference to FIG. 9.

[0150] According to one embodiment, the image quality processing unit (830) may load the updated model from the model learning unit (820) and perform image quality processing of the first image using the updated model. The image quality processing unit (830) may perform image quality processing of the first image to obtain a second image. For example, the image quality processing unit (830) may perform image quality processing of the first image using a neural network trained to process image quality. The neural network may be an inference network that implements a super resolution (SR) algorithm that can convert a low resolution image (Low Resolution, LR) into a high resolution image (High Resolution, HR). The image quality processing unit (830) may perform image quality processing of the first image using the model updated by the model learning unit (820). The image quality processing unit (830) may obtain a second image (high resolution image) by applying the first image to the updated model. As those skilled in the art will understand, super-resolution algorithms can improve image quality by increasing image resolution, allowing more details to be clearly seen. Super-resolution algorithms can be applied to specific regions of an image or to the entire image.

[0151] According to one embodiment, the cumulative quality analysis unit (840) can store the cumulative quality of the input image. The cumulative quality analysis unit (840) can operate independently from the model learning unit (820) and the image quality processing unit (830). The cumulative quality analysis unit (840) can accumulate the quality of content viewed over a medium or long period of time.

[0152] According to one embodiment, the cumulative quality analysis unit (840) may store the cumulative quality of input videos corresponding to frequently viewed content. The cumulative quality analysis unit (840) may obtain information on content with high to low viewing frequency based on the accumulated viewing history of the user over a predetermined period of time. The cumulative quality analysis unit (840) may store the cumulative quality of input videos corresponding to frequently viewed content and may not store the cumulative quality of input videos corresponding to frequently viewed content.

[0153] According to one embodiment, the cumulative quality analysis unit (840) may store the cumulative quality of an input image by type of classification information. For example, the classification information may include at least one of the content type, the content resolution, and a combination of the content type and the content resolution. For example, the cumulative quality of an input image may be accumulated by content type, by content resolution, or by a combination of the content type and the content resolution.

[0154] According to one embodiment, the cumulative quality analysis unit (840) can store the cumulative quality of input images corresponding to frequently viewed content by type of classification information. The cumulative quality of input images corresponding to preferred content is stored by content and can be stored separately by content resolution. The cumulative quality stored by type of classification information for frequently viewed content is described in FIG. 6.

[0155] The cumulative quality analysis unit (840) can provide the cumulative quality of the input image to the additional model selection unit (850).

[0156] An additional model selection unit (850) according to one embodiment may determine a model storage condition based on the cumulative quality received from the cumulative quality analysis unit (840). The model storage condition may refer to a condition of a second reference model to be stored in a non-volatile memory among second reference models updated from a previously stored first reference model. The additional model selection unit (850) may determine the model storage condition based on the cumulative quality of input images accumulated for each frequently viewed content and each resolution of the content. The model storage condition may include at least one of information on frequently viewed content, information on the resolution of the content, high-frequency points of cumulative quality, or the number of models to be stored. The model storage condition may include quality value information corresponding to high-frequency points of cumulative quality. The model storage condition may include information regarding the number of models to be stored corresponding to the number of high-frequency points of cumulative quality.

[0157] According to one embodiment, the additional model selection unit (850) may store a second reference model corresponding to the model storage condition in the second model DB (870). If the additional model selection unit (850) determines that the updated model (second reference model) from the model learning unit (820) corresponds to the model storage condition, the additional model selection unit (850) may store the second reference model in the second model DB (870). For example, the updated model may output an image with a quality that satisfies the storage condition. For example, if the quality of the input image received from the quality analysis unit (810) corresponds to a high-frequency point of cumulative quality, the additional model selection unit (850) may store the second reference model received from the model learning unit (820). The additional model selection unit (850) may store the updated second reference model to process the content with a high viewing frequency based on the cumulative quality corresponding to the content with a high viewing frequency.

[0158] According to one embodiment, the additional model selection unit (850) may store an updated model with quality values ​​corresponding to high-frequency points of accumulated quality stored for frequently viewed content and content resolution. The additional model selection unit (850) may store a second reference model generated for frequently viewed content and content resolution. The second reference model stored for frequently viewed content and content resolution is described in FIG. 7.

[0159] According to one embodiment, the first model DB (860) may be stored in a non-volatile memory of the image processing device (100) or in a database of a server. The first model DB (860) may store a first reference model previously stored in the image processing device (100).

[0160] According to one embodiment, a second model DB (870) may be stored in a non-volatile memory of an image processing device (100) or a database of a server. A second reference model corresponding to model storage conditions may be stored in the second model DB (870).

[0161] According to one embodiment, the model learning unit (820) can generate a meta model based on a first reference model stored in a first model DB (860) and a second reference model stored in a second model DB (870). The model learning unit (820) generates a target model using the first reference model and the second reference model, thereby reducing the time required to generate the target model.

[0162] The quality analysis unit (810), model learning unit (820), image quality processing unit (830), cumulative quality analysis unit (840), and additional model selection unit (850) may be software modules that classify operations performed by the processor by function or purpose, or may be configured as hardware. FIG. 9 is a drawing specifically showing the model learning unit of an image processing device according to one embodiment of the present disclosure.

[0163] Referring to FIG. 9, a model learning unit (820) according to one embodiment may include a learning DB generation unit (910), a meta model acquisition unit (920), and a transfer learning unit (930).

[0164] According to one embodiment, a learning DB generation unit (910) may generate learning data corresponding to a first image based on the quality of the first image received through the first image and quality analysis unit (810). The learning data may include a high-resolution image (correct image, label) and a low-resolution image. The learning DB generation unit (910) may generate a low-resolution image that has been deteriorated from the first image by applying a deterioration process that is the same as the degree of deterioration of the quality of the first image to the first image. The learning DB generation unit (910) may generate learning data that includes a first image used as a correct image and a low-resolution image that has been deteriorated from the first image.

[0165] According to one embodiment, the meta model acquisition unit (920) may use at least one reference model from the first model DB (860) or the second model DB (870) as a meta model. The second reference model stored in the second model DB (870) may be a model updated from the first reference model.

[0166] According to one embodiment, the meta model acquisition unit (920) may identify a model that outputs an image having a quality closest to the quality of the first image among the first reference model or the second reference model, based on the quality of the first image received through the quality analysis unit (810). The meta model acquisition unit (920) may use the reference model identified as the model having the quality closest to the quality of the first image as a meta model.

[0167] Alternatively, the meta-model acquisition unit (920) according to one embodiment may acquire a meta-model by interpolating a plurality of reference models including a first reference model and a second reference model. For example, the meta-model acquisition unit (920) may interpolate a plurality of reference models by applying weights of each reference model to parameters of the plurality of reference models. For example, the meta-model acquisition unit (920) may generate a meta-model by applying weights to each of the plurality of reference models and combining the reference models to which the weights are applied (weighted sum). The method by which the meta-model acquisition unit (920) acquires a meta-model is not limited to the above-described example.

[0168] The meta model acquisition unit (920) can transmit the generated meta model to the transfer learning unit (930).

[0169] The transfer learning unit (930) according to one embodiment can train the meta model using the training data received from the training DB generation unit (910). For example, the transfer learning unit (930) can train the meta model using the gradient descent algorithm. The gradient descent is an optimization algorithm for finding a first-order approximation, which calculates the gradient of a function and continuously moves toward a lower absolute value of the gradient to find the x value when the function value is the minimum value.

[0170] For example, the transfer learning unit (930) inputs a low-resolution image included in the learning data into the meta model, compares the image output from the meta model with the first image included in the learning data, obtains the difference between the two images as the slope of a function, and obtains the parameters of the model when the absolute value of the slope is minimized. The slope of the function may correspond to the partial derivative of the parameters of the model with respect to the difference between the two images. The transfer learning unit (930) can train the meta model by continuously updating the parameters of the meta model so that the quantitative difference between the image output from the meta model and the high-resolution image included in the learning data set is minimized.

[0171] The transfer learning unit (930) according to one embodiment may transmit the updated model to the additional model selection unit (850). If the quality of the input image received through the quality analysis unit (810) corresponds to a high-frequency point of the cumulative quality received through the cumulative quality analysis unit (840), the additional model selection unit (850) may determine that the learning data of the second reference model (updated model) received from the transfer learning unit (930) corresponds to a high-frequency point of the cumulative quality. If the additional model selection unit (850) determines that the learning data of the second reference model corresponds to a high-frequency point of the cumulative quality, the second reference model may be stored in the second model DB (870). The second reference model stored in the second model DB (870) may be used as a meta-model for image quality processing of the first image.

[0172] According to one embodiment, the meta model acquisition unit (920) can acquire a meta model for image quality processing of the first image using the first reference model and the second reference model, so that the generation speed of the meta model can be increased.

[0173] An image processing device (100) according to one embodiment can quickly generate a target model corresponding to the quality of an input image by pre-storing a second reference model corresponding to the quality of content frequently viewed by a user in a non-volatile memory. The image processing device (100) can quickly provide an upscaled image for user-preferred content by increasing the generation speed of a target model having parameters corresponding to real-time quality changes according to real-time quality changes. Fig. 10 is a flowchart illustrating an operation method of an image processing device and a server according to one embodiment of the present disclosure.

[0174] Referring to FIG. 10, the image processing device may include a first image processing device (101) and a second image processing device (102). Each of the first image processing device (101) and the second image processing device (102) may correspond to the image processing device (100) according to one embodiment.

[0175] In operation 1010, the first image processing device (101) can generate a second reference model corresponding to the quality of the input image. For example, the first image processing device (101) can generate an updated model by training the first reference model using learning data corresponding to the quality of the input image.

[0176] In operation 1020, the first image processing device (101) can transmit the second reference model to the server (200) via the communication unit. The first image processing device (101) can transmit the second reference model corresponding to the model storage conditions to the server (200) based on the accumulated quality of the input image.

[0177] In operation 1030, the server (200) may store in memory a second reference model received from the first image processing device (101) via a communication unit. The second reference model may be a quality processing model corresponding to content frequently viewed by a user of the first image processing device (101) and the cumulative quality of the content.

[0178] In operation 1040, the second image processing device (102) may determine model storage conditions corresponding to the cumulative quality of the input image. The second image processing device (102) may determine conditions of a model for processing frequently viewed content based on the cumulative quality of the input image. The model storage conditions may include at least one of information on content with a high viewing frequency of a user of the second image processing device (102), resolution information of the content, high-frequency points of cumulative quality, or the number of models to be stored.

[0179] In operation 1050, the second image processing device (102) can transmit model storage conditions to the server (200) through the communication unit.

[0180] In operation 1060, the server (200) can receive model storage conditions from the second image processing device (102) through the communication unit. The server (200) can transmit a second reference model corresponding to the model storage conditions of the second image processing device (102) to the second image processing device (102). The server (200) can provide the second reference model received from the first image processing device (101) to the second image processing device (102).

[0181] In operation 1070, the second image processing device (102) can generate a target model corresponding to the input image based on the second reference model received from the server (200).

[0182] The quality of input video is likely to be similar across content and regions. For example, cable broadcasters may transmit video with different bitrates depending on the region. For example, video transmitted by a terrestrial broadcaster to a video processing device may have a high bitrate, while video transmitted by a cable broadcaster to a video processing device may have a low bitrate. Furthermore, for example, network environments in the same region are similar, and because of this similarity, the bitrate information of videos transmitted by content providers to video processing devices may be similar.

[0183] In one embodiment, when the first image processing device (101) and the second image processing device (102) receive content from the same region, the reference model required for the first image processing device (101) and the reference model required for the second image processing device (102) are likely to be similar. Therefore, when the second reference model is stored in the server (200) for each content, the second image processing device (102) can use the second reference model created by other users (e.g., users of the first image processing device (101). The second image processing device (102) can use the second reference model created by other users by downloading or periodically updating the second reference model already secured in the server (200). Accordingly, the second image processing device (102) can more quickly learn the target model corresponding to the input image. Fig. 11 is a block diagram of an image processing device according to one embodiment of the present disclosure.

[0184] Referring to FIG. 11, an image processing device (100) according to one embodiment may include a processor (1110), a memory (1120), and a communication unit (1130).

[0185] According to one embodiment, the communication unit (1130) can transmit and receive data or signals with an external device or server. For example, the communication unit (1130) may include a Wi-Fi module, a Bluetooth module, an infrared communication module, a wireless communication module, a LAN module, an Ethernet module, a wired communication module, etc. In this case, each communication module may be implemented in the form of at least one hardware chip.

[0186] The Wi-Fi module and Bluetooth module perform communication in the Wi-Fi and Bluetooth modes, respectively. When using the Wi-Fi module or Bluetooth module, various connection information such as the SSID and session key are first transmitted and received, and after establishing a communication connection using this, various information can be transmitted and received. The wireless communication module may include at least one communication chip that performs communication according to various wireless communication standards such as Zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), and 5G (5th Generation).

[0187] According to one embodiment, the communication unit (1130) can transmit model storage conditions to the server (200). According to one embodiment, the communication unit (1130) can receive a reference model corresponding to the model storage conditions from the server (200).

[0188] According to one embodiment, a processor (1110) controls the overall operation of the image processing device (100) and the signal flow between internal components of the image processing device (100), and performs a function of processing data.

[0189] The processor (1110) may include single cores, dual cores, triple cores, quad cores, and multiples thereof. Furthermore, the processor (1110) may include multiple processors. For example, the processor (1110) may be implemented as a main processor (not shown) and a subprocessor (not shown).

[0190] Additionally, the processor (1110) may include at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a VPU (Video Processing Unit). Alternatively, according to an embodiment, the processor (1110) may be implemented in the form of a SoC (System On Chip) that integrates at least one of a CPU, a GPU, and a VPU. Alternatively, the processor (1110) may further include an NPU (Neural Processing Unit).

[0191] According to one embodiment, the memory (1120) can store various data, programs or applications for driving and controlling the image processing device (100).

[0192] Additionally, the program stored in the memory (1120) may include one or more instructions. The program (one or more instructions) or application stored in the memory (1120) may be executed by the processor (1110).

[0193] A processor (1110) according to one embodiment may include a quality analysis unit (810), a model learning unit (820), and an image quality processing unit (830) of FIG. 8, and may perform operations of the quality analysis unit (810), the model learning unit (820), and the image quality processing unit (830).

[0194] In one embodiment, a processor (1110) stores the cumulative quality of an input image based on the viewing frequency of each content by executing one or more instructions stored in a memory (1120). In one embodiment, the processor (1110) determines a model storage condition based on the viewing frequency and cumulative quality of each content. In one embodiment, the processor (1110) acquires a reference model corresponding to the model storage condition and stores the reference model in the memory. In one embodiment, the processor (1110) trains the stored reference model using learning data corresponding to the quality of the first image, thereby generating a target model corresponding to the first image.

[0195] An image processing device (100) according to one embodiment can quickly generate a target model corresponding to the quality of an input image by pre-storing a reference model corresponding to the quality of content frequently viewed by a user in a non-volatile memory. The image processing device (100) can quickly provide an upscaled image for a user's preferred content by increasing the generation speed of a target model having parameters corresponding to real-time quality changes according to real-time quality changes.

[0196] A model storage condition according to one embodiment may include at least one of content information with a high viewing frequency, resolution information of the content, high-frequency points of cumulative quality, or the number of model storages.

[0197] According to one embodiment, a processor (1110) may store a second reference model in a non-volatile memory by executing one or more instructions stored in a memory (1120), so that the second reference model learned in response to the quality of an input image in response to a model storage condition corresponds to a first reference model stored in advance.

[0198] According to one embodiment, the processor (1110) may store the second reference model in a non-volatile memory by executing one or more instructions stored in the memory (1120), as the quality of the input image corresponds to a high-frequency point of the cumulative quality.

[0199] According to one embodiment, the processor (1110) may store the second reference model in a non-volatile memory by executing one or more instructions stored in the memory (1120), as the second reference model corresponds to a model learned in response to the quality of content with a high viewing frequency.

[0200] According to one embodiment, a processor (1110) may store a reference model by type of classification information by executing one or more instructions stored in a memory (1120). According to one embodiment, the classification information may include at least one of a type of content, a type of OTT content, a type of broadcast channel, a type of game content, a resolution of the content, and a combination of a type of content and a resolution of the content.

[0201] According to one embodiment, the processor (1110) may store the cumulative quality of the input image by type of classification information. According to one embodiment, the classification information may include at least one of the following: type of content, type of OTT content, type of broadcast channel, type of game content, resolution of content, and a combination of type of content and resolution of content.

[0202] A processor (1110) according to one embodiment can store the cumulative quality of an input image corresponding to content with a high viewing frequency by executing one or more instructions stored in a memory (1120).

[0203] According to one embodiment, a processor (1110) can generate a target model by executing one or more instructions stored in a memory (1120), thereby learning in response to the quality of the first image based on a model close to the quality of the first image among a first reference model stored in advance and a second reference model learned from the first reference model.

[0204] According to one embodiment, a processor (1110) can obtain a second image whose quality has been processed from a first image based on a target model by executing one or more instructions stored in a memory (1120).

[0205] According to one embodiment, the processor (1110) may control the communication unit (1130) to transmit model storage conditions to the server (200) by executing one or more instructions stored in the memory (1120). According to one embodiment, the processor (1110) may store the reference model in the memory upon receiving the reference model corresponding to the model storage conditions from the server (200). FIG. 12 is a detailed block diagram of an image processing device according to one embodiment of the present disclosure.

[0206] The image processing device (1200) of FIG. 12 can correspond to the image processing device (100) of FIG. 11.

[0207] Referring to FIG. 12, an image processing device (1200) according to one embodiment may include a tuner unit (1240), a processor (1210), a display unit (1220), a communication unit (1250), a detection unit (1230), an input / output unit (1270), a video processing unit (1280), an audio processing unit (1285), an audio output unit (1260), a memory (1290), and a power supply unit (1295).

[0208] The communication unit (1250) of FIG. 12 may correspond to the communication unit (1130) of FIG. 11, the processor (1210) of FIG. 12 may correspond to the processor (1110) of FIG. 11, and the memory (1290) of FIG. 12 may correspond to the memory (1120) of FIG. 11. Therefore, the same content as described above will be omitted.

[0209] According to one embodiment, the display unit (1220) converts image signals, data signals, OSD signals, control signals, etc. processed by the processor (1210) to generate driving signals. The display unit (1220) may be implemented as a PDP, LCD, OLED, flexible display, etc., and may also be implemented as a 3D display. In addition, the display unit (1220) may be configured as a touch screen and may be used as an input device in addition to an output device.

[0210] A tuner unit (1240) according to one embodiment can select and tune only the frequency of a channel to be received by an image processing device (1200) among many radio wave components through amplification, mixing, resonance, etc. of a broadcast signal received wired or wirelessly. The broadcast signal includes audio, video, and additional information (e.g., EPG (Electronic Program Guide)).

[0211] The tuner unit (1240) can receive broadcast signals from various sources, such as terrestrial broadcasting, cable broadcasting, satellite broadcasting, and Internet broadcasting. The tuner unit (1240) can also receive broadcast signals from sources, such as analog broadcasting or digital broadcasting.

[0212] According to one embodiment, the communication unit (1250) may receive a control signal or a control command, etc. from an external control device. For example, the communication unit (1250) may include an IR module capable of transmitting and receiving signals with the external control device according to the IR communication standard. Specifically, the communication unit (1250) may receive a control signal or a control command, etc. corresponding to a user input (e.g., a key or button input of the control device, etc.) from the control device.

[0213] According to one embodiment, a detection unit (1230) detects a user's voice, a user's image, or a user's interaction, and may include a microphone (1231), a camera unit (1232), and a light receiving unit (1233).

[0214] The microphone (1231) receives the user's spoken voice. The microphone (1231) can convert the received voice into an electrical signal and output it to the processor (1210). The user's voice may include, for example, a voice corresponding to a menu or function of the image processing device (1200). For example, the microphone (1231) can receive a user's voice corresponding to a display rotation command, convert the received voice into an electrical signal, and output it to the processor (1210).

[0215] The camera unit (1232) can receive an image (e.g., a series of frames) corresponding to a user's motion including a gesture within the camera recognition range. The processor (1210) can use the recognition result of the received motion to select a menu displayed on the image processing device (1200) or perform a control corresponding to the motion recognition result. For example, the processor (1210) can receive an image from the camera unit (1232), recognize a user's motion corresponding to the rotation of the display from the received image, and rotate the display in response to the recognition result.

[0216] The optical receiver (1233) receives an optical signal (including a control signal) from an external control device through an optical window (not shown) of a bezel of the display unit (1220). The optical receiver (1233) can receive an optical signal corresponding to a user input (e.g., touch, pressing, touch gesture, voice, or motion) from the control device. A control signal can be extracted from the received optical signal under the control of the processor (1210).

[0217] The input / output unit (1270) according to one embodiment can receive video (e.g., moving picture, etc.), audio (e.g., voice, music, etc.), and additional information (e.g., EPG, etc.) from the outside of the image processing device (1200). The input / output unit (1270) can include any one of a High-Definition Multimedia Interface (HDMI), a Mobile High-Definition Link (MHL), a Universal Serial Bus (USB), a Display Port (DP), a Thunderbolt, a Video Graphics Array (VGA) port, an RGB port, a D-subminiature (D-SUB), a Digital Visual Interface (DVI), a component jack, and a PC port.

[0218] The processor (1210) controls the overall operation of the image processing device (1200) and the signal flow between the internal components of the image processing device (1200), and performs a function of processing data. When there is a user input or a preset stored condition is satisfied, the processor (1210) can execute the OS (Operating System) stored in the memory (1290) and various applications.

[0219] The processor (1210) may include a RAM that stores signals or data input from the outside of the image processing device (1200) or is used as a storage area corresponding to various tasks performed in the image processing device (1200), a ROM that stores a control program for controlling the image processing device (1200), and a processor.

[0220] The video processing unit (1280) performs processing on video data received by the image processing device (1200). The video processing unit (1280) can perform various image processing operations, such as decoding, scaling, noise filtering, frame rate conversion, and resolution conversion on the video data.

[0221] The audio processing unit (1285) processes audio data. The audio processing unit (1285) may perform various processing operations, such as decoding, amplification, and noise filtering, on audio data. Meanwhile, the audio processing unit (1285) may include multiple audio processing modules to process audio corresponding to multiple contents.

[0222] The audio output unit (1260) outputs audio included in a broadcast signal received through the tuner unit (1240) under the control of the processor (1210). The audio output unit (1260) can output audio (e.g., voice, sound) input through the communication unit (1250) or the input / output unit (1270). In addition, the audio output unit (1260) can output audio stored in the memory (1290) under the control of the processor (1210). The audio output unit (1260) can include at least one of a speaker, a headphone output terminal, or an S / PDIF (Sony / Philips Digital Interface:) output terminal.

[0223] The power supply unit (1295) supplies power input from an external power source to components inside the image processing device (1200) under the control of the processor (1210). In addition, the power supply unit (1295) can supply power output from one or more batteries (not shown) located inside the image processing device (1200) to the internal components under the control of the processor (1210).

[0224] The memory (1290) can store various data, programs, or applications for driving and controlling the image processing device (1200) under the control of the processor (1210). The memory (1290) can include a broadcast reception module (not shown), a channel control module, a volume control module, a communication control module, a voice recognition module, a motion recognition module, an optical reception module, a display control module, an audio control module, an external input control module, a power control module, a power control module for an external device connected wirelessly (e.g., Bluetooth), a voice database (DB), or a motion database (DB). The modules and database of the memory (1290), not shown, can be implemented in the form of software to perform a broadcast reception control function, a channel control function, a volume control function, a communication control function, a voice recognition function, a motion recognition function, an optical reception control function, a display control function, an audio control function, an external input control function, a power control function, or a power control function for an external device connected wirelessly (e.g., Bluetooth). The processor (1210) can perform each function using the software stored in the memory (1290).

[0225] Meanwhile, the block diagrams of the image processing device (100, 1200) illustrated in FIGS. 11 and 12 are block diagrams for one embodiment. Each component of the block diagrams may be integrated, added, or omitted depending on the specifications of the image processing device (100, 1200) actually implemented. That is, two or more components may be combined into one component, or one component may be subdivided into two or more components, as needed. In addition, the functions performed by each block are for explaining an embodiment, and the specific operations or devices thereof do not limit the scope of the present invention.

[0226] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0227] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

Claims

1. In the image processing device (100), A memory (1120) storing one or more instructions; and comprising one or more processors (1100) coupled to the above memory (1120) and including a processing circuit; The image processing device (100) executes the one or more instructions individually or collectively by the one or more processors (1100), Based on the viewing frequency of the content, the cumulative quality of the content including multiple input images is stored, Based on the above viewing frequency and the above cumulative quality, the model storage conditions are determined, Obtain a reference model corresponding to the above model storage conditions, Store the above reference model in the above memory (1120), An image processing device (100) that generates a target model corresponding to the first image by training a reference model stored in the memory (1120) using learning data corresponding to the quality of the first image.

2. In paragraph 1, The above model storage condition includes at least one of content information with a high viewing frequency, resolution information of the content, high-frequency points of the cumulative quality, or the number of model storages. High-viewing frequency content refers to content that has been viewed a certain number of times within a certain interval. An image processing device (100) in which a high frequency point of the above cumulative quality represents the largest quality value included in the above cumulative quality.

3. In paragraph 1 or 2, The above memory (1120) is a non-volatile memory, By executing one or more of the above instructions individually or in combination, the image processing device (100) additionally, Generate a second reference model learned from a first reference model stored in advance in response to the quality of the above multiple input images, An image processing device (100) that stores the second reference model in the memory (1120) as the second reference model corresponds to the model storage condition.

4. In paragraph 3, By executing one or more of the above instructions individually or in combination, the image processing device (100) additionally, An image processing device (100) that stores the second reference model in the memory (1120) based on the high frequency points of the above cumulative quality.

5. In clause 3 or 4, By executing one or more of the above instructions individually or in combination, the image processing device (100) additionally, An image processing device (100) that stores the second reference model learned in response to the quality of the content with a high viewing frequency in the memory (1120).

6. In paragraph 1, By executing one or more of the above instructions individually or in combination, the image processing device (100) additionally, Save the above reference model by type of classification information, An image processing device (100), wherein the above classification information includes at least one of the type of content, the type of OTT content, the type of broadcast channel, the type of game content, the resolution of the content, and a combination of the type of content and the resolution of the content.

7. In any one of paragraphs 1 to 6, By executing one or more of the above instructions individually or in combination, the image processing device (100) additionally, The cumulative quality of the above content is stored by type of classification information, An image processing device (100), wherein the above classification information includes at least one of the type of content, the type of OTT content, the type of broadcast channel, the type of game content, the resolution of the content, and a combination of the type of content and the resolution of the content.

8. In any one of paragraphs 1 to 7, By executing one or more of the above instructions individually or in combination, the image processing device (100) additionally, Store the cumulative quality of multiple input videos corresponding to content with high viewing frequency, A video processing device (100) in which content with a high viewing frequency indicates content that has been viewed a set number of times or more within a set interval.

9. In any one of paragraphs 1 to 8, By executing one or more of the above instructions individually or in combination, the image processing device (100) additionally, Identifying a model that outputs an image having a quality close to that of the first image among the first reference model that has been previously stored and the second reference model learned from the first reference model, An image processing device (100) that generates the target model by training the identified model in response to the quality of the first image.

10. In any one of paragraphs 1 to 9, By executing one or more of the above instructions individually or in combination, the image processing device (100) additionally, An image processing device (100) that obtains a second image whose image quality has been processed from the first image based on the target model.

11. In any one of paragraphs 1 to 10, Including the Communication Department (1130), By executing one or more of the above instructions individually or in combination, the image processing device (100) additionally, Controlling the communication unit (1130) to transmit the above model storage conditions to the server, An image processing device (100) that receives a reference model corresponding to the model storage conditions from the server and stores the reference model in the memory (1120).

12. In the operating method of the image processing device (100), A step (310) of storing the cumulative quality of content including multiple input images based on the viewing frequency of the content; A step (320) of determining model storage conditions based on the above viewing frequency and the above accumulated quality; A step of obtaining a reference model corresponding to the above model storage conditions; Step (330) of storing the above reference model in memory (1120); and A method comprising a step (340) of generating a target model corresponding to the first image by training a reference model stored in the memory (1120) using learning data corresponding to the quality of the first image.

13. In paragraph 12, The above model storage condition includes at least one of content information with a high viewing frequency, resolution information of the content, high-frequency points of the cumulative quality, or the number of model storages. High-viewing frequency content refers to content that has been viewed a certain number of times within a certain interval. A method wherein the high frequency point of the above cumulative quality represents the largest quality value included in the above cumulative quality.

14. In clause 12 or 13, The above memory (1120) is a non-volatile memory, The step (330) of storing the above reference model in memory (1120) is A step of generating a second reference model learned from a first reference model previously stored in the quality of the plurality of input images; and A method comprising the step of storing the second reference model in the memory (1120) as the second reference model corresponds to the model storage condition.

15. In any one of paragraphs 12 to 14, The step (340) of generating a target model corresponding to the first image is as follows: A step of identifying a model that outputs an image having a quality close to that of the first image among a first reference model that has been previously stored and a second reference model learned from the first reference model; and A method comprising the step of generating the target model by learning the identified model in response to the quality of the first image.

Images