Server and operating method thereof, and image processing device and operating method thereof

Abstract

This server comprises: a communication interface; a memory for storing one or more instructions; and at least one processor which is operatively connected to the memory and the communication interface, and which executes one or more instructions so as to control the communication interface such that: a first image is acquired by compressing an original image; quality information about the first image is acquired by analyzing the image quality of the first image; meta model information about a meta model used for image quality processing for the first image is acquired on the basis of the quality information; and the meta model information and the first image are transmitted to an image processing device displaying a second image on the basis of the meta model information and the first image.

Description

Server and its operating method, image processing device and its operating method

[0001] The present disclosure relates to a server that transmits a low-quality image to an image processing device and an operating method thereof, and an image processing device that processes a low-quality image received from a server to output a high-quality image and an operating method thereof.

[0002] With the advancement of deep learning technology, various learning-based upscaling methods are being developed. Learning-based upscaling methods exhibit excellent performance when the quality characteristics of the training images are similar to those of the input images being processed. However, if the characteristics of the image to be processed differ from the input quality assumed during training, image quality deteriorates significantly.

[0003] To address these issues, research is underway on on-device learning, which processes and adapts AI models based on input data. In the field of on-device learning, a paper on image processing and image quality improvement (ZSSR, CVPR 2018, Zero-Shot Super-Resolution using Deep Internal Learning, hereinafter referred to as Paper 1) was recently published.

[0004] Paper 1, called ZSSR (Zero-Shot Super Resolution), is a technology that constructs a database (DB) using input images tailored to the deterioration characteristics of the input image, and then uses a model trained using the DB to enlarge the image. However, ZSSR has the problem of high learning complexity and difficulty in applying it to videos with significant image quality fluctuations because it creates a new database from scratch for each input image and trains the model using this database.

[0005] To improve these problems, another paper (MetaSR, ECCV 2021, Fast Adaptation to Super-Resolution Networks via Meta-Learning, hereinafter referred to as Paper 2) was published. Paper 2 is a technology that learns an initial meta-model from an external database and finds a model that fits the characteristics of the input image through transfer learning to reduce the learning computational complexity of ZSSR. However, since the technology of Paper 2 uses only one meta-model, there are performance limitations in including all the characteristics of various input images in one meta-model. In addition, in environments that use low-capacity networks such as edge devices, these limitations of the meta-model become a factor that limits the performance of on-device learning.

[0006] Papers 1 and 2 construct a learning database based on input images, demonstrating improved image quality for still images containing buildings with repetitive contours or periodic textures. However, in reality, many images, in addition to those assumed by existing methods, are degraded during the shooting, transmission, and compression processes. These images often lose high-frequency components that serve as hints for image quality restoration, and repetitive elements within the images are often difficult to detect. Therefore, constructing a learning database based solely on images presents limitations, resulting in poor performance.

[0007] Furthermore, existing methods were developed to improve the image quality of still images, making them difficult to apply to video. Models trained independently for each image can exhibit variations in restoration performance due to differences in learning convergence and the characteristics of the training database. Consequently, applying independent models to each frame can lead to variations in image clarity, potentially resulting in flicker distortion, a temporal unevenness in image quality.

[0008] A server according to one embodiment may include a communication interface, a memory storing one or more instructions, and at least one processor operably connected to the memory and the communication interface and configured to execute the one or more instructions.

[0009] The at least one processor can obtain a first image by compressing the original image by executing the one or more instructions.

[0010] The at least one processor can analyze the image quality of the first image by executing the one or more instructions, thereby obtaining quality information of the first image.

[0011] The at least one processor can obtain meta model information for a meta model used for image quality processing of the first image based on the quality information by executing the one or more instructions.

[0012] The at least one processor can control the communication interface to transmit the meta model information and the first image to an image processing device that displays a second image based on the meta model information and the first image by executing the one or more instructions.

[0013] An image processing device according to one embodiment may include a communication interface, a display, a memory storing one or more instructions, and at least one processor operably connected to the memory, the display, and the communication interface and executing the one or more instructions.

[0014] The at least one processor can control the communication interface to receive first image and meta model information from the server by executing the one or more instructions.

[0015] The at least one processor can generate a meta model based on the meta model information by executing the one or more instructions.

[0016] The at least one processor may train the meta model using a learning data set corresponding to the first image to obtain a trained meta model by executing the one or more instructions.

[0017] The at least one processor can obtain a second image obtained by processing the first image based on the trained meta model by executing the one or more instructions.

[0018] The at least one processor can control the display to display the second image by executing the one or more instructions.

[0019] A method of operating a server according to one embodiment may include a step of obtaining a first image by compressing an original image.

[0020] The operating method of the above server may include a step of analyzing the image quality of the first image and obtaining quality information of the first image.

[0021] The operating method of the above server may include a step of obtaining meta model information for a meta model used for image quality processing of the first image based on the quality information.

[0022] The operating method of the above server may include a step of transmitting the meta model information and the first image to an image processing device that displays a second image based on the meta model information and the first image.

[0023] The above and other aspects, features, and advantages of specific embodiments of the present disclosure will become more apparent through the following description with reference to the drawings.

[0024] FIG. 1 is a diagram illustrating an image quality processing system according to one or more embodiments of the present disclosure.

[0025] FIG. 2 is a diagram illustrating quality information of various images according to one or more embodiments of the present disclosure in a graph.

[0026] FIG. 3 is a flowchart illustrating a method of operating a server according to one or more embodiments of the present disclosure.

[0027] FIG. 4 is a diagram illustrating a configuration of a server according to one or more embodiments of the present disclosure.

[0028] FIG. 5 is a diagram illustrating the structure of a first neural network according to one or more embodiments of the present disclosure.

[0029] FIG. 6 is a graph showing quality values ​​for a plurality of frame images included in video content according to one or more embodiments of the present disclosure.

[0030] FIG. 7 is a diagram illustrating how a meta model is obtained based on at least one reference model according to one or more embodiments of the present disclosure.

[0031] FIG. 8 is a flowchart illustrating an operation method of an image processing device according to one or more embodiments of the present disclosure.

[0032] FIG. 9 is a diagram showing the configuration of an image processing device according to one or more embodiments of the present disclosure.

[0033] FIG. 10 is a diagram for explaining the operation of a model learning unit according to one or more embodiments of the present disclosure.

[0034] FIG. 11 is a diagram for explaining an operation of a learning DB generation unit according to one or more embodiments of the present disclosure to acquire images of a category similar to a first image.

[0035] FIG. 12 is a diagram for explaining an operation of a learning DB generation unit generating a learning data set according to one or more embodiments of the present disclosure.

[0036] FIG. 13 is a diagram illustrating a method for reflecting deterioration occurring during a compression process according to one or more embodiments of the present disclosure into a learning image.

[0037] FIG. 14 is a block diagram illustrating a configuration of a server according to one or more embodiments of the present disclosure.

[0038] FIG. 15 is a block diagram showing the configuration of an image processing device according to one or more embodiments of the present disclosure.

[0039] FIG. 16 is a block diagram showing the configuration of an image processing device according to one or more embodiments of the present disclosure.

[0040] In this disclosure, the expression “at least one of a, b, or c” may refer to “a,” “b,” “c,” “a and b,” “a and c,” “b and c,” “all of a, b, and c,” or variations thereof. The terms used in this specification will be briefly described, and the present invention will be described in detail.

[0041] The terms used in this invention have been selected from widely used, current terms, taking into account the functions of the invention. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, in which case their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this invention should not be defined simply as names, but rather based on their inherent meanings and the overall content of the invention.

[0042] When a part of the specification is said to "include" a component, this does not exclude other components, but rather implies the inclusion of other components, unless otherwise specifically stated. Furthermore, terms such as "part," "module," etc., used throughout the specification refer to a unit that processes at least one function or operation, which may be implemented in hardware, software, or a combination of hardware and software.

[0043] Below, with reference to the attached drawings, embodiments of the present invention are described in detail so that those skilled in the art can easily implement the present invention. However, the present invention can be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, parts irrelevant to the description have been omitted to clearly explain the present invention, and similar parts have been designated with similar reference numerals throughout the specification.

[0044] In one or more embodiments of this specification, the term “user” means a person who controls a system, function, or operation, and may include a developer, administrator, or installer.

[0045] Additionally, in one or more embodiments of the present specification, an 'image' or 'picture' may represent a still image, a moving image composed of a plurality of consecutive still images (or frames), or a video.

[0046] FIG. 1 is a diagram illustrating an image quality processing system according to one or more embodiments of the present disclosure.

[0047] Referring to FIG. 1, a server (100) according to one or more embodiments may be a content provider that provides content to an image processing device (200). A content provider may refer to a terrestrial broadcaster, cable broadcaster, OTT service provider, or IPTV service provider that provides various content to consumers. However, the present invention is not limited thereto.

[0048] A server (100) according to one or more embodiments of the present disclosure may transmit various contents, such as video contents, image contents, audio contents, and text contents, to an image processing device (200). For example, the server (100) may stream image data to an image processing device (200) so that the image processing device (200) may play the image data in real time. In addition to image data, the server (100) may stream various types of data, such as audio data and text data, to the image processing device (200).

[0049] A server (100) according to one or more embodiments of the present disclosure may provide video content, such as various movies or dramas, through television programs or VOD services to an image processing device (200). The video content may include multiple frame images.

[0050] Video content may be captured by a capture device, compressed and transmitted by a server (100) according to one or more embodiments of the present disclosure, and restored and output by an image processing device (200). At this time, the video content may have various qualities depending on deterioration that occurs during the process of acquiring, transmitting, and storing the video content.

[0051] In one or more embodiments of the present disclosure, the server (100) may compress image content (video content). For example, the server (100) may obtain and store multiple compressed image data having different resolutions or different bit rates for the same image content.

[0052] In one or more embodiments of the present disclosure, the server (100) may transmit a compressed first image (101) to an image processing device (200).

[0053] In one or more embodiments of the present disclosure, the image processing device (200) may be an electronic device capable of processing and outputting an image. The image processing device (200) according to one or more embodiments of the present disclosure may be implemented in various forms including a display. For example, the image processing device (200) may be implemented in various electronic devices such as a TV, a mobile phone, a tablet PC, a digital camera, a camcorder, a laptop computer, a desktop, an e-book reader, a digital broadcasting terminal, a PDA (Personal Digital Assistant), a PMP (Portable Multimedia Player), a navigation device, an MP3 player, a wearable device, and the like.

[0054] An image processing device (200) according to one or more embodiments may obtain a second image (102, output image) by performing image quality processing on a first image (101, input image) received from a server (100). For example, the image processing device (200) may obtain a high-resolution (or high-quality) output image by upscaling a low-resolution (or low-quality) input image using an image quality processing model.

[0055] A quality processing model according to one or more embodiments may include a model that is trained using a learning data set corresponding to the first image (101), and an acquired meta model based on the quality of the first image (101).

[0056] A server (100) according to one or more embodiments may obtain quality information of a compressed first image (101) and, based on the quality information of the first image (101), obtain meta model information. The server (100) may transmit the meta model information and the compressed first image to an image processing device (200).

[0057] An image processing device (200) according to one or more embodiments may generate a meta model based on meta model information received from a server (100) and train the meta model using a learning data set corresponding to a first image (101). The image processing device (200) may process the image quality of the first image (101) using the trained meta model, thereby obtaining a high-quality (or high-resolution) second image (102).

[0058] Hereinafter, with reference to the drawings, the operations of the server (100) and the image processing device (200) according to one embodiment will be described in detail.

[0059] FIG. 2 is a diagram illustrating quality information of various images according to one or more embodiments of the present disclosure in a graph.

[0060] Figure 2 is a graph that shows the results of analyzing the image quality of various images as a quality plane, and shows the results of analyzing the quality of videos captured through a image quality analyzer as a two-dimensional graph.

[0061] Referring to FIG. 2, the horizontal axis of the graph (201) represents the Kernel Sigma value, and the vertical axis represents the compression quality (Quality Factor, QF) of the image. The Kernel Sigma value represents the blur quality of the image. The larger the Kernel Sigma value, the greater the blur, and the smaller the Kernel Sigma value, the less the blur. QF represents the degree of deterioration due to compression. The smaller the QF value, the more severe the deterioration due to compression, and the larger the QF value, the less deterioration due to compression.

[0062] In graph (201), symbols of the same shape represent the quality values ​​of images with the same resolution. As illustrated in graph (201), even for images with the same resolution and symbols, the quality values ​​of the images may be distributed in various ways. This is because even for images with the same resolution, the images may have various qualities depending on the deterioration that occurs during the acquisition, transmission, and storage of the images.

[0063] Therefore, adaptive image quality processing (image processing) is required based on the image quality information. The server (100) according to one or more embodiments can obtain a meta model adaptive to the first image based on the quality information of the first image. The image processing device (200) according to one or more embodiments can generate an image quality processing model adaptive to the first image based on the meta model obtained from the server (100) and the quality information of the first image, and perform image quality processing of the first image using the image quality processing model. Accordingly, the image quality processing performance of the first image can be improved.

[0064] FIG. 3 is a flowchart illustrating a method of operating a server according to one or more embodiments of the present disclosure.

[0065] Referring to FIG. 3, a server (100) according to one or more embodiments can compress an original image to obtain a first image (S310).

[0066] According to one embodiment, a server (100) can obtain and store multiple compressed image data having different resolutions or different bit rates for the same original image.

[0067] For example, compressed video data may include video data with resolutions of 8k, 4k, FHD, and HD for the same content. Alternatively, compressed video data may include multiple compressed video data with bit rates of 40Mbps, 30Mbps, 20Mbps, and 10Mbps for the same original video. However, this is not limited thereto.

[0068] A server (100) according to one or more embodiments may determine, among a plurality of compressed image data, image data (first image) to be transmitted to the image processing device (200) according to the state of a network connected to the image processing device (200) according to one or more embodiments. The state of the network may be determined, for example, according to the degree of traffic generation in the transmission / reception path between the server (100) and the image processing device (200), but is not limited thereto.

[0069] Alternatively, the server (100) may determine image data (first image) to be transmitted to the image processing device (200) from among a plurality of compressed image data at the request of the image processing device.

[0070] A server (100) according to one or more embodiments can analyze the image quality of a first image and obtain quality information (S320).

[0071] For example, the server (100) can determine the image quality or picture quality of the first image. The image quality can indicate the degree of image degradation. The server (100) can evaluate or determine at least one of the compression degradation of the first image, the sharpness degree, the degree of blur, the degree of noise, and the resolution of the image.

[0072] According to one or more embodiments, the server (100) may analyze or evaluate the image quality of a first image using a first neural network trained to analyze or evaluate the image quality of the first image. For example, the first neural network may be a neural network trained to evaluate the image quality of an image or video using an Image Quality Assessment (IQA) technique, a Video Quality Assessment (VQA) technique, or the like.

[0073] For example, the server (100) may obtain a kernel sigma value indicating the blur quality of an image and a quality factor (QF) indicating the compression quality of an image as quality information using the first neural network. However, the present invention is not limited thereto.

[0074] According to one or more embodiments, the server (100) may obtain quality information of the first image based on quality information for each of the plurality of frame images, when the first image is one of a plurality of frame images included in the video content.

[0075] For example, the server (100) may determine whether the quality value of the first image is abnormal based on the quality values ​​of each of the plurality of frame images, and if so, may not use the corresponding value. The server (100) may determine whether the quality value of the first image is abnormal based on the average value and standard deviation of the quality values ​​of the plurality of frame images. This will be described in detail later with reference to FIG. 6.

[0076] The server (100) according to one or more embodiments can obtain meta model information (S330) based on the quality information obtained in step 320 (S320).

[0077] For example, the server (100) can obtain a meta model using at least one reference model. The reference model is a quality processing model that has been previously learned using training images, and may include first reference models stored in the image processing device (200) and second reference models stored in the server (100).

[0078] The server (100) according to one or more embodiments may generate second reference models for quality processing of the first image. For example, the server (100) may generate the second reference models by training a quality processing model using training images having similar quality information to the first image.

[0079] Alternatively, the first reference models that have been learned can be stored in the image processing device (200) using learning images having various quality information.

[0080] A server (100) according to one or more embodiments may compare quality information of learning images corresponding to each of a plurality of reference models with quality information of a first image, and search for a reference model corresponding to a learning image having quality information similar to quality information of the first image.

[0081] A server (100) according to one or more embodiments may search for a plurality of reference models, apply weights to each of the plurality of reference models, and perform a weighted sum of the plurality of reference models to obtain a meta-model. In this case, the weights applied to each of the plurality of reference models may be determined based on the difference between the quality information of the training image corresponding to the reference model and the quality information of the first image.

[0082] Meta model information according to one or more embodiments may include information on at least one reference model for obtaining a meta model, weight information applied to at least one reference model, an obtained meta model, etc.

[0083] For example, information about at least one reference model may include identification information of the first reference model when the meta model is obtained using the first reference model. Alternatively, information about at least one reference model may include identification information of the second reference model when the meta model is obtained using the second reference model.

[0084] A server (100) according to one or more embodiments may transmit meta model information and a first image to an image processing device (200) (S340).

[0085] For example, when the server (100) obtains a meta model based on the second reference models, it may transmit the obtained meta model or the second reference models to the image processing device (200). Alternatively, when the server (100) obtains a meta model based on the first reference models, it may transmit identification information (e.g., model number, etc.) for the first reference models to the image processing device (200).

[0086] Additionally, the server (100) can transmit weight information applied to each of a plurality of reference models for obtaining a meta model to the image processing device (200).

[0087] Additionally, the server (100) can transmit quality information about the first image acquired from the server to the image processing device (200).

[0088] Figure 4 is a diagram showing the configuration of a server according to one embodiment.

[0089] Referring to FIG. 4, a server (100) according to one or more embodiments may include an image compression unit (410), an image quality analysis unit (420), and a meta model acquisition unit (430).

[0090] The image compression unit (410) according to one or more embodiments may obtain and store a plurality of compressed image data having different resolutions or different bit rates for an original image. For example, the compressed image data may include image data having resolutions of 8k, 4k, FHD, and HD for the same content. Alternatively, the compressed image data may include a plurality of compressed image data having bit rates of 40Mbps, 30Mbps, 20Mbps, and 10Mbps for the same original image. However, the present invention is not limited thereto.

[0091] Depending on the state of the network connected between the server (100) and the image processing device (200) according to one or more embodiments, the image data (first image) to be transmitted to the image processing device (200) among the plurality of compressed image data may be determined. Alternatively, based on a request from the image processing device (200), the image data (first image) to be transmitted to the image processing device (200) may be determined.

[0092] The image quality analysis unit (420) according to one or more embodiments may analyze or evaluate the image quality or quality of the first image to be transmitted to the image processing device (200). The image quality may indicate the degree of image degradation. The image quality analysis unit (420) may evaluate or determine at least one of compression degradation of the first image, sharpness degree, blur degree, noise degree, and image resolution of the first image.

[0093] The image quality analysis unit (420) according to one or more embodiments may analyze or evaluate the image quality of the first image using a first neural network trained to analyze or evaluate the image quality of the first image. For example, the first neural network may be a neural network trained to evaluate the image quality of an image or video using an Image Quality Assessment (IQA) technology, a Video Quality Assessment (VQA) technology, or the like.

[0094] For example, the first neural network may be a neural network trained to input a first image and output a kernel sigma value representing the blur quality of the first image and a quality factor (QF) representing the compression quality of the image.

[0095] Referring to FIG. 5, a first neural network according to one or more embodiments will be described in detail.

[0096] FIG. 5 is a diagram illustrating the structure of a first neural network according to one or more embodiments.

[0097] Referring to FIG. 5, the first neural network (500) may be a neural network based on a CNN (Convolution Neural Network), a DCNN (Deep Convolution Neural Network), or a Capsnet.

[0098] For example, the first neural network (500) can be trained to receive various data as input and discover or acquire on its own a method for analyzing the input data, a method for classifying the input data, and / or a method for extracting features necessary for generating result data from the input data. The first neural network (500) can be created into an artificial intelligence model with desired characteristics by applying a learning algorithm to a plurality of learning data. This learning can be performed on the server (100) itself or through a separate server / system.

[0099] Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, and the learning algorithms in the embodiments are not limited to the above-described examples unless otherwise specified.

[0100] For example, the first neural network (500) can be trained as a data inference model through supervised learning that uses learning data as input values. Alternatively, the first neural network (500) can be trained as a data inference model through unsupervised learning that discovers criteria for judging the image quality by independently learning the types of data necessary to judge the image quality without any guidance. Alternatively, the first neural network (500) can be trained as a data inference model through reinforcement learning that utilizes feedback on whether the result of inferring the image quality according to learning is correct.

[0101] In one or more embodiments, the first neural network (500) may include an input layer, a hidden layer, and an output layer. In one or more embodiments, the hidden layer may include multiple hidden layers. The first neural network (500) may be a deep neural network (DNN) including two or more hidden layers. A deep neural network (DNN) is a neural network that performs operations through multiple layers, and the depth of the network may increase depending on the number of internal layers that perform operations. The deep neural network (DNN) operation may include a convolutional neural network (CNN) operation, etc.

[0102] The first neural network (500) according to one or more embodiments may be trained using a training database that includes each training image and quality values ​​corresponding to each training image as a training data set. For example, the training data set may include degraded images and quality values ​​of the degraded images generated by compressing, blurring, or adding noise to high-resolution images in various ways. That is, the first neural network (500) may be trained so that when a degraded image is input to the first neural network (500), the quality values ​​of the degraded image are output.

[0103] For example, a first image including R, G, and B channels (RGB 3ch) may be input to the first neural network (500) illustrated in FIG. 5. Alternatively, a first image including Y, U, and V channels (YUV 3ch) may be input.

[0104] A first neural network (500) according to one or more embodiments may receive a first image including R, G, B channels (RGB 3ch) or Y, U, V channels (YUV 3ch), and perform a convolution operation by applying one or more kernels or filters to the first image to extract a feature map. For example, the first neural network (500) may apply 32 3X3 filters to the first image to output 32 channels. The first neural network (500) may generate a result value by multiplying and summing weight values ​​included in a kernel while scanning the target of the convolution operation from left to right and from top to bottom one pixel at a time. The data to be the target of the convolution operation may be scanned while moving one pixel at a time, or may be scanned while moving two pixels or more. The number of pixels through which the input data moves during the scanning process is called the stride, and the size of the output feature map can be determined based on the size of the stride.

[0105] A first neural network (500) according to one or more embodiments may have a Single Input Multi Output structure in which two types of quality values ​​are output for one input image.

[0106] In addition, the first neural network (500) according to one or more embodiments may have a structure in which intermediate layers for extracting features are used in common to reduce the complexity of the network and the output is separated at the last stage to output image quality elements.

[0107] A first neural network (500) according to one or more embodiments may obtain a 128-channel vector through pooling and convert it into a 256-channel vector through a linear network. Thereafter, the first neural network (500) may reduce the 256-channel vector to one dimension to obtain a final result. In one or more embodiments, the first neural network (500) may output a quality value of an input image as two defined quality values.

[0108] The first neural network (500) according to one or more embodiments may output blur sigma (or Kernel Sigma) and compression quality (Compression QF) as result values. Kernel Sigma and QF may implicitly express the deterioration of a compressed image. However, the present invention is not limited thereto, and the first neural network according to one or more embodiments may obtain various types of quality values ​​as results in addition to blur sigma (or Kernel Sigma) and compression quality (Compression QF) values.

[0109] The image quality analysis unit (420) according to one or more embodiments may obtain a statistically stable quality value for the first image based on quality values ​​for some or all of the content including the first image. For example, if the content is video content, the content may include multiple frame images, and the first image may be any one of the multiple frame images.

[0110] The image quality analysis unit (420) according to one or more embodiments may obtain a quality value for each of a plurality of frame images. The image quality analysis unit (420) may determine a quality value of the first image based on the quality values ​​of the plurality of frame images. This will be described with reference to FIG. 6.

[0111] FIG. 6 is a graph showing quality values ​​for a plurality of frame images included in video content according to one or more embodiments.

[0112] The x-axis of the graph in Fig. 6 represents the number of multiple frame images, and the y-axis represents the quality value of the frame images (e.g., blur quality value).

[0113] The image quality analysis unit (420) according to one or more embodiments may determine a variable range of a quality value by setting an upper bound and a lower bound of the quality value based on an average value and a standard deviation of blur quality values ​​of a plurality of frame images. If the quality value of the first image is greater than the upper bound or less than the lower bound, the image quality analysis unit (420) may not use the quality value and may replace it with another value. For example, the image quality analysis unit (420) may use an average value of the quality values ​​of the plurality of frame images as the quality value of the first image. Alternatively, the image quality analysis unit (420) may use a quality value calculated based on a quality value of a previous frame image of the first image and a quality value of a subsequent frame image of the first image as the quality value of the first image.

[0114] However, it is not limited to this, and the image quality analysis unit (420) can perform image quality analysis on the entire content in various ways, and if the quality value of the first image is determined to be abnormal, the quality value of the first image can be corrected.

[0115] The image quality analysis unit (420) of the server (100) according to one or more embodiments performs image quality analysis on the entire content including a plurality of frame images, and when there is an abnormality in the quality value of one of the plurality of frame images, the quality value is corrected, thereby improving the accuracy of the image quality analysis.

[0116] Additionally, the server (100) can transmit statistics such as the average quality value or quality value variable range of the content to the image processing device (200).

[0117] Again, referring to FIG. 3, the meta model acquisition unit (430) can generate meta model information for quality processing of the first image based on the quality information of the first image acquired from the quality analysis unit (420).

[0118] For example, the meta model acquisition unit (430) can acquire a meta model using at least one reference model. The reference model is a quality processing model that has been previously learned using training images, and may include first reference models stored in the image processing device (200) and second reference models stored in the server (100). The first reference models that have been previously learned using training images having various quality information may be stored in the image processing device (200).

[0119] The meta-model acquisition unit (430) according to one or more embodiments may generate second reference models for quality processing of the first image. For example, the meta-model acquisition unit (430) may generate the second reference models by training a quality processing model using training images having similar quality information as the first image.

[0120] A method for obtaining a meta model using quality information of a first image and at least one reference model by a meta model obtaining unit (430) according to one or more embodiments will be described below with reference to FIG. 7.

[0121] FIG. 7 is a diagram illustrating how a meta model is obtained based on at least one reference model according to one or more embodiments.

[0122] A meta model for image quality processing of a first image according to one or more embodiments can be generated based on a reference model, which is a pre-learned image quality processing model.

[0123] The graph in Fig. 7 represents a quality plane graph. A quality plane graph is a graph that represents the quality of an image with two quality factors, where the horizontal axis represents quality factor 1 and the vertical axis represents quality factor 2.

[0124] A quality plane graph according to one or more embodiments may include N points in a grid form. For example, training images corresponding to each of the N points may include training images degraded to have the corresponding quality. That is, training images having a quality value corresponding to the coordinate value of each point in the graph of FIG. 7 may refer to training images. For example, a first point (Pt1) may refer to training images having the coordinate value (x1, y1) of the first point as a quality value, and a second point (Pt2) may refer to training images having the coordinate value (x2, y1) of the second point as a quality value.

[0125] A device for generating a reference model (reference model generating device) can generate training images for each of N points by degrading training images before degradation. The reference model generating device can generate the reference model by training an image quality processing model using a training data set including training images before degradation and training images after degradation. For example, the reference model generating device can generate reference model 1 corresponding to the first point (Pt1) by training an image quality processing model using training images obtained by degrading the training images before degradation and the training images before degradation to have a quality value of the first point (Pt1) as a training data set.

[0126] Reference Model 1 may be a quality processing model trained to restore post-deterioration training images corresponding to the quality value of the first point (Pt1) to pre-deterioration training images. Furthermore, Reference Model 1 may be a model suitable for processing the quality of images having the quality value of the first point (Pt1).

[0127] In addition, the reference model generation device can generate a reference model 2 corresponding to the second point (Pt2) by training the image quality processing model using the pre-deterioration learning images and the learning images obtained by deteriorating the pre-deterioration learning images to have a quality value of the second point (Pt2) as a learning data set.

[0128] Reference Model 2 may be a quality processing model trained to restore post-deterioration training images corresponding to the quality value of the second point (Pt2) to pre-deterioration training images. Furthermore, Reference Model 2 may be a model suitable for processing the quality of images having the quality value of the second point (Pt2).

[0129] A reference model generation device according to one embodiment can generate a reference model corresponding to each of N points in a grid shape as illustrated in FIG. 7. The reference model generation device can determine the positions of quality values ​​of target learning images as the positions of each of the N points in a grid shape through uniform sampling, thereby allowing the N reference models to be trained using learning images having quality values ​​at uniform intervals.

[0130] However, the present invention is not limited thereto, and the device for generating a reference model may determine a quality value corresponding to the reference model based on the distribution of quality values ​​of the training images. For example, the device for generating a reference model may analyze training images to obtain quality values ​​of the training images, and determine a representative quality sampling location through the statistical distribution of the quality values ​​of the training images. For example, the device for generating a reference model may determine a representative sampling location using a K-means clustering algorithm. This method is an algorithm that finds a point with the minimum error when the distribution of data is expressed as K representative points.

[0131] The reference model generation device can group the quality value distribution of training images into a predetermined number of clusters, for example, K, and determine the quality value that minimizes the variance of the distance difference in each cluster. The reference model generation device can train the reference model using training images after image quality deterioration with the determined quality values ​​and the corresponding training images before image quality deterioration as a training data set. In this case, since the reference model can be trained using training images with high statistical quality values, the number of reference models can be reduced. In addition, the reference model obtained in this way can be more usable when generating a meta model in the future. In addition, when generating a meta model using the reference model obtained in this way, computational complexity and memory usage can be reduced.

[0132] The learned reference models may be stored in a server or image processing device according to one or more embodiments.

[0133] Meanwhile, the server (100) according to one or more embodiments may generate reference models using the same method as described above. For example, the server (100) may generate second reference models for performing quality processing of the first image. The server (100) may generate the second reference models by training the quality processing model using training images having similar quality information to the first image. The second reference models may be quality processing models trained so that post-degradation training images having similar quality information to the first image are restored to pre-degradation training images. However, the present invention is not limited thereto.

[0134] The meta model acquisition unit (430) according to one or more embodiments can compare the quality information of learning images corresponding to each of a plurality of reference models with the quality information of the first image, and search for a reference model corresponding to a learning image having similar quality information to the quality information of the first image.

[0135] The meta model acquisition unit (430) can search for a reference model trained with training images having quality values ​​close to the quality value of the first image. For example, as illustrated in FIG. 7, the meta model acquisition unit (430) can calculate the difference between the quality value of the first image and the quality values ​​of the training images used to train the reference models as a distance, and search for a reference model that is close in order of distance.

[0136] In addition, the meta model acquisition unit (430) can search for reference models learned with training images having quality values ​​whose difference from the quality value of the first image is within a reference value among a plurality of reference models. Alternatively, the meta model acquisition unit (430) can search for reference models learned with training images having quality values ​​of a preset number in order of the smallest difference (closest distance) from the quality value of the first image among a plurality of reference models.

[0137] As illustrated in FIG. 7, when the first image has a quality value at a point indicated by a star shape, the meta model acquisition unit (430) can search for a reference model learned with learning images having a quality value close to the quality value of the first image, i.e., a point close to the star shape on the graph of FIG. 7.

[0138] The meta model acquisition unit (430) can search for or select reference model 1 corresponding to the first point (Pt1) close to the star shape, reference model 2 corresponding to the second point (Pt2), reference model 3 corresponding to the third point (Pt3), and reference model 4 corresponding to the fourth point (Pt4).

[0139] The meta-model acquisition unit (430) according to one or more embodiments may acquire a meta-model by interpolating a plurality of searched reference models. Interpolating a plurality of reference models may mean interpolating parameters of known reference models and using the interpolated parameters as parameters of the meta-model. The meta-model acquisition unit (430) may acquire a weight (interpolation coefficient) by using the difference between the quality value of the first image and the quality value corresponding to the reference model (the distance between the position of the star shape and the position of the quality value corresponding to the reference model).

[0140] For example, the meta model acquisition unit (430) can acquire a meta model by interpolating reference models as in the following mathematical expression 1.

[0141]

[0142] [Mathematical Formula 1]

[0143] Metamodel = W1 * Reference Model 1 + W2 * Reference Model 2 + … + WN * Reference Model N

[0144] Here, reference models 1 to N may represent parameters of the reference model. W1 to WN are weights applied to each reference model, and the weights may be determined as values ​​inversely proportional to the difference between the quality values ​​corresponding to the selected reference models and the quality values ​​of the first image (the distance between the position of the star shape and the position of the quality value corresponding to the reference model). In addition, the sum of the weights may be 1. However, the present invention is not limited thereto, and the meta-model acquisition unit (430) may use various methods to interpolate a plurality of reference models to acquire a meta-model. For example, the meta-model acquisition unit (430) may acquire a meta-model from a reference model using various interpolation methods such as linear interpolation, spline interpolation, cubic interpolation, bilinear interpolation that extends linear interpolation to two dimensions, and bicubic interpolation that extends cubic interpolation to two dimensions.

[0145] The meta model acquisition unit (430) according to one or more embodiments may transmit information about the acquired meta model (meta model information) and the first image to the image processing device (200). In addition, the meta model acquisition unit (430) may transmit quality information of the first image to the image processing device (200).

[0146] Meta model information according to one or more embodiments may include information on at least one reference model for obtaining a meta model, weight information applied to at least one reference model, an obtained meta model, etc.

[0147] For example, when a meta model is acquired using a first reference model, the meta model information may include identification information of the first reference model and weight information applied to the first reference model. Alternatively, when a meta model is acquired using a second reference model, the meta model information may include second reference model information and weight information applied to the second reference model.

[0148] Meta model information and quality information of the first image according to one or more embodiments may be transmitted to the image processing device (200) together with identification information of the first image or time information (e.g., a time stamp) of the first image.

[0149] FIG. 8 is a flowchart illustrating an operation method of an image processing device according to one or more embodiments of the present disclosure.

[0150] Referring to FIG. 8, an image processing device (200) according to one or more embodiments can receive a first image and meta model information (S810).

[0151] An image processing device (200) according to one or more embodiments can generate a meta model based on meta model information (S820).

[0152] The image processing device (200) according to one or more embodiments may load a first reference model previously stored in the image processing device (200) corresponding to the identification information of the first reference model when the meta model information includes identification information of the first reference model. In addition, the image processing device (200) may store the second reference model or meta model when the meta model information includes information about the second reference model or meta model in the image processing device (200).

[0153] The image processing device (200) can generate a meta model by applying weights included in the meta model information to the first reference model or the second reference model. For example, the image processing device (200) can generate a meta model by applying weights to each of a plurality of reference models and combining the reference models to which the weights have been applied (weighted sum). However, the present invention is not limited thereto.

[0154] An image processing device (200) according to one or more embodiments can train a meta model generated in step 820 (S820) using a learning data set corresponding to a first image (S830).

[0155] For example, the image processing device (200) can use the first image to acquire a learning data set corresponding to the first image. The image processing device (200) can identify a category of the first image and select images belonging to the identified category from learning data stored in a database. The learning data may include high-definition images. The learning data may be stored in an external database or in the internal memory of the image processing device (200).

[0156] An image processing device (200) according to one or more embodiments may generate an image with degraded image quality by performing at least one of the following methods: compressing, blurring, adding noise, or down-sampling the selected learning images.

[0157] An image processing device (200) according to one or more embodiments may train a meta model by using selected training images and images with degraded image quality obtained by processing the training images for quality as a training data set. For example, the image processing device (200) may input images with degraded image quality included in a training data set into a meta model, and update parameters of the meta model so that the difference between the images output from the meta model and high-quality training images included in the training data set is minimized.

[0158] An image processing device (200) according to one or more embodiments can obtain a second image obtained by processing the quality of a first image based on a learned meta model (S840).

[0159] The learned meta-model (updated model) according to one or more embodiments may include a neural network trained to process the quality of an input image. For example, the learned meta-model may be a model trained to process a low-resolution or low-quality image into a high-resolution or high-quality image. In one embodiment, the second image may be a higher-resolution or higher-quality image than the first image.

[0160] FIG. 9 is a diagram showing the configuration of an image processing device according to one or more embodiments of the present disclosure.

[0161] Referring to FIG. 9, an image processing device (200) according to one or more embodiments may include an image quality processing unit (910) and a model learning unit (920).

[0162] The image quality processing unit (910) can perform image quality processing on the first image (101) received from the server (100) to obtain a second image (102). For example, the image quality processing unit (910) can perform image quality processing on the first image using a second neural network trained to process image quality. The second neural network can be an inference network that implements a super resolution (SR) algorithm that can convert a low resolution image (Low Resolution, LR) into a high resolution image (High Resolution, HR). The image quality processing unit (910) can obtain a high resolution image (second image) by processing the image quality of the first image using SR technology using deep learning.

[0163] The model learning unit (920) according to one or more embodiments may perform on-device learning operations. The on-device learning operations may include an operation of adaptively learning a quality processing model for processing the quality of an input image to an input image.

[0164] The model learning unit (920) can generate a meta model to be applied to the first image based on the meta model information received from the server (100). The model learning unit (920) can generate an updated model by learning (transfer learning) the generated meta model using a learning data set corresponding to the first image.

[0165] The operation of the model learning unit (920) according to one or more embodiments to generate a meta model and an updated model will be described in detail later with reference to FIG. 10.

[0166] The image quality processing unit (910) can load the updated model generated by the model learning unit (920) and perform image quality processing of the first image using the updated model.

[0167] The on-device learning operation according to one or more embodiments may be activated (operation ON) or deactivated (operation OFF). Whether the on-device learning operation according to one embodiment is activated or deactivated may vary depending on the model specifications, capacity, performance, etc. of the image processing device (200). For example, if the image processing device (200) has a built-in large-capacity memory and a high-performance CPU, the image processing device (200) may activate the on-device learning operation and perform image quality processing of the first image using an image quality processing model that is appropriately updated for the first image received from the server (100). Alternatively, the image processing device (200) may determine whether to activate the on-device learning operation based on a user input.

[0168] The model learning unit (920) according to one or more embodiments may adaptively learn an image quality processing model for the first image based on the activation of the on-device learning operation, thereby obtaining an updated model. The image quality processing unit (910) may perform image quality processing on the first image using the updated model obtained by the model learning unit (920), thereby obtaining a second image.

[0169] If the on-device learning operation according to one or more embodiments is deactivated, the model learning unit (920) may not acquire an updated model. The image quality processing unit (910) may perform image quality processing of the first image using any one of the previously stored reference models.

[0170] Alternatively, the model learning unit (920) may obtain a meta model based on meta model information according to one or more embodiments, but may omit the process of learning (transfer learning) using a learning data set corresponding to the first image. The image quality processing unit (910) may perform image quality processing of the first image using the meta model.

[0171] FIG. 10 is a diagram for explaining the operation of a model learning unit according to one or more embodiments.

[0172] Referring to FIG. 10, a model learning unit (920) according to one or more embodiments may include a data parsing unit (1010), a model storage unit (1020), a meta model generation unit (1030), a transfer learning unit (1040), and a learning database (DB) generation unit (1050).

[0173] A model learning unit (920) according to one or more embodiments may receive a first image and meta model information. The meta model information may be input to a data parsing unit (1010).

[0174] The data parsing unit (1010) can distinguish different data by parsing the meta model information received from the server (100). For example, the data parsing unit (1010) can distinguish the identification information of the first reference model, information about the second reference model, meta model information, weight information, etc. included in the meta model information.

[0175] The data parsing unit (1010) according to one or more embodiments may transmit the second reference model information to the second model storage unit (1022) when the meta model information includes information about the second reference model. Alternatively, the data parsing unit (1010) may store the meta model in the second model storage unit (1022) when the meta model information includes the meta model.

[0176] The model learning unit (920) according to one or more embodiments may include a model storage unit (1020) for storing reference models, a first model storage unit (1021) for storing previously learned reference models, and a second model storage unit (1022) for storing reference models or meta-models received from the server (100). However, the present invention is not limited thereto, and the second model storage unit (1022) may not be included depending on the model specifications, capacity, performance, etc. of the image processing device (200).

[0177] The data parsing unit (1010) according to one or more embodiments may, if the meta model information includes identification information of the first reference model, transmit the identification information of the first reference model to the first model storage unit (1021). The first model storage unit (1021) may load the first reference model corresponding to the received identification information among the previously stored first reference models into the meta model generation unit (1030).

[0178] The data parsing unit (1010) according to one or more embodiments may transmit weight information applied to a reference model included in the meta model information to the meta model generation unit (1030).

[0179] A meta model generation unit (1030) according to one or more embodiments may generate a meta model based on at least one reference model loaded from a first model storage unit (1021) or a second model storage unit (1022) and weight information applied to at least one reference model.

[0180] For example, the meta model generation unit (1030) can generate a meta model by applying a weight to each of a plurality of reference models and combining the reference models to which the weights are applied (weighted sum).

[0181] Alternatively, the meta model generation unit (1030) may obtain the meta model from the second model storage unit (1022).

[0182] The meta model generation unit (1030) can transmit the generated meta model to the transfer learning unit (1040).

[0183] The transfer learning unit (1040) according to one or more embodiments can train the meta model using a learning data set received from the learning DB generation unit (1050).

[0184] The learning DB generation unit (1050) according to one or more embodiments may obtain a learning data set corresponding to the first image using the first image. To this end, the learning DB generation unit (1050) may identify a category of the first image. For example, the learning DB generation unit (1050) may analyze the first image and identify the category of the first image using a probability value. The learning DB generation unit (1050) may identify the category with the highest probability value as the category of the first projection, and select images belonging to the identified category from the learning data stored in the database (1060). The learning data may include high-definition images. The learning data may be stored in an external database or in an internal memory.

[0185] The learning DB generation unit (1050) according to one or more embodiments may acquire a preset number of images from among images belonging to the same category as the first image. Alternatively, the learning DB generation unit (1050) may identify a preset number of categories in the order of highest probability value among the categories of the first image, and acquire images belonging to the identified categories in proportion to the probability value. For example, if the learning DB generation unit (1050) determines that the probability that an object included in the first image is a dog is 70% and that the probability that it is a cat is 30%, the learning DB generation unit (1050) may acquire dog images and cat images from among the learning data at a ratio of 7:3.

[0186] The learning DB generation unit (1050) according to one or more embodiments can degrade images belonging to an identified category to obtain images with degraded image quality. The learning DB generation unit (1050) can process the image quality of images belonging to the identified category so that it corresponds to the image quality of the first image. For example, the learning DB generation unit (1050) can generate images with degraded image quality by performing at least one of compression degrading, blurring, adding noise, or down-sampling on images belonging to the identified category.

[0187] The learning DB generation unit (1050) according to one or more embodiments can use an image belonging to an identified category and an image with deteriorated image quality obtained by processing the image quality as a learning data set.

[0188] According to one embodiment, the learning DB generation unit (1050) can transmit the learning data set to the transfer learning unit (1040).

[0189] According to one embodiment, the transfer learning unit (1040) can learn the meta model using the learning data set received from the learning DB generation unit (1050).

[0190] For example, the transfer learning unit (1040) can train the meta model using the gradient descent algorithm. The gradient descent is an optimization algorithm for finding a first-order approximation, which calculates the gradient of a function and continuously moves toward a lower absolute value of the gradient to find the x value when the function value is the minimum value.

[0191] The transfer learning unit (1040) according to one or more embodiments inputs a degraded image included in a learning data set into a meta model, compares an image output from the meta model with an image belonging to an identified category included in the learning data set, obtains the difference between the two images as a slope of a function, and obtains the parameters of the model when the absolute value of the slope is minimized. That is, the transfer learning unit (1040) can train the meta model by continuously updating the parameters of the meta model so that the quantitative difference between the image output from the meta model and the high-quality image included in the learning data set is minimized.

[0192] The transfer learning unit (1040) according to one or more embodiments can stabilize the meta model by considering cases where abrupt changes in image quality occur when applying different meta models to multiple images. For example, the transfer learning unit (1040) can adjust the performance deviation of the updated meta model for each frame image. The transfer learning unit (1040) can stabilize the meta model by using a method of averaging the parameters of the meta models of the frame images. The transfer learning unit (1040) can average the meta models using a simple moving average or an exponential moving average method. However, the present invention is not limited thereto.

[0193] FIG. 11 is a diagram for explaining an operation of a learning DB generation unit according to one or more embodiments of the present disclosure to acquire images of a category similar to a first image.

[0194] Referring to FIG. 11, the learning DB generation unit (1050) can selectively collect images having similar characteristics to the content characteristics of the first image from an external database (1120) to generate a learning DB.

[0195] The learning DB generation unit (1050) according to one or more embodiments can identify the category to which the first image belongs.

[0196] The learning DB generation unit (1050) according to one or more embodiments may identify the category of the first image using the third neural network (1110). The third neural network (1110) according to one embodiment may be an algorithm that receives an image and classifies the category of the image from the input image, or a set of algorithms, software that executes the set of algorithms, and / or hardware that executes the set of algorithms.

[0197] The third neural network (1110) according to one or more embodiments may utilize a softmax regression function to obtain various classes or categories as results. The softmax function may be used when there are multiple correct answers (classes) to be classified, i.e., when predicting multiple classes. The softmax function may receive a k-dimensional vector as input and estimate the probability for each class, when the total number of classes is k. The third neural network (1110) according to one embodiment may be a neural network trained to receive a k-dimensional vector as input and make the probability for each class obtained therefrom the same as the correct answer set. However, the present invention is not limited thereto, and the third neural network (1110) may be implemented with various types of algorithms that can classify the category of an image from an input image.

[0198] The third neural network (1110) according to one or more embodiments may obtain, as a result, a probability value for the category or class of the input first image. For example, the third neural network (1110) may obtain, as a result value, a vector expressing the probability that the category of the first image is a human face, a dog, a cat, and a building as 0.5, 0.2, 0.2, and 0.1, respectively.

[0199] The learning DB generation unit (1050) according to one or more embodiments may identify the category with the highest probability value as the category of the first image. For example, in the above example, the learning DB generation unit (1050) may identify the category of the first image as a human face, which is the category with the highest vector value.

[0200] The learning DB generation unit (1050) according to one or more embodiments may obtain images having similar content characteristics to the first image, i.e., images included in the same category or a similar category as the first image. The learning DB generation unit (1050) according to one or more embodiments may obtain images included in a similar category to the first image from an external database (1120). However, the present invention is not limited thereto, and the learning DB generation unit (1050) may also obtain images included in a similar category to the first image from among learning images stored in the internal memory of the image processing device (200), rather than from the external database (1120).

[0201] In one or more embodiments, an external database (1120) or internal memory may store images having various types of categories, labeled with an index or tag for each category of the images.

[0202] Referring to FIG. 11, the learning DB creation unit (1050) can acquire one or more images identified by an index of a category similar to the first image from an external database (1120) and create a new database (1130) including these images.

[0203] The learning DB generation unit (1050) according to one or more embodiments may identify only the category with the highest probability value among the categories of the first image and acquire images belonging to the identified category. For example, in the above example, the learning DB generation unit (1050) may acquire only the facial image of the person with the highest probability value from the external database (1120).

[0204] Alternatively, the learning DB generation unit (1050) may identify only a predetermined number of categories in the order of high probability values ​​among the categories of the first image, and obtain images belonging to the identified categories in proportion to the probability values. For example, in the above example, the learning DB generation unit (1050) may identify only three categories in the order of high probability values. For example, the learning DB generation unit (1050) may identify human faces, dogs, and cats as categories of the input images, and obtain human face images, dog images, and cat images from the external database (1120) at a ratio of 5:2:2, respectively.

[0205] The learning DB generation unit (1050) according to one or more embodiments may also include the first image in the new database (1130).

[0206] Figure 11 assumes that the learning DB generation unit (1050) has identified a human face, which is the category with the highest probability value, as the category of the input image. The learning DB generation unit (1050) can acquire N different human face images from an external database (1120) and create a new database (1130) containing them. Here, the N human face images may be different images.

[0207] FIG. 12 is a diagram for explaining an operation of a learning DB generation unit generating a learning data set according to one or more embodiments of the present disclosure.

[0208] Referring to FIG. 12, the learning DB generation unit (1050) according to one or more embodiments may degrade the image quality of images included in the new database (1130).

[0209] The learning DB generation unit (1050) according to one or more embodiments can degrade the images included in the new database (1130) to match the quality characteristics of the first image.

[0210] The learning DB generation unit (1050) according to one or more embodiments may receive quality information of a first image and degrade the collected images to have corresponding quality information. In this case, the quality information of the first image may be quality information obtained by analyzing or evaluating the image quality of the first image by the server (100). Alternatively, it may be quality information obtained by analyzing or evaluating the image quality of the first image by the image processing device (200) according to one embodiment.

[0211] Quality information of the first image according to one or more embodiments may include Kernel Sigma, which indicates the degree of blurring of the first image, and QF, which indicates the degree of compression degradation of the first image. The learning DB generation unit (1050) may degrade images included in the new database (1130) so that they have the same degree of blurring and compression degradation as the first image.

[0212] For example, the learning DB generation unit (1050) may perform filtering to degrade images included in the new database (1130). The learning DB generation unit (1050) may use a two-dimensional kernel to apply blur degradation to images. Alternatively, the learning DB generation unit (1050) may process box blur to model motion degradation. Alternatively, the learning DB generation unit (1050) may use a Gaussian filter to apply shape or optical blur.

[0213] The learning DB generation unit (1050) according to one or more embodiments can adjust the coefficients of the filter according to the blur of the first image.

[0214] Degradation is performed through a commonly known spatial filtering method, which can be similar to the operation of a low-pass filter in signal processing. Specifically, degradation can be performed through a convolution operation with a 2D Gaussian kernel.

[0215] The learning DB generation unit (1050) according to one or more embodiments can use the images included in the new database (1130) and the deteriorated images (1210) generated by deteriorating the image quality of the images as a learning data set.

[0216] FIG. 13 is a diagram illustrating a method for reflecting deterioration occurring during a compression process according to one or more embodiments of the present disclosure into a learning image.

[0217] Multimedia data, including video, is massive in volume and requires a large bandwidth for transmission. For example, uncompressed video with a resolution of 4K or higher requires a high bandwidth that makes medium- to long-distance transmission impossible. A 4K 60FPS uncompressed video with a resolution of 3840x2160, the standard resolution for UHD broadcasting, requires a very high bandwidth of 11,384 Bbps per second. To transmit such large amounts of data, it is essential to encode the video using a compression coding technique. Video can be compressed using various compression formats. For example, video can be compressed using various compression formats such as JPEG, MPEG2, H.264, and HEVC. During the compression process, information may be lost, which may cause distortion.

[0218] Encoded video data is generated in a specific format defined by each video codec and transmitted to a decoding device, which decodes the video sequence and outputs the video data. Compressed video may degrade due to information loss during the decoding process when the video is restored.

[0219] The learning DB generation unit (1050) according to one or more embodiments can generate a compressed deteriorated image by reflecting deterioration occurring during the compression process among various types of deterioration in the learning image.

[0220] The learning DB generation unit (1050) according to one or more embodiments may generate a compressed degraded image by compressing and degrading images included in the new database (1130). To this end, the learning DB generation unit (1050) may generate a compressed degraded image by encoding / decoding images included in the new database (1130). For example, the learning DB generation unit (1050) may degraded still images by using a JPEG compression method for still images. In addition, the learning DB generation unit (1050) may degraded videos by using a compression method such as MPEG2, H.264, or HEVC for videos.

[0221] Figure 13 sequentially illustrates the JPEG encoding and decoding process for an image. Referring to Figure 13, raw image data can be encoded into a JPEG compressed image through color conversion, frequency conversion (DCT), quantization, and arithmetic coding, in that order. The encoded image can be restored through decoding, dequantization, inverse DCT, and inverse color conversion processes.

[0222] The learning DB generation unit (1050) according to one or more embodiments can obtain a compressed degraded image by JPEG encoding and decoding the image to be degraded in the order shown in FIG. 13.

[0223] Since the entropy coding performed during the encoding / decoding process is a lossless compression method, no quality degradation occurs during the entropy coding and entropy decoding processes. Therefore, the learning DB generation unit (1050) according to one embodiment can obtain a compressed degraded image by performing only the methods indicated by reference numeral 1310, omitting the entropy coding and entropy decoding processes for the image to be degraded.

[0224] A learning DB generation unit (1050) according to one or more embodiments can position an image to be degraded in the position of raw image data, perform color conversion, frequency conversion (DCT), and quantization on the image to be degraded, and perform dequantization, inverse transformation (Inverse DCT), and inverse color transformation on the quantized image to obtain a compressed degraded image.

[0225] FIG. 14 is a block diagram illustrating a configuration of a server according to one or more embodiments of the present disclosure.

[0226] Referring to FIG. 14, a server according to one or more embodiments may include a communication unit (110, e.g., a communication interface), a processor (120), and a memory (130).

[0227] The communication unit (110) according to one or more embodiments may transmit and receive data or signals with an external device or server. For example, the communication unit (110) may include a Wi-Fi module, a Bluetooth module, an infrared communication module, a wireless communication module, a LAN module, an Ethernet module, a wired communication module, etc. In this case, each communication module may be implemented in the form of at least one hardware chip.

[0228] The Wi-Fi module and Bluetooth module perform communication in the Wi-Fi and Bluetooth modes, respectively. When using the Wi-Fi module or Bluetooth module, various connection information such as the SSID and session key are first transmitted and received, and after establishing a communication connection using this, various information can be transmitted and received. The wireless communication module may include at least one communication chip that performs communication according to various wireless communication standards such as Zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), and 5G (5th Generation).

[0229] A communication unit (110) according to one or more embodiments may receive a request for a first image from an image processing device (200). In addition, the communication unit (110) according to one embodiment may transmit the first image, meta model information, and quality information of the first image to the image processing device (200) according to one or more embodiments.

[0230] A processor (120) according to one or more embodiments controls the overall operation of the server (100) and the signal flow between internal components of the server (100), and performs a function of processing data.

[0231] The processor (120) may include single cores, dual cores, triple cores, quad cores, and multiples thereof. Furthermore, the processor (120) may include multiple processors. For example, the processor (120) may be implemented as a main processor (not shown) and a subprocessor (not shown).

[0232] Additionally, the processor (120) may include at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a VPU (Video Processing Unit). Alternatively, according to an embodiment, the processor (120) may be implemented in the form of a SoC (System On Chip) that integrates at least one of a CPU, a GPU, and a VPU. Alternatively, the processor (120) may further include an NPU (Neural Processing Unit).

[0233] The memory (130) according to one or more embodiments may store various data, programs or applications for driving and controlling the server (100).

[0234] Additionally, the program stored in the memory (130) may include one or more instructions. The program (one or more instructions) or application stored in the memory (130) may be executed by the processor (120).

[0235] A processor (120) according to one or more embodiments may include an image compression unit (410), an image quality analysis unit (420), and a meta model acquisition unit (430) of FIG. 4, and may perform operations of the image compression unit (410), the image quality analysis unit (420), and the meta model acquisition unit (430).

[0236] The processor (120) according to one or more embodiments may obtain a first image by compressing an original image by executing one or more instructions stored in the memory (130). The processor (120) may obtain a plurality of compressed image data having different resolutions or different bit rates for the same original image and store the obtained compressed image data in the memory (130). For example, the compressed image data may include image data having a resolution of 8k, 4k, FHD, or HD for the same content. Alternatively, the compressed image data may include a plurality of compressed image data having bit rates of 40Mbps, 30Mbps, 20Mbps, or 10Mbps for the same original image. However, the present invention is not limited thereto.

[0237] A processor (120) according to one or more embodiments may determine, among a plurality of compressed image data, image data (first image) to be transmitted to the image processing device (200) according to the state of a network connected to the image processing device (200) according to one embodiment. The state of the network may be determined, for example, according to the degree of traffic generation in the transmission / reception path between the server (100) and the image processing device (200), but is not limited thereto.

[0238] Alternatively, the processor (120) may determine image data (first image) to be transmitted to the image processing device (200) from among a plurality of compressed image data at the request of the image processing device.

[0239] The processor (120) according to one or more embodiments may analyze the image quality of the first image and obtain quality information by executing one or more instructions stored in the memory (130). For example, the processor (120) may determine the image quality or quality of the first image. The image quality of the image may indicate the degree of image deterioration. The processor (120) may evaluate or determine at least one of compression deterioration of the first image, sharpness degree, blur degree, noise degree, and image resolution of the first image.

[0240] The processor (120) according to one or more embodiments may analyze or evaluate the image quality of a first image using a first neural network trained to analyze or evaluate the image quality of a first image. For example, the first neural network may be a neural network trained to evaluate the image quality of an image or video using an Image Quality Assessment (IQA) technique, a Video Quality Assessment (VQA) technique, or the like.

[0241] For example, the processor (120) may obtain a kernel sigma value indicating the blur quality of an image and a quality factor (QF) indicating the compression quality of an image as quality information using the first neural network. However, the present invention is not limited thereto.

[0242] According to one or more embodiments, the processor (120) may obtain quality information of the first image based on quality information for each of the plurality of frame images, when the first image is one of a plurality of frame images included in the video content.

[0243] For example, the processor (120) may determine whether the quality value of the first image is abnormal based on the quality values ​​of each of the plurality of frame images, and if so, may not use the corresponding value. The processor (120) may determine whether the quality value of the first image is abnormal based on the average value and standard deviation of the quality values ​​of the plurality of frame images. This has been described in detail with reference to FIG. 6, and thus a detailed description thereof will be omitted.

[0244] A processor (120) according to one or more embodiments may obtain meta model information based on the obtained quality information by executing one or more instructions stored in the memory (130).

[0245] For example, the processor (120) can obtain a meta model using at least one reference model. The reference model is a quality processing model that has been previously learned using a learning image, and may include first reference models stored in the image processing device (200) and second reference models stored in the server (100).

[0246] The processor (120) according to one or more embodiments may generate second reference models for quality processing of the first image. For example, the processor (120) may generate the second reference models by training a quality processing model using training images having similar quality information as the first image.

[0247] Alternatively, the first reference models that have been learned can be stored in the image processing device (200) using learning images having various quality information.

[0248] A processor (120) according to one or more embodiments may compare quality information of learning images corresponding to each of a plurality of reference models with quality information of a first image, and search for a reference model corresponding to a learning image having quality information similar to the quality information of the first image.

[0249] A processor (120) according to one or more embodiments may search for a plurality of reference models, apply a weight to each of the plurality of reference models, and perform a weighted sum of the plurality of reference models to obtain a meta model. In this case, the weight applied to each of the plurality of reference models may be determined based on the difference between the quality information of the training image corresponding to the reference model and the quality information of the first image.

[0250] Meta model information according to one or more embodiments may include information on at least one reference model for obtaining a meta model, weight information applied to at least one reference model, an obtained meta model, etc.

[0251] For example, information about at least one reference model may include identification information of the first reference model when the meta model is obtained using the first reference model. Alternatively, information about at least one reference model may include identification information of the second reference model when the meta model is obtained using the second reference model.

[0252] A processor (120) according to one or more embodiments may control a communication unit (110) to transmit meta model information and a first image to an image processing device (200) by executing one or more instructions stored in the memory (130).

[0253] For example, when the communication unit (110) obtains a meta model based on the second reference models, it may transmit the obtained meta model or the second reference models to the image processing device (200). Alternatively, when the communication unit (110) obtains a meta model based on the first reference models, it may transmit identification information (e.g., model number, etc.) for the first reference models to the image processing device (200).

[0254] Additionally, the communication unit (110) can transmit weight information applied to each of a plurality of reference models for obtaining a meta model to the image processing device (200).

[0255] Additionally, the communication unit (110) can transmit quality information about the first image acquired from the server to the image processing device (200).

[0256] FIG. 15 is a block diagram showing the configuration of an image processing device according to one or more embodiments of the present disclosure.

[0257] Referring to FIG. 15, an image processing device (200) according to one or more embodiments may include a communication unit (210), a processor (220), a memory (230), and a display (240).

[0258] The communication unit (210) according to one or more embodiments may transmit and receive data or signals with an external device or server. For example, the communication unit (210) may include a Wi-Fi module, a Bluetooth module, an infrared communication module, a wireless communication module, a LAN module, an Ethernet module, a wired communication module, etc. In this case, each communication module may be implemented in the form of at least one hardware chip.

[0259] The Wi-Fi module and Bluetooth module perform communication in the Wi-Fi and Bluetooth modes, respectively. When using the Wi-Fi module or Bluetooth module, various connection information such as the SSID and session key are first transmitted and received, and after establishing a communication connection using this, various information can be transmitted and received. The wireless communication module may include at least one communication chip that performs communication according to various wireless communication standards such as Zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), and 5G (5th Generation).

[0260] The communication unit (210) according to one or more embodiments may request a first image (image content or video content) from the server (100). In addition, the communication unit (210) according to one embodiment may receive the first image, meta model information, and quality information of the first image from the server (100).

[0261] A processor (220) according to one or more embodiments controls the overall operation of the image processing device (200) and the signal flow between internal components of the image processing device (200), and performs a function of processing data.

[0262] The processor (220) may include single cores, dual cores, triple cores, quad cores, and multiples thereof. Furthermore, the processor (220) may include multiple processors. For example, the processor (220) may be implemented as a main processor and a sub processor.

[0263] Additionally, the processor (220) may include at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a VPU (Video Processing Unit). Alternatively, according to an embodiment, the processor (220) may be implemented in the form of a SoC (System On Chip) that integrates at least one of a CPU, a GPU, and a VPU. Alternatively, the processor (220) may further include an NPU (Neural Processing Unit).

[0264] The memory (230) according to one or more embodiments may store various data, programs or applications for driving and controlling the image processing device (200).

[0265] Additionally, the program stored in the memory (230) may include one or more instructions. The program (one or more instructions) or application stored in the memory (230) may be executed by the processor (220).

[0266] A processor (220) according to one or more embodiments may include an image quality processing unit (910) and a model learning unit (920) of FIG. 9, and may perform operations of the image quality processing unit (910) and the model learning unit (920).

[0267] The processor (220) according to one or more embodiments may generate a meta model based on meta model information received from the server (100) by executing one or more instructions stored in the memory (130). For example, if the meta model information includes identification information of a first reference model, the processor (220) may load a first reference model previously stored in the memory (230) corresponding to the identification information of the first reference model. In addition, if the meta model information includes information about a second reference model or meta model, the processor (220) may store the second reference model or meta model in the memory (230).

[0268] The processor (220) can generate a meta model by applying weights included in the meta model information to the first reference model or the second reference model. For example, the processor (220) can generate a meta model by applying weights to each of a plurality of reference models and combining the weighted reference models (weighted sum). However, the present invention is not limited thereto.

[0269] A processor (220) according to one or more embodiments may train a meta model using a learning data set corresponding to a first image by executing one or more instructions stored in the memory (130).

[0270] For example, the processor (220) may use the first image to acquire a learning data set corresponding to the first image. The processor (220) may identify a category of the first image and select images belonging to the identified category from learning data stored in a database. The learning data may include high-definition images. The learning data may be stored in an external database or in the internal memory (230).

[0271] The processor (220) according to one or more embodiments may generate an image with degraded image quality by performing at least one of the following methods: decompressing, blurring, adding noise, or down-sampling the selected learning images.

[0272] The processor (220) according to one or more embodiments may train a meta model by using selected training images and images with degraded image quality obtained by processing the training images for quality as a training data set. For example, the processor (220) may input images with degraded image quality included in the training data set into the meta model, and update the parameters of the meta model so that the difference between the images output from the meta model and the high-quality training images included in the training data set is minimized.

[0273] A processor (220) according to one or more embodiments may obtain a second image obtained by processing the quality of a first image based on a learned meta model by executing one or more instructions stored in the memory (130).

[0274] The learned meta-model (updated model) according to one or more embodiments may include a neural network trained to process the quality of an input image. For example, the learned meta-model may be a model trained to process a low-resolution or low-quality image into a high-resolution or high-quality image. In one embodiment, the second image may be a higher-resolution or higher-quality image than the first image.

[0275] A display (240) according to one or more embodiments converts image signals, data signals, OSD signals, control signals, etc. processed by a processor (220) to generate a driving signal. The display (240) may be implemented as a PDP, LCD, OLED, flexible display, etc., and may also be implemented as a 3D display. In addition, the display (240) may be configured as a touch screen and may be used as an input device in addition to an output device.

[0276] A display (240) according to one or more embodiments may display a second image in which a first image is image-quality-processed.

[0277] FIG. 16 is a block diagram showing the configuration of an image processing device according to one or more embodiments of the present disclosure.

[0278] The image processing device (1600) of FIG. 16 may be an embodiment of the image processing device (200) of FIG. 1 and FIG. 15.

[0279] Referring to FIG. 16, an image processing device (1600) according to one or more embodiments may include a tuner unit (1640), a processor (1610), a display unit (1620), a communication unit (1650), a detection unit (1630), an input / output unit (1670), a video processing unit (1680), an audio processing unit (1685), an audio output unit (1660), a memory (1690), and a power supply unit (1695).

[0280] The communication unit (1650) of FIG. 16 may correspond to the communication unit (210) of FIG. 15, the processor (1610) of FIG. 16 may correspond to the processor (220) of FIG. 15, the memory (1690) of FIG. 16 may correspond to the memory (230) of FIG. 15, and the display unit (1620) of FIG. 16 may correspond to the display (240) of FIG. 15. Therefore, the same content as described above will be omitted.

[0281] A tuner unit (1640) according to one or more embodiments can select and tune only the frequency of a channel to be received by an image processing device (1600) among many radio wave components through amplification, mixing, resonance, etc. of a broadcast signal received wired or wirelessly. The broadcast signal includes audio, video, and additional information (e.g., EPG (Electronic Program Guide)).

[0282] The tuner unit (1640) can receive broadcast signals from various sources, such as terrestrial broadcasting, cable broadcasting, satellite broadcasting, and Internet broadcasting. The tuner unit (1640) can also receive broadcast signals from sources, such as analog broadcasting or digital broadcasting.

[0283] The communication unit (1650) according to one or more embodiments may receive a control signal or a control command, etc. from an external control device. For example, the communication unit (1650) may include an IR module capable of transmitting and receiving signals with the external control device according to the IR communication standard. Specifically, the communication unit (1650) may receive a control signal or a control command, etc. corresponding to a user input (e.g., a key or button input of the control device, etc.) from the control device.

[0284] A sensing unit (1630) according to one or more embodiments detects a user's voice, a user's image, or a user's interaction, and may include a microphone (1631), a camera unit (1632), and a light receiving unit (1633).

[0285] The microphone (1631) receives the user's spoken voice. The microphone (1631) can convert the received voice into an electrical signal and output it to the processor (1610). The user's voice may include, for example, a voice corresponding to a menu or function of the image processing device (1600). For example, the microphone (1631) can receive a user's voice corresponding to a display rotation command, convert the received voice into an electrical signal, and output it to the processor (1610).

[0286] The camera unit (1632) can receive an image (e.g., a series of frames) corresponding to a user's motion including a gesture within the camera recognition range. The processor (1610) can use the recognition result of the received motion to select a menu displayed on the image processing device (1600) or perform a control corresponding to the motion recognition result. For example, the processor (1610) can receive an image from the camera unit (1632), recognize a user's motion corresponding to the rotation of the display from the received image, and rotate the display in response to the recognition result.

[0287] The optical receiver (1633) receives an optical signal (including a control signal) from an external control device through an optical window (not shown) of a bezel of the display unit (1620). The optical receiver (1633) can receive an optical signal corresponding to a user input (e.g., touch, pressing, touch gesture, voice, or motion) from the control device. A control signal can be extracted from the received optical signal under the control of the processor (1610).

[0288] The input / output unit (1670) according to one or more embodiments may receive video (e.g., moving images, etc.), audio (e.g., voice, music, etc.), and additional information (e.g., EPG, etc.) from the outside of the image processing device (1600). The input / output unit (1670) may include any one of a High-Definition Multimedia Interface (HDMI), a Mobile High-Definition Link (MHL), a Universal Serial Bus (USB), a Display Port (DP), a Thunderbolt, a Video Graphics Array (VGA) port, an RGB port, a D-subminiature (D-SUB), a Digital Visual Interface (DVI), a component jack, and a PC port.

[0289] The processor (1610) controls the overall operation of the image processing device (1600) and the signal flow between the internal components of the image processing device (1600), and performs a function of processing data. The processor (1610) can execute an OS (Operating System) and various applications stored in the memory (1690) when there is a user input or when a preset and stored condition is satisfied.

[0290] The processor (1610) may include a RAM that stores signals or data input from the outside of the image processing device (1600) or is used as a storage area corresponding to various tasks performed in the image processing device (1600), a ROM that stores a control program for controlling the image processing device (1600), and a processor.

[0291] The video processing unit (1680) performs processing on video data received by the image processing device (1600). The video processing unit (1680) can perform various image processing operations, such as decoding, scaling, noise filtering, frame rate conversion, and resolution conversion on the video data.

[0292] The audio processing unit (1685) processes audio data. The audio processing unit (1685) may perform various processing operations, such as decoding, amplification, and noise filtering, on audio data. Meanwhile, the audio processing unit (1685) may include multiple audio processing modules to process audio corresponding to multiple contents.

[0293] The audio output unit (1660) outputs audio included in a broadcast signal received through the tuner unit (1640) under the control of the processor (1610). The audio output unit (1660) can output audio (e.g., voice, sound) input through the communication unit (1650) or the input / output unit (1670). In addition, the audio output unit (1660) can output audio stored in the memory (1690) under the control of the processor (1610). The audio output unit (1660) can include at least one of a speaker, a headphone output terminal, or an S / PDIF (Sony / Philips Digital Interface:) output terminal.

[0294] The power supply unit (1695) supplies power input from an external power source to components within the image processing device (1600) under the control of the processor (1610). In addition, the power supply unit (1695) can supply power output from one or more batteries located within the image processing device (1600) to the internal components under the control of the processor (1610).

[0295] The memory (1690) can store various data, programs, or applications for driving and controlling the image processing device (1600) under the control of the processor (1610). The memory (1690) can include a broadcast reception module, a channel control module, a volume control module, a communication control module, a voice recognition module, a motion recognition module, an optical reception module, a display control module, an audio control module, an external input control module, a power control module, a power control module for an external device connected wirelessly (e.g., Bluetooth), a voice database (DB), or a motion database (DB). The modules and database of the memory (1690) can be implemented in the form of software to perform a broadcast reception control function, a channel control function, a volume control function, a communication control function, a voice recognition function, a motion recognition function, an optical reception control function, a display control function, an audio control function, an external input control function, a power control function, or a power control function for an external device connected wirelessly (e.g., Bluetooth). The processor (1610) can perform each function using the software stored in the memory (1690).

[0296] Meanwhile, the block diagrams of the server (100) and the image processing device (200, 1600) illustrated in FIGS. 14 to 16 are block diagrams for one embodiment. Each component of the block diagrams may be integrated, added, or omitted depending on the specifications of the server (100) and the image processing device (200, 1600) actually implemented. That is, two or more components may be combined into one component, or one component may be subdivided into two or more components, as needed. In addition, the functions performed by each block are for explaining embodiments, and the specific operations or devices thereof do not limit the scope of the present invention.

[0297] A server according to one or more embodiments may include a communication unit, a memory storing one or more instructions, and at least one processor executing the one or more instructions.

[0298] The at least one processor can obtain a first image by compressing the original image by executing the one or more instructions.

[0299] The at least one processor can analyze the image quality of the first image by executing the one or more instructions, thereby obtaining quality information of the first image.

[0300] The at least one processor can obtain meta model information for a meta model used for image quality processing of the first image based on the quality information by executing the one or more instructions.

[0301] The at least one processor can control the communication unit to transmit the meta model information and the first image to an image processing device by executing the one or more instructions.

[0302] The above first image may include a plurality of frame images included in the video content.

[0303] The at least one processor can obtain quality information of a frame image at a first point in time among the plurality of frame images based on overall quality information of the plurality of frame images by executing the one or more instructions.

[0304] The at least one processor can set a quality upper limit value and a quality lower limit value based on a quality average value and a standard deviation of the plurality of frame images by executing the one or more instructions.

[0305] The at least one processor can determine a quality value of the frame image of the first time point based on the quality upper limit value and the quality lower limit value by executing the one or more instructions.

[0306] The at least one processor can control the communication unit to transmit the quality information to the image processing device by executing the one or more instructions.

[0307] The above meta model can be obtained based on at least one reference model.

[0308] The above meta model information may include information about the at least one reference model and information about a weight applied to the at least one reference model.

[0309] The at least one processor can determine the at least one reference model and the weight to be applied to the at least one reference model based on the quality information by executing the one or more instructions.

[0310] The at least one processor may determine the weight based on a difference between quality information corresponding to the at least one reference model and quality information for the first image by executing the one or more instructions.

[0311] The at least one reference model may include a first reference model pre-stored in the server or a second reference model pre-stored in the image processing device.

[0312] The at least one processor can generate the first reference model corresponding to the first image based on the first image by executing the one or more instructions.

[0313] The at least one processor may select the first reference model or the second reference model based on the quality information by executing the one or more instructions.

[0314] The at least one processor can control the communication unit to transmit information about the selected first reference model or identification information of the selected second reference model to the image processing device by executing the one or more instructions.

[0315] An image processing device according to one or more embodiments may include a communication unit, a memory storing one or more instructions, and at least one processor executing the one or more instructions.

[0316] The at least one processor can control the communication unit to receive first image and meta model information from the server by executing the one or more instructions.

[0317] The at least one processor can generate a meta model based on the meta model information by executing the one or more instructions.

[0318] The at least one processor can train the meta model using a learning data set corresponding to the first image by executing the one or more instructions.

[0319] The at least one processor can obtain a second image obtained by processing the first image based on the learned meta model by executing the one or more instructions.

[0320] The at least one processor can generate the meta model based on at least one reference model information included in the meta model information and weight information applied to the at least one reference model by executing the one or more instructions.

[0321] The at least one processor can obtain quality information of the first image by executing the one or more instructions.

[0322] The at least one processor can generate the learning data set by degrading images classified into the same category as the first image based on quality information of the first image by executing the one or more instructions.

[0323] A method of operating a server according to one or more embodiments may include a step of obtaining a first image by compressing an original image.

[0324] The operating method of the above server may include a step of analyzing the image quality of the first image and obtaining quality information of the first image.

[0325] The operating method of the above server may include a step of obtaining meta model information for a meta model used for image quality processing of the first image based on the quality information.

[0326] The operating method of the above server may include a step of transmitting the meta model information and the first image to an image processing device.

[0327] The above first image may include a plurality of frame images included in the video content.

[0328] The step of obtaining quality information of the first image may include a step of obtaining quality information of a frame image at a first point in time among the plurality of frame images based on overall quality information of the plurality of frame images.

[0329] The step of obtaining quality information of the first image may include a step of setting a quality upper limit value and a quality lower limit value based on a quality average value and a standard deviation of the plurality of frame images.

[0330] The step of obtaining quality information of the first image may include a step of determining a quality value of the frame image of the first time point based on the quality upper limit value and the quality lower limit value.

[0331] The above operating method may further include a step of transmitting the quality information to the image processing device.

[0332] The above meta model can be obtained based on at least one reference model.

[0333] The above meta model information may include information about the at least one reference model and information about a weight applied to the at least one reference model.

[0334] The step of obtaining the meta model information may include a step of determining the at least one reference model and a weight applied to the at least one reference model based on the quality information.

[0335] The step of determining a weight to be applied to the at least one reference model may include a step of determining the weight based on a difference between quality information corresponding to the at least one reference model and quality information for the first image.

[0336] The at least one reference model may include a first reference model pre-stored in the server or a second reference model pre-stored in the image processing device.

[0337] The operating method of the above server may further include a step of generating the first reference model corresponding to the first image based on the first image.

[0338] The step of obtaining the above meta model information may include a step of selecting the first reference model or the second reference model based on the quality information.

[0339] The step of transmitting the above meta model information may include a step of transmitting information about the selected first reference model or identification information of the selected second reference model to the image processing device.

[0340] A server according to one or more embodiments may analyze the image quality of not only a first image (input image) but also the entire content including the first image to determine quality information of the first image, thereby improving the accuracy of image quality analysis.

[0341] By obtaining an adaptive image quality processing model for an input image and performing image quality processing on the input image using the obtained image quality processing model, the image quality processing performance of an image processing device can be improved.

[0342] The operating method of the server and the operating method of the image processing device according to one or more embodiments may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the present invention or may be those known to and usable by those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0343] Additionally, the operating method of the server and the operating method of the image processing device according to the disclosed embodiments may be provided as a computer program product. The computer program product may be traded as a product between a seller and a buyer.

[0344] A computer program product may include a software program and a computer-readable storage medium on which the software program is stored. For example, a computer program product may be available from a manufacturer of an electronic device or an electronic market (e.g., Google Play). TMThe product may include a product in the form of a software program (e.g., a downloadable app) distributed electronically through a store, app store, etc. For electronic distribution, at least a portion of the software program may be stored on a storage medium or temporarily created. In this case, the storage medium may be a storage medium of a manufacturer's server, an electronic market server, or a relay server that temporarily stores the software program.

[0345] In a system comprising a server and a client device, the computer program product may include a storage medium of the server or a storage medium of the client device. Alternatively, if a third device (e.g., a smartphone) exists that is communicatively connected to the server or the client device, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include a software program itself that is transmitted from the server to the client device or the third device, or from the third device to the client device.

[0346] In this case, one of the server, the client device, and the third device may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, two or more of the server, the client device, and the third device may execute the computer program product to perform the method according to the disclosed embodiments in a distributed manner.

[0347] For example, a server (e.g., a cloud server or an artificial intelligence server, etc.) may execute a computer program product stored on the server, thereby controlling a client device in communication with the server to perform a method according to the disclosed embodiments.

[0348] Although the embodiments have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements made by those skilled in the art using the basic concept of the present invention defined in the following claims also fall within the scope of the present invention.

Claims

1. In the server (100), Communication interface (110); A memory (130) storing one or more instructions; and Operable connected to the above memory and the above communication interface, and executing one or more instructions to compress the original image and obtain a first image, By analyzing the image quality of the first image, quality information of the first image is obtained, Based on the above quality information, meta model information for the meta model used for image quality processing of the first image is obtained, A server including at least one processor (120) that controls the communication interface to transmit the meta model information and the first image to an image processing device that displays a second image based on the meta model information and the first image.

2. In paragraph 1, The above first image includes multiple frame images of video content, The at least one processor (120) executes the one or more instructions, A server that obtains quality information of a first frame image at a first point in time among the plurality of frame images based on overall quality information of the plurality of frame images.

3. In paragraph 2, The at least one processor (120) executes the one or more instructions, Based on the quality average value and standard deviation of the above multiple frame images, the quality upper limit value and the quality lower limit value are set, A server that determines a quality value of the first frame image based on the above quality upper limit value and quality lower limit value.

4. In any one of paragraphs 1 to 3, The at least one processor (120) executes the one or more instructions, A server that controls the communication interface to transmit the above quality information to the image processing device.

5. In any one of paragraphs 1 to 4, The above meta model is obtained based on at least one reference model, The above meta model information is, Contains information about at least one reference model and information about weights applied to the at least one reference model, The at least one processor (120) executes the one or more instructions, A server that determines at least one reference model and a weight applied to the at least one reference model based on the quality information.

6. In paragraph 5, The at least one processor (120) executes the one or more instructions, A server that determines the weights based on the difference between quality information corresponding to at least one reference model and quality information for the first image.

7. In paragraph 5 or 6, At least one of the above reference models, A server including a first reference model previously stored in the server or a second reference model previously stored in the image processing device.

8. In paragraph 7, The at least one processor (120) executes the one or more instructions, A server that generates a first reference model corresponding to the first image based on the first image.

9. In paragraph 7 or 8, The at least one processor (120) executes the one or more instructions, Based on the above quality information, the first reference model or the second reference model is selected, Based on the selection of the first reference model, the communication interface is controlled to transmit information about the first reference model to the image processing device, A server that controls the communication interface to transmit identification information of the second reference model to the image processing device based on the selection of the second reference model.

10. In the image processing device (200), Communication interface (210); display; A memory (230) storing one or more instructions; and Operable connected to the above memory, the above display and the above communication interface, and executing one or more instructions, Controlling the communication interface (210) to receive first image and meta model information from the server (100), Based on the above meta model information, a meta model is generated, To obtain a trained meta model, the meta model is trained using a learning data set corresponding to the first image, Based on the above trained meta model, a second image is obtained by processing the first image for quality, An image processing device comprising at least one processor (220) that controls the display to display the second image.

11. In paragraph 10, The at least one processor (220) executes the one or more instructions, An image processing device that generates the meta model based on reference model information included in the above meta model information and weight information applied to the above reference model information.

12. In paragraph 11, The at least one processor (120) executes the one or more instructions, Obtain quality information of the first image above, An image processing device that generates the learning data set by deteriorating images classified into the same category as the first image based on quality information of the first image.

13. In the operating method of the server (100), A step of compressing the original image to obtain a first image; A step of analyzing the image quality of the first image to obtain quality information of the first image; A step of obtaining meta model information for a meta model used for image quality processing of the first image based on the above quality information; and A method of operating a server, comprising the step of transmitting the meta model information and the first image to an image processing device that displays a second image based on the meta model information and the first image.

14. In paragraph 13, The above first image includes multiple frame images of video content, The step of obtaining quality information of the first image is: A method of operating a server, comprising the step of obtaining quality information of a first frame image at a first point in time among the plurality of frame images based on overall quality information of the plurality of frame images.

15. In paragraph 14, The step of obtaining quality information of the first image is: A step of setting a quality upper limit value and a quality lower limit value based on the quality average value and standard deviation of the plurality of frame images; and A method of operating a server, comprising the step of determining a quality value of the first frame image based on the quality upper limit value and the quality lower limit value.

Images