Method for condensing training dataset, and image processing device

The method of clustering and updating pixel values in learning data sets addresses the inefficiencies in existing upscaling technologies by compressing HR data sets for edge devices, enhancing performance and resource management in image processing.

WO2024248321A1PCT designated stage expired Publication Date: 2024-12-05SAMSUNG ELECTRONICS CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
PCT/KR2024/005078
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-08-04
Filing Date
2024-04-16
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Existing upscaling technologies for images face performance deterioration when the quality characteristics of the input image differ from those assumed during training, and current methods for compressing learning data sets are inefficient for edge devices with limited computational and memory resources.

Method used

A method involving clustering of learning data sets to generate a compressed High Resolution (HR) data set by selecting images, updating pixel values based on neural network losses, and iteratively refining the compressed HR data set to maintain similar performance to the original large data set, reducing the required data size and memory usage.

Benefits of technology

This approach allows for efficient compression of learning data sets, reducing the data size while maintaining upscaling performance, thus optimizing resource usage in edge devices and improving image processing efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure KR2024005078_05122024_PF_FP_ABST
    Figure KR2024005078_05122024_PF_FP_ABST
Patent Text Reader

Abstract

A method for condensing a training dataset, and an image processing device are provided. The method for condensing a training dataset comprises the steps of: generating a cluster set by clustering the training dataset; generating, for each cluster belonging to the cluster set, an initial condensed high resolution (HR) dataset by selecting some images from among images belonging to the each cluster; obtaining a first loss of a first neural network model on the basis of the training dataset, and obtaining a second loss of a second neural network model on the basis of the initial condensed HR dataset; and generating a condensed HR dataset by updating pixels of each of images belonging to the initial condensed HR dataset on the basis of the first loss and the second loss.
Need to check novelty before this filing date? Find Prior Art

Description

Method and image processing device for compressing a learning data set

[0001] The disclosed embodiment relates to a method for compressing a learning data set and an image processing device, and more specifically, to a method for compressing a learning data set and an image processing device for processing low-quality images using a compressed learning data set.

[0002] Recently, as the display screen size and resolution of image processing devices that process images have rapidly increased, upscaling technology that converts low-resolution (LR) images into high-resolution (HR) images is being developed.

[0003] Furthermore, with the advancement of deep learning technology, various forms of learning-based upscaling technologies are being developed. Learning-based upscaling technologies exhibit excellent performance when the quality of the training images is similar to the quality characteristics of the input images being processed. However, if the characteristics of the image to be processed differ from the input quality assumed during training, image quality performance deteriorates significantly.

[0004] To solve these problems, research is being conducted on upscaling technology through on-device learning, which processes and adapts AI models to the input images.

[0005] For example, edge devices such as TVs or mobile devices can generate a learning dataset with similar quality characteristics to the input image in real time, use this dataset to train an AI model, and then upscale the input image using the trained AI model. To improve the performance of on-device learning, a sufficiently large learning dataset is required, but the size of the available learning dataset is limited by the available computational and memory resources of the edge device.

[0006] Accordingly, research is being conducted to utilize training data sets of sufficient size for on-device learning. For example, high-quality external image patches with similar content characteristics to the input image can be stored in external memory, and then image patches with similar quality characteristics to the input image can be selected and used as a training data set. However, this method of generating a training data set for on-device learning is inefficient for edge devices with limited computational resources and internal memory because it stores high-quality external image patches in external memory without additional processing.

[0007] Meanwhile, unlike in the fields of image processing and upscaling, research is being conducted in the field of image classification to compress and utilize a small learning data set from a large learning data set to efficiently perform exploration and learning of network architecture.

[0008] Paper 1 (Dataset Condensation with Gradient Matching, ICLR 2021) is a technique for compressing a small training dataset so that the performance of a network trained on a large training dataset is similar to that of a network trained on a small training dataset. The process of compressing a small training dataset from a large training dataset is performed through learning. Specifically, to compress a large training dataset, the small training dataset is iteratively updated so that the gradients of the network parameters calculated from the large training dataset are similar to the gradients of the network parameters calculated from the small training dataset.

[0009] However, the data set compression method used in the field of image classification is a technique for compressing a data set consisting of pairs of data and labels, and there is a problem that it is difficult to apply it to compressing a data set in the field of upscaling consisting of pairs of low-resolution images and high-resolution images.

[0010] A method for compressing a learning data set according to one embodiment of the present disclosure includes the steps of: generating a cluster set by clustering a learning data set; generating an initial compressed HR (High Resolution) data set by selecting some images from among images belonging to each cluster for each cluster belonging to the cluster set; obtaining a first loss of a first neural network model based on the learning data set and obtaining a second loss of a second neural network model based on the initial compressed HR data set; and generating a compressed HR data set by updating each pixel of images belonging to the initial compressed HR data set based on the first loss and the second loss.

[0011] An image processing device according to one embodiment of the present disclosure includes a memory storing one or more instructions, and one or more processors executing one or more instructions stored in the memory. The one or more processors obtain a metamodel for image quality processing of an input image by executing the one or more instructions. The one or more processors obtain a compressed HR (High Resolution) data set corresponding to a learning data set. The one or more processors obtain an image quality-degraded data set from the compressed HR data set using an input image. The one or more processors obtain an image quality-processed output image based on a metamodel learned using the compressed HR data set and the image quality-degraded data set.

[0012] FIG. 1 is a drawing for explaining how an image processing device outputs an image whose quality has been processed, according to one embodiment.

[0013] Figure 2 is a block diagram of an image processing device and server according to one embodiment.

[0014] Figure 3 is an internal block diagram of a processor of a server according to one embodiment.

[0015] FIG. 4 is a diagram illustrating a clustered learning data set according to a clustering operation of a server according to one embodiment.

[0016] FIG. 5 is an internal block diagram illustrating a data set compression unit of a server according to one embodiment.

[0017] FIG. 6 is a diagram illustrating an operation of a server compressing a learning data set according to one embodiment.

[0018] FIG. 7a is a diagram showing an initial compressed HR data set and a compressed HR data set according to a compression operation of a server according to one embodiment.

[0019] FIG. 7b is a diagram specifically illustrating an initial compressed HR data set and a compressed HR data set according to a compression operation of a server according to one embodiment.

[0020] FIG. 8a is a flowchart illustrating a method for compressing a learning data set according to one embodiment.

[0021] FIG. 8b is a specific flowchart illustrating a method for compressing a learning data set according to one embodiment.

[0022] FIG. 8c is a flowchart illustrating a method for compressing a learning data set according to one embodiment.

[0023] FIG. 9a is a diagram illustrating a process in which a server compresses a learning data set by category according to one embodiment.

[0024] FIG. 9b is a diagram illustrating a process by which a server compresses a learning data set by content type, according to one embodiment. FIG. 10 is a flowchart illustrating a method for compressing a learning data set by category, according to one embodiment.

[0025] Fig. 11 is an internal block diagram of a processor of an image processing device according to one embodiment.

[0026] Figure 12a is a diagram for explaining a model learning unit according to one embodiment.

[0027] Figure 12b is a diagram for explaining a model learning unit according to one embodiment.

[0028] FIG. 13 is a diagram for explaining the operation of a degradation processing unit according to one embodiment to process the quality of a learning data set.

[0029] Figure 14a is a diagram for explaining a model learning unit according to one embodiment.

[0030] FIG. 14b is a diagram for explaining the operation of a learning data selection unit including an AI model, according to one embodiment.

[0031] Figure 14c is a diagram for explaining a model learning unit according to one embodiment.

[0032] FIG. 15 is a flowchart illustrating a method for processing image quality of an input image according to one embodiment.

[0033] FIG. 16 is a diagram for explaining an operation of a streaming server compressing a learning data set and an operation of an image processing device storing learning data according to one embodiment.

[0034] FIG. 17 is a diagram for explaining an operation of an image processing device according to one embodiment of the present invention to compress a learning data set according to a background process.

[0035] In this disclosure, the expression “at least one of a, b or c” may refer to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “all of a, b and c”, or variations thereof.

[0036] Below, embodiments of the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

[0037] The terms used in this disclosure are described as currently common terms, taking into account the functions mentioned herein. However, these terms may mean various other terms depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Therefore, the terms used in this disclosure should not be interpreted solely based on their names, but rather based on the meanings of the terms and the overall content of this disclosure.

[0038] Additionally, the terminology used in this disclosure is for the purpose of describing specific embodiments only and is not intended to limit the present disclosure.

[0039] Throughout the specification, when a part is said to be "connected" to another part, this includes not only the cases where it is "directly connected" but also the cases where it is "electrically connected" with another element in between.

[0040] As used herein, and particularly in the claims, the terms "above" and "above" and similar referents may refer to both the singular and the plural. Furthermore, unless the order of steps in a method according to the present disclosure is explicitly specified, the steps described may be performed in any appropriate order. The present disclosure is not limited by the order in which the steps are described.

[0041] The appearances of phrases such as “in some embodiments” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment.

[0042] Some embodiments of the present disclosure may be represented by functional block configurations and various processing steps. Some or all of these functional blocks may be implemented by various hardware and / or software configurations that perform specific functions. For example, the functional blocks of the present disclosure may be implemented by one or more microprocessors or by circuit configurations for a given function. Furthermore, for example, the functional blocks of the present disclosure may be implemented in various programming or scripting languages. The functional blocks may be implemented by algorithms that execute on one or more processors. Furthermore, the present disclosure may employ conventional techniques for electronic configuration, signal processing, and / or data processing. Terms such as “mechanism,” “element,” “means,” and “configuration” may be used broadly and are not limited to mechanical and physical configurations.

[0043] Additionally, the connecting lines or connecting members between components depicted in the drawings are merely exemplary representations of functional connections and / or physical or circuit connections. In an actual device, connections between components may be represented by various functional connections, physical connections, or circuit connections that may be replaced or added.

[0044] Additionally, terms such as “part”, “module”, etc. described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software, or a combination of hardware and software.

[0045] Additionally, the term "user" in this specification refers to a person who uses the image processing device, and may include a consumer, evaluator, viewer, administrator, or installer. Furthermore, the term "manufacturer" in this specification may refer to a manufacturer that manufactures the image processing device and / or components included in the image processing device.

[0046] In the present disclosure, compressing a training dataset may mean updating pixels of images belonging to the training dataset. 'Condensing of a training dataset' may be expressed as 'synthesis of a training dataset', 'updating of a training dataset', or 'training of training data'. For example, compression may mean that new images that are completely different from images belonging to previous training datasets are generated as pixels of images belonging to the training dataset are repeatedly updated.

[0047] In addition, in the present disclosure, a small training dataset can be obtained from a large training dataset according to a compression operation of the training dataset. In this case, the characteristics of the large training dataset may be similar to the characteristics of the small training dataset. The similarity between the characteristics of the large training dataset and the characteristics of the small training dataset may mean that the performance of a neural network model trained with the large training dataset is similar to the performance of a neural network model trained with the small training dataset. In the present disclosure, similar performance between neural network models may mean that the parameters of each neural network model are similar, or that the gradients of each neural network model are similar. In the present disclosure, the term "similar" may include not only being similar to each other, but also being identical to each other.

[0048] In the present disclosure, a small training dataset may be referred to as a condensed dataset compressed from a large training dataset or a synthetic dataset synthesized from a large training dataset.

[0049] In the present disclosure, the learning data set may correspond to a high-resolution (HR) data set containing high-resolution images. In the present disclosure, the high-resolution data set may be expressed as "high-resolution" in the sense that it includes images with a higher resolution than a low-resolution data set generated by degrading the high-resolution data set.

[0050] In the present disclosure, the compressed data set may correspond to a compressed high-resolution data set. In the present disclosure, the compressed high-resolution data set may be referred to as a compressed HR data set.

[0051] In the present disclosure, the learning data set is exemplified as including an image, but is not limited thereto, and may further include patches cut out from the image to have semantic information, etc.

[0052] The present disclosure will be described in detail with reference to the attached drawings below.

[0053] FIG. 1 is a drawing for explaining how an image processing device outputs an image whose quality has been processed, according to one embodiment.

[0054] Referring to FIG. 1, the image processing device (100) may be an electronic device capable of processing and outputting an image. In an embodiment, the image processing device (100) may be implemented as various types of electronic devices including a display.

[0055] The image processing device (100) may be fixed or mobile, and may be a digital TV capable of receiving digital broadcasting, but is not limited thereto. The image processing device (100) may include at least one of a desktop, a smartphone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a laptop personal computer (PC), a netbook computer, a digital camera, a PDA (Personal Digital Assistants), a PMP (Portable Multimedia Player), a camcorder, a navigation device, a wearable device, a smart watch, a home network system, a security system, and a medical device.

[0056] The image processing device (100) may be implemented as a flat display device, a curved display device having a screen with a curvature, or a flexible display device whose curvature can be adjusted. The output resolution of the image processing device (100) may have various resolutions, such as, for example, HD (High Definition), Full HD, Ultra HD, or a resolution clearer than Ultra HD.

[0057] The image processing device (100) can output video. The video may be composed of multiple frames. The video may include items such as television programs provided by content providers or various movies or dramas provided through VOD services. A content provider may refer to a terrestrial broadcaster, cable broadcaster, OTT service provider, or IPTV service provider that provides various content, including video, to consumers. The video is captured, compressed, transmitted, and then restored and output by the image processing device (100). Due to limitations in the physical characteristics of the device used to capture the video and limited bandwidth, information may be lost, resulting in image distortion. Distorted video may result in reduced quality.

[0058] In one embodiment, the image processing device (100) can process the image quality of an image. As described above, the image processing device (100) can process the image quality of an image that is distorted due to information loss due to limitations in the physical characteristics of the device and limited bandwidth. In one embodiment, the image processing device (100) can obtain an output image (120) in which an input image (110) has been image-processed.

[0059] In one embodiment, the image processing device (100) may utilize artificial intelligence (AI) technology to process the image quality. In one embodiment, the image processing device (100) may be an edge device in which artificial intelligence is connected to an electronic device that outputs an image to a user. In one embodiment, the image processing device (100) may process the image quality using on-device AI technology. On-device AI technology refers to a technology in which AI technology is executed on an edge device, and an edge device refers to an electronic device that can process data in real time on the device itself. In one embodiment, the image processing device (100) may process the image quality more quickly because the image processing device (100) collects, calculates, and processes information on its own without going through a cloud server according to the on-device AI technology.

[0060] AI technology can be comprised of machine learning (deep learning) and component technologies utilizing machine learning. AI technology can be implemented using algorithms. Here, an algorithm or set of algorithms for implementing AI technology is called a neural network. A neural network can receive input data, perform operations for analysis and classification, and output result data.

[0061] In one embodiment, the on-device AI technology may be performed by at least one processor included in the image processing device (100). In one embodiment, the on-device AI technology may also be referred to as an on-device learning system.

[0062] In one embodiment, the image processing device (100) may obtain a model to process the image quality of an input image (110) using on-device AI technology, obtain a learning data set suitable for the characteristics of the input image (110), and perform transfer learning on the model using the learning data set. The image processing device (100) may generate a meta model adaptive to the input image (110) using on-device AI technology. The meta model is a model that has learned a 'method of learning' and may refer to a neural network model that can quickly learn or adapt to new data. The technology of teaching the 'method of learning' to such a meta model may be expressed as meta-learning. For example, the meta model may be quickly learned to fit the input image (110).

[0063] In one embodiment, the image processing device (100) can evaluate the image quality of an image. The image quality may refer to the quality of the image or the degree of image deterioration. In one embodiment, the image processing device (100) can evaluate the input image (110) to obtain at least one of the compression quality, blur quality, resolution, and noise of the input image (110). In one embodiment, the image processing device (100) can evaluate the image quality of the input image (110) to obtain a learning data set that matches the characteristics of the input image (110).

[0064] In one embodiment, the image processing device (100) can train the meta model using a training data set. In one embodiment, the training data set used in the image processing device (100) may be a small training data set (20) compressed from a large training data set (10). In one embodiment, instead of using the large training data set (10), the image processing device (100) may use a small training data set (20) having similar characteristics to the large training data set (10). In one embodiment, the small training data set (20) may be composed of a smaller number of images than the large training data set (10). In one embodiment, the image processing device (100) can train the meta model using the small training data set (20), thereby reducing the amount of training data set required for training the meta model in an edge device with limited computational resources and memory resources. Additionally, the image processing device (100) can reduce the amount of memory used in an edge device and reduce the amount of power consumed for calculation.

[0065] In the present disclosure, a small training dataset (20) may be referred to as a 'condensed dataset' compressed from a large training dataset (10), or a 'synthetic dataset' synthesized from a large training dataset (10). In the present disclosure, compressing a training dataset may mean that the number of images belonging to the training dataset is reduced, and the pixels of the images are repeatedly updated. For example, as the training dataset is compressed, the number of images belonging to the training dataset may be reduced from 100 to 1. For example, the pixels of one image may be repeatedly updated so that the characteristics of 100 images belonging to the training dataset are reflected in one image. For example, 100 images may belong to a large learning data set (10), and one image that is repeatedly updated may belong to a small learning data set (20). Meanwhile, in the present disclosure, images belonging to a large learning data set (10) or a small learning data set (20) may be high-resolution images.

[0066] In one embodiment, the characteristics of a large learning data set (10) and the characteristics of a small learning data set (20) may be similar. For example, the characteristics of 100 images belonging to a large learning data set (10) and the characteristics of 1 image belonging to a small learning data set may be similar. In the present disclosure, the similarity of the characteristics of a large learning data set (10) and the characteristics of a small learning data set (20) may mean that the performance of a first neural network model trained with a large learning data set (10) is similar to the performance of a second neural network model trained with a small learning data set (20). For example, when the parameters of a first neural network model trained with a large learning data set (10) are similar to the parameters of a second neural network model trained with a small learning data set (20), the performances of the two neural network models may be expressed as being similar. For example, the parameters of a neural network model may include the weights and gradients of the neural network model. For example, if the gradient of a first neural network model is similar to the gradient of a second neural network model, the performance of the two neural network models can be expressed as similar.

[0067] In the present disclosure, the term "similar" may encompass not only things that are similar to each other, but also things that are identical to each other. In the present disclosure, the neural network model may be an inference network that implements a super-resolution (SR) algorithm capable of upscaling a low-resolution (LR) image into a high-resolution (HR) image.

[0068] Meanwhile, in the present disclosure, a compressed high-resolution image belonging to a small learning data set (20) may have different data from a high-resolution image before compression belonging to a large learning data set (10). For example, as pixels of a high-resolution image before compression are repeatedly updated, a new image different from the high-resolution image before compression may be generated. For example, a compressed high-resolution image may have different pixel information, contrast value, luminance information, shape, embedding vector, etc. from a high-resolution image before compression. However, in the present disclosure, the characteristics of a small learning data set (20) and the characteristics of a large learning data set (10) can be expressed as being similar in the sense that the upscaling performance of a second neural network model trained with a compressed high-resolution image in a small learning data set (20) and the upscaling performance of a first neural network model trained with a high-resolution image in a large learning data set (10) are similar.

[0069] In one embodiment, the operation of compressing the learning data set may be performed in the server (1000). The server (1000) may store a large learning data set (10) required for learning the meta model in the image processing device (100). In one embodiment, the server (1000) may generate a small learning data set (20) based on the large learning data set (10). For example, the server (1000) may generate a compressed HR (High Resolution) data set having compressed high-resolution images based on high-resolution images belonging to the large learning data set (10). For example, the small learning data set (20) may correspond to the compressed HR (High Resolution) data set. In one embodiment, the server (1000) may provide necessary data to an edge device in an offline environment. The edge device may collect data from the server (1000) and perform data processing and analysis.

[0070] In one embodiment, the server (1000) can generate an initial compressed HR data set by selecting some images from among the images belonging to a large learning data set (10). Here, the initial compressed HR data set can be composed of images that are randomly extracted from among the images belonging to the large learning data set (10). In one embodiment, the server (1000) can generate a compressed HR data set composed of compressed images by repeatedly updating the initial compressed HR data set composed of the randomly extracted images. The server (1000) can generate a compressed HR data set corresponding to a small learning data set (20) by extracting a small number of images from the images belonging to the large learning data set (10) and performing an update on the small number of images. In one embodiment, the server (1000) can repeatedly update the compressed HR data set so that the characteristics of the compressed HR data set are similar to the characteristics of the large learning data set (10) corresponding to the high-resolution data set.

[0071] In one embodiment, the server (1000) may approximate the upscaling performance of a first neural network model trained using the large training data set (10) and the upscaling performance of a second neural network model trained using the compressed HR data set to generate a compressed HR data set having characteristics similar to those of the large training data set (10). For example, the server (1000) may approximate the parameters of the two neural network models so that the upscaling performance of the second neural network model approximates the upscaling performance of the first neural network model. For example, the server (1000) may calculate the parameters of the first neural network model and the parameters of the second neural network model, and iteratively update the compressed HR data set so that the difference between the parameters of the first neural network model and the parameters of the second neural network model is minimized.

[0072] For example, the server (1000) may repeatedly update each pixel of images in the compressed HR data set, which is training data for the second neural network model, so that the second gradient of the second neural network model approximates the first gradient of the first neural network model. This is described in detail in FIGS. 5 and 6 .

[0073] In one embodiment, the server (1000) may generate a high-resolution image-low-resolution image pair to train a neural network model using a training data set. For example, the server (1000) may perform at least one of compression degradation, blurring degradation, resolution adjustment, or noise addition on a high-resolution image to obtain a low-resolution image with degraded image quality. In one embodiment, the server (1000) may degrade high-resolution images belonging to a large training data set (10) to generate a low-resolution data set having low-resolution images to train a first neural network model. The large training data set (10) may have a higher resolution than the low-resolution data set. In one embodiment, the server (1000) may degrade compressed high-resolution images belonging to a compressed HR data set to generate a compressed LR data set having compressed low-resolution images to train a second neural network model. The compressed HR data set may have a higher resolution than the degraded data set. In one embodiment, the server (1000) may update the compressed LR data set to correspond to the updated compressed HR data set whenever the compressed HR data set is updated.

[0074] In one embodiment, the server (1000) may provide the image processing device (100) with a small amount of learning data set (20) having characteristics similar to those of the large amount of learning data set (10). For example, the server (1000) may provide the image processing device (100) with a compressed HR data set corresponding to the large amount of learning data set (10).

[0075] In one embodiment, the image processing device (100) can store the compressed HR data set generated by the server (1000) in the memory of the image processing device (100). In one embodiment, the image processing device (100) can obtain a compressed HR data set corresponding to a small amount of learning data set (20) instead of a large amount of learning data set (10). The image processing device (100) can obtain a data set with degraded image quality corresponding to the input image (110) from the compressed HR data set. For example, the image processing device (100) can obtain the data set with degraded image quality from the compressed HR data set so as to have an image quality corresponding to the image quality of the input image (110). For example, the image processing device (100) can analyze the input image (110) to obtain the quality of the input image (110) in order to obtain a learning data set suitable for the input image (110). For example, the image processing device (100) can evaluate the input image (110) to obtain the degree of degradation of at least one of the compression quality, blur quality, resolution, and noise of the input image (110). For example, the image processing device (100) can obtain a data set with deteriorated quality by performing at least one of compression degradation, blurring degradation, resolution adjustment, or noise addition on the compressed HR data set. The compressed HR data set can have a higher resolution than the data set with deteriorated quality.

[0076] In one embodiment, the image processing device (100) can train a meta model based on a compressed HR data set and a data set with degraded image quality. The image processing device (100) can train the meta model by updating the parameters of the meta model so that the difference between the output value obtained by applying the degraded image quality data set to the meta model and the compressed HR data set is minimized. In one embodiment, the performance of the meta model trained using the compressed HR data set, which is a small training data set (20), may be similar to the performance of the meta model trained using a large training data set (10). For example, the parameters of the meta model trained using the compressed HR data set may be similar to or identical to the parameters of the meta model trained using the large training data set (10).

[0077] In one embodiment, instead of training a meta model with a large learning data set (10) in an image processing device (100), the meta model can be trained with a compressed HR data set corresponding to a small data set (20). As the image processing device (100) trains the meta model with the compressed HR data set, the upscaling performance of the meta model can be maintained. Accordingly, the amount of learning data set required for training an AI model for performing upscaling in the image processing device (100) can be reduced, and the amount of external memory used and the power consumption for operations in the image processing device (100) can be reduced.

[0078] Alternatively, in one embodiment, the operation of compressing the learning data set may be performed by the image processing device (100). The image processing device (100) may compress the learning data set while the on-device AI technology is not running, thereby generating a compressed data set. In this case, the image processing device (100) may store a large learning data set and a compressed data set, which is a small learning data set.

[0079] In one embodiment, the image processing device (100) can load a meta model learned by on-device AI technology, and perform image quality processing by applying an input image (110) to the learned meta model, thereby obtaining an output image (120) with image quality processing.

[0080] Figure 2 is a block diagram of an image processing device and server according to one embodiment.

[0081] Referring to FIG. 2, the server (1000) may include a processor (1001) and memory (1002).

[0082] The memory (1002) can store one or more instructions. The memory (1002) can store at least one program executed by the processor (1001). In one embodiment, the memory (1002) can store at least one program that executes an operation of compressing a large training data set into a small compressed data set. In one embodiment, the memory (1002) can include an external database that stores a training data set for on-device AI operation of the image processing device (100). In one embodiment, the memory (1002) can store a large training data set and a compressed HR data set to be provided to the image processing device (100). In one embodiment, the memory (1002) can store at least one neural network model and / or predefined operation rules or AI models.

[0083] The memory (1002) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a magnetic memory, a magnetic disk, and an optical disk.

[0084] The server (1000) may include one or more processors (1001). The processor (1001) may control the overall operation of the server (1000). The processor (1001) may control the server (1000) to perform an operation of compressing a learning data set by executing one or more instructions stored in the memory (1002).

[0085] In one embodiment, the processor (1001) generates a cluster set (410) by executing one or more instructions to cluster the training data set (310, 611). The processor (1001) generates an initial compressed HR data set (601) by selecting some images from among the images belonging to each cluster in each cluster belonging to the cluster set (410). The processor (1001) obtains a first loss (631) of a first neural network model (630) based on the training data set (310, 611), and obtains a second loss (641) of a second neural network model (640) based on the initial compressed HR data set (601). The processor (1001) generates a compressed HR data set (621) by updating each pixel of images belonging to the initial compressed HR data set (601) based on the first loss (631) and the second loss (641).

[0086] In one embodiment, the server (1000) may generate a compressed HR data set, which is a small learning data set (20) having similar characteristics to a large learning data set (10). The number of compressed high-resolution images included in the small learning data set (20) may be less than the number of high-resolution images included in the large learning data set (10). The amount of data included in the small learning data set (20) may be less than the amount of data included in the large learning data set (10). Accordingly, the amount of learning data required for learning an AI model for performing upscaling in the image processing device (100) is reduced, and the amount of external memory used and the amount of power consumed for computation in the image processing device (100) may be reduced.

[0087] In one embodiment, the processor (1001) may perform preprocessing (1100) on the learning data set (310, 611) based on the amount of information of each image belonging to the learning data set (310, 611) by executing one or more instructions.

[0088] In one embodiment, the processor (1001) may perform quality processing on the training data set (310, 611) to generate a low-resolution data set (612) and quality processing on the compressed HR data set (621) to generate a compressed LR data set (622) by executing one or more instructions. The processor (1001) may calculate a first loss (631) which is the difference between the training data set (310, 611) and an output value (635) obtained by inputting the low-resolution data set (612) into a first neural network model (630). The processor (1001) may calculate a second loss (641) which is the difference between the compressed LR data set (622) and an output value (645) obtained by inputting the second neural network model (640).

[0089] In one embodiment, the processor (1001) may calculate a first gradient for minimizing a first loss (631) of a first neural network model (630) by executing one or more instructions. The processor (1001) may calculate a second gradient for minimizing a second loss (641) of a second neural network model (640). The processor (1001) may update each pixel of images in the compressed HR data set (621) so that a difference between the first gradient and the second gradient is reduced.

[0090] In one embodiment, the processor (1001) may calculate a matching loss (650), which is a difference between a first gradient and a second gradient, by executing one or more instructions. The processor (1001) may calculate a matching gradient for minimizing the matching loss (650). The processor (1001) may update the compressed HR data set (621) by applying the matching gradient to each pixel of the images in the compressed HR data set (621). The matching gradient may represent a slope of the matching loss (650) for each pixel of the images in the compressed HR data set (621).

[0091] The first gradient may represent the slope of the first loss (631) with respect to the parameters of the first neural network model (630). The second gradient may represent the slope of the second loss (641) with respect to the parameters of the second neural network model (640).

[0092] The first loss (631) and / or the second loss (641) may be characterized by not being a Mean Absolute Error.

[0093] In one embodiment, the processor (1001) may perform image quality processing on a compressed HR data set (621) by executing one or more instructions to generate a compressed LR (Low Resolution) data set (622).

[0094] In one embodiment, the processor (1001) may update parameters of the second neural network model (640) by executing one or more instructions to apply the updated compressed HR data set (621) and compressed LR data set (622) to the second neural network model (640).

[0095] In one embodiment, the processor (1001) may calculate a second loss (641) between an output value (645) obtained by applying the compressed HR data set (621) to the second neural network model (640) and the compressed LR data set (622) by executing one or more instructions. The processor (1001) may calculate a second gradient for minimizing the second loss (641). The processor (1001) may update the parameters of the second neural network model (640) by applying the second gradient to the parameters of the second neural network model (640). The second gradient may represent a slope of the second loss (641) with respect to the parameters of the second neural network model (640).

[0096] The initial parameters of the second neural network model (640) may be characterized as being the same as the initial parameters of the first neural network model (630).

[0097] The parameters of the second neural network model (640) learned using the compressed HR data set (621) may be characterized as being approximate to the parameters of the first neural network model (630) learned using the learning data set (310, 611).

[0098] In one embodiment, the processor (1001) may classify the training data set (310, 611) based on categories by executing one or more instructions. The processor (1001) may obtain a multi-compressed data set (940) by compressing the training data set (310, 611) for each category.

[0099] In one embodiment, the processor (1001) may determine a number of images to be selected from among images belonging to each cluster based on user input by executing one or more instructions. In one embodiment, the determined number of images may be equal to the number of images belonging to the compressed HR data set (621).

[0100] In one embodiment, the image processing device (100) may include a processor (101) and a memory (102).

[0101] The memory (102) can store one or more instructions. The memory (102) can store at least one program executed by the processor (101). The memory (102) can store at least one neural network and / or predefined operation rules or AI models. In addition, the memory (102) can store data input to or output from the image processing device (100).

[0102] The memory (102) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a magnetic memory, a magnetic disk, and an optical disk.

[0103] In one embodiment, the memory (102) may store a compressed HR data set corresponding to the learning data set.

[0104] In one embodiment, the memory (102) may store a plurality of compressed HR data sets compressed from a training data set having categories corresponding to categories of an input image. In one embodiment, the memory (102) may store a plurality of compressed HR data sets compressed from a training data set having content identical to the content of the input image. In one embodiment, the plurality of compressed HR data sets may be referred to as multiple compressed data sets.

[0105] In one embodiment, the memory (102) may store compressed high-quality images for each cluster belonging to the cluster set among the high-quality images within the learning data set.

[0106] In one embodiment, the memory (102) may store multiple reference models. The multiple reference models may be image quality processing models trained using training images with different image quality values. In an embodiment, the memory (102) may store image quality values ​​of training images used to train each of the multiple reference models together with the reference models.

[0107] The processor (101) may include one or more processors. The processor (101) may control the overall operation of the image processing device (100). The processor (101) may control the image processing device (100) to function by executing one or more instructions stored in the memory (102).

[0108] In one embodiment, the processor (101) may obtain a meta model for image quality processing of an input image (110) by executing one or more instructions. In one embodiment, the processor (101) may obtain a compressed HR (High Resolution) data set (621) corresponding to a learning data set (310, 611). In one embodiment, the processor (101) may obtain an image quality deteriorated data set (1302) from the compressed HR data set (621, 940) using the input image (110). In one embodiment, the processor (101) may obtain an image quality-processed output image (120) based on a meta model learned using the compressed HR data set (621) and the image quality deteriorated data set (1302).

[0109] In one embodiment, the characteristics of the compressed HR data set, which is a small-scale learning data set, may be similar to the characteristics of the large-scale learning data set (10). In other words, even if the meta-model is trained with the compressed HR data set instead of training the meta-model with the large-scale learning data set (10) in the image processing device (100), the upscaling performance of the image processing device (100) can be maintained. Accordingly, the amount of learning data set required for training the AI ​​model for performing upscaling in the image processing device (100) can be reduced, and the amount of external memory used and the power consumption for operations in the image processing device (100) can be reduced.

[0110] In one embodiment, the processor (101) can obtain a degraded image quality data set (1302) by degrading the image quality of the compressed HR data set (621) to have an image quality corresponding to the image quality of the input image (110) by executing one or more instructions.

[0111] In one embodiment, the processor (101) can identify the category of the input image (110) by executing one or more instructions. In one embodiment, the processor (101) can obtain a compressed HR data set corresponding to the same category as the category of the input image (110) from among multiple condensed datasets (940) in which the learning data sets (310, 611) are compressed by category by executing one or more instructions.

[0112] In one embodiment, the processor (101) can identify the type of content of the input image (110) through metadata of the input image (110) by executing one or more instructions. In one embodiment, the processor (101) can obtain a compressed HR data set corresponding to the same type of content as the content of the input image (110) from among multiple condensed datasets (960) in which the learning data set (310, 611) is compressed by type of content by executing one or more instructions.

[0113] In one embodiment, the parameters of the meta model learned using the compressed HR data set (621) and the image quality deteriorated data set (1302) may be characterized as being approximate to the parameters of the meta model learned using the learning data set (310, 611) and the low-resolution data set (612) whose image quality is deteriorated from the learning data set (310, 611).

[0114] In one embodiment, the compressed HR data set may be characterized in that the high-quality images in the training data set include compressed high-quality images for each cluster belonging to the cluster set.

[0115] In one embodiment, the processor (101) may receive an input image and a compressed data set corresponding to the input image from the streaming server (2000) by executing one or more instructions. In one embodiment, the processor (101) may obtain a data set corresponding to a category of the input image from the compressed data set and the compressed HR data set received from the streaming server (2000).

[0116] Hereinafter, with reference to FIGS. 3 to 10, an operation of a server (1000) according to one embodiment to compress a learning data set will be described.

[0117] Figure 3 is an internal block diagram of a processor of a server according to one embodiment.

[0118] Referring to FIG. 3, a processor (1001) of a server (1000) may include a preprocessing unit (1100), a clustering unit (1200), and a data set compression unit (1300). The processor (1001) may perform operations of each of the preprocessing unit (1100), the clustering unit (1200), and the data set compression unit (1300) by executing one or more instructions stored in a memory (1002). In one embodiment, the processor (1001) may generate a compressed data set (320) based on a learning data set (310). The learning data set (310) corresponds to the large-scale learning data set (10) of FIG. 1, and may include a high-resolution image.

[0119] In one embodiment, the preprocessing unit (1100) may perform preprocessing on the learning data set (310) based on the amount of information of each high-resolution image included in the learning data set (310). The preprocessing unit (1100) may preemptively remove images that have little influence on improving the upscaling performance of the neural network model among the high-resolution images included in the learning data set (310). For example, the preprocessing unit (1100) may measure the amount of information of each high-resolution image included in the learning data set (310) and distinguish between texture data with a low contrast value and edge data with a high contrast value. Texture data may refer to complex data with a low contrast value and a large amount of information, and edge data may refer to data with a high contrast value and a small amount of information. The preprocessing unit (1100) may preemptively remove texture data that has an adaptive influence on improving the upscaling performance of the neural network model among the high-resolution images included in the learning data set (310).

[0120] In one embodiment, the preprocessing unit (1100) can distinguish between a high-resolution image including texture data and a high-resolution image including edge data by measuring statistical characteristics of a high-resolution image and then removing high-resolution images whose measured values ​​are below a certain value. Alternatively, in one embodiment, the preprocessing unit (1100) can distinguish between a high-resolution image including texture data and a high-resolution image including edge data by measuring statistical characteristics of a high-resolution image and then listing the measured values ​​in order of size and removing the lower N values ​​(N is an arbitrary natural number).

[0121] In one embodiment, the pre-processing (1100) may be omitted.

[0122] In one embodiment, the clustering unit (1200) can generate a cluster set by clustering the preprocessed learning data set (310). In order to efficiently compress the learning data set (310), the server (1000) can group high-resolution images with similar characteristics into each cluster rather than compressing the entire learning data set at once, and then perform compression on each cluster. In one embodiment, the server (1000) can obtain a compressed HR (High Resolution) data set by compressing high-resolution images belonging to each cluster belonging to each cluster in the cluster set.

[0123] In one embodiment, the clustering unit (1200) may cluster the learning data set (310) based on pixel information, contrast value, luminance information, shape, embedding vector, etc. of the high-resolution images included in the learning data set (310). For example, the clustering unit (1200) may group high-resolution images that have similar pixel information, contrast value, luminance information, shape, embedding vector, etc., among the high-resolution images included in the learning data set (310), into each cluster. For example, the clustering unit (1200) may group images having edge data into one cluster, as shown in FIG. 5A, and group images having texture data into one cluster.

[0124] In one embodiment, the clustering unit (1200) may cluster the learning data set (310) using a conventional algorithm, such as the K-means clustering algorithm. In one embodiment, the number of clusters belonging to a cluster set may be determined based on a user's input. For example, the clustering unit (1200) may cluster into k clusters, each having i high-resolution images, based on the user's input (i and k are natural numbers). For example, the clustering unit (1200) may group the data into 128 clusters, each having 100 high-resolution images.

[0125] In one embodiment, the clustering unit (1200) may be omitted.

[0126] In one embodiment, the data set compression unit (1300) can generate a compressed data set (320) based on the learning data set (310). In one embodiment, the data set compression unit (1300) can generate a compressed data set (320) having compressed high-resolution images by compressing high-resolution images belonging to the learning data set (310).

[0127] In one embodiment, the data set compression unit (1300) can compress the cluster set clustered by the clustering unit (1200). The data set compression unit (1300) can individually perform compression for each cluster belonging to the cluster set. The data set compression unit (1300) can generate compressed high-resolution images by compressing high-resolution images belonging to each cluster for each cluster.

[0128] In one embodiment, the data set compression unit (1300) can determine the number of compressed high-resolution images among the high-resolution images belonging to each cluster based on a user input. In one embodiment, the user can select m images among the i high-resolution images belonging to each cluster. The data set compression unit (1300) can generate an initial compressed HR data set with the m high-resolution images selected among the i high-resolution images belonging to each cluster. The data set compression unit (1300) can generate a compressed HR data set having m compressed high-resolution images for each cluster by repeatedly updating the initial compressed HR data set having m high-resolution images for each cluster. For example, the data set compression unit (1300) can perform compression for 128 clusters, each cluster having 100 (i=100) high-resolution images. For example, the user can decide to have 1 (m=1) compressed high-resolution image for each cluster having 100 high-resolution images. For example, the data set compression unit (1300) can compress 100 high-resolution images belonging to each cluster into one compressed high-resolution image. For example, the data set compression unit (1300) can compress a learning data set (310) composed of 12,800 high-resolution images into one high-resolution image for each of 128 clusters, thereby generating a compressed data set (320) having 128 compressed high-resolution images. In one embodiment, the data set compression unit (1300) can reduce the number of high-resolution images belonging to each cluster by compressing each cluster.

[0129] In one embodiment, the data set compression unit (1300) may perform an initialization operation. In one embodiment, the data set compression unit (1300) may select some of the images belonging to the learning data set (310) to generate an initial compressed HR data set. In one embodiment, the data set compression unit (1300) may select some images for each cluster belonging to the cluster set to generate an initial compressed HR data set. In one embodiment, the data set compression unit (1300) may extract m high-resolution images for each cluster from the learning data set (310) to generate an initial compressed HR data set having m high-resolution images for each cluster.

[0130] In one embodiment, the data set compression unit (1300) may degrade the compressed HR data set to generate a compressed LR (Low Resolution) data set. In one embodiment, the data set compression unit (1300) may perform at least one of compression degrading, blurring degrading, resolution adjustment, or noise addition on the compressed HR data set to generate the compressed LR data set. In one embodiment, the compressed LR data set may include a low-resolution image having a lower resolution than a high-resolution image included in the compressed HR data set. In one embodiment, the data set compression unit (1300) may obtain a pair of compressed high-resolution images-compressed low-resolution images, which are training data for training a second neural network model, based on the training data set (310). For example, the second neural network model may be an inference network that implements a super-resolution (SR) algorithm that can convert a low-resolution (LR) image into a high-resolution (HR) image.

[0131] In addition, in one embodiment, the data set compression unit (1300) can generate a low-resolution data set by degrading the learning data set (310) through an initialization operation. In one embodiment, the data set compression unit (1300) can obtain a high-resolution image-low-resolution image pair, which is learning data for learning the first neural network model, based on the learning data set (310). For example, the first neural network model can be an inference network that implements a super-resolution (SR) algorithm that can convert a low-resolution (LR) image into a high-resolution (HR) image. For example, the first neural network model and the second neural network model can be a single neural network model with the same parameters, and can be neural network models that are learned with different learning data.

[0132] In one embodiment, the data set compression unit (1300) can update the compressed HR data set by comparing a first neural network model trained using the training data set (310) and a second neural network model trained using the compressed HR data set. In one embodiment, the data set compression unit (1300) can obtain a compressed data set (320) including the updated compressed HR data set.

[0133] In one embodiment, the data set compression unit (1300) may iteratively update the compressed HR data set so that the performance of the first neural network model trained with the training data set (310) is similar to the performance of the second neural network model trained with the compressed HR data set. In one embodiment, the data set compression unit (1300) may apply the training data set (310) to the first neural network model to calculate a first gradient, and apply the compressed HR data set to the second neural network model to calculate a second gradient.

[0134] Here, the gradient of a neural network model represents the slope of the loss function, which is the derivative of the loss function of the neural network model with respect to its parameters. The gradient can be used in the gradient descent algorithm to adjust the parameters of the neural network model in a direction that minimizes the loss function of the neural network model. For example, the direction of the gradient indicates the direction in which the parameters should move to minimize the loss function, and the magnitude of the gradient can indicate how much the parameters should be updated.

[0135] For example, the first gradient may represent the slope of the first loss with respect to the parameters of the first neural network model. The second gradient may represent the slope of the second loss with respect to the parameters of the second neural network model.

[0136] In one embodiment, the data set compression unit (1300) may update pixels of images in the compressed HR data set based on the first gradient and the second gradient. In one embodiment, the data set compression unit (1300) may calculate a matching loss, which is a difference between the first gradient of the first neural network model and the second gradient of the second neural network model. In one embodiment, the data set compression unit (1300) may adjust pixels of the compressed HR data set in a direction in which the matching loss is minimized. For example, the data set compression unit (1300) may calculate a matching gradient for minimizing the matching loss. For example, the matching gradient may represent a slope of the matching loss for each pixel of the images in the compressed HR data set. The matching gradient may be used to adjust pixels of images in the compressed HR data set in a direction in which the matching loss is minimized, in a manner similar to a gradient descent algorithm. For example, the gradient may be similar to the way in which the loss is inclination with respect to the parameters of a neural network model, and is used to adjust the parameters of a neural network model in a direction that minimizes the loss using a gradient descent algorithm.

[0137] In one embodiment, the data set compression unit (1300) can calculate pixels of images belonging to the compressed HR data set to be updated to minimize matching loss through a matching gradient. The data set compression unit (1300) can update pixels of images in the compressed HR data set through the matching gradient. In one embodiment, the data set compression unit (1300) can generate a compressed HR data set, which is a small training data set having similar characteristics to the large training data set (310).

[0138] In one embodiment, when a meta model is trained using a dataset consisting of 256 high-quality images of 32x32 size, the performance of the neural network model may be similar to that of the same meta model when trained using a compressed HR dataset consisting of 64 compressed high-quality images of 32x32 size. Accordingly, the memory usage may be reduced by approximately four times, and the amount of computation required to update the neural network model may also be reduced by approximately four times. The above-described figures are provided for convenience of explanation and are not limited to the above-described figures.

[0139] In one embodiment, the server (1000) generates a compressed HR data set, which is a small-scale learning data set similar to the characteristics of a large-scale learning data set (310), so that the memory usage of the learning data required for the image processing device (100) to learn the meta model can be reduced. In one embodiment, the image processing device (100) can maintain the upscaling performance of the image processing device (100) even if it learns the meta model with a small-scale learning data set instead of learning the meta model with a large-scale learning data set. Accordingly, the amount of learning data set required for learning an AI model for performing upscaling in the image processing device (100) can be reduced, and the usage of external memory and power consumption for operations in the image processing device (100) can be reduced.

[0140] FIG. 4 is a diagram illustrating a clustered learning data set according to a clustering operation of a server according to one embodiment.

[0141] Referring to FIG. 4, the server (1000) can preprocess a learning data set (310) containing hundreds of thousands of high-resolution images through a preprocessing unit (1100). The server (1000) can cluster the learning data set (310) into 128 clusters, each cluster having 100 high-resolution images, through a clustering unit (1200). By clustering the learning data set (310), the server (1000) can generate a cluster set (410) having high-resolution images for each cluster.

[0142] For example, a cluster set (410) may include Cluster 0 (411), Cluster 1 (412), and Cluster 127 (413). Cluster 0 (411) may have 100 high-resolution images having similar characteristics among the high-resolution images belonging to the learning data set (310). Cluster 1 (412) may have 100 high-resolution images having similar characteristics among the high-resolution images belonging to the learning data set (310). Cluster 127 (413) may have 100 high-resolution images having similar characteristics among the high-resolution images belonging to the learning data set (310).

[0143] The server (1000) can cluster high-resolution images belonging to the learning data set (310) based on statistical characteristics of the high-resolution images through the clustering unit (1200). For example, in FIG. 4, Cluster 0 (411) may include high-resolution images including horizontal edge data, Cluster 1 (412) may include high-resolution images including vertical edge data, and Cluster 127 (413) may include high-resolution images including texture data.

[0144] Meanwhile, the number of clusters belonging to the cluster set (410) may vary depending on user input and is not limited to 128. In addition, the number of high-resolution images belonging to each cluster is an exemplary number for convenience of explanation and is not limited to 100.

[0145] The server (1000) can provide a cluster set (410) corresponding to the clustered learning data set (310) to the data set compression unit (1300).

[0146] FIG. 5 is an internal block diagram illustrating a data set compression unit of a server according to one embodiment. FIG. 6 is a diagram illustrating an operation of a server according to one embodiment to compress a learning data set. In the present disclosure, a learning data set (611) may correspond to a large-scale learning data set (10) of FIG. 1 or a learning data set (310) of FIG. 3. In addition, the learning data set (611) may correspond to a cluster set (410) in which images belonging to the learning data set (611) are grouped by cluster. In addition, a compressed HR data set (621) may correspond to a small-scale learning data set (20) of FIG. 1 or a compressed data set (320) of FIG. 3. Meanwhile, in the present disclosure, the learning data set (611) may have higher-resolution images than images belonging to the low-resolution data set (612), and thus may be referred to as a 'high-resolution data set.'

[0147] Referring to FIGS. 5 and 6, the data set compression unit (1300) of the server (1000) may include an initialization unit (510), a gradient matching unit (520), a compressed HR data set update unit (530), a compressed LR data set update unit (540), and a parameter update unit (550). The gradient matching unit (520), the compressed HR data set update unit (530), the compressed LR data set update unit (540), and the parameter update unit (550) may be referred to as an iteration unit because they operate repeatedly.

[0148] In one embodiment, the initialization unit (510) may perform a first initialization operation for generating an initial compressed HR data set (601) based on a learning data set (611). The initialization unit (510) may perform a second initialization operation for obtaining an initial compressed LR data set based on the initial compressed HR data set (601) and obtaining a low-resolution data set (612) based on the learning data set (611). The initialization unit (510) may perform a third initialization operation for initializing parameters of a neural network model.

[0149] In one embodiment, the initialization unit (510) may generate an initial compressed HR data set (601) based on a learning data set as a first initialization operation. In one embodiment, the initialization unit (510) may select some images from images belonging to the learning data set (611) to generate the initial compressed HR data set (601). The initialization unit (510) may initialize a compressed HR data set (621).

[0150] In one embodiment, the initialization unit (510) may determine the number of compressed high-resolution images among the high-resolution images belonging to each cluster based on user input. In one embodiment, the user may select m images among the i high-resolution images belonging to each cluster (m is a natural number). The initialization unit (510) may generate an initial compressed HR data set (601) using m high-resolution images randomly selected among the i high-resolution images belonging to each cluster.

[0151] In one embodiment, if the initial compressed HR data set (601) is determined to have m high-resolution images for each cluster, the number of final compressed high-resolution images for each cluster can be determined. For example, the initialization unit (510) can perform compression for 128 clusters, each cluster having 100 (i=100) high-resolution images. For example, the user can decide to have 1 (m=1) compressed high-resolution image for each cluster having 100 high-resolution images. For example, the initialization unit (510) can compress the 100 high-resolution images belonging to each cluster into 1 compressed high-resolution image. For example, the initialization unit (510) can cluster a learning data set (611) consisting of 12,800 high-resolution images into 128 clusters, and compress 100 high-resolution images belonging to each cluster into one high-resolution image for each cluster, thereby generating a compressed HR data set (621) having 128 compressed high-resolution images. For example, since one compressed high-resolution image having similar characteristics to the 100 high-resolution images can be generated for each cluster, the number of data can be reduced.

[0152] In one embodiment, the initialization unit (510) may generate a high-resolution image-low-resolution image pair, which is training data for training a neural network model, as a second initialization operation. In one embodiment, the initialization unit (510) may degrade the training data set (611) to obtain a low-resolution data set (612). In one embodiment, the initialization unit (510) may obtain a low-resolution data set (612) in which the training data set corresponding to the training data set (611) is degrade-processed. In one embodiment, the initialization unit (510) may obtain a compressed LR data set (622) in which the compressed HR data set (621) is degrade-processed. In one embodiment, the degrade processing may include at least one of compression degrade, blurring degrade, down-sampling, or noise addition to the image. In one embodiment, compression degradation includes a process of encoding / decoding an image, and may degrade a still image using a JPEG compression method.

[0153] In one embodiment, the initialization unit (510) may initialize the parameters of the neural network model as the third initialization operation. For example, the neural network model may be an inference network capable of upscaling a low-resolution (LR) image into a high-resolution (HR) image. For example, the parameters of the neural network model may include weights, gradients, biases, etc. In the present disclosure, the first neural network model (630) trained using a training data set and the second neural network model (640) trained using a compressed data set may be the same neural network model derived from a single neural network model. In FIG. 6, the first neural network model (630) and the second neural network model (640) are each briefly illustrated as having two layers, but are not limited thereto.

[0154] In one embodiment, the gradient matching unit (520), the compressed HR data set update unit (530), the compressed LR data set update unit (540), and the parameter update unit (550) may iteratively operate so that the parameters of the second neural network model (640) trained with the compressed data set approximate the parameters of the first neural network model (630) trained with the learning data set.

[0155] In one embodiment, the gradient matching unit (520) may match the second gradient of the second neural network model (640) to the first gradient of the first neural network model (630) so that the performance of the second neural network model (640) trained with the compressed HR data set (621) becomes similar to the performance of the first neural network model (630) trained with the training data set (611).

[0156] In one embodiment, the gradient matching unit (520) may apply the learning data set (611) and the low-resolution data set (612) to the first neural network model (630) to calculate a first gradient. In one embodiment, the gradient matching unit (520) may apply the compressed HR data set (621) and the compressed LR data set (621) to the second neural network model (640) to calculate a second gradient.

[0157] In one embodiment, the gradient of the neural network model can be used in the gradient descent algorithm. Gradient descent is an optimization algorithm for finding a first-order approximation. It is a method of finding the parameter (x value) when the loss function (y value) is at a minimum by differentiating the loss function with respect to the parameters of the neural network model to obtain the gradient of the loss function and continuously moving toward the direction with a lower absolute value of the gradient. In one embodiment, the gradient of the neural network model can be updated to the parameters of the neural network model through a backward pass.

[0158] For example, the parameters of a neural network model can be updated via mathematical expression 1.

[0159]

[0160] In mathematical expression 1, w (t+1) are the updated parameters of the neural network model, and w t is the current parameter of the neural network model, lr is the learning rate, Loss is the loss function, is the gradient, which is the slope of the loss function (Loss) with respect to the parameters (w) of the neural network model. The gradient is the value obtained by differentiating the parameters (w) of the neural network model with respect to the loss function (Loss). The direction of the gradient indicates the direction in which the parameters (w) should move to minimize the loss function (Loss), and the magnitude of the gradient can indicate how much the parameters (w) should be updated. The parameters of the neural network model can be calculated by reflecting the gradient on the current parameters of the neural network model.

[0161] For example, the gradient matching unit (520) can obtain the first loss (631) by calculating the difference between the output value (635) obtained by applying the low-resolution data set (612) to the first neural network model (630) and the learning data set (611). The gradient matching unit (520) can obtain the first gradient by differentiating the first loss (631) with the parameters of the first neural network model (630). The gradient matching unit (520) can calculate the first gradient for minimizing the first loss (631). The gradient matching unit (520) can reflect the calculated first gradient in the parameters of the first neural network model (630) to update the parameters of the first neural network model (630). For example, the gradient matching unit (520) can obtain the second loss (641) by calculating the difference between the output value (645) obtained by applying the compressed LR data set (622) to the second neural network model (640) and the compressed HR data set (621). The gradient matching unit (520) can obtain the second gradient by differentiating the second loss (641) with the parameters of the second neural network model (640). The gradient matching unit (520) can calculate the second gradient for minimizing the second loss (641). The gradient matching unit (520) can reflect the calculated second gradient in the parameters of the second neural network model (640) to update the parameters of the second neural network model (640). Meanwhile, the initial parameters of the first neural network model (630) and the initial parameters of the second neural network model (640) may be the same.

[0162] In one embodiment, the gradient matching unit (520) can update the compressed HR data set (621) based on the first gradient and the second gradient. In one embodiment, the gradient matching unit (520) can calculate a matching loss (650), which is a difference between the first gradient and the second gradient. For example, the gradient matching unit (520) can calculate a matching gradient so that the matching loss (650) is minimized. The matching gradient can be used to adjust pixels of images in the compressed HR data set (621) in a direction that minimizes the matching loss, in a manner similar to a gradient descent algorithm.

[0163] For example, the compressed HR data set (621) can be updated through mathematical expression 2.

[0164]

[0165] In mathematical expression 2, p (t+1) is the updated pixel value of the image in the compressed HR data set (621), and p t is the current pixel value of the image in the compressed HR data set (621), lr is the learning rate, and Matching Loss is the matching loss (650). is a matching gradient, which is the slope of the matching loss (650) for the pixel value (p) of the image in the compressed HR data set (621). The matching gradient is a value obtained by differentiating the pixel value (p) of the image in the compressed HR data set (621) with respect to the matching loss (650). The direction of the matching gradient indicates the direction in which the pixel value (p) of the image in the compressed HR data set (621) should move to minimize the matching loss (650), and the size of the matching gradient may indicate how much the pixel value (p) of the image in the compressed HR data set (621) should be updated. The pixel value of the image in the compressed HR data set (621) may be calculated by reflecting the matching gradient to the current pixel value of the image in the compressed HR data set (621).

[0166] In one embodiment, the gradient matching unit (520) can differentiate the matching loss (650) by the pixels of the images in the compressed HR data set (621) to obtain the matching gradient of the matching loss (650), and can obtain the pixel (x value) of the compressed HR data set (621) when the matching loss (y value) has a minimum value by moving the absolute value of the matching gradient toward a lower side. In one embodiment, the gradient matching unit (520) can reflect the matching gradient to the pixels of the images in the compressed HR data set (621) to update the pixel values ​​of the images in the compressed HR data set (621). In one embodiment, the matching gradient can be updated to the compressed HR data set (621) through backward pass.

[0167] Meanwhile, in one embodiment, the size of the matching gradient may be equal to the number of pixels in each image in the compressed HR data set (621). For example, if the number of pixels is 10x10, the size of the matching gradient may be equal to 10x10.

[0168] In one embodiment, the compressed HR data set update unit (530) can update the compressed HR data set (621) based on the matching gradient. In one embodiment, the compressed HR data set update unit (530) can obtain the pixel value (x value) of the compressed HR data set (621) to be updated by moving in the direction in which the matching gradient (y value) decreases so that the matching loss (650) is minimized. In one embodiment, the compressed HR data set update unit (530) can generate the compressed HR data set (621) including the images having the updated pixel by reflecting the matching gradient to the current pixel of the images in the compressed HR data set (621). For example, the compressed HR data set update unit (530) can update the pixels of the images in the compressed HR data set (621) by adding a value obtained by multiplying the matching gradient by a learning rate to the pixels of the images in the compressed HR data set.

[0169] In one embodiment, when the matching loss (650), which is the difference between the first gradient and the second gradient, is minimized, the upscaling performance of the first neural network model (630) and the upscaling performance of the second neural network model (640) may be similar.

[0170] In one embodiment, the matching gradient is a gradient differentiated with respect to the pixels of the image with respect to the gradient obtained by differentiating the loss function (631, 641) with respect to the parameters of the neural network model, and thus may be a gradient differentiated twice from the loss function. The matching gradient may also be expressed by the following equation 1.

[0171] [Formula 1]

[0172]

[0173] In formula 1, represents the first gradient, which is the slope of the first loss (631), represents the second gradient, which is the slope of the second loss (641).

[0174] Typically, the loss function may be the mean absolute error (MAE) that takes an absolute value, or the mean squared error (MSE) that takes a square value. However, in one embodiment, the server (1000) needs to calculate the matching gradient that is differentiated twice from the loss function in order to update the compressed HR data set (621). Therefore, it may be preferable to apply the mean squared error (MSE) instead of the mean absolute error (MAE) as the loss function of the neural network model according to one embodiment. Therefore, the loss function of the neural network model according to one embodiment may not be the mean absolute error (MAE).

[0175] In one embodiment, the compressed HR data set (621) acquired through the compressed HR data set update unit (530) may be different from the initialized compressed HR data set (621) acquired through the initialization unit (510). For example, the high-definition image belonging to the compressed HR data set (621) acquired through the compressed HR data set update unit (530) may be different from any one of the pixel information, contrast value, luminance information, shape, or embedding vector of the high-definition image belonging to the initialized compressed HR data set (621) acquired through the initialization unit (510).

[0176] In one embodiment, the compressed LR data set update unit (540) can degrade the updated compressed HR data set (621) to update the compressed LR data set (622). In one embodiment, the compressed LR data set update unit (540) can obtain the compressed LR data set (622) by performing at least one of compression degrading, blurring degrading, resolution adjustment (e.g., down sampling), or noise addition on the compressed HR data set (621). In one embodiment, the compressed LR data set update unit (540) includes an encoding / decoding process and can degrade a still image using a JPEG compression method.

[0177] In one embodiment, the parameter update unit (550) may apply the updated compressed HR data set (621) and the updated compressed LR data set (622) to the second neural network model (640) to update the parameters of the second neural network model (640). In one embodiment, the parameter update unit (550) may calculate a second loss (641) between an output value obtained by applying the updated compressed HR data set (621) to the second neural network model (640) and the updated compressed LR data set (622). In one embodiment, the parameter update unit (550) may calculate a second gradient for the parameters of the second neural network model (640) based on the second loss (641) and update the parameters of the second neural network model (640). In one embodiment, the parameter update unit (550) may update the parameters such that the second loss (641) is minimized. In one embodiment, the parameter update unit (550) can train the second neural network model (640) by updating the parameters of the second neural network model (640).

[0178] Meanwhile, in one embodiment, the parameters of the second neural network model (640) may also be updated with the parameters of the first neural network model (630). For example, the parameters of the first neural network model (630) may not be updated with parameters based on the training data set, but may be updated with parameters of the second neural network model (640).

[0179] In one embodiment, the server (1000) may repeat the process of updating the second neural network model (640) and the process of updating the compressed HR data set (621). In one embodiment, the server (1000) may obtain a second gradient by differentiating the second loss (641) of the second neural network model (640) with respect to the parameters of the second neural network model (640), obtain parameters for minimizing the second loss (641), and update the second neural network model (640). In one embodiment, the server (1000) may obtain a matching gradient by differentiating the matching loss (650) with respect to the compressed HR data set (621), and obtain updated pixel values ​​of an image within the compressed HR data set for minimizing the matching loss (650).

[0180] FIG. 7a is a diagram illustrating an initial compressed HR data set and a compressed HR data set according to a compression operation of a server according to one embodiment. FIG. 7b is a diagram specifically illustrating an initial compressed HR data set and a compressed HR data set according to a compression operation of a server according to one embodiment.

[0181] Referring to FIG. 7A, the server (1000) can cluster preprocessed high-resolution images within a learning data set through a clustering unit (1200). The server (1000) can generate a cluster set (410) clustered into 128 clusters, each of which has 100 high-resolution images (see FIG. 4). The server (1000) can generate a compressed HR data set (702) including compressed high-resolution images for each cluster (411, 412, 413) belonging to the cluster set (410) through a data set learning unit (1300).

[0182] The server (1000) can generate an initial compressed HR data set (701) from each cluster (411, 412, 413) belonging to the cluster set (410) through the initialization unit (510). The server (1000) can determine the number (m) of high-resolution images that will constitute the initial compressed HR data set (701) for each cluster (411, 412, 413) based on a user input. The server (1000) can generate a compressed HR data set (702) having the same number of compressed high-resolution images as the number of high-resolution images of the initial compressed HR data set (701) through the iteration units (520, 530, 540, 550).

[0183] The server (1000) can select one initial high-resolution image (m=1) from among 100 high-resolution images belonging to each cluster (411, 412, 413) for each of 128 clusters (411, 412, 413). The server (1000) can generate an initial compressed HR data set (701) having one initial high-resolution image for each cluster (411, 412, 413) through the initialization unit (510). For example, the initial compressed HR data set (701) can have 128 initial high-resolution images belonging to each of 128 clusters, as shown in FIG. 7b.

[0184] The server (1000) can generate one compressed high-resolution image by repeatedly updating one initial high-resolution image for each of 128 clusters (411, 412, 413). The server (1000) can generate 128 compressed HR data sets (702) by updating 128 initial high-resolution images belonging to the initial compressed HR data set (701) through the iteration unit. For example, the compressed HR data set (702) can have 128 compressed high-resolution images belonging to each of 127 clusters, as shown in FIG. 7b. For example, the server (1000) can extract one initial compressed high-resolution image from 100 high-resolution images belonging to Cluster 0 (411) and generate one compressed high-resolution image through iteration. For example, the parameters of the second neural network model (640) trained with one compressed high-resolution image can be approximated to the parameters of the first neural network model (630) trained with 100 high-resolution images belonging to Cluster 0 (411).

[0185] In one embodiment, the high-resolution images belonging to the cluster set (410) may include images that match the initial compressed high-resolution images generated through the initialization unit (510). Since the server (1000) selects some images from among the high-resolution images belonging to the cluster set (410) to generate the initial compressed high-resolution images, the initial compressed high-resolution images may match the high-resolution images belonging to the cluster set (410).

[0186] Meanwhile, in one embodiment, the initial compressed high-resolution image may have different data than the compressed high-resolution image acquired through the iteration unit. Since the server (1000) updates the pixels of the initial compressed high-resolution image through the iteration unit, the initial compressed high-resolution image and the compressed high-resolution image may differ. For example, the compressed high-resolution image may have different pixel information, contrast values, luminance information, shape, embedding vectors, etc. from the initial high-resolution image.

[0187] FIG. 8a is a flowchart illustrating a method for compressing a learning data set according to one embodiment. FIG. 8a will be described in conjunction with FIGS. 3 and 6.

[0188] Referring to FIG. 8A, a method for compressing a learning data set according to one embodiment may include steps 710 to 740. In one embodiment, steps 710 to 740 may be executed by at least one processor included in the server (1000). In one embodiment, the method for compressing a learning data set is not limited to that illustrated in FIG. 8A, and may further include steps not illustrated in FIG. 8A.

[0189] In step 710, the server (1000) according to one embodiment can generate a cluster set (410) by clustering the learning data set (310). For example, in order to efficiently compress the learning data set (310), the server (1000) can generate a cluster set (410) by clustering high-resolution images having similar characteristics without compressing the entire learning data set at once. The server (1000) can perform compression on the cluster set (410). For example, the server (1000) can perform compression on each cluster belonging to the cluster set (410). For example, the server (1000) can compress high-resolution images for each cluster in which high-resolution images having similar characteristics are grouped. The server (1000) can obtain a compressed high-resolution data set (compressed HR data set) having compressed high-resolution images for each cluster.

[0190] In one embodiment, the server (1000) may cluster the learning data set (310) based on pixel information, contrast values, luminance information, shape, embedding vectors, etc. of high-resolution images included in the learning data set (310). In one embodiment, the server (1000) may cluster the learning data set (310) using a conventional algorithm, such as a K-means clustering algorithm.

[0191] In one embodiment, the number of clusters may be determined based on user input. For example, the server (1000) may cluster the data into k clusters, each having i high-resolution images (where i and k are natural numbers). For example, the server (1000) may cluster the data into 128 clusters, each having 100 high-resolution images. For example, the server (1000) may generate a cluster set (410) in which the training data set (310) is clustered into 128 clusters.

[0192] According to one embodiment, the server (1000) may perform preprocessing on the learning data set (310) based on the amount of information of each of the high-resolution images belonging to the learning data set (310) before performing step 710. The cluster set (410) may be clustered based on the preprocessed learning data set (310).

[0193] In step 720, the server (1000) according to one embodiment may generate an initial compressed HR (High Resolution) data set by selecting some images from among the images belonging to each cluster in the cluster set (410). In the present disclosure, the cluster set (410) may correspond to the learning data set (611).

[0194] According to one embodiment, the server (1000) may select some of the images belonging to each cluster according to the first initialization operation. According to one embodiment, the server (1000) may determine the number of images to be extracted from each cluster based on a user input. In one embodiment, the number of images extracted from each cluster is the same as the number of images belonging to the initial compressed HR data set (601), and may be the same as the number of images belonging to the compressed HR data set (621). For example, the server (1000) may determine the number of final compressed high-resolution images based on a user input.

[0195] According to one embodiment, the server (1000) may degrade a high-resolution image belonging to the learning data set (611) according to the second initialization operation to obtain a low-resolution data set (612) having a low-resolution image. The server (1000) may degrade an initial compressed HR data set (601) according to the second initialization operation to obtain an initial compressed LR data set. The server (1000) may perform at least one of compression degrading, blurring degrading, resolution adjustment, or noise addition on the initial compressed HR data set (601) to obtain the initial compressed LR data set.

[0196] According to one embodiment, the server (1000) may initialize parameters of the first neural network model and parameters of the second neural network model according to the third initialization operation. For example, the parameters may include weights, gradients, biases, etc.

[0197] In step 730, the server (1000) according to one embodiment may input the learning data set (611) into the first neural network model (630) to obtain the first loss (631), and input the initial compressed HR data set (601) into the second neural network model (640) to obtain the second loss (641).

[0198] For example, a neural network model may be an inference network that implements a super-resolution (SR) algorithm that can transform low-resolution (LR) images into high-resolution (HR) images. For example, the first and second neural network models may be a single neural network model with identical parameters, but trained with different training data.

[0199] The server (1000) can obtain the first loss (631) by calculating the difference between the output value (635) obtained by applying the low-resolution data set (612) to the first neural network model (630) and the learning data set (611).

[0200] The server (1000) can obtain the second loss (641) by calculating the difference between the output value (645) obtained by applying the initial compressed LR data set to the second neural network model (640) and the initial compressed HR data set.

[0201] In addition, through the iteration process, the server (1000) can obtain the second loss (641) by calculating the difference between the output value (645) obtained by applying the compressed LR data set (622) to the second neural network model (640) and the compressed HR data set (621).

[0202] In step 740, the server (1000) according to one embodiment can generate a compressed HR data set (621) by updating each pixel of images belonging to the initial compressed HR data set (601) based on the first loss (631) and the second loss (641). The server (1000) can update the compressed HR data set (621) based on the first loss (631) and the second loss (641) that are repeatedly produced through an iteration process. By updating the compressed HR data set (621), the server (1000) can generate a compressed HR data set (621) having similar characteristics to the learning data set (611). The meaning that the compressed HR data set (621) has similar characteristics to the learning data set (611) may mean that the parameters of the second neural network model (640) learned using the compressed HR data set (621) are similar to the parameters of the first neural network model (630) learned using the learning data set (611).

[0203] According to one embodiment, the server (1000) may update the compressed HR data set (621) so that the upscaling performance of the first neural network model (630) trained using the training data set (611) and the upscaling performance of the second neural network model (640) trained using the compressed HR data set (621) are similar. For example, when the gradients of the two neural network models are similar, the upscaling performances of the two neural network models can be expressed as being similar.

[0204] According to one embodiment, the server (1000) may calculate a first gradient of a first neural network model (630) trained using a training data set (611). The server (1000) may calculate a second gradient of a second neural network model (640) trained using a compressed HR data set (621). The server (1000) may update the compressed HR data set (621) so that the difference between the first gradient and the second gradient is reduced. For example, the server (1000) may update the compressed HR data set until the difference between the first gradient and the second gradient is minimized.

[0205] The first gradient may represent the slope of the first loss (631) with respect to the weight of the first neural network model (630). For example, the first gradient may be It can be expressed as. Here, Loss1 is the first loss (631), and w1 may be the weight of the first neural network model (630). The first loss (631) according to one embodiment may be characterized by not being a mean absolute error. For example, a mean squared error that can be differentiated twice, etc. may be applied to the first loss (631).

[0206] The second gradient may represent the slope of the second loss (641) with respect to the weights of the second neural network model (640). For example, the second gradient may be It can be expressed as. Here, Loss2 is the second loss (641), and w2 may be the weight of the second neural network model (640). The second loss (641) according to one embodiment may be characterized by not being a mean absolute error. For example, a mean squared error that can be differentiated twice, etc. may be applied to the second loss (641).

[0207] According to one embodiment, the server (1000) can calculate a matching loss (650), which is the difference between the first gradient and the second gradient. For example, the matching loss (650) can be expressed as Loss1-Loss2. The server (1000) can adjust the pixels of the images belonging to the compressed HR data set (621) in a direction in which the matching loss (650) is minimized. The server (1000) can calculate a matching gradient to minimize the matching loss (650). The matching gradient can be used to adjust the pixels of the images in the compressed HR data set (621) in a direction in which the matching loss (650) is minimized, in a manner similar to a gradient descent algorithm.

[0208] The matching gradient may represent the slope of the matching loss (650) for each pixel of the images in the compressed HR data set (621). For example, the matching gradient may be can be expressed as . Here, the matching loss is the matching loss (650) which is the difference between the first loss (631) and the second loss (641), and p can be the pixel of each image in the compressed HR data set (621).

[0209] In one embodiment, the server (1000) can obtain a matching gradient, which is a slope for the matching loss (650), by differentiating the matching loss (650) with respect to the pixels of images in the compressed HR data set (621). Similarly to a gradient descent algorithm, the server (1000) can obtain a pixel (x value) of the compressed HR data set (621) when the matching loss (y value) has a minimum value by moving the absolute value of the matching gradient toward a lower value.

[0210] In one embodiment, the server (1000) can update pixel values ​​of images in the compressed HR data set (621) by reflecting the matching gradient to the pixels of the images in the compressed HR data set (621) (see Equation 2). In one embodiment, the matching gradient can be updated to the compressed HR data set (621) through backward pass.

[0211] According to one embodiment, the server (1000) may update the compressed HR data set (621) by repeating steps 720, 730, and 740 so that the matching loss (650), which is the difference between the first gradient and the second gradient, is minimized. According to one embodiment, the server (1000) may generate the compressed HR data set (621) having similar characteristics to the learning data set (611). In one embodiment, the second gradient of the second neural network model (640) trained using the compressed HR data set (621) may mean that it is approximate to the first gradient of the first neural network model (630) trained using the learning data set (611). In one embodiment, the number of images belonging to the compressed HR data set (621) may be less than the number of images belonging to the learning data set (611). Accordingly, the amount of learning data set required for learning an AI model for performing upscaling in an image processing device (100) is reduced, and the amount of external memory used and power consumption for calculation in the image processing device (100) can be reduced.

[0212] FIG. 8b is a specific flowchart illustrating a method for compressing a learning data set according to one embodiment. FIG. 8b will be described in conjunction with FIGS. 3 and 6.

[0213] Referring to FIG. 8B, a method for compressing a learning data set according to one embodiment may include steps 750 to 790. In one embodiment, steps 750 to 790 may be executed by at least one processor included in the server (1000). In one embodiment, the method for compressing a learning data set is not limited to that illustrated in FIG. 8B, and may further include steps not illustrated in FIG. 8B.

[0214] In step 750, the server (1000) according to one embodiment may generate an initial compressed HR data set (601) by selecting some of the images belonging to each cluster in the cluster set (410). Step 750 corresponds to step 720 of FIG. 8A.

[0215] In step 762, the server (1000) according to one embodiment may perform image quality processing on the learning data set (611) to obtain a low-resolution data set (612). For example, the server (1000) may perform at least one of compression degradation, blurring degradation, resolution adjustment, or noise addition on the learning data set (611) to obtain a low-resolution data set (612) with image quality degradation.

[0216] In step 764, the server (1000) according to one embodiment can obtain a first loss (631) of the first neural network model (630) based on the training data set (611) and the low-resolution data set (612). For example, the server (1000) can calculate the first loss (631), which is the difference between the output value (635) obtained by inputting the low-resolution data set (612) into the first neural network model (630) and the training data set (611).

[0217] In step 766, the server (1000) according to one embodiment may calculate a first gradient for minimizing the first loss (631). For example, the first gradient may represent a slope of the first loss (631) with respect to a parameter of the first neural network model (630). For example, the parameter may be a weight of the first neural network model (630). For example, the first gradient may be It can be expressed as . Here, Loss1 may be the first loss (631), and w1 may be the weight of the first neural network model (630).

[0218] In step 763, the server (1000) according to one embodiment may perform image quality processing on the initial compressed HR data set (601) to obtain an initial compressed LR data set. For example, the server (1000) may perform at least one of compression degradation, blurring degradation, resolution adjustment, or noise addition on the initial compressed HR data set (601) to obtain an image quality-degraded initial compressed LR data set.

[0219] In step 765, the server (1000) according to one embodiment may obtain a second loss (641) of the second neural network model (640) based on the initial compressed HR data set (601) and the initial compressed LR data set. For example, the server (1000) may calculate the second loss (641), which is the difference between the output value (645) obtained by inputting the initial compressed LR data set into the second neural network model (640) and the initial compressed HR data set (601).

[0220] In addition, the server (1000) according to one embodiment can calculate the second loss (641) based on the compressed HR data set (621) and the compressed LR data set (622) through an iteration process. For example, the server (1000) can calculate the difference between the output value (645) obtained by applying the compressed LR data set (622) to the second neural network model (640) and the compressed HR data set (621) to obtain the second loss (641).

[0221] In step 767, the server (1000) according to one embodiment may calculate a second gradient for minimizing the second loss (641) of the second neural network model (640). For example, the second gradient may represent the slope of the second loss (641) with respect to the parameters of the second neural network model (640). For example, the parameters may be weights of the second neural network model (640). For example, the second gradient may be It can be expressed as . Here, Loss2 may be the second loss (641), and w2 may be the weight of the second neural network model (640).

[0222] In step 770, the server (1000) according to one embodiment may calculate a matching loss (650), which is the difference between the first gradient and the second gradient. For example, the matching loss (650) may be expressed as Loss1-Loss2.

[0223] In step 780, the server (1000) according to one embodiment may calculate a matching gradient to minimize the matching loss (650). The server (1000) may adjust pixels of images belonging to the compressed HR data set (621) in a direction in which the matching loss (650) is minimized. The server (1000) may calculate a matching gradient to minimize the matching loss (650). The matching gradient may be used to adjust pixels of images within the compressed HR data set (621) in a direction in which the matching loss (650) is minimized, in a manner similar to a gradient descent algorithm.

[0224] The matching gradient may represent the slope of the matching loss (650) for each pixel of the images in the compressed HR data set (621). For example, the matching gradient may be can be expressed as . Here, the matching loss is the matching loss (650) which is the difference between the first loss (631) and the second loss (641), and p can be a pixel of each image in the compressed HR data set (621). The server (1000) can obtain the pixel (x value) of the compressed HR data set (621) when the matching loss (y value) has the minimum value by moving the absolute value of the matching gradient toward the lower side.

[0225] In step 790, the server (1000) according to one embodiment may update each pixel of the images belonging to the compressed HR data set (621) based on the matching gradient. The server (1000) may update the pixel values ​​of the images in the compressed HR data set (621) by reflecting the matching gradient to the pixels of the images in the compressed HR data set (621) (see Equation 2). In one embodiment, the matching gradient may be updated to the compressed HR data set (621) through backward pass.

[0226] FIG. 8c is a flowchart illustrating a method for compressing a learning data set according to one embodiment. FIG. 8c will be described in conjunction with FIGS. 3 and 6.

[0227] Referring to FIG. 8C, a method for compressing a learning data set according to one embodiment may include steps 810 to 860. In one embodiment, steps 810 to 860 may be executed by at least one processor included in the server (1000). In one embodiment, the method for compressing a learning data set is not limited to that illustrated in FIG. 8C, and may further include steps not illustrated in FIG. 8C.

[0228] In step 810, the server (1000) according to one embodiment can generate a cluster set (410) by clustering the learning data set (310). Step 810 corresponds to step 710 of FIG. 8A.

[0229] In step 820, the server (1000) according to one embodiment may generate an initial compressed HR (High Resolution) data set by selecting some images from among the images belonging to each cluster in the cluster set (410). In the present disclosure, the cluster set (410) may correspond to the learning data set (611). Step 820 corresponds to step 720 of FIG. 8A.

[0230] In step 830, the server (1000) according to one embodiment may input the learning data set (611) into the first neural network model (630) to obtain the first loss (631), and input the initial compressed HR data set (601) into the second neural network model (640) to obtain the second loss (641). Step 830 corresponds to step 730 of FIG. 8A.

[0231] In step 840, the server (1000) according to one embodiment can generate a compressed HR data set (621) by updating each pixel of images belonging to the initial compressed HR data set (601) based on the first loss (631) and the second loss (641). Step 840 corresponds to step 740 of FIG. 8A.

[0232] In step 850, the server (1000) according to one embodiment may perform image quality processing on the updated compressed HR data set (621) to generate a compressed LR (Low Resolution) data set (622). The server (1000) according to one embodiment may perform at least one of compression degradation, blurring degradation, resolution adjustment, or noise addition on the compressed HR data set (621) to obtain the compressed LR data set (622).

[0233] In step 860, the server (1000) according to one embodiment can update the parameters of the second neural network model (640) by applying the compressed HR data set (621) and the compressed LR data set (622) to the second neural network model (640).

[0234] According to one embodiment, the server (1000) may apply the updated compressed HR data set (621) and the updated compressed LR data set (622) to the second neural network model (640) to update the parameters of the second neural network model (640). According to one embodiment, the server (1000) may calculate a second loss (641) between the output value (645) obtained by applying the updated compressed HR data set (621) to the second neural network model (640) and the updated compressed LR data set (622). According to one embodiment, the server (1000) may calculate a second gradient for the parameters of the second neural network model (640) based on the second loss (641). According to one embodiment, the server (1000) may update the parameters of the second neural network model (640) based on the second gradient through a gradient descent algorithm or the like.

[0235] FIG. 9a is a diagram illustrating a process in which a server compresses a learning data set by category according to one embodiment.

[0236] Referring to FIG. 9A, a server (1000) according to one embodiment can classify categories of a large learning data set (910). By classifying categories of the learning data set (910), the server (1000) can generate a classified learning data set (930).

[0237] For example, the server (1000) may include an image classifier (920). The server (1000) may classify a large amount of learning data set (910) into categories through the image classifier (920). The server (1000) may identify categories of the large amount of learning data set (910) through the image classifier (920). The image classifier (920) may be an algorithm that receives an image and classifies a category of the image from the input image, or a set of algorithms, software that executes a set of algorithms, and / or hardware that executes a set of algorithms.

[0238] For example, the server (1000) can identify the category of the learning data set (910) and classify it into categories such as human face, character, general, etc. The server (1000) can generate a learning data set (930) in which the learning data set (910) is classified by category.

[0239] According to one embodiment, a server (1000) may compress images belonging to a learning data set (930) classified by category to generate multiple compressed data sets (940). According to one embodiment, a server (1000) may compress high-resolution images belonging to a learning data set (930) classified by category to generate multiple compressed data sets compressed by category. The server (1000) may obtain multiple compressed data sets (940) corresponding to multiple compressed data sets.

[0240] For example, the server (1000) may perform preprocessing, clustering, and data set compression for each of a plurality of learning data sets (930) classified by category. The server (1000) may execute a preprocessing unit (1100), a clustering unit (1200), and a data set compression unit (1300) for each of a plurality of learning data sets (930) classified by category, thereby generating a multi-compressed data set (940).

[0241] For example, the server (1000) can perform preprocessing, clustering, and data set compression on a learning data set corresponding to a category of a human face, thereby obtaining a compressed data set compressed by category.

[0242] For example, the server (1000) can perform preprocessing, clustering, and data set compression on a learning data set corresponding to a category of text, thereby obtaining a compressed data set compressed by category.

[0243] For example, the server (1000) can perform preprocessing, clustering, and data set compression on a learning data set corresponding to a general category to obtain a compressed data set compressed by category.

[0244] For example, the server (1000) can generate a multi-compressed data set (940) that includes multiple compressed data sets compressed by category.

[0245] In one embodiment, the image processing device (100) stores a multi-compressed data set (940) compressed by category in the memory (102), and can selectively acquire a compressed data set corresponding to a category related to the input image. This will be described in detail in FIG. 14A.

[0246] FIG. 9b is a diagram illustrating a process in which a server compresses a learning data set for each type of content according to one embodiment.

[0247] Referring to FIG. 9B, a server (1000) according to one embodiment can classify a large learning data set (910) by content type. In one embodiment, the server (1000) can generate a classified learning data set (950) by classifying the content of the learning data set (910) by type.

[0248] For example, the type of content refers to the type of content or the genre of content, and the type of content can include various types such as content for video calls, sports content, movie content, music content, game content, animation content, content provided through SNS for real-time chat, content provided through web browsers, 2D content, and 3D content.

[0249] According to one embodiment, a server (1000) can classify a large learning data set (910) by content type through an image classifier (920). The server (1000) can identify the content type of the large learning data set (910) through the image classifier (920). The image classifier (920) may be an algorithm that receives an image and classifies the type of content of the input image from the input image, or a set of algorithms, software that executes the set of algorithms, and / or hardware that executes the set of algorithms.

[0250] For example, the server (1000) can identify the type of content of the learning data set (910) and classify the images belonging to the learning data set (910) into types such as content for video calls, sports content, movie content, music content, game content, animation content, content provided by SNS for real-time chatting, content provided by a web browser, 2D content, and 3D content. The server (1000) can generate a learning data set (950) in which the learning data set (910) is classified by category.

[0251] According to one embodiment, a server (1000) may compress images belonging to a learning data set (950) classified by content type to generate multiple compressed data sets (960). According to one embodiment, a server (1000) may compress high-resolution images belonging to a learning data set (950) classified by content type to generate multiple compressed data sets compressed by content type. The server (1000) may obtain multiple compressed data sets (960) corresponding to the multiple compressed data sets. For example, the server (1000) may perform preprocessing, clustering, and data set compression on each of the multiple learning data sets (950) classified by content type.

[0252] For example, the server (1000) can perform preprocessing, clustering, and data set compression on a learning data set corresponding to video call content to obtain a compressed data set in which images corresponding to the video call content are compressed. For example, the server (1000) can perform preprocessing, clustering, and data set compression on a learning data set corresponding to sports content to obtain a compressed data set in which images corresponding to sports content are compressed. For example, the server (1000) can perform preprocessing, clustering, and data set compression on a learning data set corresponding to movie content to obtain a compressed data set in which images corresponding to movie content are compressed.

[0253] In one embodiment, the image processing device (100) stores a multi-compressed data set (940) compressed by category in the memory (102), and can selectively acquire a compressed data set corresponding to content related to the input image. This will be described in detail in FIG. 14c.

[0254] Meanwhile, the server (1000) according to one embodiment can classify the type of content without using the image classifier (920). For example, the server (1000) can classify the type of content through the properties, type, and genre of the application. Furthermore, for example, the server (1000) can identify the content corresponding to each image by identifying the metadata of the images belonging to the learning data set (910).

[0255] FIG. 10 is a flowchart illustrating a method for compressing a learning data set by category, according to one embodiment.

[0256] Referring to FIG. 10, in step 1010, a server (1000) according to one embodiment can classify a learning data set (910) based on categories.

[0257] For example, the server (1000) can identify categories of a large learning data set (910) through an image classifier (920). The server (1000) can classify the large learning data set (910) by category.

[0258] In step 1020, a server (1000) according to one embodiment can obtain a multi-compressed data set (940) by compressing a learning data set belonging to each category.

[0259] For example, the server (1000) may perform preprocessing, clustering, and data set compression on each of a plurality of high-resolution data sets belonging to each of a plurality of categories. The server (1000) may generate a multi-compressed data set (940) in which each of a plurality of learning data sets classified into a plurality of categories is independently compressed. For example, the server (1000) may include a plurality of compressed data sets that are independently compressed for each category.

[0260] Alternatively, the server (1000) may classify a large learning data set (910) by content type. In one embodiment, the server (1000) may generate a classified learning data set (950) by classifying the content of the learning data set (910) by type.

[0261] Hereinafter, with reference to FIGS. 11 to 15, a process of processing image quality of an input image using a compressed data set by an image processing device (100) according to one embodiment will be described.

[0262] Fig. 11 is an internal block diagram of a processor of an image processing device according to one embodiment.

[0263] Referring to FIG. 11, the processor (101) of the image processing device (100) may include an image quality determination unit (210), a model learning unit (220), and an image quality processing unit (230). The processor (101) may perform the operations of the image quality determination unit (210), the model learning unit (220), and the image quality processing unit (230) by executing one or more instructions stored in the memory (102).

[0264] In one embodiment, the image quality determination unit (210) can determine the image quality or quality of an input image. The image quality can indicate the degree of image deterioration. After an image is acquired through a capture device, it undergoes processing, compression, storage, transmission, restoration, etc., resulting in information loss and deterioration. The image quality determination unit (210) can analyze the image to determine the degree of image deterioration.

[0265] In one embodiment, the image quality determination unit (210) can analyze the input image in real time to determine at least one of image compression degradation, image sharpness degree, blur degree, noise degree, and image resolution.

[0266] In one embodiment, the image quality determination unit (210) may evaluate the image quality of an input image using a third neural network model trained to evaluate the image quality of an input image. In one embodiment, the third neural network model may be a neural network trained to evaluate the image quality of a video and / or image using an Image Quality Assessment (IQA) technique and / or a Video Quality Assessment (VQA) technique.

[0267] In one embodiment, the image quality determination unit (210) can analyze the input image and transmit the image quality of the input image obtained to the model learning unit (220).

[0268] In one embodiment, the model learning unit (220) can learn a meta model using the image quality value (quality value) for the input image obtained by the image quality determination unit (210). By using the input image as a learning data set, the model learning unit (220) can resolve the domain gap problem, in which the image quality of the input image changes rapidly.

[0269] In one embodiment, the model learning unit (220) can receive an input image. Additionally, the model learning unit (220) can receive the image quality of the input image from the image quality determination unit (210).

[0270] In one embodiment, the model learning unit (220) may train a meta-model using a training data set. Instead of using a large training data set, the model learning unit (220) may use a small training data set with similar characteristics to the large training data set. The small training data set may refer to a compressed HR data set compressed from the large training data set.

[0271] In one embodiment, the model learning unit (220) may obtain a compressed HR data set corresponding to the learning data set. In one embodiment, the model learning unit (220) may obtain a data set with degraded image quality corresponding to an input image from the compressed HR data set. In one embodiment, the model learning unit (220) may degrade the image quality of the images in the compressed HR data set so that the image quality of the images corresponds to the image quality of the input image. For example, the model learning unit (220) may degrade, blur, down-sample, or add noise to the images in the compressed HR data set to generate a low-resolution image.

[0272] In one embodiment, the model learning unit (220) can learn a meta model based on a compressed HR data set and a data set obtained by processing the compressed HR data set to correspond to an input image.

[0273] In one embodiment, the model learning unit (220) can train the meta model by updating the parameters of the meta model so that the difference between the output value obtained by applying the image quality deterioration data set to the meta model and the compressed HR data set is minimized. A meta model trained using a learning data set corresponding to an input image may be referred to as a transfer model.

[0274] In one embodiment, the performance of a meta-model trained using a compressed HR data set, which is a small training data set, may be similar to the performance of a meta-model trained using a large training data set. For example, the parameters of a meta-model trained using a compressed HR data set may be similar to or identical to the parameters of a meta-model trained using a training data set.

[0275] In one embodiment, the image quality processing unit (230) can load and use the meta model, i.e., the transfer model, acquired and trained by the model learning unit (220). The image quality processing unit (230) can process the image quality of the input image using the transfer model. In one embodiment, the image quality processing unit (230) can be a fourth neural network model trained to process the image quality of the input image. For example, the fourth neural network model can be an inference network that implements a super-resolution (SR) algorithm that can convert a low-resolution (LR) image into a high-resolution (HR) image. The fourth neural network model can perform the same function as the first neural network model (630 of FIG. 6) or the second neural network model (640 of FIG. 6), but is not limited thereto. The image quality processing unit (230) can obtain a high-resolution image by processing the image quality of the input image using SR technology using deep learning.

[0276] In one embodiment, the image processing device (100) may train the meta-model with a small compressed HR data set instead of training the meta-model with a large training data set. In one embodiment, the upscaling performance of the meta-model trained with a small compressed HR data set may be similar to the upscaling performance of the meta-model trained with a large training data set. Accordingly, the amount of training data set required for training the AI ​​model for performing upscaling in the image processing device (100) may be reduced, and the amount of external memory used and the power consumption for computation in the image processing device (100) may be reduced.

[0277] Referring to FIG. 12a, the model learning unit (220) may include a learning data collection unit (227), a degradation processing unit (221), and a transfer learning unit (225).

[0278] In one embodiment, the model learning unit (220) can obtain a meta model. The model learning unit (220) can use an input image and a compressed HR data set (621) as a learning data set. The model learning unit (220) can generate a low-resolution data set by degrading the learning data set in accordance with the image quality of the input image. The model learning unit (220) can train a meta model using the learning data set and the low-resolution data set as a high-resolution-low-resolution pair, thereby generating an adaptive transfer model for the input image.

[0279] In one embodiment, the learning data collection unit (227) can obtain an input image and a compressed HR data set (621). The learning data collection unit (227) can store the input image and the compressed HR data set (621) as a learning data set. The learning data collection unit (227) can store the learning data set in a memory (102 of FIG. 2). For example, the learning data set can correspond to the learning data set (1301) of FIG. 13. The compressed HR data set (621) can be stored in an external database or stored in the internal memory (102). The compressed HR data set (621) can be provided from a server (1000 of FIG. 2).

[0280] In one embodiment, the learning data collection unit (227) may obtain a compressed HR data set (621), which is a small learning data set, instead of a large learning data set. For example, the large learning data set may correspond to the learning data set (310) of FIG. 3 or the learning data set (611) of FIG. 6. For example, the compressed HR data set (621) may correspond to the compressed data set (320) of FIG. 3. For example, the compressed HR data set (621) may have a smaller number of high-resolution images than the large learning data set. In one embodiment, the learning data collection unit (227) may reduce the amount of memory (102) used and the amount of computation by using the compressed HR data set (621) as a learning data set for learning the meta model. In one embodiment, the learning data collection unit (227) may provide the stored learning data set to the deterioration processing unit (221). Additionally, in one embodiment, the learning data collection unit (227) can provide the stored learning data set to the transfer learning unit (225).

[0281] In one embodiment, the degradation processing unit (221) may receive a learning data set from the learning data collection unit (227). The degradation processing unit (221) may perform image quality processing on the learning data set that stores the input image and the compressed HR data set (621). The degradation processing unit (221) may obtain an image quality deteriorated data set by performing image quality processing on the learning data set based on the image quality of the input image, i.e., the quality value.

[0282] The degradation processing unit (221) can degrade the image quality of images belonging to the compressed HR data set (621) so that they have image quality corresponding to the image quality of the input image. The degradation processing unit (221) can degrade the input image so that they have image quality corresponding to the image quality of the input image. For example, the image quality corresponding to the image quality of the input image may be a quality value determined by the image quality determination unit (210 of FIG. 11) as deterioration information lost during the process of compression, storage, transmission, restoration, etc. of the input image. For example, the degradation processing unit (221) can generate images with degraded image quality by performing at least one of compression degradation, blurring, adding noise, or down sampling on images belonging to the compressed HR data set (621). In addition, the deterioration processing unit (221) can generate an image with deteriorated image quality by performing at least one of the following methods: compressing the input image, blurring, adding noise, or down-sampling the input image. The deterioration processing unit (221) can generate an image with deteriorated image quality data set including images with deteriorated image quality.

[0283] In one embodiment, the degradation processing unit (221) can transmit the image quality degradation data set to the transfer learning unit (225).

[0284] In one embodiment, the transfer learning unit (225) may receive a learning data set including a compressed HR data set (621) and an input image from the learning data collection unit (227). The transfer learning unit (225) may receive a data set with degraded image quality from the degraded processing unit (221). The transfer learning unit (225) may use the learning data set and the data set with degraded image quality as high-resolution-low-resolution pairs of learning data.

[0285] In one embodiment, the transfer learning unit (225) can acquire a meta model. For example, the transfer learning unit (225) can acquire a meta model from the meta model acquisition unit (223 of FIG. 12B). The transfer learning unit (225) can train the meta model using a learning data set and a data set with degraded image quality. In one embodiment, the transfer learning unit (225) can train the meta model using a gradient descent algorithm.

[0286] In one embodiment, the transfer learning unit (225) inputs a data set with degraded image quality from a learning data set into a meta model, compares an image output from the meta model with the learning data set, calculates the difference between the two as the slope of a loss function, and obtains the parameters of the meta model when the absolute value of the slope is minimized. That is, the transfer learning unit (225) can acquire a transfer model by continuously updating the parameters of the meta model so that the quantitative difference between the image output from the meta model and the compressed high-resolution image included in the learning data set is minimized, thereby training the meta model.

[0287] In one embodiment, the transfer learning unit (225) can train a meta-model using various known learning algorithms. The transfer learning unit (225) can selectively apply learning hyperparameters (learning rate, batch size, termination condition, etc.) and optimization algorithms (SGD, Adam, Adamp sec), etc., depending on system constraints, such as memory, computing power, etc.

[0288] In this way, according to the embodiment, the transfer learning unit (225) can generate a transfer model that is adaptively trained to fit the input image. The meta model updated by the transfer learning unit (225) can be loaded into the image quality processing unit (230) and used for image quality processing.

[0289] Figure 12b is a drawing for explaining a model learning unit (220) according to one embodiment.

[0290] Referring to Fig. 12b, the model learning unit (220) may further include a meta model acquisition unit (223) in addition to the learning data collection unit (227), the degradation processing unit (221), and the transfer learning unit (225). In Fig. 12b, any overlapping content with Fig. 12a will be omitted for explanation.

[0291] In one embodiment, the model learning unit (220) can acquire a meta model based on the image quality of the input image. The model learning unit (220) can train the meta model using the input image and the compressed HR data set (621) as learning data sets, thereby generating a transfer model adaptive to the input image.

[0292] In one embodiment, the meta model acquisition unit (223) can acquire a meta model based on the image quality, i.e., the quality value, of the input image. If on-device learning is performed from a random initial model without the meta model acquisition unit (223), a long learning time is required. However, according to an embodiment, a model suitable for the quality of the input image can be selected in real time through the meta model acquisition unit (223), and a meta model can be quickly generated using the selected model.

[0293] In one embodiment, the metamodel acquisition unit (223) may acquire a metamodel using multiple reference models. The reference model is a quality processing model that has been previously learned using training images and may be stored in a memory (102) or a reference model database. The manufacturer may create multiple reference models in advance and store them in the image processing device (100).

[0294] In one embodiment, multiple reference models may be trained using training images each having a quality value at a uniform interval. Alternatively, in another embodiment, the quality value corresponding to the reference model may be determined based on the quality value distribution of the training images. In one embodiment, the meta model acquisition unit (223) may compare the quality of the images used to train each of the multiple reference models with the quality of the input image, and search for a reference model trained using training images having a quality similar to that of the input image.

[0295] In one embodiment, if there are multiple searched reference models, the meta model acquisition unit (223) may obtain a meta model by interpolating the multiple reference models. In one embodiment, the meta model acquisition unit (223) may assign a weight to each of the multiple searched reference models and obtain a meta model by performing a weighted sum on each of the weighted reference models.

[0296] In one embodiment, the meta model acquisition unit (223) can transmit the acquired meta model to the transfer learning unit (225).

[0297] In one embodiment, the transfer learning unit (225) can train the meta model acquired by the meta model acquisition unit (223) using the learning data set received from the learning data collection unit (227) and the deterioration-processed data set received from the deterioration processing unit (221). In one embodiment, the transfer learning unit (225) can train the meta model using a gradient descent algorithm.

[0298] FIG. 13 is a diagram for explaining the operation of a degradation processing unit according to one embodiment to process the quality of a learning data set.

[0299] Referring to FIG. 13, the degradation processing unit (221) can degrade the image quality by degrading the compressed HR data set (621) and input image included in the learning data set (1301). The learning data set (1301) can be stored in a new database.

[0300] In one embodiment, the deterioration processing unit (221) can deteriorate the compressed high-resolution images included in the compressed HR data set (621) to match the quality value of the input image. Furthermore, in one embodiment, the deterioration processing unit (221) can deteriorate the input image to match the quality value of the input image.

[0301] In one embodiment, the degradation processing unit (221) may receive degradation elements and quality values ​​of an image from the image quality determination unit (210), for example, IQA, and degrade the collected images to have quality values ​​corresponding thereto.

[0302] For example, if the image quality determination unit (210) analyzes the image and obtains the image quality value as Kernel Sigma indicating the degree of blur of the image and Compression Quality indicating the degree of compression degradation of the image, the degradation processing unit (221) can degrade the compressed high-resolution images included in the compressed HR data set (621) using the same blur and image compression method. In addition, the degradation processing unit (221) can degrade the input image by performing blur processing and image compression on the input image.

[0303] In one embodiment, the deterioration processing unit (221) may perform filtering to deteriorate an image. For example, the deterioration processing unit (221) may use a two-dimensional kernel to apply blur deterioration to an image.

[0304] In one embodiment, the degradation processing unit (221) can generate a degradation-degraded data set (1302) by degrading the image quality of the input image and the compressed high-resolution images of the compressed HR data set (621) included in the learning data set (1301).

[0305] The learning data set (1301) and the image quality deteriorated data set (1302) are transferred to the transfer learning unit (225) and can be used as learning data for the meta model.

[0306] FIG. 14a is a diagram for explaining a model learning unit (220-1) according to one embodiment. FIG. 14b is a diagram for explaining the operation of a learning data selection unit including an AI model according to one embodiment.

[0307] Referring to FIG. 14a, the model learning unit (220-1) may further include a learning data selection unit (1410) in addition to the learning data collection unit (227), the degradation processing unit (221), and the transfer learning unit (225).

[0308] In one embodiment, the model learning unit (220-1) can identify the category of the input image and selectively obtain a compressed HR data set having a category corresponding to the category of the input image. In one embodiment, the model learning unit (220-1) can obtain a multi-compressed data set (940 of FIG. 9A) in which the learning data set is compressed by category. In one embodiment, the model learning unit (220-1) can identify the category of the input image and select only the compressed data set corresponding to the category of the input image from among the multi-compressed data sets (940) from an external database. In one embodiment, the model learning unit (220-1) can select a compressed data set having a category corresponding to the category of the input image from among the multi-compressed data sets (940) stored in the external database and store the selected compressed data set in a new database (e.g., 1402). The multi-compressed data set (940) may be stored in an external database (e.g., memory (1002) of the server (1000)). Among the multi-compressed data sets (940), a compressed data set having a category corresponding to a category of an input image may be stored in the internal memory (102) of the image processing device (100).

[0309] In one embodiment, the learning data selection unit (1410) can identify the category of the input image. In one embodiment, the learning data selection unit (1410) may be an algorithm that receives an image and classifies the category of the image from the input image, or a set of algorithms, software that executes the set of algorithms, and / or hardware that executes the set of algorithms. The learning data selection unit (1410) may use a softmax regression function to obtain various classes or categories as results. However, the present invention is not limited thereto, and the learning data selection unit (1410) may be implemented with various types of algorithms that can classify the category of the image from the input image.

[0310] In one embodiment, the learning data selection unit (1410) can analyze the input image to identify the category of the input image with a probability value. The learning data selection unit (1410) can identify the category with the highest probability value as the category of the input image, and select a compressed HR data set in which images belonging to the identified category are compressed from among the multiple compression data sets (940). The multiple compression data set (940) can include high-resolution images belonging to the compressed HR data set. In one embodiment, the compressed HR data set can be characterized as being compressed from a learning data set having a category corresponding to the category of the input image.

[0311] For example, the learning data selection unit (1410) can obtain a probability value for the category or class of the input image as a result. For example, referring to FIG. 14b, the learning data selection unit (1410) can determine that the probability that the category of the input image is a human face, the probability that it is text, and the probability that it is other categories (General) are 80%, 10%, and 10%, respectively.

[0312] In one embodiment, the learning data selection unit (1410) may identify the category with the highest probability value as the category of the input image. For example, referring to FIG. 14B , the learning data selection unit (1410) may identify the category of the input image as a human face, which is the category with the highest vector value. In this case, the learning data selection unit (1410) may select and acquire a compressed HR data set, in which a learning data set corresponding to the human face category is compressed, from the multi-compressed data set (940).

[0313] Alternatively, in one embodiment, the learning data selection unit (1410) may identify a category of an input image with a probability value and select a compressed HR data set from the multiple compressed data set (940) in proportion to the probability value of the identified category.

[0314] In one embodiment, the learning data selection unit (1410) can obtain a compressed HR data set in which a learning data set corresponding to the human face category is compressed, a compressed HR data set in which a learning data set corresponding to the text category is compressed, and a compressed HR data set in which a learning data set corresponding to the general category is compressed at a ratio of 8:1:1 from the multiple compressed data sets (940).

[0315] In one embodiment, the learning data selection unit (1410) can select a compressed HR data set corresponding to a category of an input image and provide it to the learning data collection unit (227).

[0316] In one embodiment, the learning data collection unit (227) can obtain a compressed HR data set corresponding to a category of an input image and an input image. The learning data collection unit (227) can store the compressed HR data set corresponding to the category of the input image and the input image as a learning data set. The learning data set can be stored in the memory (102) as a new database. In one embodiment, the learning data collection unit (227) can provide the learning data set including the compressed HR data set corresponding to the category of the input image and the input image to the deterioration processing unit (221). Furthermore, in one embodiment, the learning data collection unit (227) can provide the learning data set to the transfer learning unit (225).

[0317] In one embodiment, the degradation processing unit (221) can degrade compressed high-resolution images belonging to the identified category to obtain images with degraded image quality. In one embodiment, the degradation processing unit (221) can degrade the image quality of the compressed high-resolution images belonging to the identified category so that the image quality of the compressed high-resolution images corresponds to the image quality of the input image. In addition, the degradation processing unit (221) can degrade the input image so that the input image is degraded to the same degree of deterioration as the input image. In one embodiment, the degradation processing unit (221) can provide the degraded data set to the transfer learning unit (225).

[0318] In one embodiment, the transfer learning unit (225) may use compressed high-resolution images belonging to the identified category and degraded images obtained by quality-processing the compressed high-resolution images as training data. In addition, the transfer learning unit (225) may use input images and degraded images obtained by quality-processing the input images as training data.

[0319] In the present disclosure, since the image processing device (100) does not utilize a cloud server with infinite resources, it is necessary to utilize finite resources more efficiently. In one embodiment, the model learning unit (220-1) included in the image processing device (100) selects only images with similar content characteristics to the input image from an external database, and uses these images to train a model, thereby enabling more efficient and accurate processing of the image quality of the input image.

[0320] Figure 14c is a diagram for explaining a model learning unit according to one embodiment.

[0321] Referring to FIG. 14c, a model learning unit (220-2) according to one embodiment may further include a learning data selection unit (1420) in addition to a learning data collection unit (227), a degradation processing unit (221), and a transfer learning unit (225).

[0322] In one embodiment, the model learning unit (220-2) can identify what type of content the input image is and selectively obtain a compressed HR data set corresponding to the type of content of the input image. In one embodiment, the model learning unit (220-2) can obtain a multi-compressed data set (960 of FIG. 9B) in which the learning data set is compressed according to the type of content. In one embodiment, the model learning unit (220-2) can select a compressed data set corresponding to the same type of content as the input image from among the multi-compressed data sets (960) stored in an external database and store the selected compressed data set in a new database. The multi-compressed data set (960) can be stored in an external database (e.g., a memory (1002) of the server (1000). A compressed data set corresponding to the same type of content as the input image from among the multi-compressed data sets (960) can be stored in the internal memory (102) of the image processing device (100).

[0323] For example, the image processing device (100) can receive various types of content, such as VOD (Video On Demand) content or real-time broadcast programs provided by a content provider. Alternatively, for example, the image processing device (100) can execute various types of applications, and the image processing device (100) can receive various types of content depending on the properties of the applications.

[0324] For example, the type of content refers to the type of content or the genre of content, and the type of content can include various types such as content for video calls, sports content, movie content, music content, game content, animation content, content provided through SNS for real-time chat, content provided through web browsers, 2D content, and 3D content.

[0325] In one embodiment, the learning data selection unit (1420) can identify the metadata of the input image to identify the type of content of the input image. For example, the learning data selection unit (1420) can identify the properties, type, and genre of the application providing the input image through the metadata of the input image. The learning data selection unit (1420) can identify the type of content of the input image based on the properties, type, and genre of the application. For example, the learning data selection unit (1420) can identify whether the running application is a video call app, a movie app, a music app, etc. The learning data selection unit (1420) can identify the properties of the application to identify whether the input image corresponds to video call content, movie content, or music content.

[0326] In one embodiment, the learning data selection unit (1420) may be a classification algorithm that identifies the type of content of the image, or a set of algorithms, software that executes the set of algorithms, and / or hardware that executes the set of algorithms.

[0327] In one embodiment, the learning data selection unit (1420) can select a compressed HR data set corresponding to the same type of content as the input image and provide it to the learning data collection unit (227).

[0328] In one embodiment, the learning data collection unit (227) may acquire a compressed HR data set and an input image corresponding to the same type of content as the input image. The learning data collection unit (227) may store the compressed HR data set and the input image corresponding to the same type of content as the input image as a learning data set. The learning data set may be stored in the memory (102) as a new database. In one embodiment, the learning data collection unit (227) may provide the learning data set including the compressed HR data set and the input image corresponding to the same type of content as the input image to the deterioration processing unit (221). Furthermore, in one embodiment, the learning data collection unit (227) may provide the learning data set to the transfer learning unit (225).

[0329] In one embodiment, the degradation processing unit (221) may degrade compressed high-resolution images belonging to a compressed HR data set corresponding to the same type of content as the input image to obtain an image with degraded image quality. In one embodiment, the degradation processing unit (221) may degrade the image quality of the compressed high-resolution images belonging to the same type of content as the input image so that the image quality of the compressed high-resolution images corresponds to the image quality of the input image. In addition, the degradation processing unit (221) may degrade the input image so that the input image is degraded to the same degree of deterioration as the input image. In one embodiment, the degradation processing unit (221) may provide the degraded data set to the transfer learning unit (225).

[0330] In one embodiment, the transfer learning unit (225) may use as training data a compressed high-resolution image belonging to the same type of content as the input image, and an image with deteriorated quality obtained by quality-processing the compressed high-resolution image. In addition, the transfer learning unit (225) may use as training data the input image, and an image with deteriorated quality obtained by quality-processing the input image.

[0331] In the present disclosure, since the image processing device (100) does not utilize a cloud server with infinite resources, it is necessary to utilize finite resources more efficiently. In one embodiment, the model learning unit (220-2) included in the image processing device (100) selects only images of the same type of content as the input image from an external database, and uses these images to train a model, thereby enabling more efficient and accurate processing of the image quality of the input image.

[0332] FIG. 15 is a flowchart illustrating a method for processing image quality of an input image according to one embodiment.

[0333] Referring to FIG. 15, a method for processing image quality of an input image based on a compressed HR data set according to one embodiment may include steps 1510 to 1540. In one embodiment, steps 1510 to 1540 may be executed by at least one processor included in an image processing device (100). In one embodiment, a method for compressing a learning data set is not limited to that illustrated in FIG. 15, and may further include steps not illustrated in FIG. 15.

[0334] In step 1510, an image processing device (100) according to one embodiment can obtain a meta model for image quality processing of an input image.

[0335] In one embodiment, the image processing device (100) may obtain a quality value of an input image, compare the quality value of the input image with the quality value of a previously learned reference model, and search for a reference model having a quality value corresponding to the quality of the input image. In one embodiment, the image processing device (100) may obtain a meta model using the searched reference model.

[0336] In step 1520, an image processing device (100) according to one embodiment can obtain a compressed HR (High Resolution) data set corresponding to a learning data set.

[0337] An image processing device (100) according to one embodiment may use a compressed HR data set (621) corresponding to a small amount of learning data set instead of a large amount of learning data set as a learning data set for learning a meta model.

[0338] In one embodiment, the characteristics of the compressed HR data set (621) and the characteristics of the learning data sets (310, 611) may be similar. In one embodiment, the parameters of the meta model learned using the compressed HR data set (621) and the image quality deteriorated data set (1302) may be characterized as being approximate to the parameters of the meta model learned using the learning data set (310, 611) and the low-resolution data set (612) whose image quality is deteriorated from the learning data set (310, 611).

[0339] In one embodiment, the compressed HR data set (621) may include high-resolution images individually compressed for each cluster belonging to the cluster set (410) within the learning data set (310, 611). The compressed HR data set may be compressed for each cluster.

[0340] In one embodiment, the image processing device (100) can identify the category of the input image. In one embodiment, the image processing device (100) can obtain a compressed HR data set of a category corresponding to the category of the input image from among multiple condensed datasets compressed by category from a learning data set. Here, the learning data set may correspond to the learning data set (910) of FIG. 9A. In addition, the multiple condensed datasets may correspond to the multiple condensed data set (940) of FIG. 9A, which corresponds to a plurality of compressed data sets in which the learning data set (910) is compressed by category.

[0341] In one embodiment, the image processing device (100) can analyze an input image to identify a category of the input image using a probability value. The image processing device (100) can identify the category with the highest probability value as the category of the input image, and select a compressed HR data set in which images belonging to the identified category are compressed among the multi-compressed data sets (940).

[0342] Alternatively, in one embodiment, the image processing device (100) may identify a category of an input image with a probability value and select a compressed HR data set from the multiple compressed data set (940) in proportion to the probability value of the identified category. In one embodiment, the compressed HR data set may be characterized as being compressed from a learning data set having a category corresponding to the category of the input image. In one embodiment, the image processing device (100) may identify a category of an input image and select a compressed data set having a category corresponding to the category of the input image from the multiple compressed data set (940) stored in an external database. The compressed data set having a category corresponding to the category of the input image may be stored in the external database or may be stored in the internal memory (102) of the image processing device (100).

[0343] Alternatively, in one embodiment, the image processing device (100) can identify the metadata of the input image to identify the type of content of the input image. For example, the learning data selection unit (1420) can identify the properties, type, and genre of an application that provides the input image through the metadata of the input image. The learning data selection unit (1420) can identify the type of content of the input image based on the properties, type, and genre of the application. For example, the image processing device (100) can identify whether the running application is a video call app, a movie app, a music app, etc. The image processing device (100) can identify the properties of the application to identify whether the input image corresponds to video call content, movie content, or music content.

[0344] In one embodiment, the image processing device (100) may select a compressed HR data set from a multiple compressed data set (960) in which a learning data set is compressed by type of content. In one embodiment, the selected compressed HR data set may be characterized as being compressed from a learning data set corresponding to the same type of content as an input image. The image processing device (100) may select a compressed HR data set corresponding to the same type of content as an input image from a multiple compressed data set (960) stored in an external database and store the selected compressed HR data set in an internal memory (102). Here, the learning data set may correspond to the learning data set (910) of FIG. 9B. In addition, the multiple condensed datasets may correspond to the multiple condensed data set (960) of FIG. 9B, in which the learning data set (910) corresponds to a plurality of compressed data sets compressed by type of content.

[0345] In step 1530, the image processing device (100) according to one embodiment can obtain a data set with degraded image quality from the compressed HR data set using an input image. For example, the image processing device (100) can obtain a data set with degraded image quality from the compressed HR data set so as to have an image quality corresponding to the image quality of the input image. For example, the image processing device (100) can obtain a data set with degraded image quality by performing at least one of compression degrading, blurring degrading, resolution adjustment, or noise addition.

[0346] In step 1540, an image processing device (100) according to one embodiment can obtain an output image whose image quality has been processed based on a meta model learned using a compressed HR data set and the image quality deteriorated data set.

[0347] In one embodiment, the image processing device (100) may train the meta-model with a small compressed HR data set instead of training the meta-model with a large training data set. In one embodiment, the upscaling performance of the meta-model trained with a small compressed HR data set may be similar to the upscaling performance of the meta-model trained with a large training data set. Accordingly, the amount of training data set required for training the AI ​​model for performing upscaling in the image processing device (100) may be reduced, and the amount of external memory used and the power consumption for computation in the image processing device (100) may be reduced.

[0348] FIG. 16 is a diagram for explaining an operation of a streaming server compressing a learning data set and an operation of an image processing device storing learning data according to one embodiment.

[0349] Referring to FIG. 16, a streaming server (2000) according to one embodiment can provide content to an image processing device (1600). The streaming server (2000) according to one embodiment can collect and compress a high-resolution learning data set corresponding to the characteristics of the content to the image processing device (1600), and transmit the collected data together with the content.

[0350] In one embodiment, the streaming server (2000) may be a content provider, such as a terrestrial broadcaster, cable broadcaster, OTT service provider, or IPTV service provider, that provides various content, including video, to consumers. The streaming server (2000) may provide items such as television programs or various movies or dramas through VOD services. In one embodiment, the content may be composed of scenes comprising multiple frames.

[0351] In one embodiment, the streaming server (2000) may provide content consisting of a scene (2003) including multiple frames to the image processing device (1600). In one embodiment, the streaming server (2000) may capture, compress, and transmit the content.

[0352] In one embodiment, the streaming server (2000) can capture a scene (2003) and collect a high-resolution training data set corresponding to the characteristics of the captured scene (2003). In one embodiment, the streaming server (2000) can collect a large number of training data sets (2001) containing high-resolution images similar to the characteristics of the scene (2003). For example, the streaming server (2000) can collect a training data set (2001) containing high-resolution images of the same category as the category of the scene (2003).

[0353] In one embodiment, the streaming server (2000) may perform preprocessing, clustering, and data set compression on a learning data set (2001) to generate a compressed data set (2002). For example, the streaming server (2000) may include a preprocessing unit (2010), a clustering unit (2020), and a data set compression unit (2030). The operations of the preprocessing unit (2010), the clustering unit (2020), and the data set compression unit (2030) may correspond to the preprocessing unit (1100), the clustering unit (1200), and the data set compression unit (1300) of the server (1000) of FIG. 3.

[0354] In one embodiment, the streaming server (2000) may generate a compressed data set having similar characteristics for each of a plurality of frames included in the scene (2003). For example, the streaming server (2000) may collect high-resolution images having the same category for each of a plurality of frames included in the scene (2003) and compress the high-resolution images for each category. The compressed data set (2002) may have a smaller amount of data than the training data set (2001).

[0355] In one embodiment, a streaming server (2000) can provide content consisting of scenes (2003) and a compressed data set (2002) corresponding to the content to an image processing device (1600).

[0356] In one embodiment, the image processing device (1600) may receive content from a streaming server (2000) and simultaneously receive a compressed data set (2002) corresponding to the content.

[0357] In one embodiment, the image processing device (1600) can restore content received from the streaming server (2000) through a decoder. In one embodiment, the scene (2003) received by the image processing device (1600) from the streaming server (2000) can be an input image of the image processing device (1600).

[0358] In one embodiment, the image processing device (1600) can selectively collect images similar to the content characteristics of an input image from an extended database (1640) and create a new database (1650) having similar characteristics to the content characteristics. A learning data set for training a meta model of the image processing device (1600) can be stored in the new database (1650), and the learning data set can be composed of compressed high-resolution images. The extended database (1640) can be stored in the internal memory of the image processing device (1600) or in an external database.

[0359] In one embodiment, the extended database (1640) may store a compressed data set (2002) received from a streaming server (2000) and a compressed data set (1630) received from a server (1000 of FIG. 1). The compressed data set (1630) received from the server (1000) may correspond to the compressed data set (320) of FIG. 3 or the compressed HR data set (621) of FIG. 12A.

[0360] In one embodiment, the image processing device (1600) can identify a category of an input image using an image classifier (1620). In one embodiment, the image classifier (1620) can correspond to the learning data selection unit (1410) of FIG. 14A.

[0361] In one embodiment, the image processing device (1600) may analyze an input image using an image classifier (1620) to identify a category of the input image using a probability value. The image classifier (1620) may identify the category with the highest probability value as the category of the input image, and select a compressed HR data set in which images belonging to the identified category are compressed from an extended database (1640).

[0362] Alternatively, in one embodiment, the image processing device (1600) may use an image classifier (1620) to identify a category of an input image using a probability value, and select a compressed HR data set from an expanded database (1640) in proportion to the probability value of the identified category.

[0363] In one embodiment, the image processing device (1600) can store a compressed HR data set selected from the expanded database (1640) corresponding to the category of the input image in a new database (1650). The new database (1650) can be stored in the internal memory of the image processing device (1600). The image processing device (1600) can store the compressed HR data set in the new database (1650) through the learning data collection unit (277) of FIG. 14A.

[0364] In one embodiment, the image processing device (1600) may also include the input image in a new database (1650).

[0365] In one embodiment, a new database (1650) having categories corresponding to the categories of the input image may be stored in an external database or in the internal memory (102) of the image processing device (1600).

[0366] In one embodiment, the image processing device (1600) can receive an input image and a compressed data set (2002) corresponding to the input image from the streaming server (2000). In one embodiment, the image processing device (1600) can obtain a data set corresponding to a category of the input image from the compressed data set (2002) received from the streaming server (2000) and the compressed HR data set received from the server (1000). In one embodiment, the image processing device (1600) can select a data set corresponding to a category of the input image from an extended database (1640) storing the compressed data set (2002) received from the streaming server (2000) and the compressed HR data set received from the server (1000).

[0367] In one embodiment, the image processing device (1600) can receive a compressed data set from not only the server (1000) but also the streaming server (2000), select images of categories corresponding to the input image, and store them in a new database (1650). In one embodiment, the image processing device (1600) can obtain learning data similar to the characteristics of the input image by using various compressed data sets, and thus can learn and utilize the meta model more efficiently and process the image quality of the input image more efficiently and accurately.

[0368] FIG. 17 is a diagram for explaining an operation of an image processing device according to one embodiment of the present invention to compress a learning data set according to a background process.

[0369] Referring to FIG. 17, an image processing device (1700) according to one embodiment can operate with an on-device process (1701) and a background process (1702). The image processing device (1700) can perform image processing using on-device AI technology according to the on-device process (1701). The image processing device (1700) can compress a learning data set according to a background process (1702) that does not use the on-device AI technology. Since the operation of compressing the learning data set requires a large amount of memory and calculation, the image processing device (1700) can perform compression of the learning data set while not using the on-device AI technology.

[0370] In one embodiment, the image processing device (1700) can upscale the input image (1705) while the on-device process (1701) is being performed. The image processing device (1700) can store a learning data set (1701) corresponding to the input image (1705) in a new database (1750). For example, the image processing device (1700) can store the input image (1705) in the new database (1750). Additionally, in one embodiment, the image processing device (1700) can store a high-resolution image related to the input image (1705) from a streaming server that provided the input image (1705) in the new database (1750). The new database (1750) can be stored in the memory of the image processing device (1700).

[0371] In one embodiment, the image processing device (1700) may not perform a learning data set compression operation while the on-device process (1701) is being performed.

[0372] In one embodiment, if the image processing device (1700) determines that a sufficient amount of training data sets are stored in the new database (1750), the image processing device (1700) may compress the training data sets while the on-device AI technology is not in operation. For example, if the image processing device (1700) determines that the input image (1705) has sufficient resolution and thus does not require an upscaling operation, the image processing device (1700) may compress the training data sets without performing the on-device AI technology.

[0373] In one embodiment, the image processing device (1700) may compress a learning data set according to a background process (1702) that does not utilize on-device AI technology. In one embodiment, the image processing device (1700) may include a preprocessing unit (1710), a clustering unit (1720), and a data set compression unit (1730). The image processing device (1700) may perform preprocessing, clustering, and data set compression on a learning data set stored in a new database (1750) to generate a compressed data set (1760). The operations of the preprocessing unit (1710), the clustering unit (1720), and the data set compression unit (1730) may correspond to the preprocessing unit (1010), the clustering unit (1020), and the data set compression unit (1030) of the server (1000) of FIG. 3.

[0374] In one embodiment, the image processing device (1700) can generate a small compressed data set (1760) from a training data set stored in a new database (1750).

[0375] In one embodiment, the image processing device (1700) can generate a compressed data set (1760) by repeating an operation of updating a compressed data set previously stored in a database as an initial value, without performing learning from the beginning when the input image (1705) is received, in order to minimize the complexity of the compression operation.

[0376] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0377] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

Claims

1. In a method for compressing a learning data set (310, 611), Step (710) of generating a cluster set (410) by clustering the learning data set (310, 611); A step (720) of generating an initial compressed HR data set (601) by selecting some images from among the images belonging to each cluster in each cluster belonging to the above cluster set (410); Step (730) of obtaining a first loss (631) of a first neural network model (630) based on the above learning data set (310, 611) and obtaining a second loss (641) of a second neural network model (640) based on the above initial compressed HR data set (601); and A method comprising a step (740) of generating a compressed HR data set (621) by updating each pixel of images belonging to the initial compressed HR data set (601) based on the first loss (631) and the second loss (641).

2. In paragraph 1, The above method, A method further comprising a step of performing preprocessing (1100) on the learning data set (310, 611) based on the amount of information of each image belonging to the learning data set (310, 611).

3. In paragraph 1 or 2, The step (730) of obtaining the first loss (631) and obtaining the second loss (641) is as follows: A step of processing the image quality of the above learning data set (310, 611) to generate a low-resolution data set (612), and processing the image quality of the compressed HR data set (621) to generate a compressed LR data set (622); A step of calculating the first loss (631), which is the difference between the output value (635) obtained by inputting the low-resolution data set (612) into the first neural network model (630) and the learning data set (310, 611); and A method comprising a step of calculating the second loss (641), which is the difference between the output value (645) obtained by inputting the compressed LR data set (622) into the second neural network model (640) and the compressed HR data set (621).

4. In any one of paragraphs 1 to 3, The step (740) of generating the compressed HR data set (621) is as follows: A step (766) of calculating a first gradient for minimizing the first loss (631) of the first neural network model (630); Step (767) of calculating a second gradient for minimizing the second loss (641) of the second neural network model (640); and A method comprising the step of updating each pixel of the images in the compressed HR data set (621) so that the difference between the first gradient and the second gradient is reduced.

5. In paragraph 4, The step (740) of generating the compressed HR data set (621) is as follows: A step (770) of calculating a matching loss (650), which is the difference between the first gradient and the second gradient; A step (780) of calculating a matching gradient to minimize the above matching loss (650); and A step (790) of updating the compressed HR data set (621) by applying the matching gradient to each pixel of the images in the compressed HR data set (621), The above matching gradient is a method representing the slope of the matching loss (650) for each pixel of the images in the compressed HR data set (621).

6. In paragraph 4 or 5, The above first gradient represents the slope of the first loss (631) with respect to the parameters of the first neural network model (630), The second gradient is a method representing the slope of the second loss (641) with respect to the parameters of the second neural network model (640).

7. In any one of paragraphs 1 to 6, A method, characterized in that the first loss (631) and / or the second loss (641) is not a mean absolute error.

8. In any one of paragraphs 1 to 7, The above method, A method further comprising a step (850) of processing the compressed HR data set (621) for image quality to generate a compressed LR (Low Resolution) data set (622).

9. In paragraph 8, The above method, A method further comprising a step (860) of updating parameters of the second neural network model (640) by applying the updated compressed HR data set (621) and the compressed LR data set (622) to the second neural network model (640).

10. In paragraph 9, The step (860) of updating the parameters of the second neural network model (640) is as follows: A step of calculating a second loss (641) between the output value (645) obtained by applying the compressed HR data set (621) to the second neural network model (640) and the compressed LR data set (622); A step of calculating a second gradient for minimizing the second loss (641); and A step of updating the parameters of the second neural network model (640) by applying the second gradient to the parameters of the second neural network model (640), The second gradient is a method representing the slope of the second loss (641) with respect to the parameters of the second neural network model (640).

11. In any one of paragraphs 1 to 10, A method, characterized in that the initial parameters of the second neural network model (640) are identical to the initial parameters of the first neural network model (630).

12. In any one of paragraphs 1 to 11, A method characterized in that the parameters of the second neural network model (640) learned using the compressed HR data set (621) are approximate to the parameters of the first neural network model (630) learned using the learning data set (310, 611).

13. In any one of paragraphs 1 to 12, The above method, Step (1010) of classifying the above learning data set (310, 611) based on categories; and A method further comprising a step (1020) of obtaining a multi-compressed data set (940) by compressing the above learning data set (310, 611) for each category.

14. In any one of paragraphs 1 to 13, The step (720) of generating an initial compressed HR data set by selecting some images from among the images belonging to each cluster in each cluster belonging to the above cluster set (410) is as follows. A method comprising the step of determining a number of images to be selected from among the images belonging to each cluster based on user input.

15. In the image processing device (100), A memory (102) storing one or more instructions; and comprising one or more processors (101) for executing one or more instructions stored in the memory (102); The above one or more processors (101) execute the above one or more instructions, Obtain (1510) a meta model for image quality processing of an input image (110), Obtain (1520) a compressed HR (High Resolution) data set (621) corresponding to the learning data set (310, 611), Using the above input image (110), a data set (1302) with deteriorated image quality is obtained (1530) from the compressed HR data set (621), An image processing device (100) that obtains (1540) an output image (120) whose image quality has been processed based on a meta model learned using the compressed HR data set (621) and the image quality deteriorated data set (1302).

Citation Information

Patent Citations

  • Method and system for processing super-resolution video

    KR102467091B1

  • Compression-informed video super-resolution

    WO2022231643A1

  • KR20220055970A