Method and device for improving image quality based on super-resolution neural network

Abstract

The invention provides a method and device for improving image quality based on a super-resolution neural network. According to one embodiment of the present disclosure, there is provided a method for improving image quality optimized for distortion characteristics of a target image, the method comprising the steps of: generating one or more learning data sets to which one or more distortions are added; inputting the one or more learning data sets into a degenerative encoder neural network (DEN) to obtain each learning distortion characteristic value; inputting a service data set including the image blocks of the target image into the DEN to obtain a service distortion characteristic value; calculating the similarity between each learning distortion characteristic value and the service distortion characteristic value; and selecting a learning data set most similar to the service distortion characteristic value.

Description

Technical Field

[0001] In some embodiments, the present disclosure relates to methods and apparatus for image quality enhancement based on super-resolution neural networks. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and may not constitute prior art.

[0003] Based on the image quality enhancement model, low-quality images can be converted into high-quality images. The image quality enhancement model can be trained based on machine learning based on artificial neural networks. In order to train the image quality enhancement model, sufficient data for supervised learning of distortion is required. Here, distortion means blur, noise, etc. found in low-quality images. One way to collect data for supervised learning is to manually convert low-quality data into high-quality data. This method is quite expensive. Depending on the service environment that provides the image, it may not be possible to collect data for supervised learning in the first place.

[0004] In order to generate data for supervised learning, the existing technology randomly adds distortions such as blur, noise, and compression to the target image to collect data for supervised learning. This method is a general method that can be applied to any image, but it has the problem of generating data for supervised learning that cannot focus on the distortion characteristics specifically reflected in the target image.

[0005] Meanwhile, a drawback of an image quality enhancement model trained on a dataset with various distortion characteristics reflected is that it is not adequately trained for the distortion characteristics specifically reflected in the target image, and thus it is not optimized to convert the target image into a high-quality image. Summary of the invention

[0006] Technical issues

[0007] According to one embodiment of the present disclosure, the image quality enhancement apparatus may select a training dataset having the most similar distortion characteristics to the service dataset of the target image by calculating the similarity between each distortion characteristic value of the training dataset and the distortion characteristic value of the service dataset.

[0008] According to one embodiment of the present disclosure, the image quality enhancement apparatus may select a super-resolution neural network optimized for a specific distortion based on the similarity between distortion characteristic values.

[0009] Problems to be solved by the present invention are not limited to the above-mentioned problems, and other problems not mentioned will be apparent to those of ordinary skill in the art from the following description.

[0010] Technical Solution

[0011] At least one aspect of the present disclosure provides an image quality enhancement method optimized for distortion characteristics of a target image, the image quality enhancement method comprising: generating one or more training data sets having one or more distortions; obtaining training distortion characteristic values ​​respectively by inputting the one or more training data sets into a degradation encoder neural network (DEN); obtaining service distortion characteristic values ​​by inputting a service data set including image blocks of the target image into the degradation encoder neural network; calculating the similarity between each training distortion characteristic value and the service distortion characteristic value; and selecting the training data set having the highest similarity to the service distortion characteristic value.

[0012] Another aspect of the present disclosure provides an image quality enhancement method optimized for distortion characteristics of a target image, the image quality enhancement method comprising: training one or more super-resolution neural networks (SRNs) to be optimized for specific distortions respectively by using training data sets with different distortions; calculating the similarity between a service data set and each training data set respectively applied to one or more super-resolution neural networks by using a degradation encoder neural network (DEN); selecting a super-resolution neural network trained using the training data set with the highest similarity among the one or more super-resolution neural networks; and converting the target image into a high-quality image by using the selected super-resolution neural network.

[0013] Another aspect of the present disclosure provides an image quality enhancement device optimized for distortion characteristics of a target image, the image quality enhancement device comprising a memory configured to store one or more instructions, and a processor, wherein the processor is configured to execute the one or more instructions to perform the following steps: training one or more super-resolution neural networks (SRNs) to be optimized for specific distortions by using training data sets with different distortions; calculating the similarity between a service data set and each training data set respectively applied to one or more super-resolution neural networks by using a degradation encoder neural network (DEN); selecting a super-resolution neural network trained using the training data set with the highest similarity among the one or more super-resolution neural networks; and converting the target image into a high-quality image by using the selected super-resolution neural network.

[0014] Technical Effects

[0015] In some embodiments, the present disclosure trains a super-resolution neural network based on a training data set with the highest similarity, so that a target image can be converted into a high-quality image by using a super-resolution neural network optimized for the target image.

[0016] In some embodiments, the present disclosure calculates the similarity between the distortion characteristics of each training dataset and the distortion characteristics of the service dataset, so that a super-resolution neural network trained by focusing on the distortion characteristics of the target image can be selected from pre-trained super-resolution neural networks.

[0017] The effects of the present disclosure are not limited to those described above, and other unmentioned effects will be apparent to those of ordinary skill in the art from the above description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1a is a flowchart of an image quality enhancement method according to at least one embodiment of the present disclosure.

[0019] Figure 1b is a flowchart of an image quality enhancement method according to another embodiment of the present disclosure.

[0020] Figure 2 is a diagram illustrating a process of generating a training data set according to at least one embodiment of the present disclosure.

[0021] Figure 3 is a diagram illustrating a process of training a super-resolution neural network according to at least one embodiment of the present disclosure.

[0022] Figure 4 is a diagram illustrating a process of feeding a service dataset into a degradation encoder neural network to extract distortion characteristics in accordance with at least one embodiment of the present disclosure.

[0023] Figure 5a is a diagram illustrating a process of calculating a weight based on similarity between a training dataset and a service dataset according to at least one embodiment of the present disclosure.

[0024] Figure 5b is a diagram illustrating an example training data set in accordance with at least one embodiment of the present disclosure.

[0025] Figure 6 is a block diagram of an image quality enhancement apparatus according to at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] Hereinafter, some embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the following description, the same reference numerals preferably represent the same elements, although the elements are shown in different drawings. In addition, in the following description of some embodiments, for the sake of clarity and brevity, the detailed description of related known components and functions will be omitted when it is considered to be obscure the subject matter of the present disclosure.

[0027] In addition, various ordinal numbers or letter numbers such as first, second, i), ii), a), b), etc. are only used to distinguish one component from another component, and do not imply or suggest the substance, order or sequence of the components. Throughout the specification, when a component "includes" or "comprising" a component, the component is intended to further include other components without excluding other components unless otherwise explicitly stated.

[0028] Figure 1a is a flowchart of an image quality enhancement method according to at least one embodiment of the present disclosure.

[0029] Reference Figure 1a , the image quality enhancement device may generate one or more training data sets to which one or more distortions are added (S100). Here, the distortion includes blur, noise, etc. The method of adding distortion may include adding a single distortion characteristic at a specific ratio, or adding a combination of two or more distortion characteristics at a specific ratio. The training data set refers to one or more image patches included in the training image, and is classified into a single training set.

[0030] The image quality enhancement device may calculate the similarity (Score) between each distortion characteristic value of the training dataset and the distortion characteristic value of the service dataset (S102). Here, the service dataset refers to one or more image blocks constituting a target image to be subjected to image quality enhancement and classified into a single service set.

[0031] The image quality enhancement device can obtain the distortion characteristic value of each data set by using a degradation encoder neural network (DEN). Here, the degradation encoder neural network is a neural network that outputs the characteristic value of the distortion contained in the input image. The output of the degradation encoder neural network is a vector clustered by the distortion characteristics and intensity of the input image. The degradation encoder neural network can be trained based on contrastive learning. Contrastive learning is a method of training an objective by using an objective function that is designed to minimize the loss between image blocks with the same distortion and maximize the loss between image blocks with different distortions. Contrastive learning uses a first objective function (L DEN ) to train the degradation encoder neural network. The first objective function can be expressed by using Eq.

[0032] [Equation 1]

[0033]

[0034] Here, DEN(·) represents the output of the degenerate encoder neural network, N deg is the total number of different distortions to be learned, and p m,1 and p m,2are the query sample and the positive sample corresponding to the image patch to which the mth distortion is added. An image patch is one of the multiple images constituting the dataset. ～m,n are negative samples corresponding to image patches to which non-mth distortions are added, and N neg is the total number of negative samples. When training a degradation encoder neural network by using contrastive learning, the larger the number of negative samples, the more stable training can be achieved.

[0035] The image quality enhancement device can input one or more training data sets and service data sets into the degradation encoder neural network respectively. Based on the output of the degradation encoder neural network, the image quality enhancement device can obtain the distortion characteristic value of each of the one or more training data sets and the distortion characteristic value of the service data set.

[0036] The similarity between the distortion characteristic value of any training dataset and the distortion characteristic value of the service dataset can be calculated by using Equation 2.

[0037] [Equation 2]

[0038]

[0039] here, is the total number of representative samples of the training dataset, and is the total number of representative samples of the serving dataset.

[0040] Since it is inefficient to calculate similarity based on all samples in a dataset, representative samples of each dataset can be extracted in advance. Methods for extracting representative samples of data include randomly sampling from the dataset or selecting representative images of each cluster by using K-means clustering.

[0041] In Equation 2, , which is the average of the distortion characteristic values ​​of one or more samples selected from each training dataset, A value representing the distortion characteristic of each training data set (hereinafter referred to as a training distortion characteristic value) may be used. Such a training distortion characteristic value may be interpreted as a value representing the distortion characteristic that can be best reconstructed by a super-resolution neural network to be trained by using the corresponding training data set. In addition, A value that can be used to represent the distortion characteristic of the service data set (hereinafter referred to as the service distortion characteristic value). The image quality enhancement device can pre-calculate the training distortion characteristic value for each training data set in one or more training data sets, and generate a lookup table in which the calculated training distortion characteristic values ​​are arranged in the form of a table.

[0042] The image quality enhancement device can select a training data set having the highest similarity with the service distortion characteristic value (S104). The reason for selecting the training data set having the highest similarity is that the selected training data set can be used to train a super-resolution neural network optimized for the distortion characteristics of the target image.

[0043] The image quality enhancement apparatus may train a single super-resolution neural network (SRN) based on the selected training data set ( S106 ).

[0044] The image quality enhancement apparatus may convert the target image into a high-quality image by using a super-resolution neural network trained based on the selected training data set ( S108 ).

[0045] Figure 1b is a flowchart of an image quality enhancement method according to another embodiment of the present disclosure.

[0046] Reference Figure 1b , the image quality enhancement device may obtain one or more super-resolution neural networks optimized for different distortions (S150). The image quality enhancement device may generate one or more training data sets optimized for different distortions. The image quality enhancement device may use the training data sets to train one or more super-resolution neural networks. Each of the one or more super-resolution neural networks may be equivalent to a super-resolution neural network optimized for a specific distortion.

[0047] The image quality enhancement device may calculate the similarity between each training data set applied to one or more super-resolution neural networks and the service data set by using a degradation encoder neural network (S152). For example, the image quality enhancement device may obtain a training distortion characteristic value and a service distortion characteristic value by using a degradation encoder neural network, the training distortion characteristic value being a value representing the distortion characteristic of the training data set applied to each super-resolution neural network, and the service distortion characteristic value being a value representing the distortion characteristic of the service data set. Here, the training distortion characteristic value may be an average value of the output of the degradation encoder neural network for one or more samples selected from each training data set, and the service distortion characteristic value may be an average value of the output of the degradation encoder neural network for one or more samples selected from the service data set. The image quality enhancement device may pre-calculate the training distortion characteristic values ​​of the training data sets respectively applied to the super-resolution neural networks and store them in the form of a lookup table. The image quality enhancement device may calculate the similarity based on the difference between the training distortion characteristic value and the service distortion characteristic value corresponding to each super-resolution neural network.

[0048] The image quality enhancement apparatus may select a super-resolution neural network to which a training data set having the highest similarity to a service data set is applied from among the one or more pre-trained super-resolution neural networks ( S154 ).

[0049] The image quality enhancement device may convert the target image into a high-quality image by using the selected super-resolution neural network (S156).

[0050] Figure 2 is a diagram illustrating a process of generating a training data set according to at least one embodiment of the present disclosure.

[0051] Reference Figure 2 , the image quality enhancement device may add distortion to the original copy 200 of the training image. The image quality enhancement device may generate a training data set 202 having blur added to the original copy 200 of the training image, a training data set 204 having noise added to the original copy 200, and a training data set 206 having both blur and noise added. The image quality enhancement device may generate multiple training data sets 202, 204, and 206 by combining blur and noise of different intensities.

[0052] The image quality enhancement apparatus may train a super-resolution neural network based on a training dataset focusing on the distortion characteristics of the target image by generating a training dataset having added distortion similar to the distortion characteristics of the target image.

[0053] Figure 3 is a diagram illustrating a process of training a super-resolution neural network according to at least one embodiment of the present disclosure.

[0054] Reference Figure 3 , the image quality enhancement device can input the training image 300 with arbitrary distortion added to the degradation encoder neural network 30 to obtain the training distortion characteristic value 302, and input the training image 300 to the super-resolution neural network 32 to obtain the output image 306.

[0055] The image quality enhancement device may calculate a second objective function based on the difference between the output image 306 and the desired image 308. The second objective function is a function for training a super-resolution neural network (SRN). The second objective function may have a weight 304 applied thereto, which is derived based on the training distortion characteristic value 302. The second objective function may be expressed as shown in Equation 3.

[0056] [Equation 3]

[0057]

[0058] here, is a low-quality training image 300 with arbitrary distortion added, is the output image 306 of the degraded encoder neural network, is the desired high-quality image 308 used as the target of image quality enhancement, and the subscript i represents the i-th sample in the training batch. i It refers to the weight.

[0059] The weight may be determined based on the similarity between the training image 300 in the training dataset and the target image in the serving dataset, and may be expressed by using Equation 4.

[0060] [Equation 4]

[0061]

[0062] Here, N bc is the total number of samples in the training batch, N sp is the total number of representative samples of the serving dataset.

[0063] Since it is computationally heavy to consider all images in the serving dataset when calculating the weights, one may have obtained N sp Samples. Extract N sp Methods for obtaining samples may include random sampling from the serving dataset or by selecting and sampling representative images for each cluster using a K-means clustering algorithm.

[0064] It can be interpreted as a value representing the distortion characteristics of the service data set. The image quality enhancement device can pre-calculate the value of the distortion characteristics of the service data set for a specific service data set. And store it in a lookup table.

[0065] The image quality enhancement apparatus may train a super-resolution neural network by using a second objective function generated based on the operation weights 304 and the difference between the output image 306 and the desired image 308. As described above, according to at least one embodiment of the present disclosure, the similarity between the training dataset and the service dataset may be reflected as a weight in the error back propagation process.

[0066] Figure 4 is a diagram illustrating a process of feeding a service dataset into a degradation encoder neural network to extract distortion characteristics in accordance with at least one embodiment of the present disclosure.

[0067] Reference Figure 4, the image quality enhancement device may input the sample 400 extracted from the service data set into the degradation encoder neural network before training the super-resolution neural network, thereby obtaining a service distortion characteristic value 402. The service distortion characteristic value 402 may be represented as a vector value of (1×N), where N is a natural number. The image quality enhancement device may calculate an average value 404 of the service distortion characteristic value based on the corresponding service distortion characteristic value 402. The average value 404 of the service distortion characteristic value may be a value representing the distortion characteristic of the service data set.

[0068] Figure 5a is a diagram illustrating a process of calculating a weight based on similarity between a training dataset and a service dataset according to at least one embodiment of the present disclosure.

[0069] Figure 5b is a diagram illustrating an example training data set in accordance with at least one embodiment of the present disclosure.

[0070] Reference Figure 5a , the image quality enhancement device may input each image block 500 of the training data set into the degradation encoder neural network, thereby obtaining a training distortion characteristic value 502 for training each image. Here, the image blocks 500 of the training data set may have different distortions added to them, such as Figure 5b The training distortion characteristic value 502 may be represented as a vector value of (1×N), where N is a natural number. The image quality enhancement apparatus may calculate the weight 506 of the training data set based on the similarity between the training distortion characteristic value 502 and the average value 504 of the service distortion characteristic value.

[0071] In the present disclosure, similarity can be exploited in two ways. The first way is as follows Figure 3 The weight of each sample in the training data set is calculated based on similarity and combined with the error back propagation process of the super-resolution neural network.

[0072] The second method of using similarity is to select one of multiple super-resolution neural networks or multiple training datasets based on similarity, as mentioned above. Figure 1a and Figure 1b If the data to be provided to the service is specifically determined, it is not necessary to train all super-resolution neural networks, and it may be efficient to train only one super-resolution neural network based on the training dataset with the highest similarity, as Figure 1a As shown. For example, in order to select a dataset whose distortion is most similar to the distortion characteristic value of the target image, the present disclosure can calculate the similarity of each of the multiple datasets. By comparing the calculated similarities between them, the dataset corresponding to the highest similarity can be selected to train a single super-resolution neural network.

[0073] Figure 6 is a block diagram of an image quality enhancement apparatus according to at least one embodiment of the present disclosure.

[0074] Reference Figure 6 , the image quality enhancement device 600 includes both or one of a memory 602 and a processor 604 .

[0075] The memory 602 may store a program that can be executed by the processor 604 so as to perform the image quality enhancement method according to at least one embodiment of the present disclosure. For example, the program may include multiple instructions that can be executed by the processor 604, and the multiple instructions can be executed by the processor 604 to perform the image quality enhancement method.

[0076] The memory 602 may include at least one of a volatile memory and a nonvolatile memory. The volatile memory may include, for example, a static random access memory (SRAM) or a dynamic random access memory (DRAM), and the nonvolatile memory may include, for example, a flash memory.

[0077] The processor 604 may include at least one configuration capable of executing at least one instruction. The processor 604 may execute the instructions stored in the memory 602, and by executing the instructions, the processor 604 may perform the image quality enhancement method according to the present disclosure.

[0078] Various illustrative implementations of the devices and methods described herein can be realized by digital electronic circuitry, integrated circuitry, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof.

[0079] Various illustrative implementations of the systems and methods described herein may be implemented by digital electronic circuits, integrated circuits, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include those implemented in one or more computer programs executable on a programmable system. The programmable system includes at least one programmable processor, at least one input device, and at least one output device connected to a storage system to receive data and instructions from the storage system and to send data and instructions to the storage system, wherein the programmable processor may be a dedicated processor or a general-purpose processor. A computer program (also referred to as a program, software, software application, or code) includes instructions for a programmable processor and is stored in a "computer-readable recording medium".

[0080] Computer-readable recording media include any type of recording device on which data that can be read by a computer system is recordable. Examples of computer-readable recording media include non-volatile or non-transitory media such as ROM, CD-ROM, magnetic tape, floppy disk, memory card, hard disk, optical / magnetic disk, storage device, etc. Computer-readable recording media may also include transient media such as data transmission media. In addition, computer-readable recording media may be distributed in computer systems connected via a network, where computer-readable codes may be stored and executed in a distributed manner.

[0081] Although the corresponding steps in the sequence diagram / flow chart are described as being performed sequentially, they only illustrate the technical ideas of some embodiments of the present disclosure. Therefore, a person of ordinary skill in the relevant field can perform the steps by changing the order described in the sequence diagram / flow chart or by performing two or more steps in parallel, so the steps in the sequence diagram / flow chart are not limited to the time sequence shown.

[0082] Various implementations of the systems and techniques described herein can be implemented by programmable computers. Here, the computer includes a programmable processor, a data storage system (including a volatile memory, a non-volatile memory or any other type of storage system or a combination thereof) and at least one communication interface. For example, a programmable computer can be one of a server, a network device, a set-top box, an embedded device, a computer expansion module, a personal computer, a laptop computer, a personal data assistant (PDA), a cloud computing system, and a mobile device.

[0083] Although the exemplary embodiments of the present disclosure have been described for illustrative purposes, it will be appreciated by those skilled in the art that various modifications, additions, and substitutions are possible without departing from the concept and scope of the claimed invention. Therefore, for the sake of brevity and clarity, the exemplary embodiments of the present disclosure have been described. The scope of the technical ideas of the embodiments of the present disclosure is not limited by the illustrations. Therefore, it will be appreciated by those of ordinary skill that the scope of the claimed invention is not limited by the embodiments explicitly described above, but by the claims and their equivalents.

[0084] (reference numerals)

[0085] 600: Image quality enhancement device

[0086] 602: Memory

[0087] 604: Processor

[0088] CROSS-REFERENCE TO RELATED APPLICATIONS

[0089] This application claims priority to Korean Patent Application No. 10-2022-0056193 filed on May 6, 2022, the disclosure of which is incorporated herein by reference in its entirety.

Claims

1. An image quality enhancement method for optimizing the distortion characteristics of a target image, the image quality enhancement method comprising the following steps: generating one or more training data sets having one or more distortions; Obtaining training distortion characteristic values ​​respectively by inputting the one or more training data sets into the degradation encoder neural network DEN; Obtaining a service distortion characteristic value by inputting a service data set including an image block of the target image into the degradation encoder neural network; calculating a similarity between each of the training distortion characteristic values ​​and the service distortion characteristic value; as well as A training data set having the highest similarity to the service distortion characteristic value is selected.

2. The image quality enhancement method according to claim 1, further comprising the following steps: Training a super-resolution neural network SRN based on the selected training dataset; as well as The target image is converted into a high-quality image by using the super-resolution neural network.

3. The image quality enhancement method according to claim 1, wherein: The training distortion characteristic value is an average of outputs of the degradation encoder neural network for at least one or more samples selected from each training data set, and The service distortion characteristic value is an average value of the output of the degradation encoder neural network for at least one or more samples selected from the service data set.

4. The image quality enhancement method according to claim 1, wherein: The similarity is calculated based on a difference between each of the training distortion characteristic values ​​and the service distortion characteristic value. 5 . A computer-readable recording medium storing instructions which, when executed by a computer, cause the computer to perform the image quality enhancement method according to claim 1 .

6. An image quality enhancement method for optimizing the distortion characteristics of a target image, the image quality enhancement method comprising the following steps: Training one or more super-resolution neural networks (SRN) to be optimized for specific distortions respectively by using training datasets with different distortions; calculating a similarity between a serving dataset and each of the training datasets respectively applied to the one or more super-resolution neural networks by using a degradation encoder neural network DEN; selecting, among the one or more super-resolution neural networks, a super-resolution neural network trained using a training data set having a highest similarity; as well as The target image is converted into a high-quality image by using the selected super-resolution neural network.

7. The image quality enhancement method according to claim 6, wherein: The step of calculating the similarity comprises the following steps: obtaining, by using the degradation encoder neural network, training distortion characteristic values ​​and service distortion characteristic values, each of which represents a distortion characteristic of each training data set applied to each of the super-resolution neural networks, and service distortion characteristic values ​​which are values ​​representing a distortion characteristic of the service data set; and The similarity is calculated based on a difference between each of the training distortion characteristic values ​​and the service distortion characteristic value.

8. The image quality enhancement method according to claim 7, wherein: The training distortion characteristic value is an average of outputs of the degradation encoder neural network for at least one or more samples selected from each training data set, and The service distortion characteristic value is an average value of the output of the degradation encoder neural network for at least one or more samples selected from the service data set.

9. A computer-readable recording medium storing instructions which, when executed by a computer, cause the computer to perform the image quality enhancement method according to claim 6.

10. An image quality enhancement device for optimizing distortion characteristics of a target image, the image quality enhancement device comprising: a memory configured to store one or more instructions; as well as processor, The processor is configured to execute the one or more instructions to perform the following steps: Training one or more super-resolution neural networks (SRN) to be optimized for specific distortions respectively by using training datasets with different distortions; calculating a similarity between a serving dataset and each of the training datasets respectively applied to the one or more super-resolution neural networks by using a degradation encoder neural network DEN; selecting a super-resolution neural network trained using a training data set having the highest similarity among the one or more super-resolution neural networks; and The target image is converted into a high-quality image by using the selected super-resolution neural network.

Images