Image processing device, operating method thereof, and neural network training method

Abstract

An image processing device and a neural network training method are provided. The image processing device comprises a memory for storing one or more instructions, and at least one processor for executing the one or more instructions stored in the memory. The processor: acquires a neural network model for image quality processing of an input image by executing the one or more instructions; calculates a plurality of gradients obtained by partially differentiating weights of the neural network model with respect to a loss of the neural network model, by applying, to the neural network model, training data corresponding to the input image; removes at least one from among the plurality of gradients by applying, to the plurality of gradients, a gradient mask having gradient pruning information; trains the neural network model by updating the weights of the neural network model on the basis of one or more remaining gradients from among the plurality of gradients; and acquires an image-quality-processed output image from the input image on the basis of the trained neural network model.

Description

Image processing device, its operation method, and neural network learning method

[0001] The disclosed embodiments relate to an image processing device, an operating method of the image processing device, and a method for training a neural network. More specifically, the present invention relates to an image processing device that processes low-resolution images using a trained neural network, and a method for training a neural network.

[0002] Recently, as the display screen size and resolution of image processing devices that process images have rapidly increased, super-resolution (SR) technology that converts low-resolution (LR) images into high-resolution (HR) images is being developed.

[0003] Furthermore, with the advancement of deep learning technology, various forms of learning-based high-resolution technologies are being developed. Learning-based high-resolution technology trains an AI model (Artificial Intelligence model) in the cloud and then performs inference using the trained AI model. Learning-based high-resolution technology shows excellent performance when the quality characteristics of the training images and the input images to be actually processed are similar. However, if the characteristics of the actual images to be processed differ from the input quality assumed during training, the image quality performance deteriorates significantly, causing a problem called the domain gap. The domain gap problem can occur due to a mismatch in the characteristics of the training images used to train the AI ​​model and the images to be actually processed.

[0004] To address this domain gap issue, research is being conducted on upscaling technology through on-device learning, which trains and adapts AI models based on actual input images. For example, an AI model can be integrated into an edge device such as a TV or mobile device, and the input image can be used as training data for the AI ​​model in real time, and the trained AI model can be used to upscale the input image. On-device learning can refer to the operation of training the AI ​​model installed on the device in real time by collecting and computing information on the edge device itself without going through a cloud server. An edge device refers to an electronic device that can process data in real time on the device itself.

[0005] However, since edge devices have limited computational resources and internal memory, it is essential to reduce the amount of computation and resource usage required for AI model training, and shorten the time required for training.

[0006] Meanwhile, edge devices with AI models have limited computational resources, so integer (INT) operations are preferred over floating-point (FP) operations. Floating-point operations refer to operations on real-number data represented as real numbers that include the part after the decimal point, while integer operations refer to operations on integer data represented as integers that do not represent the part after the decimal point. Unlike floating-point operations, integer operations do not handle precision after the decimal point, so the amount of computation may be less than floating-point operations. In this case, for an edge device to perform integer operations, a quantization process that converts floating-point data to integer data may be involved. Since the quantization process approximates continuous values ​​(e.g., floating-point data) to discrete values ​​(e.g., integer data), quantization errors, which are errors that occur during the approximation process, may exist. Training AI models based on integer operations can lead to data loss and reduced accuracy due to quantization errors, resulting in lower performance compared to training AI models based on floating-point operations. Therefore, research is being conducted to minimize computational load while preventing performance degradation during AI model training on edge devices.

[0007] An image processing device according to one embodiment of the present disclosure includes a memory storing one or more instructions, and at least one processor executing the one or more instructions stored in the memory. The processor obtains a neural network model for quality processing of an input image by executing the one or more instructions. The processor applies learning data corresponding to the input image to the neural network model, thereby calculating a plurality of gradients obtained by partially differentiating weights of the neural network model with respect to a loss of the neural network model. The processor removes at least one gradient among the plurality of gradients by applying a gradient mask having gradient pruning information to the plurality of gradients. The processor trains the neural network model by updating weights of the neural network model based on one or more remaining gradients among the plurality of gradients. The processor obtains an output image whose quality has been processed from the input image based on the trained neural network model.

[0008] An operating method of an image processing device according to one embodiment of the present disclosure includes the steps of: obtaining a neural network model for image quality processing of an input image; applying learning data corresponding to the input image to the neural network model, thereby calculating a plurality of gradients obtained by partially differentiating weights of the neural network model with respect to a loss of the neural network model; removing at least one gradient among the plurality of gradients by applying a gradient mask having gradient pruning information to the plurality of gradients; training the neural network model by updating weights of the neural network model based on at least one remaining gradient among the plurality of gradients; and obtaining an image quality-processed output image from the input image based on the trained neural network model.

[0009] A method for training a neural network according to one embodiment of the present disclosure includes the steps of: obtaining a plurality of low-resolution training images that have been image quality deteriorated by a plurality of image quality degradation types corresponding to training images; training the neural network model by applying the training images and the plurality of low-resolution training images to the trained neural network model; calculating a plurality of gradients of the trained neural network model by applying the training images and at least one low-resolution training image to the trained neural network model; and generating a gradient mask for removing at least one gradient among the plurality of calculated gradients.

[0010] The present disclosure can be readily understood by the following detailed description and its accompanying drawings, wherein reference numerals refer to structural elements.

[0011] FIG. 1 is a drawing for explaining how an image processing device outputs an image whose quality has been processed, according to one embodiment.

[0012] FIG. 2 is a diagram for explaining the learning process of a meta model according to one embodiment.

[0013] Figure 3 is an internal block diagram of an image processing device according to one embodiment.

[0014] FIG. 4 is a drawing for explaining an operation of an image processing device according to one embodiment to determine the image quality of an input image.

[0015] FIG. 5 is a diagram illustrating an operation of an image processing device according to one embodiment of the present invention to adapt a meta model in response to an input image. FIG. 6 is a diagram illustrating a convolution operation in a convolution layer of a meta model according to one embodiment of the present invention.

[0016] FIG. 7 is a diagram for explaining the relationship between a filter and a gradient in a convolution layer of a meta model according to one embodiment.

[0017] Figure 8 is a diagram for explaining the result of slope pruning according to one embodiment.

[0018] Fig. 9 is a flowchart showing an operation method of an image processing device according to one embodiment.

[0019] Fig. 10 is an internal block diagram of an image processing device according to one embodiment.

[0020] FIG. 11 is a diagram for explaining the quantization operation of an image processing device according to one embodiment.

[0021] FIG. 12 is a diagram for explaining the quantization operation of an image processing device according to one embodiment.

[0022] Fig. 13 is a flowchart showing an operation method of an image processing device according to one embodiment.

[0023] Figure 14 is a block diagram of a server according to one embodiment.

[0024] FIG. 15 is a flowchart illustrating a method for a server to learn a meta model according to one embodiment.

[0025] FIG. 16 is a diagram illustrating a process for a server to learn a meta model according to one embodiment.

[0026] FIG. 17 is a flowchart illustrating a method for a server to obtain a slope mask according to one embodiment.

[0027] FIG. 18a is a diagram illustrating a process in which a server, according to one embodiment, calculates a gradient through a learned neural network.

[0028] FIG. 18b is a diagram illustrating a process by which a server obtains a slope mask according to one embodiment.

[0029] FIG. 19 is a flowchart illustrating a method by which a server learns a meta model according to one embodiment.

[0030] Fig. 20 is a block diagram of an image processing device according to one embodiment.

[0031] Fig. 21 is a flowchart showing an operation method of an image processing device and a server according to one embodiment.

[0032] In this disclosure, the expression “at least one of a, b or c” may refer to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “all of a, b and c”, or variations thereof.

[0033] Below, embodiments of the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

[0034] The terms used in this disclosure are described as currently common terms, taking into account the functions mentioned herein. However, these terms may mean various other terms depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Therefore, the terms used in this disclosure should not be interpreted solely based on their names, but rather based on the meanings of the terms and the overall content of this disclosure.

[0035] Additionally, the terminology used in this disclosure is for the purpose of describing specific embodiments only and is not intended to limit the present disclosure.

[0036] Throughout the specification, when a part is said to be "connected" to another part, this includes not only the cases where it is "directly connected" but also the cases where it is "electrically connected" with another element in between.

[0037] As used herein, and particularly in the claims, the terms "above" and "above" and similar referents may refer to both the singular and the plural. Furthermore, unless the order of steps in a method according to the present disclosure is explicitly specified, the steps described may be performed in any appropriate order. The present disclosure is not limited by the order in which the steps are described.

[0038] The appearances of phrases such as “in some embodiments” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment.

[0039] Some embodiments of the present disclosure may be represented by functional block configurations and various processing steps. Some or all of these functional blocks may be implemented by various hardware and / or software configurations that perform specific functions. For example, the functional blocks of the present disclosure may be implemented by one or more microprocessors or by circuit configurations for a given function. Furthermore, for example, the functional blocks of the present disclosure may be implemented in various programming or scripting languages. The functional blocks may be implemented by algorithms that execute on one or more processors. Furthermore, the present disclosure may employ conventional techniques for electronic configuration, signal processing, and / or data processing. Terms such as “mechanism,” “element,” “means,” and “configuration” may be used broadly and are not limited to mechanical and physical configurations.

[0040] Additionally, the connecting lines or connecting members between components depicted in the drawings are merely exemplary representations of functional connections and / or physical or circuit connections. In an actual device, connections between components may be represented by various functional connections, physical connections, or circuit connections that may be replaced or added.

[0041] The processor of the present disclosure may include various processing circuits and / or multiple processors. For example, the term “processor” as used herein, including in the claims, may include various processing circuits, including at least one processor. At least one processor, one or more processors, may be configured to perform various functions described herein, individually and / or collectively, in a distributed fashion. As used herein, “processor,” “at least one processor,” and “one or more processors” may be configured to perform various functions. However, these terms encompass, without limitation, situations where one processor performs some of the functions and other processor(s) perform other parts of the functions, and situations where a single processor may perform all of the functions. Furthermore, the at least one processor may include a combination of processors that perform various functions of the disclosed functions in a distributed manner. At least one processor may execute program instructions to achieve or perform various functions.

[0042] Additionally, terms such as “part”, “module”, etc. described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software, or a combination of hardware and software.

[0043] Additionally, the term "user" in this specification refers to a person who uses the image processing device, and may include a consumer, evaluator, viewer, administrator, or installer. Furthermore, the term "manufacturer" in this specification may refer to a manufacturer that manufactures the image processing device and / or components included in the image processing device.

[0044] In the present disclosure, 'image' may refer to a still image, a picture, a frame, a moving image composed of a plurality of consecutive still images, or a video.

[0045] In this disclosure, "neural network" refers to a representative example of an artificial neural network model that mimics brain neurons, and is not limited to an artificial neural network model using a specific algorithm. A neural network may also be referred to as a deep neural network.

[0046] Additionally, in the present disclosure, a "parameter" refers to a value used in the computational process of each layer forming a neural network, and may be used, for example, when applying an input value to a predetermined computational formula. A parameter is a value set as a result of training, and may be updated through separate training data as needed.

[0047] FIG. 1 is a drawing for explaining how an image processing device outputs an image whose quality has been processed, according to one embodiment.

[0048] Referring to FIG. 1, the image processing device (100) may be an electronic device capable of processing and outputting an image. In an embodiment, the image processing device (100) may be implemented as various types of electronic devices including a display. For example, the image processing device (200) may be implemented as various electronic devices such as a TV, a mobile phone, a tablet PC, a digital camera, a camcorder, a laptop computer, a desktop, an e-book reader, a digital broadcasting terminal, a PDA (Personal Digital Assistant), a PMP (Portable Multimedia Player), a navigation device, an MP3 player, a wearable device, etc.

[0049] The image processing device (100) can output video. The video may be composed of multiple frames. The video may include items such as television programs provided by content providers or various movies or dramas provided through VOD services. The content provider may refer to a terrestrial broadcaster, cable broadcaster, OTT service provider, or IPTV service provider that provides various content, including video, to consumers. The video is captured, compressed, transmitted, and then restored and output by the image processing device (100). Due to limitations in the physical characteristics of the device used to capture the video and limited bandwidth, information may be lost, resulting in image distortion. This image distortion may degrade the image quality.

[0050] An image processing device (100) according to one embodiment can process the image quality of an image. The image processing device (100) can process the image quality of video content whose quality has been degraded due to image distortion. For example, the image processing device (100) can obtain an output image (120) by performing image quality processing on an input image (110). For example, the image processing device (100) can obtain a high-resolution (or high-quality) output image by upscaling a low-resolution (or low-quality) input image using an image quality processing model.

[0051] According to one embodiment, the image quality processing model may include an inference network that implements a high-resolution (SR) algorithm capable of converting a low-resolution image into a high-resolution image. For example, the image quality processing model may include a neural network model trained using training data corresponding to the input image (110) from a neural network model acquired based on the quality of the input image (110).

[0052] An image processing device (100) according to one embodiment can train an image quality processing model installed in the image processing device (100) in real time within the device itself. The image quality processing model according to one embodiment may be referred to as a meta model. A meta model may refer to a neural network model that can quickly learn or generalize to new data. The meta model may be pre-trained using training data corresponding to various quality information stored in the cloud and then used to process the image quality of an actual image in the image processing device (100). The image processing device (100) can reduce the domain gap problem by training the meta model using actual images as training data. Since the meta model is a neural network that has been pre-trained for various image quality degradation scenarios, the training process for adjusting the parameters of the meta model based on the input image (110) can be simplified. For example, the number of iterative operations in the gradient descent method for adjusting the parameters of the meta model can be reduced. The image processing device (100) can generate an adaptive meta model for the input image (110) through on-device learning. In the present disclosure, the process in which the image processing device (100) learns a meta model using an input image (110) may be referred to as meta model adaptive learning.

[0053] To train a neural network model, training data consisting of pairs of low-resolution and high-resolution images can be used. The neural network model can be trained by updating its parameters to reduce the difference (training error or loss) between the low-resolution images in the training data processed by the neural network model and the output images and the high-resolution images in the training data. The parameters of the neural network model can be updated through multiple iterations. The training process of a neural network model includes forward propagation and back propagation. For example, the back propagation process of a neural network model involves calculating the difference between the predicted results of the neural network model using the training data and the actual results, calculating the gradient based on the difference, and updating the parameters of the neural network. Accordingly, the back propagation process may require three times more computational effort than the forward propagation process.

[0054] In one embodiment of the present disclosure, the image processing device (100) can reduce the backpropagation operation process through gradient pruning. The image processing device (100) can calculate the gradient for the meta model by applying learning data corresponding to the input image (110) to the meta model. The image processing device (100) can remove some gradients through gradient pruning. The image processing device (100) can update the parameters of the meta model using the remaining gradients that are not removed. The image processing device (100) can quickly adapt and learn the meta model to the input image (110) through gradient pruning.

[0055] For example, the image processing device (100) can generate gradient filters having a number corresponding to the number of weight filters by applying an input image to a neural network model having a predetermined number of weight filters. The image processing device (100) can retain gradient filters having a large generated gradient value and remove gradient filters having a small gradient value. The gradient filter is used to update the weight filter, and a gradient filter having a large gradient value can have a large degree of updating of the weight. The greater the degree to which the weight is updated in the direction of minimizing the loss, the higher the high-resolution performance of the neural network model can be improved.

[0056] The image processing device (100) can selectively update the weights of the neural network by considering the degree to which they contribute to improved high-resolution performance, thereby reducing the computational load of the backpropagation process. This will be discussed in detail in FIGS. 3 to 9.

[0057] In addition, in one embodiment of the present disclosure, the image processing device (100) can learn the meta model based on integer operations during the adaptive learning process for the meta model performed using the input image (110). The image processing device (100) can perform operations on the meta model based on weights expressed as integer data. For example, the image processing device (100) can perform a convolution operation using integer weights. Accordingly, the image processing device (100) can reduce the amount of computation compared to operations performed using real-valued weights. In addition, the quantization error occurring in the process of converting real-valued data into integer data can be minimized. This will be described in detail with reference to FIGS. 10 to 13.

[0058] Hereinafter, with reference to FIG. 2, an operation in which a meta model is first learned based on learning data in a server (1000) and adaptively learned based on an input image (110) in an image processing device (100) according to one embodiment will be described.

[0059] FIG. 2 is a diagram for explaining the learning process of a meta model according to one embodiment.

[0060] Referring to FIG. 2, the learning process of the meta model can be performed in the server (1000) and the image processing device (100). The learning process in the server (1000) can be expressed as primary learning, a training phase, or meta-learning. The learning process in the image processing device (100) can be expressed as secondary learning, a test phase, adaptive learning, or meta-adaptation.

[0061] A meta-model is a neural network model that can rapidly learn or generalize to new data, as it "preliminarily learns from the learning process performed during the test phase." The technique of teaching these meta-models "how to learn" can be expressed as meta-learning.

[0062] The technique of training a meta-model to quickly process new data can be referred to as meta-adaptation. For example, the meta-model may be a neural network model that adapts to new data. For example, the meta-model may be trained to process input images (110). Adapting the meta-model can be expressed as training the meta-model to adapt to the input images (110).

[0063] In the first learning step, a learned meta model (220) can be acquired from the initial meta model (210). In the second learning step, a meta model (230) adapted to the input image (110) can be acquired in response to the learned meta model (220). In one embodiment, the first learning step may be performed by a server (1000), and the second learning step may be performed by an image processing device (100). However, this is merely an example, and the two devices may be implemented as a single device.

[0064] The server (1000) can acquire an initial meta model (210). The initial meta model (210) can have initialized weights. The server (1000) can train the initial meta model (210) based on a training image (201) and a plurality of quality degradation kernels (202) used to degrade the quality of the training image (201) in a plurality of quality degradation types. For example, the server (1000) can generate a plurality of images degraded in different ways from the training image (201).

[0065] The server (1000) can obtain multiple transformed neural networks by applying multiple image quality degradation kernels (202) to the learning video (201). The server (1000) can learn various image quality degradation scenarios through the multiple transformed neural networks, and can repeat learning for the meta-model so as to have minimal errors in processing various image quality degradation scenarios. The server (1000) can obtain a learned meta-model (220) by performing meta-learning using the learning video (201). The learned meta-model (220) can have optimal weights that can process various image quality degradation scenarios. The learned meta-model (220) can be referred to as a meta-learned neural network.

[0066] The server (1000) can input a learning image (201) into the learned meta model (220) and produce a gradient mask (203). The server (1000) can simulate the meta learning process to produce a gradient from the learned meta model (220). The gradient may be a vector representation of a partial derivative with respect to parameters of a neural network model. The server (1000) can identify the degree to which the produced gradient contributes to the output of the learned meta model (220) and produce a gradient mask (203) for pruning the gradient. The gradient mask (203) can represent information for removing or maintaining the gradient according to the degree to which it contributes to the output of the learned meta model (220). For example, the gradient mask (203) can have a vector value for removing a gradient that contributes little to the output of the learned meta model (220) and a vector value for maintaining a gradient that contributes highly. The degree to which a weight contributes to the output of the meta model (220) can increase as the slope according to the weight increases.

[0067] The image processing device (100) can load a meta model that has been learned in advance from the server (1000) and load a gradient mask. The image processing device (100) can obtain a meta model (230) from the server (1000). The image processing device (100) can adjust the weights of the neural network by training the obtained meta model (230) with training data corresponding to an actual input image. The image processing device (100) can obtain an input image (110) and generate a low-resolution image (250) that has been deteriorated according to the degree of deterioration of the input image (110). The meta model (230) can be adapted to the input image (110) by being repeatedly trained a predetermined number of times with training data including the low-resolution image (250) and the input image (110).

[0068] In the present disclosure, the meta model (230) can be expressed as being adapted to the input image (110) in the sense that the meta model (230) is trained using the input image (110). In the present disclosure, the meta model (230) trained using training data corresponding to the input image can be referred to as an 'adapted meta model', an 'adaptive meta model', or a 'transfer model'. The image processing device (1000) can prune the gradient produced by the meta model (230) using the gradient mask (203). For example, the image processing device (100) can remove gradients that have a low degree of contribution to the output of the meta model (230) and maintain gradients that have a high degree of contribution. The image processing device (100) can adjust the weights of the meta model (230) based on the pruned gradients, thereby reducing the amount of computation.

[0069] The image processing device (100) can train a meta model (230) based on integer data. For example, the image processing device (100) can perform a convolution operation using integer weights, thereby reducing the amount of computation and minimizing quantization errors.

[0070] An image processing device (100) can generate an output image (120) with improved resolution of an input image (110) by using an adaptive meta model (230) to the input image (110).

[0071] Figure 3 is an internal block diagram of an image processing device according to one embodiment.

[0072] Referring to FIG. 3, an image processing device (100) according to one embodiment may include an image quality determination unit (310), a model adaptation unit (320), and an image quality processing unit (340). The image quality determination unit (310), the model adaptation unit (320), the gradient pruning unit (322), and the image quality processing unit (340) may be implemented by at least one processor. The image quality determination unit (310), the model adaptation unit (320), the gradient pruning unit (322), and the image quality processing unit (340) may operate according to at least one instruction stored in a memory.

[0073] In one embodiment, the image quality determination unit (310) can analyze or evaluate the image quality or quality of the input image (110). The image quality can indicate the degree of image deterioration. After the image is acquired through the image processing device (100), information is lost during the process of processing, compression, storage, transmission, restoration, etc., resulting in deterioration. The image quality determination unit (310) can evaluate or determine at least one of the compression deterioration of the input image (110), the compression degree of the input image, the degree of blur, the degree of noise, and the resolution of the image. In one embodiment, the image quality determination unit (310) can provide the image quality of the input image (110) to the model adaptation unit (320).

[0074] In one embodiment, the model adaptation unit (320) may generate learning data corresponding to the input image (110). The learning data may include an input image (110) corresponding to a high-resolution image (correct image, label) and a low-resolution image corresponding to the quality of the input image (110). For example, the model adaptation unit (320) may obtain a low-resolution image by degrading the input image (110) to have a degree of deterioration corresponding to the quality of the input image (110). For example, the quality determination unit (310) may obtain a low-resolution image by performing at least one of compression quality, blur quality, resolution, or noise on the input image (110). The low-resolution image may be an image that has been degraded once more by the degree of deterioration that has occurred while the input image (110) is processed, compressed, stored, transmitted, restored, etc. in the image processing device (100).

[0075] In one embodiment, the model adaptation unit (320) may acquire a metamodel. For example, the model adaptation unit (320) may acquire a metamodel corresponding to the image quality of the input image (110), but is not limited thereto. The metamodel may correspond to the metamodel (230) of FIG. 2.

[0076] In one embodiment, the model adaptation unit (320) may train a meta model using training data consisting of an input image (110) and a low-resolution image. In one embodiment, the model adaptation unit (320) may train a meta model (230) so that the meta model can process image quality according to the input image (110).

[0077] In one embodiment, the model adaptation unit (320) can obtain a result image (or predicted image) output from the meta model (230) by inputting a low-resolution image included in the learning data into the meta model (230) through a forward propagation process.

[0078] In one embodiment, the model adaptation unit (320) may compare the output image and the input image (110) included in the learning data, and adjust the parameters of the meta model (230) so that the difference between the two images is minimized. The parameters may include weights. The model adaptation unit (320) may calculate a loss, which is a difference between the output image and the input image (110). The model adaptation unit (320) may calculate the gradient of the meta model (230) by partially differentiating the loss with respect to the parameters of the meta model (230). The gradient may have a value for adjusting the parameters of the neural network in a direction that minimizes the loss (or learning error) of the neural network in a gradient descent algorithm.

[0079] In one embodiment, the slope can be expressed by mathematical expression 1.

[0080]

[0081] Here, g represents the gradient of the meta-model, J represents the loss of the meta-model, and w represents the weights of the meta-model. The gradient can be a partial derivative of the meta-model weights with respect to the loss. For example, the direction of the gradient determines the direction in which the weights move to minimize the loss, and the magnitude of the gradient can determine the degree of weight update.

[0082] In one embodiment, the model adaptation unit (320) can repeatedly update parameters until the loss is minimized. The model adaptation unit (320) can optimize the parameters of the meta-model. In one embodiment, the parameters of the meta-model updated by the model adaptation unit (320) can be expressed as in the following mathematical expression 1.

[0083]

[0084] In mathematical expression 2, w i+1 is the updated weight of the meta model, and w iis the current weight of the meta model, lr is the learning rate of the meta model, is the slope of the meta model (corresponding to Equation 1). The weight of the meta model can be updated by reflecting the slope to the current weight of the meta model.

[0085] Meanwhile, in one embodiment, the meta model (230) may include a convolutional neural network (CNN) composed of multiple convolutional layers. The convolutional neural network (CNN) may perform convolutional operations through multiple convolutional layers. Each of the convolutional layers of the CNN may include one or more filters, and each filter may have weights. Meanwhile, the filters may be referred to as weighted filters.

[0086] In one embodiment, the model adaptation unit (320) may update the weights of the meta-model with the calculated gradient. The gradient may be used to update the weights of the filters used in each convolutional layer of the meta-model (230).

[0087] In one embodiment, the model adaptation unit (320) may produce a number of gradients corresponding to the number of filters used in each convolutional layer. For example, the gradients may include multiple gradients corresponding to the multiple filters used in each convolutional layer. Each of the multiple gradients may be used to update the weights of the multiple filters.

[0088] In one embodiment, the model adaptation unit (320) can remove at least one gradient from among the plurality of gradients by applying a gradient mask having gradient pruning information to the plurality of gradients through the gradient pruning unit (322). At least one gradient from among the plurality of gradients is removed through the gradient mask, and the remaining gradients may be maintained without being removed. The gradient mask may include information for masking, removing, or pruning the plurality of gradients.

[0089] In one embodiment, the gradient mask may be a binary mask having a binary value indicating whether or not to remove a gradient used to update the weights of the metamodel. For example, the gradient mask may have a vector value of 0 corresponding to a gradient to be removed, and a vector value of 1 corresponding to a gradient to be maintained. The gradient mask may be stored in an external database or in the internal memory of the image processing device (100).

[0090] In one embodiment, the model adaptation unit (320) may train the metamodel (230) by removing some of the plurality of gradients through a gradient mask and updating the parameters of the metamodel (230) with the remaining gradients. In one embodiment, the model adaptation unit (320) may omit the process of updating the parameters of the metamodel (230) corresponding to the removed gradients. The parameters of the metamodel (230) may be updated based on the gradients that remain without being removed based on the gradient mask. In the present disclosure, since gradient pruning omits some of the backpropagation process, it may also be referred to as backward pruning.

[0091] The weights updated by the model adaptation unit (320) according to one embodiment can be expressed as in the following mathematical expression 3.

[0092]

[0093] In mathematical expression 3, w (l) are the weights corresponding to each filter for the lth convolutional layer, lr is the learning rate, and m (l) is the gradient mask for the lth convolutional layer, and g (l) is the gradient for the lth convolutional layer. Gradient mask m (l) may contain information for removing the gradient used to update each filter.

[0094] In one embodiment, the gradient mask m (l) The vector value of the multiple slopes g (l) can be multiplied by m (l) g (l) can represent the remaining gradient after removing it based on the gradient mask. The weights of the filter for the lth convolutional layer w (l) is the remaining gradient m that is not removed based on the gradient mask. (l) g (l) can be updated based on.

[0095] In one embodiment, multiple slopes g (l) Is can be expressed as . Here, k x is the width of the gradient data (or the number of columns of the gradient matrix), and k y is the height of the gradient data (or the number of rows of the gradient matrix), and c out represents the number of gradients (or the number of gradient filters). In one embodiment, the gradient mask m (l) The size of (or the number of rows of the gradient mask matrix) is C out can be expressed as

[0096] Meanwhile, the filter for the lth convolutional layer is can be expressed as c out represents the number of output channels (or the number of feature map channels). Meanwhile, the number of input channels is illustrated as 1 for convenience, but is not limited thereto.

[0097] In one embodiment, the gradient mask m (l) Size of (C) out ) is the slope g (l) The number of (C) out ) may be the same.

[0098] In one embodiment, the model adaptation unit (320) can remove at least one gradient for updating at least one filter among the plurality of filters through a gradient mask, and retain the remaining gradients for updating the remaining filters among the plurality of filters. The model adaptation unit (320) can train the metamodel by updating the remaining filters based on the remaining gradients.

[0099] In one embodiment, the gradient mask may include information for maintaining gradients that contribute significantly to improving the high-resolution performance of the meta-model. For example, gradients with large gradient values ​​may contribute significantly to improving the high-resolution performance. For example, gradients with large gradient values ​​have a large degree of weight update, so the weights of the meta-model can be quickly optimized. For example, the gradient mask may include information for maintaining gradients with large gradient values ​​and removing gradients with small gradient values. The gradient mask may include information for removing gradients corresponding to a minimum value of n by numerically quantifying multiple gradients (where n is a natural number). Alternatively, for example, the gradient mask may include information for removing gradients below a threshold value by numerically quantifying multiple gradients. For example, the numerical values ​​of multiple gradients may be obtained by summing data in a gradient data matrix for each gradient filter. Multiple gradients may correspond to multiple gradient filters. The detailed process of generating the gradient mask will be described in FIGS. 17 to 18B.

[0100] In one embodiment, the image processing device (100) can rapidly learn a meta-model through gradient pruning. By performing learning through gradient pruning, the image processing device (100) can reduce the computational load of the backpropagation process and reduce memory usage. The image processing device (100) can acquire an adaptive meta-model for an input image (110) through gradient pruning.

[0101] In one embodiment, the image quality processing unit (330) may utilize the meta model learned by the model adaptation unit (320). The image quality processing unit (330) may process the image quality of the input image (110) using the updated meta model. The image quality processing unit (330) may process the image quality of the input image (110) using a meta model adaptive to the input image (110), thereby obtaining a high-resolution image.

[0102] Meanwhile, in one embodiment, the image processing device (100) can learn a neural network model based on integer operations. By performing learning based on integer operations, the image processing device (100) can reduce the amount of computation compared to real-valued operations and minimize quantization errors. This will be discussed in detail with reference to FIGS. 10 to 13.

[0103] Meanwhile, although FIG. 3 illustrates the image quality determination unit (310), the model adaptation unit (320), the gradient pruning unit (322), and the image quality processing unit (340) individually, the image quality determination unit (310), the model adaptation unit (320), the gradient pruning unit (322), and the image quality processing unit (340) may be implemented through a single processor. In this case, the image quality determination unit (310), the model adaptation unit (320), the gradient pruning unit (322), and the image quality processing unit (340) may be implemented as a dedicated processor, or may be implemented through a combination of a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU) and software. In addition, in the case of a dedicated processor, it may include a memory for implementing an embodiment of the present disclosure, or may include a memory processing unit for utilizing an external memory. The quality judgment unit (310), model adaptation unit (320), gradient pruning unit (322), and image quality processing unit (340) may be composed of multiple processors.

[0104] FIG. 4 is a diagram illustrating an operation of an image processing device according to one embodiment to determine the image quality of an input image. The operation of FIG. 4 can be performed through the image quality determination unit (310) of FIG. 3.

[0105] Referring to FIG. 4, an image processing device (100) according to one embodiment can determine the image quality of an input image (110) and generate a low-resolution image (250) based on the input image (110).

[0106] In one embodiment, the image processing device (100) may include a kernel prediction network (410), which is a neural network trained to evaluate the image quality of an input image (110). In one embodiment, the image processing device (100) may evaluate the image quality of the input image (110) through the kernel prediction network (410). The kernel prediction network (410) may analyze or evaluate the image quality of the input image (110) using a neural network trained to analyze or evaluate the image quality of the input image (110). For example, the neural network may be a neural network trained to evaluate the image quality of an image or video using an Image Quality Assessment (IQA) technology, a Video Quality Assessment (VQA) technology, or the like.

[0107] In one embodiment, the kernel prediction network (410) may output a predicted kernel (420). For example, the kernel prediction network (410) may analyze an image to determine the degree of degradation that has occurred while the image is being processed, compressed, stored, transmitted, restored, etc. For example, the predicted kernel (420) may have information regarding at least one of the compressed image quality, the blur quality, the resolution, or the noise of the input image (110).

[0108] In one embodiment, the image processing device (100) may obtain a low-resolution image (250) based on the image quality of the input image (110) obtained by analyzing the input image (110). For example, the image processing device (100) may obtain a low-resolution image (250) by applying a kernel (420) predicted through a kernel prediction network (410) to the input image (110). For example, the image processing device (100) may obtain a low-resolution image (250) by performing at least one of compression image quality, blur image quality, resolution, or noise on the input image (110).

[0109] In one embodiment, the image processing device (100) can generate training data including an input image (110) and a low-resolution image (250). For example, in FIG. 4, the image processing device (100) can determine that the resolution of an image received by the image processing device (100) has been reduced by a factor of two as it has been converted from FHD (Full HD) to UHD (Ultra HD) through the kernel prediction network (410). The image processing device (100) can generate a low-resolution image (250) by reducing the input image (110) by a factor of 1 / 2 so that image quality degradation processing corresponding to the input image (110) is applied.

[0110] FIG. 5 is a diagram illustrating an operation of an image processing device according to one embodiment of the present invention to adapt a metamodel in response to an input image. The operation of FIG. 5 may be performed through the model adaptation unit (320) of FIG. 3.

[0111] Referring to FIG. 5, the image processing device (100) can obtain a neural network model (510). Here, the neural network model (510) can correspond to the meta model (230) of FIG. 2.

[0112] An image processing device (100) according to one embodiment can train a neural network model (510) using learning data consisting of an input image (110) and a low-resolution image (250) whose image quality has been processed corresponding to the input image (110).

[0113] A neural network model (510) according to one embodiment may be a convolutional neural network (CNN) having a plurality of convolutional layers. Each of the convolutional layers may include one or more filters, and each filter may have a weight value. For example, the image processing device (100) may perform a convolution operation between a low-resolution image (250) and filters used in the convolutional layers of the neural network model (510) to output a resultant image (520). For example, the convolutional layers may include at least one convolutional layer, for example, a first convolutional layer (610), a second convolutional layer (620), and a third convolutional layer (630).

[0114] A low-resolution image (250) can be input to a first convolution layer (610). 5X5X5 displayed in the first convolution layer (610) exemplifies convolution processing for a single input image, a low-resolution image (250), using five filters of 5X5 size. For example, since five filters are used in the first convolution layer (610), five feature maps (615) can be generated as a result of convolution processing using the five filters. The feature maps (615) output from the first convolution layer (610) can be input to a second convolution layer (620). The second convolution layer (620) can correspond to the lth convolution layer. The feature map (not shown) output from the second convolution layer (620) can be input to the third convolution layer (630). The third convolution layer (630) can generate a result image (520), which is one output data, using one filter of size 5X5.

[0115] Meanwhile, in one embodiment, the neural network model (510) may further include an activation layer (not shown). For example, the activation layer may include a sigmoid function, a Tanh function, a ReLU (Rectified Linear Unit) function, etc. to impart non-linear characteristics to each feature map.

[0116] The image processing device (100) can calculate the gradient (640) to minimize the loss (530), which is the difference between the result image (520) and the input image (110), through a backpropagation process. For example, the direction of the gradient (640) indicates the movement direction of the weights of the filters used in the convolution layers to minimize the loss (530), and the magnitude of the gradient (640) can indicate the degree of update of the weights of the filters. The image processing device (100) can update the parameters of the neural network model (510) in the direction in which the gradient of the calculated loss (530) decreases through the gradient descent method, and repeat this until the loss (530) is minimized.

[0117] The gradient may include multiple gradients corresponding to multiple filters used in each convolutional layer. For example, the gradient (640) may include multiple gradients (641, 642, 643, 644, 645). Each of the multiple gradients (641, 642, 643, 644, 645) may correspond to five filters used in the first convolutional layer (610). For example, the five filters may be the multiple filters (731, 732, 733, 734, 735) illustrated in FIG. 6.

[0118] Referring to FIGS. 6 and 7, the convolution operation process of the convolution layer according to one embodiment will be further described. FIG. 6 is a diagram for explaining the convolution operation in the convolution layer of the meta model according to one embodiment. FIG. 7 is a diagram for explaining the relationship between a filter and a gradient in the convolution layer of the meta model according to one embodiment.

[0119] Referring to FIG. 6, one feature map (750) can be generated through a multiplication operation and an addition operation between the parameters of a filter (730) having a size of 5X5 used in the first convolution layer (610) and the sample values ​​in the corresponding input data (705). Since five filters are used in the first convolution layer (610), five feature maps can be generated through a convolution operation process using the five filters.

[0120] In FIG. 6, I1 to I49 indicated in the input data (705) represent samples (which may be referred to as pixels) of the input data (705), and F1 to F25 indicated in the filter (730) represent weights of the filter (730). In addition, M1 to M9 indicated in the feature map (750) represent samples of the feature map. For example, the feature map (750) may correspond to the feature map (615) of FIG. 6.

[0121] In the convolution operation process, a product operation is performed on each of the sample values ​​of I1, I2, I3, I4, I5 I8, I9, I10, I11, I12, I15, I16, I17, I18, I19, I22, I23, I24, I25, I26, I29, I30, I31, I32, and I33 of the input data (705) and each of F1 to F25 of the filter (730), and a value obtained by combining (e.g., adding) the result values ​​of the product operation can be assigned as the value of M1 of the feature map (750). The stride of the convolution operation can be 1.

[0122] A feature map (750) having a predetermined size can be obtained by performing a convolution operation between sample values ​​in the input data (705) and samples of the filter (730) while the filter (730) moves according to the stride until it reaches the last sample of the input data (705). For example, a feature map (750) having a size of 5X5 can be obtained. A loss function can be obtained based on the difference between the feature map (750) and the result image (250).

[0123] According to the present disclosure, through training of a neural network model (510), the values ​​of the weights of the neural network model (510), for example, the weights of the filters (730) used in the convolutional layers of the neural network model (510) (for example, F1 to F25 of the filters) can be optimized.

[0124] The convolution layers included in the neural network model (510) can perform processing according to the convolution operation process described in relation to FIG. 5, but the convolution operation process described in FIG. 5 is only an example and is not limited thereto.

[0125] Referring to FIG. 7, a plurality of gradients (641, 642, 643, 644, 645) may be generated to update a plurality of filters (731, 732, 733, 734, 735) in a first convolutional layer (610). For example, each of the plurality of gradients (641, 642, 643, 644, 645) may be a gradient of a loss function with respect to the weights of the plurality of filters (731, 732, 733, 734, 735). For example, the first gradient (641) may be a gradient of a loss function with respect to the weights of the first filter (731). For example, the second gradient (642) may be a gradient of a loss function with respect to the weights of the second filter (732). For example, the third slope (643) may be the slope of the loss function with respect to the weight of the third filter (733). For example, the fourth slope (644) may be the slope of the loss function with respect to the weight of the fourth filter (734). For example, the fifth slope (645) may be the slope of the loss function with respect to the weight of the fifth filter (735).

[0126] In one embodiment, the plurality of slopes (641, 642, 643, 644, 645) can be expressed as . In addition, multiple filters (731, 732, 733, 734, 735) can be expressed as . The sizes of the plurality of slopes (641, 642, 643, 644, 645) can correspond to the sizes of the plurality of filters (731, 732, 733, 734, 735). For example, each slope or each filter It can have a size. The number of multiple gradients (641, 642, 643, 644, 645) can be equal to the number of multiple filters (731, 732, 733, 734, 735). For example, the number of gradients or the number of filters can be c outcan be expressed as . For example, since five filters of size 5x5 are used in the first convolution layer (610), five gradients of size 5x5 can be produced to update the five filters.

[0127] The image processing device (100) can apply a gradient mask (650) having gradient pruning information to the calculated gradient (640). The gradient mask (650) can include gradient pruning information for pruning multiple gradients. The gradient mask (650) is exemplified as being expressed as a column vector, but is not limited thereto. The gradient mask (650) can have a vector value of 0 corresponding to gradients (642, 644, 645) to be removed, and can have a vector value of 1 corresponding to gradients (641, 643) to be maintained. The slopes to be removed (642, 644, 645) are slopes that have a low contribution to improving the performance of the neural network model (510), and the slopes to be maintained (641, 643) are slopes that have a high contribution to improving the performance of the neural network model (510).

[0128] The size (or number of rows) of the gradient mask (650) is equal to the number of filters of the neural network model (510) and may be equal to the number of gradients (640). For example, the number of gradient masks (650) may be c out can be expressed as For example, the gradient mask (650) It may include five vector values ​​for whether to remove each of the plurality of gradients (641, 642, 643, 644, 645) corresponding to each of the plurality of filters (731, 732, 733, 734, 735).

[0129] The image processing device (100) can remove the second slope (642), the fourth slope (644), and the fifth slope (645) among the plurality of slopes (641, 642, 643, 644, 645) based on the slope mask (650), and maintain the first slope (641) and the third slope (643). For example, among the plurality of slopes (641, 642, 643, 644, 645), the second slope (642), the fourth slope (644), and the fifth slope (645) can be removed, and the first slope (641) and the third slope (643) can be maintained. The first slope (641) and the third slope (643) can be included in the pruned slope (660).

[0130] The image processing device (100) may partially update the weights of the neural network model (510) based on the pruned gradient (660). For example, the image processing device (100) may update the first filter (731) and the third filter (733), and not update the second filter (732), the fourth filter (734), and the fifth filter (735). Since the update processes of the second filter (732), the fourth filter (734), and the fifth filter (735) are omitted, the backpropagation process may be simplified.

[0131] The image processing device (100) can obtain an updated first filter (831) based on the first slope (641) and obtain an updated third filter (833) based on the third slope (643). The first filter (831) and the third filter (833) can be repeatedly updated through iteration, and can be updated until the weights of the first filter (831) and the weights of the third filter (833) are optimized.

[0132] Referring back to FIG. 5, the image processing device (100) preserves the gradient that contributes to improving the high-resolution performance of the neural network model (510) and repeatedly updates only the weights of the filter corresponding to the preserved gradient, thereby enabling the neural network model (510) to be trained quickly. In one embodiment, the image processing device (100) can quickly train a neural network model (510) that is adaptive to the input image (110) through gradient pruning, and since the amount of computation for the backpropagation process that updates the weights through the gradient is reduced, the memory usage can be reduced.

[0133] Meanwhile, the convolution layers of the neural network model (510) are not limited to the first convolution layer (610), the second convolution layer (620), and the third convolution layer (630) illustrated. For example, the neural network model (510) may include L convolution layers. Each of the first convolution layer (610), the second convolution layer (620), and the third convolution layer (630) may be referred to as an lth convolution layer (L and l are natural numbers). The plurality of filters may refer to filters belonging to the lth convolution layer.

[0134] Meanwhile, in the present disclosure, the gradient descent method is exemplified as an algorithm for training a neural network model (510), but is not limited thereto. The image processing device (100) can train the neural network model (510) using various known learning algorithms. The image processing device (100) can selectively apply learning hyperparameters (learning rate, batch size, termination condition, etc.) and optimization algorithms (stochastic gradient descent (SGD), Adam, Adamp sec), etc., depending on the constraints of the system, such as memory, computational unit, power, etc.

[0135] Figure 8 is a diagram for explaining the result of slope pruning according to one embodiment.

[0136] Referring to Fig. 8, in 801, J represents a loss function, A represents activation, and G represents a gradient. The activation may be an intermediate output that is calculated to produce a gradient according to the chain rule of backpropagation. The loss function, activation, and gradient are each represented as a convolution tensor, and the convolution tensor refers to a high-dimensional extended matrix to which a convolution operation is applied.

[0137] J at 802 prune represents the pruned loss function, and A prune represents pruned activation, and G prune represents a pruned gradient. W represents a weight updated based on the pruned gradient. Since the image processing device (100) does not use the gradient removed by the gradient mask, the process of calculating a part of the activation used for gradient calculation may be omitted. Accordingly, the amount of memory for storing the activation, which is an intermediate output, may be reduced. In addition, since the image processing device (100) does not use the gradient removed by the gradient mask, the process of calculating a part of the loss function used for gradient calculation may be omitted. Accordingly, the amount of calculation for calculating the loss function may be reduced.

[0138] Fig. 9 is a flowchart showing an operation method of an image processing device according to one embodiment.

[0139] In step 910, the image processing device (100) according to one embodiment may acquire a meta model. For example, the image processing device (100) may acquire a meta model corresponding to the image quality of the input image (110), but is not limited thereto. The meta model may correspond to the meta model (230) of FIG. 2 or the neural network model (510) of FIG. 5.

[0140] In step 920, an image processing device (100) according to one embodiment can calculate multiple gradients of a meta model using learning data corresponding to an input image (110).

[0141] An image processing device (100) according to one embodiment can train a meta model using learning data including an input image (110) and a low-resolution image degraded from the input image (110). The meta model can perform a convolution operation between the low-resolution image and filters used in convolution layers of the meta model to output a feature map of the resulting image. Each convolution layer has one or more filters, and each filter can have a weight.

[0142] The image processing device (100) can obtain a result image output from the meta model (230) by inputting a low-resolution image included in the learning data into the meta model (230). The image processing device (100) can compare the result image and the input image (110) to calculate the loss, which is the difference between the two images. The image processing device (100) can calculate the gradient of the meta model so that the loss is minimized. The gradient can be expressed as in Mathematical Expression 1. The gradient can have multiple gradients that are obtained by partially differentiating the loss function with respect to each of the weights of the neural network.

[0143] The image processing device (100) can calculate a plurality of gradients corresponding to each of the plurality of filters belonging to each of the convolution layers. The plurality of gradients can be used to update the weights of each of the plurality of filters used in each of the convolution layers. The image processing device (100) can optimize the weights of the plurality of filters used in the convolution layers of the meta model.

[0144] The image processing device (100) can calculate a number of gradients corresponding to the number of filters used in the convolution layers. The image processing device (100) can calculate a plurality of gradients that can update each of the plurality of filters. In one embodiment, the number of the plurality of filters belonging to the lth convolution layer may be equal to the number of the plurality of gradients for the lth convolution layer.

[0145] In one embodiment, the image processing device (100) may perform gradient pruning according to step 930 and then update the weights of each of the plurality of filters based on the remaining gradients.

[0146] In step 930, the image processing device (100) according to one embodiment can remove at least one slope among the plurality of slopes by applying a slope mask having slope pruning information to the plurality of slopes. The image processing device (100) according to one embodiment can preserve the remaining slopes among the plurality of slopes without removing them.

[0147] In one embodiment, the gradient mask may include gradient pruning information for masking, removing, or pruning multiple gradients. For example, the gradient pruning information may include information on which of the multiple filters within each convolutional layer has a gradient to remove for updating a filter, or which of the gradients to keep for updating a filter.

[0148] An image processing device (100) according to one embodiment may remove at least one slope corresponding to at least one filter among a plurality of filters based on a slope mask, and maintain the remaining slopes corresponding to the remaining filters among the plurality of filters. Accordingly, the image processing device (100) may update the weights of the remaining filters corresponding to the remaining slopes without updating the weight of at least one filter corresponding to the removed at least one slope.

[0149] In one embodiment, the gradient mask (650) may have a vector value for removing a gradient that has a low contribution to improving high-resolution performance among a plurality of filters belonging to the meta model, and a vector value for maintaining a gradient that has a high contribution among the plurality of filters. For example, the gradient mask may have a vector value of 0 corresponding to a gradient to be removed among the plurality of gradients, and a vector value of 1 corresponding to a gradient to be maintained. For example, a gradient with a large gradient value may have a high contribution to improving high-resolution performance.

[0150] In one embodiment, the size of the gradient mask for the lth convolutional layer may be equal to the number of multiple gradients for updating each of the multiple filters belonging to the lth convolutional layer.

[0151] In step 940, the image processing device (100) according to one embodiment can train the meta model by updating the weights of the meta model based on the remaining slopes among the plurality of slopes.

[0152] In one embodiment, the image processing device (100) can obtain a pruned gradient through a gradient mask. The pruned gradient may include the remaining gradients excluding the gradients removed through the gradient mask. The image processing device (100) can update the weights of the remaining filters corresponding to the remaining gradients maintained by the gradient mask. The image processing device (100) can train the meta-model by updating the weights of the filters based on the pruned gradients. This is described in Mathematical Expression 3.

[0153] In step 950, an image processing device (100) according to one embodiment can obtain an output image whose quality has been processed from an input image based on an updated meta model.

[0154] An image processing device (100) according to one embodiment can process the image quality of an input image (110) using an updated meta model. The image processing device (100) can obtain a high-resolution output image (120) by applying an input image (110) to a meta model that is adaptive to the input image (110).

[0155] In one embodiment, the image processing device (100) can reduce the amount of gradient data to be calculated in the backpropagation process while continuing the forward propagation process, and can reduce the amount of computation required to update the weights. The image processing device (100) removes gradients that have a low contribution to improving the high-resolution performance of the metamodel and does not update the weights of the filter corresponding to the removed gradients, thereby reducing the amount of computation required for the backpropagation process and reducing memory usage.

[0156] In one embodiment, the image processing device (100) preserves the gradient that contributes to improving the high-resolution performance of the meta model and repeatedly updates only the weights of the filter corresponding to the preserved gradient, thereby enabling rapid image quality processing of the input image (110).

[0157] Fig. 10 is an internal block diagram of an image processing device according to one embodiment.

[0158] Referring to FIG. 10, an image processing device (100) according to one embodiment may include an image quality determination unit (310), a model adaptation unit (320-1), and an image quality processing unit (340). In one embodiment, the model adaptation unit (320-1) may include a gradient pruning unit (322) and a quantization unit (324). The image processing device (100) illustrated in FIG. 10 is the same as the image processing device (100) illustrated in FIG. 3, except that the model adaptation unit (320-1) further includes a quantization unit (324). The quantization unit (324) may be implemented with at least one processor. The quantization unit (324) may operate according to at least one instruction stored in a memory.

[0159] In one embodiment, the image processing device (100) trains a metamodel in real time within the device, which may require significant computational resources and computational load. For example, the metamodel training process involves a backpropagation process repeated hundreds to tens of thousands of times. Since the image processing device (100) has limited computational resources and internal memory compared to cloud servers, it can train a neural network model based on integer operations.

[0160] In one embodiment, the image processing device (100) may perform a quantization process to convert real-valued data into integer data in order to train a neural network model based on integer operations. In one embodiment, integer data refers to data that does not have a value below the decimal point, and may be expressed as a binary number such as an 8-bit integer, a 16-bit integer, or a 32-bit integer. In one embodiment, real-valued data refers to floating-point numbers that have a value below the decimal point, and the floating-point numbers may be expressed as binary numbers such as 16-bit, 32-bit, or the like. In one embodiment, real-valued data may be converted into integer data and expressed through a quantization process. For example, when one real-valued data is converted into 8-bit integer data, one real-valued data may be expressed as two integer data, for example, a value of the data and a scale of the data.

[0161] In one embodiment, the model adaptation unit (320-1) can calculate a gradient through a forward propagation and backward propagation process, and the calculated gradient may be real-valued data. The quantization unit (324) can obtain integer data by going through a quantization process for the real-valued gradient. The model adaptation unit (320-1) can perform a convolution operation using the integer data obtained through the quantization unit (324) and train the meta-model. Accordingly, the image processing device (100) can reduce the amount of computation by training the meta-model connected to the image processing device (100) with integer data, compared to training with real-valued data. In the present disclosure, a meta-model trained with integer data may be referred to as an integer neural network model. Data of a meta-model trained with integer data may be expressed as integer data, not real-valued data.

[0162] In one embodiment, the memory of the image processing device (100) can store integer weights (see FIG. 11). The image processing device (100) can quantize the real-valued gradient produced through the quantization unit (324) into an integer gradient. The image processing device (100) can update the quantized integer gradient to the integer weight stored in the memory. The image processing device (100) can perform a convolution operation for the next iteration using the updated integer weight. The image processing device (100) according to one embodiment can train a meta-model using integer data acquired through the quantization unit (324). Since the image processing device (100) according to one embodiment can train a meta-model using integer weights, the amount of computation can be reduced.

[0163] Alternatively, in one embodiment, the memory of the image processing device (100) may store real-valued weights (see FIG. 12). The image processing device (100) may update the real-valued slopes obtained through the quantization unit (324) to real-valued weights without quantizing them. The image processing device (100) may quantize the updated real-valued weights into integer weights. The image processing device (100) may perform a convolution operation for the next iteration using the quantized integer weights. The image processing device (100) according to one embodiment may train a meta-model using integer data obtained through the quantization unit (324).

[0164] Meanwhile, the quantization unit (324) according to one embodiment may perform scaling on the gradient as a preprocessing for updating the gradient of the weights. For example, the quantization unit (324) may scale the range of the gradient so that the range of the gradient is identical to the range of the weights. Here, scaling refers to an operation of adjusting the range of data.

[0165] According to one embodiment, when integer weights are stored in the memory of the image processing device (100), the image processing device (100) may perform scaling on the integer gradient as a preprocessing for updating the integer gradient to the integer weight (see FIG. 11). In one embodiment, since the integer weights are stored in the memory of the image processing device (100), a quantization process for converting the real-valued gradient to the integer gradient is involved. In this case, the precision of the integer gradient converted from the real-valued gradient may decrease due to quantization error. Accordingly, when scaling the range of the gradient to correspond to the range of the weight, the distribution distortion of the gradient data may increase. In one embodiment, the image processing device (100) updates the weights using the gradient with increased data distribution distortion, which may cause distribution distortion of the weight data. When training a meta-model based on weights with data distribution distortion, data distribution distortion and quantization error may increase.

[0166] In contrast, according to one embodiment, when real-valued weights are stored in the memory of the image processing device (100), scaling of the real-valued gradients may be performed as preprocessing for updating the real-valued gradients in the real-valued weights (see FIG. 12). In one embodiment, since real-valued weights are stored in the memory of the image processing device (100), a quantization process for the real-valued gradients may be omitted. In this case, no quantization error exists and the precision of the real-valued gradients may be maintained relatively high. Accordingly, when scaling the range of the gradients to correspond to the range of the weights, the distribution distortion of the gradient data may be minimized. In one embodiment, the image processing device (100) may update the weights using the gradients with minimized data distribution distortion, and perform quantization on the updated weights to train the meta-model. Accordingly, the data distribution distortion and quantization error of the image processing device (100) may be minimized.

[0167] In one embodiment, the specific operation method of the quantization unit (324) is different when integer weights are stored in the memory of the image processing device (100) and when real-valued weights are stored, and this is described in FIG. 11 and FIG. 12, respectively.

[0168] FIG. 11 is a diagram for explaining the quantization operation of an image processing device according to one embodiment.

[0169] Referring to FIG. 11, an image processing device (100) according to one embodiment can perform a quantization operation through a model adaptation unit (320-1).

[0170] In one embodiment, the memory (1100) of the image processing device (100) may store an integer weight (1104). For example, the integer weight (1104) may be integer data consisting of 8 bits. The range of the integer weight (1104) may be [-1, 1], and the precision may be 0.008. However, the bits, range, and precision of the integer weight (1104) are not limited to the example of FIG. 11.

[0171] In operation S1110, the image processing device (100) can quantize the real-valued slope (1101) produced by the image processing device (100) into an integer slope (1102).

[0172] The real number gradient (1101) may be real number data consisting of 32 bits. The range of the real number gradient (1101) may be [-0.002, 0.002], and the precision may be 9e-13. However, the bits, range, and precision of the real number gradient (1101) are not limited to the example of FIG. 11.

[0173] The integer slope (1102) may be integer data consisting of 8 bits. The range of the integer slope (1102) may be [-0.002, 0.002], and the precision may be 1e-5. However, the bits, range, and precision of the integer slope (1102) are not limited to the example of FIG. 11.

[0174] Since the integer gradient (1102) is quantized from the real gradient (1101), it may have lower precision than the real gradient (1101). The range of the integer gradient (1102) may be the same as the range of the real gradient (1101).

[0175] In operation S1120, the image processing device (100) may perform scaling on the integer gradient (1102) as a preprocessing for updating the integer gradient (1102) to the integer weight (1104). For example, the image processing device (100) may adjust the range of the integer gradient (1102) so that the range of the integer gradient (1102) is the same as the range of the integer weight (1104). Here, scaling refers to an operation of adjusting the range of data. For example, scaling may include a min-max normalization method that scales so that the minimum and maximum of the range of data are 0 and 1, a normalization method that scales so that all values ​​are expressed between -1 and 1, and a standardization method that standardizes so that the data has a normal distribution with a mean of 0 and a variance of 1.

[0176] The image processing device (100) can obtain a scaled integer slope (1103) by scaling the range of the integer slope (1102). For example, in FIG. 11, the scaling is illustrated as a normalization method that scales so that all values ​​are expressed between -1 and 1, but is not limited thereto.

[0177] The scaled integer gradient (1103) may be integer data consisting of 8 bits. The range of the scaled integer gradient (1103) is [-1, 1], which is the same as the range of the integer weight (1104), and the precision may be 0.008. However, the bits, range, and precision of the integer gradient (1102) are not limited to the example of FIG. 11.

[0178] In one embodiment, when scaling the range of the integer gradient (1102) to match the range of the integer weight (1104), if the range difference between the two data is large, the precision of the integer gradient (1102) may decrease. For example, the precision of the scaled integer gradient (1103) may decrease significantly compared to the precision of the integer gradient before scaling. Accordingly, the data distribution of the integer gradient may be distorted.

[0179] In operation S1130, the image processing device (100) can update the integer gradient (1103) in the integer weight (1104) stored in the memory (1100). The image processing device (100) can obtain the updated integer weight (1105) and perform a convolution operation for the next iteration (e.g., i+1) using the updated integer weight (1105).

[0180] In operation S1140, the image processing device (100) can perform a convolution operation of the forward propagation process using the updated integer weights (1105). The image processing device (100) can train a meta-model using training data corresponding to an input image using the integer weights (1105). The image processing device (100) can obtain a meta-model based on integer operations. Accordingly, the amount of computation for iterative learning for the image processing device (100) to train the meta-model can be reduced.

[0181] However, as examined in operation S1120, if the difference between the range of the integer slope (1102) and the range of the integer weight (1104) is large, the distribution of the integer slope (1102) may be distorted in the process of scaling the range of the integer slope (1102) to match the range of the integer weight (1104). If the distribution of the integer slope (1102) is distorted, the distribution of the integer weight (1104) may also be distorted. In addition, since the integer slope (1102) is quantized from the real slope (1101) and has lower precision than the real slope (1101), the amount of precision decrease may increase with scaling, and the quantization error problem may increase.

[0182] Hereinafter, with reference to FIG. 12, a method of operating an image processing device (100) according to one embodiment to learn a meta model based on integer data and resolve data distribution distortion problems and quantization error problems will be described.

[0183] FIG. 12 is a diagram for explaining the quantization operation of an image processing device according to one embodiment.

[0184] Referring to FIG. 12, an image processing device (100) according to one embodiment can perform a quantization operation through a model adaptation unit (320-1).

[0185] In one embodiment, the memory (1200) of the image processing device (100) may store a real-valued weight (1203). For example, the real-valued weight (1203) may be real-valued data consisting of 32 bits. The range of the real-valued weight (1203) may be [-1, 1], and the precision may be 4e-10. However, the bits, range, and precision of the real-valued weight (1203) are not limited to the example of FIG. 12.

[0186] In operation S1210, the image processing device (100) can perform scaling on the real-valued gradient (1201) as a preprocessing for updating the real-valued gradient (1201) produced by the image processing device (100) to the real-valued weight (1203).

[0187] The real number gradient (1201) may be real number data consisting of 32 bits. The range of the real number gradient (1201) may be [-0.002, 0.002], and the precision may be 9e-13. However, the bits, range, and precision of the real number gradient (1201) are not limited to the example of FIG. 12.

[0188] The image processing device (100) can adjust the range of the real-valued slope (1201) so that the range of the real-valued slope (1201) is the same as the range of the real-valued weight (1203). Here, scaling refers to an operation of adjusting the range of data. For example, scaling may include a min-max normalization method that scales so that the minimum and maximum of the range of data are 0 and 1, a normalization method that scales so that all values ​​are expressed between -1 and 1, and a standardization method that standardizes so that it has a normal distribution with a mean of 0 and a variance of 1.

[0189] The image processing device (100) can scale the range of the real-valued gradient (1201) to obtain a scaled real-valued gradient (1202). For example, in FIG. 12, the scaling is illustrated as a normalization method that scales all values ​​to be expressed between -1 and 1, but is not limited thereto.

[0190] The scaled real-valued gradient (1202) may be real-valued data consisting of 32 bits. The range of the scaled real-valued gradient (1202) is [-1, 1], which is the same as the range of the real-valued weight (1203), and the precision may be 4e-10. However, the bits, range, and precision of the real-valued gradient (1201) are not limited to the example of FIG. 12.

[0191] In one embodiment, in the process of scaling the range of the real-valued gradient (1201) to match the range of the real-valued weight (1203), the precision of the scaled real-valued gradient (1202) is high, so that the data distribution distortion of the real-valued gradient can be minimized. For example, the precision of the scaled real-valued gradient (1202) can be higher than the precision of the scaled integer gradient (1103) that has undergone quantization and scaling in FIG. 11.

[0192] In operation S1220, the image processing device (100) can update the scaled real-valued gradient (1202) to the real-valued weight (1203) stored in the memory (1200). The image processing device (100) can obtain the updated real-valued weight (1204).

[0193] In operation S1230, the image processing device (100) can quantize the updated real-valued weight (1204) into an integer weight (1205).

[0194] The updated real-valued weight (1204) may be real-valued data consisting of 32 bits, but the bits of the updated real-valued weight (1204) are not limited thereto.

[0195] The integer weight (1205) may be integer data consisting of 8 bits, but the bits of the integer weight (1205) are not limited thereto.

[0196] In operation S1240, the image processing device (100) can perform a convolution operation of the forward propagation process for the next iteration (e.g., i+1) using the integer weight (1205). The image processing device (100) can train a meta model with training data corresponding to an input image using the integer weight (1205). The image processing device (100) can obtain a meta model based on integer operations. Accordingly, the amount of computation for iterative learning for the image processing device (100) to train the meta model can be reduced.

[0197] In one embodiment, since the memory of the image processing device (100) stores the real-valued weights (1203), the quantization process for the real-valued gradient (1201) can be omitted. In this case, there is no quantization error, and the precision of the real-valued gradient (1201) can be maintained relatively high. For example, it can be higher than the precision of the scaled integer gradient (1103) that has undergone quantization and scaling in FIG. 11.

[0198] Accordingly, when scaling the range of the real-valued gradient (1201) to correspond to the range of the real-valued weight (1203), the distribution distortion of the data of the real-valued gradient (1201) can be minimized. In one embodiment, the image processing device (100) can update the real-valued weight (1203) using the real-valued gradient (1201) with minimized data distribution distortion, and perform quantization on the updated real-valued weight (1204) to train the meta-model. Accordingly, the data distribution distortion and quantization error of the image processing device (100) can be minimized.

[0199] An image processing device (100) according to one embodiment can reduce quantization errors while training a neural network model based on integer operations. Accordingly, the image processing device (100) can reduce the amount of computation required to train a neural network model while preventing performance degradation of an AI model.

[0200] An image processing device (100) according to one embodiment can perform not only the inference of a neural network model based on integer operations through the above-described quantization process, but also the learning of a neural network model based on integer operations.

[0201] Fig. 13 is a flowchart illustrating an operating method of an image processing device according to one embodiment. The operating method of the image processing device (100) of Fig. 13 corresponds to the operation of the image processing device (100) according to one embodiment disclosed in Fig. 12.

[0202] In step 1310, an image processing device (100) according to one embodiment may obtain a meta model. Step 1310 may correspond to step 910 of FIG. 9.

[0203] In step 13320, the image processing device (100) according to one embodiment may calculate multiple gradients of the meta model using learning data corresponding to the input image (110). Step 1320 may correspond to step 920 of FIG. 9.

[0204] In one embodiment, the gradient (640) produced by the image processing device (100) may be real-valued data. The image processing device (100) may perform a quantization process based on the real-valued gradient produced in steps 1330 to 1360, and then train a meta-model using integer data. The image processing device (100) may acquire a meta-model based on integer operations.

[0205] In step 1330, in one embodiment, the image processing device (100) can remove at least one of the plurality of real-valued gradients by applying a gradient mask having gradient pruning information to the plurality of real-valued gradients. Step 1330 may correspond to step 930 of FIG. 9.

[0206] In step 1340, in one embodiment, the image processing device (100) may scale the plurality of real-valued gradients so that the range of the plurality of real-valued gradients corresponds to the range of the weights of the meta-model. The plurality of real-valued gradients may correspond to the real-valued gradient (1201) of FIG. 12.

[0207] The image processing device (100) can perform scaling on the real-valued gradient (1201) as a preprocessing for updating the real-valued gradient (1201) produced by the image processing device (100) to the real-valued weight (1203). The image processing device (100) can adjust the range of the real-valued gradient (1201) so that the range of the real-valued gradient (1201) is the same as the range of the real-valued weight (1203). Here, scaling means an operation of adjusting the range of data. The image processing device (100) can scale the range of the real-valued gradient (1201) to obtain a scaled real-valued gradient (1202).

[0208] In one embodiment, in the process of scaling the range of the real-valued gradient (1201) to match the range of the real-valued weight (1203), the precision of the scaled real-valued gradient (1202) is high, so that the data distribution distortion of the real-valued gradient can be minimized. For example, the precision of the scaled real-valued gradient (1202) can be higher than the precision of the scaled integer gradient (1103) that has undergone quantization and scaling in FIG. 11.

[0209] In one embodiment, the image processing device (100) may perform gradient pruning according to steps 930 and 950 of FIG. 9, then scale the gradient using the remaining gradient, update the weights, quantize the updated weights, and train a metamodel based on the quantized weights. In this case, the plurality of gradients may be pruned gradients.

[0210] Alternatively, in one embodiment, the image processing device (100) may scale the plurality of slopes calculated according to step 920 of FIG. 9, update the weights of the plurality of filters, quantize the updated weights, and train the meta-model based on the quantized weights.

[0211] In step 1350, in one embodiment, the image processing device (100) may update some of the real-valued weights based on the scaled multiple real-valued gradients. The real-valued weights may correspond to the real-valued weights (1203) of FIG. 12. The image processing device (100) may update the scaled real-valued gradients (1202) to the real-valued weights (1203) stored in the memory (1200). The image processing device (100) may obtain the updated real-valued weights (1204).

[0212] In step 1360, in one embodiment, the image processing device (100) may quantize some of the updated real-valued weights into integer weights. Some of the updated real-valued weights may be updated weights corresponding to the remaining gradients after performing gradient pruning. The real-valued weights may correspond to the real-valued weights (1204) of FIG. 12, and the integer weights may correspond to the integer weights (1205).

[0213] In step 1370, in one embodiment, the image processing device (100) can train the meta-model by performing a convolution operation based on integer weights. The integer weights may be quantized weights from weights updated in response to the remaining gradients after performing gradient pruning. For example, the image processing device (100) can perform a convolution operation of the forward propagation process for the next iteration (e.g., i+1) using the integer weights (1205). The image processing device (100) can obtain a meta-model based on integer operations. Accordingly, the amount of computational effort for iterative learning for the image processing device (100) to train the meta-model can be reduced.

[0214] The image processing device (100) can obtain an output image whose quality has been processed from an input image based on the learned meta model.

[0215] An image processing device (100) according to one embodiment of the present disclosure learns a meta model based on integer operations, and thus can reduce the amount of computation and resource usage required for learning the meta model.

[0216] According to one embodiment of the present disclosure, the image processing device (100) performs an update on real-valued weights based on real-valued gradients and then performs a quantization process, thereby reducing data distribution distortion and quantization errors. Accordingly, the image processing device (100) can reduce the amount of computation required to train the meta-model while preventing performance degradation of the meta-model.

[0217] Figure 14 is an internal block diagram of a server according to one embodiment.

[0218] Referring to FIG. 14, a server (1000) according to one embodiment may include a processor (1001) and a memory (1002).

[0219] The memory (1002) can store one or more instructions. The memory (1002) can store at least one program executed by the processor (1001). In one embodiment, the memory (1002) can include a database storing learning data for learning the meta model of the image processing device (100). In one embodiment, the memory (1002) can store at least one neural network model and / or predefined operation rules.

[0220] The memory (1002) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a magnetic memory, a magnetic disk, and an optical disk.

[0221] The server (1000) may include one or more processors (1001). The processor (1001) may control the overall operation of the server (1000). The processor (1001) may control the server (1000) to perform an operation of compressing a learning data set by executing one or more instructions stored in the memory (1002).

[0222] In one embodiment, the processor (1001) obtains a plurality of low-resolution learning images (1620, 2620) that have been degraded in a plurality of quality degradation types corresponding to the learning images (201, 1610) by executing one or more instructions. The processor (1001) trains the neural network model by applying the learning images (201, 1610) and the plurality of low-resolution learning images (1620, 2620). The processor (1001) applies the learning images (201, 1610) and the plurality of low-resolution learning images (1620, 2620) to the trained neural network model, thereby calculating a plurality of gradients of the neural network model. The processor (1001) generates a gradient mask (203, 650, 1860) for removing at least one gradient among the calculated plurality of gradients.

[0223] In one embodiment, the processor (1001) may obtain a plurality of modified neural networks (1630, 2630) corresponding to each of a plurality of low-resolution training images (1620, 2620) by executing one or more instructions. The processor (1001) may apply a test image (1650) to each of the plurality of modified neural networks (1630, 2630) to produce a test loss (1690, 2690) for each of the plurality of neural networks (1630, 2630). The processor (1001) may update the weights of the neural network model so that the sum of the test losses (1690, 2690) for each of the plurality of neural networks (1630, 2630) is minimized.

[0224] In one embodiment, the processor (1001) may obtain a first low-resolution learning image (1620) with image quality degraded to a first type and a second low-resolution learning image (2620) with image quality degraded to a second type, corresponding to the learning image (201, 1610), by executing one or more instructions. The processor (1001) may apply the learning image (201, 1610) and the first low-resolution learning image (1620) to a neural network model to obtain a first neural network (1670), and apply the learning image (201, 1610) and the second low-resolution learning image (2620) to the neural network model to obtain a second neural network (2670). The processor (1001) can apply a test image (1650) to a first neural network (1670) to produce a first test loss (1690), and apply a test image (1650) to a second neural network (1670) to produce a second test loss (2690). The processor (1001) can update the weights of the neural network model based on the first test loss (1690) and the second test loss (2690).

[0225] In one embodiment, the processor (1001) may generate a gradient mask (203, 650, 1860) having a first value for removing at least one of the plurality of gradients and a second value for retaining the remainder of the plurality of gradients by executing one or more instructions.

[0226] In one embodiment, the processor (1001) may generate a gradient mask (203, 650, 1860) by executing one or more instructions to digitize a plurality of gradients, thereby having a vector value of 0 for a gradient corresponding to a minimum number n of the values ​​for the plurality of gradients, and having a vector value of 1 for the remaining gradients.

[0227] In one embodiment, the processor (1001) may generate a gradient mask (203, 650, 1860) by executing one or more instructions to digitize a plurality of gradients for each gradient filter, such that a gradient having a numerical value lower than a threshold value has a vector value of 0, and a gradient having a vector value of 1 has a vector value higher than the threshold value.

[0228] In one embodiment, the processor (1001) can quantify a plurality of slopes by executing one or more instructions, by summing data belonging to a slope data matrix for each slope filter, for a plurality of slope filters corresponding to the plurality of slopes.

[0229] In one embodiment, for a neural network model including at least one convolutional layer, the size of the gradient mask (203, 650, 1860) for the first convolutional layer may be equal to the number of multiple gradients for updating each of the multiple filters belonging to the first convolutional layer.

[0230] In one embodiment, the size of the gradient mask (203, 650, 1860) for the first convolutional layer may be equal to the number of filters of the first convolutional layer.

[0231] FIG. 15 is a flowchart illustrating a method for a server to learn a meta model according to one embodiment. FIG. 16 is a diagram illustrating a process for a server to learn a meta model according to one embodiment.

[0232] Referring to FIGS. 15 and 16, in step 1510, a server (1000) according to one embodiment can set parameters of an initial neural network (1600). For example, the initial neural network (1600) may be a neural network prior to training with a large training data set. For example, the initial neural network (1600) may be a neural network prior to meta-learning. The initial neural network (1600) may be the initial meta-model (210) of FIG. 2. The server (1000) can set initial parameters of the initial neural network (1600) to be trained. The server (1000) can generate a meta-learned neural network by training the initial neural network (1600) having the initial parameters.

[0233] In step 1520, the server (1000) according to one embodiment may acquire a plurality of modified neural networks by training a neural network using training images for a plurality of kernels. The plurality of kernels may include image quality deterioration kernels. For example, the image quality deterioration kernels may include a kernel that processes compression degradation of the training images, a kernel that processes blurring degradation, a kernel that processes down sampling, a kernel that processes noise addition, etc. The server (1000) may apply each of the plurality of kernels to the training images to generate a plurality of low-resolution training images that have been image quality deteriorated by a plurality of image quality deterioration types. The server (1000) may train a neural network using the training images and each of the plurality of low-resolution training images. The server (1000) may acquire a plurality of modified neural networks by training a neural network using the training images and each of the plurality of low-resolution training images. Each of the plurality of modified neural networks may be a neural network having a high-resolution performance for each of the plurality of kernels.

[0234] For example, referring to FIG. 16, the server (1000) can train a neural network (1630) using a training image (1610) for a first kernel (k1) (1615), and can obtain a first modified neural network (1670) having high-resolution performance for the first kernel (1615) (step 1521). The server (1000) can apply the first kernel (k1) (1615) to the training image (1610) to generate a first low-resolution training image (1620). The server (1000) can generate a first low-resolution training image (1620) with degraded image quality by performing at least one of compression degradation, blurring degradation, down sampling, and noise addition on the training image (1610). The server (1000) can train the neural network (1630) by applying the training image (1610) and the first low-resolution training image (1620) to the neural network (1630). The server (1000) can calculate the first loss (Loss_k1) (1645), which is the difference between the output image (1640) and the training image (1610) by inputting the first low-resolution training image (1620) into the neural network (1630). The server (1000) can update the parameters of the neural network (1630) until the first loss (1645) is minimized. The server (1000) can obtain the first modified neural network (1670) having the updated parameters.

[0235] Additionally, for example, the server (1000) can apply a second kernel (k2) (2615) to the training image (1610) to generate a second low-resolution training image (2620) that has been image quality-degraded. The server (1000) can train a neural network (2630) using the training image (1610) and the second low-resolution training image (2620). The server (1000) can obtain a second modified neural network (2670) having high-resolution performance for the second kernel (k2) (2615). The second modified neural network (2670) can be a neural network having updated parameters based on the second kernel (k2) (2615) (step 1522).

[0236] In one embodiment, the first kernel (1615) and the second kernel (2615) may be different types of image degradation kernels. The parameters of the first modified neural network (1670) and the second modified neural network (2670) may be different.

[0237] In the present disclosure, each operation of training a neural network (1630) using a training image (1610) for a first kernel (1615) and training a neural network (2630) using a training image (1610) for a second kernel (2615) may be referred to as an inner loop.

[0238] In step 1530, the server (1000) according to one embodiment may evaluate the performance of a plurality of modified neural networks using test images for a plurality of kernels. The server (1000) may generate a plurality of losses from the plurality of modified neural networks. The losses generated from the plurality of modified neural networks may be referred to as test losses. The server (1000) may apply each of the plurality of kernels to the test images to generate a plurality of low-resolution test images that have been degraded using a plurality of image degradation types. The server (1000) may train each of the plurality of modified neural networks using the test images and each of the plurality of low-resolution test images. The server (1000) may evaluate the performance of the plurality of modified neural networks using the test images and each of the plurality of low-resolution test images. As a result of the performance evaluation, the server (1000) may generate a test loss for each of the plurality of modified neural networks.

[0239] For example, referring to FIG. 16, the server (1000) can evaluate the performance of the first modified neural network (1670) using the test image (1650) for the first kernel (k1) (1615) (step 1531). The server (1000) can apply the first kernel (k1) (1615) to the test image (1650) to generate a first low-resolution test image (1660). The server (1000) can generate the first low-resolution test image (1660) with degraded image quality by performing at least one of compression degradation, blurring degradation, down-sampling, and noise addition on the test image (1650). The server (1000) can evaluate the performance of the first modified neural network (1670) by applying the test image (1650) and the first low-resolution test image (1660) to the first modified neural network (1670). The server (1000) can calculate the first test loss (Test Loss_k1) (1690), which is the difference between the output image (1680) and the test image (1650) by inputting the first low-resolution test image (1660) to the first modified neural network (1670). The first test loss (1690) can be performance evaluation data of the first modified neural network (1670).

[0240] Additionally, for example, the server (1000) can evaluate the performance of the second transformed neural network (2670) using the test image (1650) for the second kernel (k2) (2615) (step 1532). The server (1000) can apply the second kernel (k2) (2615) to the test image (1650) to generate a second low-resolution test image (2660). The server (1000) can apply the test image (1650) and the second low-resolution test image (2660) to the second transformed neural network (2670) to evaluate the performance of the second transformed neural network (2670). The server (1000) can calculate a second test loss (Test Loss_k2) (2690), which is the difference between the output image (2680) and the test image (1650), by inputting the second low-resolution test image (2660) into the second transformed neural network (2670). The second test loss (2690) may be performance evaluation data of the second transformed neural network (2670).

[0241] In one embodiment, the first test loss (1690) of the first modified neural network (1670) and the second test loss (2670) of the second modified neural network (2670) may be different.

[0242] In step 1540, the server (1000) according to one embodiment may sum test losses. The server (1000) may sum multiple test losses generated from multiple transformed neural networks. For example, the server (1000) may sum a first test loss (1690) generated from a first transformed neural network (1670) and a second test loss (2690) generated from a second transformed neural network (2670).

[0243] In step 1550, the server (1000) according to one embodiment may generate a trained neural network by updating the parameters of the initial neural network (1600) based on a plurality of test losses. The server (1000) may update the parameters of the initial neural network (1600) until the sum of the plurality of test losses is minimized. The server (1000) may obtain a neural network having updated parameters from the initial neural network (1600). The server (1000) may obtain a trained neural network, such as a meta-learned neural network, from the initial neural network (1600). In one embodiment, the meta-learned neural network may correspond to the meta-model (220) of FIG. 2. The meta-model (220) is a result of reflecting the results learned by a plurality of modified neural networks into the initial neural network (1600). In the present disclosure, this may be referred to as an outer loop.

[0244] In one embodiment, the meta-learned neural network may outperform the initial neural network (1600) in generating high-resolution images. Furthermore, the speed at which the meta-learned neural network adapts to actual input images in the image processing device (100) may be faster than that of the initial neural network (1600).

[0245] Figure 17 is a flowchart illustrating a method for a server to obtain a gradient mask according to one embodiment. Figure 18a is a diagram illustrating a process for a server to calculate a gradient using a trained neural network according to one embodiment. Figure 18b is a diagram illustrating a process for a server to obtain a gradient mask according to one embodiment.

[0246] Referring to FIG. 17, in step 1710, the server (1000) can apply a learning image and at least one low-resolution learning image to the learned meta model to produce multiple gradients of the learned meta model.

[0247] Referring to FIG. 18a, the learned neural network model (1830) may correspond to the neural network learned from the initial neural network (1600) in FIGS. 15 and 16. The learned neural network model (1830) may correspond to the meta model (220) of FIG. 2. The server (1000) may simulate a general meta learning process to derive a gradient (1840) from the learned neural network model (1830). For example, the server (1000) may derive a gradient (1840) of the neural network model (1830) by applying a learning image (1810) and a plurality of low-resolution learning images to the neural network model (1830). The learning image (1810) corresponds to the learning image (1610) of FIG. 16, the plurality of low-resolution learning images correspond to the plurality of low-resolution learning images (1620, 2620) of FIG. 16, and the plurality of kernels may correspond to the plurality of kernels (1615, 2615) of FIG. 16. The output image (1820) may correspond to the output image (1640) of FIG. 16.

[0248] The server (1000) uses the already trained neural network to calculate the gradient, which is an intermediate output of the backpropagation process, and generates a gradient mask, so the weights of the trained neural network model (1830) are not updated. The server (1000) can obtain the gradient (1840) and the gradient mask (1860) with only a few iterations of training on the trained neural network model (1830).

[0249] Since the learned neural network model (1830) includes a plurality of convolutional layers, the server (1000) can produce a plurality of gradients (1840) corresponding to the number of filters belonging to each of the plurality of convolutional layers. For example, the first gradient (1841) is the gradient of the loss function with respect to the weight of the first filter and may represent information for updating the weight of the first filter. The second gradient (1842), the third gradient (1843), the fourth gradient (1844), and the fifth gradient (1845) may each correspond to the second filter, the third filter, the fourth filter, and the fifth filter, respectively. The plurality of gradients (1840) may also be referred to as a plurality of gradient filters or a plurality of gradient maps.

[0250] In step 1720, the server (1000) can identify a slope among multiple slopes that contributes less to the output of the meta model.

[0251] In one embodiment, the degree of contribution to the metamodel's output may increase as the gradient value increases. For example, a gradient with a large gradient value indicates a greater degree of weight update, allowing the metamodel's weights to be optimized more quickly. The degree of contribution to the metamodel's output may indicate the degree of improvement in the metamodel's high-resolution performance.

[0252] In one embodiment, the server (1000) may quantify a plurality of gradients (1840) by gradient filters to select gradients that significantly contribute to the output of the meta model. In one embodiment, the server (1000) may identify n gradients having the minimum values ​​among the quantified plurality of gradients (1840) (where n is a natural number). The server (1000) may generate a gradient mask (1860) by setting a vector value for removing the n gradients having the minimum values. Alternatively, in one embodiment, the server (1000) may identify gradients that are lower than a threshold value among the quantified plurality of gradients (1840). The server (1000) may generate a gradient mask (1860) by setting a vector value for removing the gradients that are lower than the threshold value.

[0253] For example, referring to FIG. 18B, the server (1000) can digitize a plurality of gradients (1840) for each gradient filter. The plurality of gradients (1840) can have gradient matrix data of k rows and k columns for each gradient filter. The server (1000) can have a digitized gradient by summing data belonging to the gradient matrix data for each gradient filter through a normalization process. The digitized gradient can be represented as a matrix (1850). For example, a value obtained by summing 5X5 gradient data corresponding to a first gradient (1841) can be located in the first row (1851) of the matrix (1850). The second slope (1842) may correspond to the second row (1852), the third slope (1843) may correspond to the third row (1853), the fourth slope (1844) may correspond to the fourth row (1854), and the fifth slope (1845) may correspond to the fifth row (1855). The number of rows of the matrix (1850) may be 5, which is the same as the number of slopes. Meanwhile, in one embodiment, the method of quantifying multiple slopes (1840) is not limited to the above-described example.

[0254] For example, the server (1000) can identify that in the matrix (1850) corresponding to the numerical slope, the second row (1852), the fourth row (1854), and the fifth row (1855) correspond to a minimum of three numbers. Or, for example, the server (1000) can identify that in the matrix (1850) corresponding to the numerical slope, the second row (1852), the fourth row (1854), and the fifth row (1855) are less than or equal to a threshold value (e.g., 7).

[0255] At step 1730, the server (1000) can learn a gradient mask by setting vector values ​​to remove the identified gradient.

[0256] In one embodiment, the gradient mask (1860) may have information for maintaining gradients that contribute significantly to the output of the meta model. For example, the gradient mask (1860) may have information for maintaining gradients with large gradient values ​​and removing gradients with small gradient values. The server (1000) may generate a gradient mask (1860) having a vector value of 0 for removing at least one of the plurality of gradients (1840) and a vector value of 1 for maintaining the remaining gradients. A gradient corresponding to a vector value of 0 may have a small contribution to the output of the neural network. A gradient corresponding to a vector value of 1 may have a large contribution to the output of the neural network.

[0257] For example, the server (1000) can set the gradient mask (1860) to have a vector value of 0 at positions corresponding to the second row (1852), the fourth row (1854), and the fifth row (1855) in the matrix (1850) corresponding to the numerical gradient. The server (1000) can set the gradient mask (1860) to have a vector value of 1 at positions corresponding to the first row (1851) and the third row (1853) in the matrix (1850) corresponding to the numerical gradient.

[0258] In one embodiment, the gradient mask (1860) may be used in the meta-model learning process of the image processing device (100).

[0259] Meanwhile, in the above-described embodiment, the gradient mask is exemplified as being obtained through a channel-wise method, but is not limited thereto. For example, the gradient mask may be obtained through a layer-wise method. For example, a gradient mask for removing gradients based on the convolutional layers of a neural network may be generated through a layer-wise method. For example, the gradient mask may include information for updating some of the convolutional layers and not updating the remaining parts.

[0260] FIG. 19 is a flowchart illustrating a method by which a server learns a meta model according to one embodiment.

[0261] In step 1910, the server (1000) according to one embodiment can obtain a plurality of low-resolution learning images with degraded image quality in a plurality of image quality degradation types corresponding to the learning images.

[0262] In one embodiment, the plurality of low-resolution training images may be characterized as having image quality deterioration by performing at least one of compression deterioration, blurring deterioration, down sampling, and noise addition on the training images.

[0263] In step 1920, the server (1000) according to one embodiment can apply a learning image and a plurality of low-resolution learning images to a meta model to obtain a plurality of transformed neural networks corresponding to each of the plurality of low-resolution learning images.

[0264] According to one embodiment, a server (1000) can obtain a first low-resolution learning image with image quality degraded to a first type and a second low-resolution learning image with image quality degraded to a second type in response to a learning image.

[0265] According to one embodiment, a server (1000) can obtain a first transformed neural network by applying training images and a first low-resolution training image to a meta model. The server (1000) can obtain a second transformed neural network by applying training images and a second low-resolution training image to a meta model.

[0266] In step 1930, the server (1000) according to one embodiment can train the meta model by updating the weights of the meta model based on the test loss for each of the plurality of transformed neural networks.

[0267] According to one embodiment, the server (1000) may apply a test image to each of a plurality of transformed neural networks, thereby calculating a test loss for each of the plurality of transformed neural networks. For example, a test image may be applied to a first transformed neural network, thereby calculating a first test loss for the first transformed neural network. For example, a test image may be applied to a second transformed neural network, thereby calculating a second test loss for the second transformed neural network.

[0268] According to one embodiment, the server (1000) may update the weights of the meta-model such that the sum of the test losses for each of the multiple transformed neural networks is minimized. For example, the server (1000) may update the weights of the meta-model based on the first test loss and the second test loss.

[0269] In step 1940, the server (1000) according to one embodiment may apply a learning image and at least one low-resolution learning image to the learned metamodel, thereby calculating multiple gradients of the metamodel. Step 1940 may correspond to step 1710 of FIG. 17.

[0270] In step 1950, the server (1000) according to one embodiment may generate a gradient mask for removing at least one gradient among the generated plurality of gradients.

[0271] According to one embodiment, the server (1000) can identify a slope among multiple filters that contributes less to the output of the meta model.

[0272] A server (1000) according to one embodiment can generate a gradient mask having a first value for removing at least one of a plurality of gradients and a second value for maintaining the remaining of the plurality of gradients.

[0273] According to one embodiment, a gradient mask may have a vector value for maintaining a gradient that contributes significantly to the output of the metamodel. The larger the gradient value, the greater the contribution to the output of the metamodel.

[0274] In one embodiment, the server (1000) can provide a meta model that can be quickly learned based on actual data in the image processing device (100) by pre-training the meta model for various image quality degradation scenarios. In addition, the server (1000) can quickly learn the meta model by preserving the gradient for updating the weights of the filters that contribute to improving the high-resolution performance of the meta model and repeatedly updating only the weights of the filters corresponding to the preserved gradients.

[0275] Fig. 20 is a block diagram of an image processing device according to one embodiment.

[0276] Referring to FIG. 20, an image processing device (100) according to one embodiment may include a processor (101) and a memory (102).

[0277] The image processing device (100) may be fixed or mobile, and may be a digital TV capable of receiving digital broadcasting, but is not limited thereto. The image processing device (100) may include at least one of a desktop, a smartphone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a laptop personal computer (PC), a netbook computer, a digital camera, a PDA (Personal Digital Assistants), a PMP (Portable Multimedia Player), a camcorder, a navigation device, a wearable device, a smart watch, a home network system, a security system, and a medical device.

[0278] The image processing device (100) may be implemented as a flat display device, a curved display device having a screen with a curvature, or a flexible display device whose curvature can be adjusted. The output resolution of the image processing device (100) may have various resolutions, such as, for example, HD (High Definition), Full HD, Ultra HD, or a resolution clearer than Ultra HD.

[0279] The memory (102) can store one or more instructions. The memory (102) can store at least one program executed by the processor (101). The memory (102) can store at least one neural network and / or predefined operation rules or AI models. In addition, the memory (102) can store data input to or output from the image processing device (100).

[0280] The memory (102) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a magnetic memory, a magnetic disk, and an optical disk.

[0281] In one embodiment, the memory (102) may store a gradient mask. The gradient mask may represent information for removing or retaining gradients depending on the degree to which they contribute to the output of the learned metamodel.

[0282] In one embodiment, the memory (102) can store real-valued weights (1203).

[0283] The processor (101) may include one or more processors. The processor (101) may control the overall operation of the image processing device (100). The processor (101) may control the image processing device (100) to function by executing one or more instructions stored in the memory (102).

[0284] In one embodiment, the processor (101) obtains a neural network model for image quality processing of an input image (110) by executing one or more instructions. The processor (101) applies learning data corresponding to the input image (110) to the neural network model, thereby calculating a plurality of gradients obtained by partially differentiating weights of the neural network model with respect to a loss of the neural network model. The processor (101) removes at least one gradient among the plurality of gradients by applying a gradient mask (203, 650, 1860) having gradient pruning information to the plurality of gradients. The processor (101) trains the neural network model by updating the weights of the neural network model based on the remaining gradients among the plurality of gradients. The processor (101) obtains an output image (120) whose image quality has been processed from the input image (110) based on the trained neural network model.

[0285] In one embodiment, the slope mask (203, 650, 1860) may have information for removing at least one slope among the plurality of slopes and information for maintaining the remaining slopes excluding the at least one slope.

[0286] In one embodiment, the processor (101) may calculate a plurality of gradients for updating weights of a plurality of filters, each corresponding to a plurality of filters belonging to at least one convolution layer of the neural network model, by executing one or more instructions. The processor (101) may remove at least one gradient corresponding to at least one filter among the plurality of filters based on the gradient mask (203, 650, 1860). The processor (101) may update weights of the remaining filters based on the remaining gradients corresponding to the remaining filters among the plurality of filters.

[0287] In one embodiment, the slope mask (203, 650, 1860) may include information for removing slopes corresponding to a minimum number n by digitizing the plurality of slopes.

[0288] In one embodiment, the slope mask (203, 650, 1860) may include information for removing slopes having a numerical value below a threshold value by digitizing the plurality of slopes.

[0289] In one embodiment, for the neural network model including at least one convolution layer, the size of the gradient mask (203, 650, 1860) for the first convolution layer may be equal to the number of the plurality of gradients for updating each of the plurality of filters belonging to the first convolution layer. In one embodiment, the processor (101) executes one or more instructions,

[0290] In one embodiment, the processor (101) may adjust the range of the plurality of slopes corresponding to real-valued data by executing one or more instructions so that the range of the plurality of slopes corresponds to the range of the weights of the neural network model. The processor (101) may update the plurality of slopes having the adjusted ranges in the real-valued weights (1203). The processor (101) may convert the real-valued weights (1204) into integer weights (1205) by quantizing the updated real-valued weights (1204). The processor (101) may perform a convolution operation of the neural network model based on the converted integer weights (1205).

[0291] In one embodiment, the processor (101) may remove at least one of the plurality of real-valued gradients by applying the gradient mask to the plurality of real-valued gradients by executing one or more instructions. The processor (101) may update some of the real-valued weights based on the remaining real-valued gradients among the plurality of real-valued gradients. The processor (101) may convert some of the updated real-valued weights into integer weights.

[0292] In one embodiment, the processor (101) may obtain a quality value of the input image (110) by executing one or more instructions. Based on the quality value of the input image (110), the processor (101) may obtain a low-resolution image by performing at least one of compression degradation, blurring degradation, resolution adjustment, or noise addition on the input image (110).

[0293] Fig. 21 is a flowchart showing an operation method of an image processing device and a server according to one embodiment.

[0294] Referring to FIG. 21, in operation 2110, a server (1000) according to an embodiment may learn a meta model. The server (1000) may obtain a plurality of low-resolution learning images that have been degraded in a plurality of quality degradation types in response to a learning image. The server (1000) may apply the learning image and the plurality of low-resolution learning images to a meta model, thereby obtaining a plurality of modified neural networks corresponding to each of the plurality of low-resolution learning images. The server (1000) may learn the meta model by updating the weights of the meta model based on the test loss for each of the plurality of modified neural networks. This has been described with reference to FIGS. 15 and 16.

[0295] In operation 2120, a server (1000) according to one embodiment may apply a learning image to a meta model to derive multiple gradients of the meta model. The server (1000) may generate a gradient mask to remove some of the multiple gradients.

[0296] In operation 2130, the server (1000) according to one embodiment may provide a metamodel and a gradient mask to the image processing device (100). A metamodel learned with a specific training image may be paired with a gradient mask corresponding to the metamodel. The server (1000) may provide a plurality of metamodels and a plurality of gradient masks corresponding thereto. The server (1000) may communicate with the image processing device (100) through a communication unit or provide the metamodel and the gradient mask to the image processing device (100) through an offline upload process. In operation 2140, the image processing device (100) according to one embodiment may download the metamodel and the gradient mask from the server (1000). The image processing device (100) may communicate with the server (1000) through a communication unit or download the metamodel and the gradient mask through an offline upload process.

[0297] In one embodiment, the image processing device (100) that has acquired the meta model and the gradient mask may operate according to operation 910 of FIG. 9. Alternatively, in one embodiment, the image processing device (100) that has acquired the meta model and the gradient mask may operate according to operation 1310 of FIG. 13.

[0298] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0299] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

Claims

1. In the image processing device (100), A memory (102) storing one or more instructions; and At least one processor (101) for executing one or more instructions stored in the memory (102), The above at least one processor (101) executes the one or more instructions, Obtain a neural network model for processing the image quality of the input image (110), By applying learning data corresponding to the input image (110) to the neural network model, multiple gradients are calculated by differentiating the weights of the neural network model with respect to the loss of the neural network model. By applying a gradient mask (203, 650, 1860) having gradient pruning information to the plurality of gradients, at least one gradient among the plurality of gradients is removed, By updating the weights of the neural network model based on one or more remaining slopes among the plurality of slopes, the neural network model is trained, An image processing device (100) that obtains an output image (120) whose quality has been processed from the input image (110) based on the learned neural network model.

2. In paragraph 1, The image processing device (100) wherein the above-mentioned slope mask (203, 650, 1860) has information for removing at least one slope among the plurality of slopes and information for maintaining one or more of the remaining slopes.

3. In paragraph 1 or 2, The above at least one processor (101) executes the one or more instructions, For each of a plurality of filters belonging to at least one convolution layer of the neural network model, a plurality of gradients are calculated for updating weights of the plurality of filters, Based on the above gradient mask (203, 650, 1860), at least one gradient corresponding to at least one filter among the plurality of filters is removed, An image processing device (100) that updates the weights of the remaining filters based on the remaining slopes corresponding to the remaining filters among the plurality of filters.

4. In any one of paragraphs 1 to 3, The above gradient mask (203, 650, 1860) is an image processing device (100) that includes information for removing gradients corresponding to a minimum number n by digitizing the plurality of gradients (n is a natural number).

5. In any one of paragraphs 1 to 4, The above gradient mask (203, 650, 1860) is an image processing device (100) that includes information for removing a gradient having a value lower than a threshold value among a plurality of values ​​obtained by digitizing the plurality of gradients.

6. In any one of paragraphs 1 to 5, An image processing device (100) wherein, for the neural network model including at least one convolution layer, the size of the gradient mask (203, 650, 1860) for the first convolution layer is equal to the number of the plurality of gradients for updating each of the plurality of filters belonging to the first convolution layer.

7. In any one of paragraphs 1 to 6, The above memory (102) stores a real number weight (1203), The above at least one processor (101) executes the one or more instructions, By adjusting the range of the plurality of slopes so that the range of the plurality of slopes corresponding to the real-valued data corresponds to the range of the weights of the neural network model, adjusted plurality of slopes are obtained (S1210). Update the adjusted multiple slopes to the above real number weights (1203) (S1220), By quantizing the updated real-valued weight (1204) above, the real-valued weight (1204) is converted into an integer-valued weight (1205) (S1230). An image processing device (100) that performs a convolution operation of the neural network model (S1240) based on the converted integer weight (1205).

8. In any one of paragraphs 1 to 7, The above at least one processor (101) executes the one or more instructions, By applying the gradient mask to the plurality of real number gradients, at least one of the plurality of real number gradients is removed (1330). Based on the remaining real-valued gradients among the above multiple real-valued gradients, some of the real-valued weights are updated (1350). An image processing device (100) that converts some of the updated real-valued weights into integer weights (1360).

9. In any one of paragraphs 1 to 8, The above at least one processor (101) executes the one or more instructions, Obtain the image quality value of the above input image (110), An image processing device (100) that obtains a low-resolution image by performing at least one of compression degradation, blurring degradation, resolution adjustment, or noise addition on the input image (110) based on the image quality value of the input image (110).

10. In the learning method of the neural network model, A step of obtaining a plurality of low-resolution learning images (1620, 2620) that have been image quality deteriorated by a plurality of image quality degradation types corresponding to learning images (201, 1610); A step of training the neural network model by applying the training image (201, 1610) and the plurality of low-resolution training images (1620, 2620) to the neural network model; A step of applying the learning image (201, 1610) and at least one low-resolution learning image (1620, 2620) to the learned neural network model to produce multiple gradients of the learned neural network model; and A method comprising the step of generating a gradient mask (203, 650, 1860) for removing at least one gradient among the generated plurality of gradients.

11. In Article 10, The step of training the above neural network model is: A step of obtaining a plurality of transformed neural networks (1630, 2630) corresponding to each of the plurality of low-resolution learning images (1620, 2620); A step of applying a test image (1650) to each of the plurality of transformed neural networks (1630, 2630) to produce a plurality of test losses (1690, 2690) for each of the plurality of transformed neural networks (1630, 2630); and A method comprising the step of updating the weights of the modified neural network model so that the total sum of the plurality of test losses (1690, 2690) for each of the plurality of neural networks (1630, 2630) is minimized.

12. In clause 10 or 11, The step of obtaining the above multiple low-resolution learning images (1620, 2620) is as follows: In response to the above learning image (201, 1610), a step of obtaining a first low-resolution learning image (1620) with image quality degraded to a first type and a second low-resolution learning image (2620) with image quality degraded to a second type is included. The step of training the above neural network model is: A step of obtaining a first neural network (1670) by applying the learning image (201, 1610) and the first low-resolution learning image (1620) to the neural network model, and obtaining a second neural network (2670) by applying the learning image (201, 1610) and the second low-resolution learning image (2620) to the neural network model; A step of applying a test image (1650) to the first neural network (1670) to produce a first test loss (1690), and applying the test image (1650) to the second neural network (2670) to produce a second test loss (2690); and A method comprising the step of updating weights of the neural network model based on the first test loss (1690) and the second test loss (2690).

13. In any one of paragraphs 10 to 12, The step of generating the above slope mask (203, 650, 1860) is: A method comprising the step of generating a gradient mask (203, 650, 1860) having a first value for removing at least one of the plurality of gradients and a second value for maintaining at least one remaining gradient of the plurality of gradients.

14. In any one of paragraphs 10 to 13, The step of generating the above slope mask (203, 650, 1860) is: A method comprising the step of generating the gradient mask (203, 650, 1860) by digitizing the plurality of gradients so as to have a vector value of 0 for a gradient corresponding to a minimum number n of the numerical values ​​for the plurality of gradients and a vector value of 1 for one or more remaining gradients (n is a natural number).

15. In clause 13 or 14, The above method, A method further comprising a step of digitizing the plurality of slopes by summing data belonging to a slope data matrix for each slope filter, for a plurality of slope filters corresponding to the plurality of slopes.

Images