Image processing apparatus, operating method thereof, and neural network training method

Abstract

The invention provides an image processing apparatus and a neural network training method. The image processing apparatus includes a memory to store one or more instructions, and at least one processor to execute the one or more instructions stored in the memory. The processor: obtains a neural network model for image quality processing of an input image by executing one or more instructions; calculating a plurality of gradients obtained by partial differentiation of a weight of the neural network model with respect to a loss of the neural network model by applying training data corresponding to the input image to the neural network model; removing at least one gradient from among the plurality of gradients by applying a gradient mask having gradient pruning information to the plurality of gradients; training the neural network model by updating a weight of the neural network model based on one or more remaining gradients among the plurality of gradients; and based on the trained neural network model, acquiring an output image of which the image is subjected to quality processing from the input image.

Description

Technical Field

[0001] This disclosure relates to an image processing apparatus, a method for using the image processing apparatus, and a method for training a neural network, and more specifically, to an image processing apparatus for performing image quality processing on low-resolution images and a method for training a neural network. Background Technology

[0002] As the size and resolution of image processing devices increase, super-resolution (SR) technology can be used to convert low-resolution (LR) images into high-resolution (HR) images.

[0003] With the development of deep learning technology, various types of learning-based SR (Simulation and Reference) technologies are being developed. Learning-based SR technologies can be used to learn or train artificial intelligence (AI) models in the cloud, and then use the learned AI models to perform inference. When the quality characteristics of the training images are similar to the quality characteristics of the input images to be processed, learning-based SR technologies can exhibit excellent performance. However, when the characteristics of the images to be processed differ from the assumed input image quality during training, it can lead to a domain gap problem, where the image quality is significantly degraded. The domain gap problem may arise due to the feature mismatch between the training images used to train the AI ​​model and the images to be processed.

[0004] One approach to addressing the domain gap involves upscaling techniques, which use on-device learning to train and adapt AI models based on the actual images to be processed. For example, an AI model can be connected to an edge device such as a television (TV) or mobile device, allowing it to train in real-time using input images as training data, and then upscaling those images using the trained AI model. On-device learning can refer to the process of collecting and computing information on the edge device itself to train an AI model mounted on the device in real-time without using cloud servers. An edge device can be any electronic device capable of processing data itself in real-time.

[0005] However, since edge devices may have limited computing resources and internal memory, it may be beneficial to reduce the amount of computation and resources used to train AI models and shorten the training time.

[0006] Because the edge devices to which AI models are connected may have limited computing resources, integer (INT) operations may be superior to floating-point (FP) operations. FP operations refer to arithmetic operations performed on FP data, which are real numbers including decimal places; INT operations refer to arithmetic operations performed on INT data, which are integers excluding decimal places. Because INT operations may not handle decimal precision, the computational cost for INT operations may be less than that for FP operations. Enabling edge devices to perform INT operations may involve quantization, converting FP data into INT data. Since quantization may involve approximating continuous values ​​(e.g., FP data) with discrete values ​​(e.g., INT data), quantization errors may occur, referring to errors that occur during the approximation process. When training AI models based on INT operations, quantization errors can lead to data loss or reduced accuracy, thus potentially resulting in performance degradation compared to AI models trained on FP operations. Therefore, it may be beneficial to reduce or minimize computational cost while preventing performance degradation during AI model training performed by edge devices. Summary of the Invention

[0007] Solution to the problem

[0008] According to one aspect of this disclosure, an image processing apparatus includes: at least one processor; and a memory configured to store one or more instructions, which, when executed by the at least one processor, cause the image processing apparatus to: obtain a neural network model for performing image quality processing on an input image; compute a plurality of gradients by applying training data corresponding to the input image to the neural network model and by performing partial differentials of the weights of the neural network model with respect to the loss of the neural network model; remove at least one gradient from the plurality of gradients by applying a gradient mask including gradient pruning information to the plurality of gradients; train the neural network model by updating the weights of the neural network model based on one or more remaining gradients among the plurality of gradients; and obtain a quality-processed output image based on the input image using the trained neural network model.

[0009] According to one aspect of this disclosure, an image processing apparatus includes the following method of operation: obtaining a neural network model for performing image quality processing on an input image; calculating multiple gradients by applying training data corresponding to the input image to the neural network model and performing partial differential calculations of the weights of the neural network model relative to the loss of the neural network model; removing at least one gradient from the multiple gradients by applying a gradient mask including gradient pruning information to the multiple gradients; training the neural network model by updating the weights of the neural network model based on one or more remaining gradients among the multiple gradients; and obtaining a quality-processed output image based on the input image using the trained neural network model.

[0010] According to one aspect of this disclosure, a method for training a neural network model includes: obtaining a plurality of low-resolution training images with image quality deteriorated to various types of image quality based on training images; training a neural network model by applying the training images and the plurality of low-resolution training images to the neural network model; computing a plurality of gradients of the trained neural network model by applying the training images and the plurality of low-resolution training images to the trained neural network model; and generating a gradient mask for removing at least one gradient from the plurality of gradients. Attached Figure Description

[0011] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become clearer from the following description taken in conjunction with the accompanying drawings, in which:

[0012] Figure 1 This is a view of an image processing apparatus used to explain the image whose output image quality has been processed according to embodiments of the present disclosure.

[0013] Figure 2 This is a block diagram illustrating the process of training a meta-model according to embodiments of the present disclosure.

[0014] Figure 3 This is a block diagram of the internal structure of an image processing apparatus according to an embodiment of the present disclosure.

[0015] Figure 4 This is a diagram used to explain the operation performed by an image processing apparatus according to an embodiment of the present disclosure to determine the image quality of an input image.

[0016] Figure 5 This is a diagram used to explain the operation performed by an image processing apparatus according to an embodiment of the present disclosure to adapt a meta-model corresponding to an input image. Figure 6 This is a view used to describe the convolution operations performed in the convolutional layers of a metamodel according to embodiments of the present disclosure.

[0017] Figure 7This is a view used to describe the relationship between filters and gradients in a convolutional layer of a metamodel according to embodiments of the present disclosure.

[0018] Figure 8 This is a diagram used to explain the results of gradient pruning according to embodiments of the present disclosure.

[0019] Figure 9 This is a flowchart of an operation method of an image processing apparatus according to an embodiment of the present disclosure.

[0020] Figure 10 This is a block diagram of the internal structure of an image processing apparatus according to an embodiment of the present disclosure.

[0021] Figure 11 This is a view used to explain the quantization operation of an image processing apparatus according to embodiments of the present disclosure.

[0022] Figure 12 This is a view used to explain the quantization operation of an image processing apparatus according to embodiments of the present disclosure.

[0023] Figure 13 This is a flowchart of an operation method of an image processing apparatus according to an embodiment of the present disclosure.

[0024] Figure 14 This is a block diagram of the internal structure of a server according to an embodiment of the present disclosure.

[0025] Figure 15 This is a flowchart of a method for training a meta-model executed by a server according to an embodiment of the present disclosure.

[0026] Figure 16 This is a diagram used to explain the process of training a meta-model by a server according to embodiments of the present disclosure.

[0027] Figure 17 This is a flowchart of a method for obtaining a gradient mask performed by a server according to an embodiment of the present disclosure.

[0028] Figure 18a This is a diagram used to explain the process of calculating gradients by a trained neural network, performed by a server, according to embodiments of the present disclosure.

[0029] Figure 18b This is a diagram used to explain the process of obtaining a gradient mask performed by a server according to embodiments of the present disclosure.

[0030] Figure 19 This is a flowchart of a method for training a meta-model executed by a server according to an embodiment of the present disclosure.

[0031] Figure 20This is a block diagram of an image processing apparatus according to an embodiment of the present disclosure.

[0032] Figure 21 This is a flowchart of the operation method of the image processing apparatus and server according to embodiments of the present disclosure. Detailed Implementation

[0033] Throughout the disclosure, the expression "at least one of a, b, or c" means only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0034] Embodiments of this disclosure have been described in detail herein with reference to the accompanying drawings, enabling those skilled in the art to readily implement this disclosure. However, this disclosure may be embodied in many different forms and should not be construed as limited to the examples set forth herein.

[0035] While general terms may be used to describe some embodiments of this disclosure in consideration of its functionality, these general terms may vary depending on the intent of those skilled in the art, precedent, the emergence of new technologies, etc. Therefore, these terms must be defined based on their meaning and the content of the disclosure as a whole, rather than simply stating the terms.

[0036] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of the invention.

[0037] Throughout the disclosure, when the first element is referred to as “connected” or “coupled” to the second element, the first element may be directly connected or coupled to the second element, or may be electrically connected or coupled to the second element using an intermediate element inserted therebetween.

[0038] In the context of describing this disclosure (particularly in the context of the following claims), the terms “a” and “an”, as well as “the” and similar designations, should be interpreted to cover both the singular and the plural. Furthermore, the steps of all methods described herein may be performed in any suitable order unless otherwise indicated or clearly contradicted by the context. Embodiments of this disclosure are not limited to the described order of operations.

[0039] Therefore, the phrase "according to an embodiment" used throughout the disclosure does not necessarily refer to the same embodiment.

[0040] Embodiments of this disclosure can be described based on functional block components and various processing steps. Some or all of these functional blocks can be implemented using various numbers of hardware and / or software configurations that perform a particular function. For example, the functional blocks of this disclosure can be implemented by one or more microprocessors, or by circuit configurations for a particular function. For example, the functional blocks of this disclosure can be implemented using various programming or scripting languages. Functional blocks can be implemented as algorithms that execute in one or more processors. The terms “mechanism,” “element,” “part,” and “configuration” are used extensively and are not limited to mechanical or physical embodiments.

[0041] Furthermore, the connecting lines or connectors between components shown in the various accompanying figures are intended to represent exemplary functional relationships and / or physical or logical couplings between components. In actual devices, connections between components can be represented by a variety of replaceable or addable functional connections, physical connections, or circuit connections.

[0042] The term "processor" can include various processing circuits and / or multiple processors. For example, as used herein, including the claims, the term "processor" can include various processing circuits, including at least one processor, wherein one or more of the at least one processor, individually and / or collectively in a distributed manner, can be configured to perform the various functions described herein. As used herein, when "processor," "at least one processor," and "one or more processors" are described as being configured to perform multiple functions, these terms cover, for example, cases where one processor performs some of the functions while another processor performs other functions, and cases where a single processor can perform all of the functions. Furthermore, at least one processor can include, for example, a combination of processors performing the various described / disclosed functions in a distributed manner. At least one processor can execute program instructions to implement or perform the various functions.

[0043] The terms “unit” and “module”, as well as terms ending with the suffixes “device (-er)” and “device (-or)”, can refer to an element that performs at least one function or operation and can be implemented as hardware, software, or a combination of hardware and software.

[0044] As used herein, the term "user" can refer to a person who uses the image processing device and can include consumers, evaluators, viewers, managers, or installation engineers. The term "manufacturer" as used herein can refer to a manufacturer that manufactures the image processing device and / or components included in the image processing device.

[0045] The term "image" as used here can refer to a still image, picture, frame, or moving image or video including multiple consecutive still images.

[0046] The term "neural network" as used here can refer to a representative example of an artificial neural network model that simulates nerves or neurons in the brain, and is not limited to artificial neural network models using a specific algorithm. Neural networks can also be called deep neural networks.

[0047] The "parameter" used here can be a value used in the computation process of each layer in a neural network; for example, it can be used when input values ​​are applied to a predetermined calculation formula. This parameter can also be a value set as a training result and can be updated as needed using individual training data.

[0048] Figure 1 This is a view of an image processing apparatus used to explain the output of an image whose image quality has been processed according to embodiments of the present disclosure.

[0049] Reference Figure 1 The image processing apparatus 100 can be an electronic device capable of processing images and outputting the processed images. According to embodiments of this disclosure, the image processing apparatus 100 can be implemented using various types of electronic devices, including a display. For example, the image processing apparatus 100 can be any type of electronic device, such as a television (TV), mobile phone, tablet PC, digital camera, camcorder, laptop computer, desktop computer, e-book terminal, digital broadcasting terminal, personal digital assistant (PDA), portable multimedia player (PMP), navigation device, MP3 player, or wearable device.

[0050] Image processing apparatus 100 can output video. The video may include multiple frames. The video may include projects such as television programs provided by content providers or various movies or dramas offered through video-on-demand (VOD) services. Content providers may be terrestrial or cable broadcasting stations that provide consumers with various content, including video, or set-top box (OTT) service providers or Internet Protocol television (IPTV) service providers. The video is captured, compressed, and sent to image processing apparatus 100, where it is reconstructed and output. For example, information may be lost due to limitations in the physical characteristics of the equipment used to capture the video and limited bandwidth, which may result in image distortion. The quality of the image may be degraded due to image distortion.

[0051] The image processing apparatus 100 according to embodiments of the present disclosure can process the image quality of an image. The image processing apparatus 100 can process the image quality of video content with image quality degraded due to image distortion. For example, the image processing apparatus 100 can obtain an output image 120 by performing image quality processing on an input image 110. For example, the image processing apparatus 100 can obtain a high-resolution (or high-quality) output image by upscaling a low-resolution (or low-quality) input image using an image quality processing model.

[0052] Image quality processing models according to embodiments of the present disclosure may include an inference network that implements a super-resolution (SR) algorithm capable of converting low-resolution images into high-resolution images. For example, the image quality processing model may include a neural network model trained using training data corresponding to the input image 110 from a neural network model obtained based on the quality of the input image 110.

[0053] The image processing apparatus 100 according to embodiments of the present disclosure can train itself in real time an image quality processing model included in the image processing apparatus 100. The image quality processing model according to embodiments of the present disclosure can be referred to as a meta-model. A meta-model can refer to a neural network model capable of quickly learning or generalizing new data. The meta-model can be pre-trained using learning data corresponding to various quality information stored in the cloud, and then the meta-model can be used to process the image quality of actual images in the image processing apparatus 100. The image processing apparatus 100 can reduce the domain gap problem by training the meta-model using actual images as training data. Because the meta-model can be a neural network pre-trained for various image quality degradation scenarios, the training process of adjusting the parameters of the meta-model based on the input image 110 can be simplified. For example, the number of iterations in the gradient descent method for adjusting the meta-model parameters can be reduced. The image processing apparatus 100 can generate an adaptive meta-model in the input image 110 through on-device learning. According to embodiments, the process performed by the image processing apparatus 100 to train the meta-model using the input image 110 can be referred to as meta-model adaptive learning.

[0054] To train a neural network model, training data consisting of a pair of images (e.g., a low-resolution image and a high-resolution image) can be used. The neural network model can be trained through a process in which its parameters are updated to reduce the difference (e.g., training error or loss) between the output image obtained by processing the low-resolution image included in the training data and the high-resolution image included in the training data. The parameters of the neural network model can be updated by performing multiple iterations. The process of training a neural network model can include forward propagation and backpropagation. For example, the backpropagation process of a neural network model can include calculating the difference between the predictions of the neural network model using the training data and the actual results of the neural network model, calculating the gradient based on this difference, and updating the parameters of the neural network. Therefore, the backpropagation process may require three times the computational cost of the forward propagation process.

[0055] According to embodiments of this disclosure, the image processing apparatus 100 can use gradient pruning to reduce the backpropagation process. The image processing apparatus 100 can compute the gradient of the meta-model by applying training data corresponding to the input image 110 to the meta-model. The image processing apparatus 100 can use gradient pruning to remove a portion of the gradient. The image processing apparatus 100 can use the remaining gradients that were not removed to update the parameters of the meta-model. The image processing apparatus 100 can use gradient pruning to quickly adapt the meta-model to the input image 110 and train the meta-model.

[0056] For example, the image processing apparatus 100 can calculate the number of gradient filters corresponding to the number of weight filters by applying the input image to a neural network model with a predetermined number of weight filters. The image processing apparatus 100 can retain gradient filters with large calculated gradient values ​​and remove gradient filters with small gradient values. Gradient filters can be used to update weight filters, and gradient filters with large gradient values ​​can update weights to a greater extent. The greater the degree to which the weights are updated in the direction that minimizes the loss, the greater the improvement in the high-resolution performance of the neural network model.

[0057] Because the image processing device 100 can selectively update the weights of the neural network by considering the contribution of each weight to improving high-resolution performance, the computational cost in backpropagation processing can be reduced. (Refer to...) Figures 3 to 9 An example of this situation is described in detail.

[0058] According to embodiments of this disclosure, the image processing apparatus 100 can train a meta-model based on integer (INT) operations during an adaptive learning process that uses the input image 110 to perform integer (INT) operations. The image processing apparatus 100 can perform arithmetic operations on the meta-model based on weights represented as INT data. For example, the image processing apparatus 100 can perform convolution operations using INT weights. Therefore, the computational load can be reduced compared to when the image processing apparatus 100 performs operations using floating-point (FP) weights. Furthermore, quantization errors occurring during the conversion of FP data to INT data can be minimized. (Refer to...) Figures 10 to 13 An example of this situation is described in detail.

[0059] The following reference Figure 2 An example of operation according to an embodiment is described, in which the meta-model is first trained based on training data in server 1000 and adaptively trained based on input image 110 in image processing device 100.

[0060] Figure 2 This is a block diagram illustrating the process of training a meta-model according to embodiments of the present disclosure.

[0061] Reference Figure 2 The process of training the meta-model can be performed by the server 1000 and the image processing device 100. The training process in the server 1000 can be represented as primary training, a training phase, or meta-learning. The training process in the image processing device 100 can be represented as secondary training, a testing phase, adaptive learning, or meta-adaptation.

[0062] A metamodel can be a model that has pre-learned the training process performed during the testing phase through previous model training processes, and can refer to a neural network model that can quickly learn or generalize new data. The technique or process of training such a metamodel using a training process can be called meta-learning.

[0063] The process or technique used to train a meta-model so that it can be quickly processed based on new data can be called meta-adaptation. For example, a meta-model can be a neural network model that adapts to new data. For example, a meta-model can be trained so that it can be quickly processed based on input image 110, and in the sense that the meta-model is trained to adapt to input image 110, the meta-model can be called adapted.

[0064] In the initial training phase, the trained meta-model 220 can be obtained based on the initial meta-model 210. In the secondary training phase, a meta-model 230 adapted to the input image 110 corresponding to the trained meta-model 220 can be obtained. According to embodiments of this disclosure, the initial training phase can be performed by the server 1000, and the secondary training phase can be performed by the image processing apparatus 100. However, this is merely an example, and the embodiments are not limited thereto. For example, in some embodiments, the initial and secondary training phases can be implemented using either the image processing apparatus or the server 1000, or by any other device or apparatus.

[0065] Server 1000 can obtain an initial meta-model 210. The initial meta-model 210 can have initialized weights. Server 1000 can train the initial meta-model 210 based on the training image 201 and multiple image quality degradation kernels 202 used to degrade the image quality of the training image 201 into various types of degraded image quality. For example, server 1000 can generate multiple images that degrade in a manner different from that of the training image 201.

[0066] Server 1000 can obtain multiple modified neural networks by applying each of multiple image quality degradation kernels 202 to the training image 201. Server 1000 can learn various image quality degradation scenarios through these modified neural networks and can perform repeated training of the meta-model to minimize error when handling various image quality degradation scenarios. Server 1000 can obtain a trained meta-model 220 by performing meta-learning using the training image 201. The trained meta-model 220 can have optimal weights capable of handling various image quality degradation scenarios. The trained meta-model 220 can be referred to as a meta-learning neural network.

[0067] Server 1000 can compute gradient mask 203 by inputting training image 201 into trained meta-model 220. Server 1000 can compute gradients from trained meta-model 220 by simulating a meta-learning process. Gradients can be represented as vectors representing partial differentiations with respect to the parameters of the neural network model. Server 1000 can generate gradient mask 203 for pruning gradients by identifying the degree of contribution of the computed gradients to the output of trained meta-model 220. Gradient mask 203 can represent information for removing or preserving gradients based on their contribution to the output of trained meta-model 220. For example, gradient mask 203 can have vector values ​​for removing gradients that contribute less to the output of trained meta-model 220, and vector values ​​for preserving gradients that contribute more to the output of trained meta-model 220. The contribution of weights to the output of meta-model 220 can increase with increasing gradients based on weights.

[0068] Image processing apparatus 100 can load a pre-trained meta-model from server 1000 and can load gradient mask 203. Image processing apparatus 100 can obtain meta-model 230 from server 1000. Image processing apparatus 100 can adjust the weights of the neural network by training the obtained meta-model 230 using training data corresponding to the actual input image. Image processing apparatus 100 can obtain input image 110 and can generate low-resolution image 250 by degrading input image 110 according to the degree of degradation of input image 110. Meta-model 230 can be adapted to input image 110 by repeatedly training it a predetermined number of times using training data including low-resolution image 250 and input image 110.

[0069] According to an embodiment, when the meta-model 230 can be trained using the input image 110, this can be referred to as a meta-model 230 adapted to the input image 110. For example, a meta-model 230 trained using training data corresponding to the input image can be referred to as an "adapted meta-model," "adaptive meta-model," or "transfer model." The image processing device 100 can prune gradients computed from the meta-model 230 using gradient mask 203. For example, the image processing device 100 can remove gradients that contribute less to the output of the meta-model 230 and retain gradients that contribute more to the output of the meta-model 220. Because the image processing device 100 can adjust the weights of the meta-model 230 based on the pruned gradients, the computational cost can be reduced.

[0070] The image processing device 100 can train the meta-model 230 based on integer data. For example, because the image processing device 100 uses integer weights to perform convolution operations, it can reduce computation and minimize quantization error.

[0071] The image processing device 100 can generate an output image 120 by using a meta-model 230 adapted to the input image 110 to improve the resolution of the input image 110.

[0072] Figure 3 This is a block diagram of the internal structure of an image processing apparatus 100 according to an embodiment of the present disclosure.

[0073] Reference Figure 3The image processing apparatus 100 according to embodiments of the present disclosure may include an image quality determiner 310, a model adaptation unit 320, and an image quality processor 330. The image quality determiner 310, model adaptation unit 320, gradient pruning unit 322, and image quality processor 330 may be implemented as at least one processor. The image quality determiner 310, model adaptation unit 320, gradient pruning unit 322, and image quality processor 330 may operate according to at least one instruction stored in a memory.

[0074] According to embodiments of this disclosure, the image quality determiner 310 can analyze or evaluate the image quality or quality of the input image 110. Image quality can indicate the degree of image degradation. After an image is acquired using the image processing apparatus 100, information may be lost or degraded when the image undergoes processes such as processing, compression, storage, transmission, and recovery. The image quality determiner 310 can evaluate or determine at least one of the following: compression degradation, degree of compression, degree of blurring, noise level, or resolution of the input image 110. According to embodiments of this disclosure, the image quality determiner 310 can provide or indicate the image quality of the input image 110 to the model adaptation unit 320.

[0075] According to embodiments of this disclosure, model adaptation unit 320 can generate training data corresponding to input image 110. The training data may include input image 110 corresponding to a high-resolution image (e.g., an answer image or label) and a low-resolution image corresponding to the image quality of input image 110. For example, model adaptation unit 320 can obtain a low-resolution image by degrading input image 110 to have a degree of degradation relative to the image quality of input image 110. For example, image quality determiner 310 can obtain a low-resolution image by giving the input image at least one of compressed image quality, blurred image quality, reduced resolution, and increased noise. The low-resolution image may be an image in which input image 110 has been further degraded to the same degree of degradation that occurs when input image 110 undergoes processing, compression, storage, transmission, and recovery such as by image processing device 100.

[0076] According to embodiments of this disclosure, the model adaptation unit 320 can obtain a meta-model. For example, the model adaptation unit 320 can obtain a meta-model corresponding to the image quality of the input image 110. However, the embodiments are not limited thereto. In the embodiments, the meta-model can be... Figure 2 The meta-model 230 corresponds to this.

[0077] According to embodiments of the present disclosure, model adaptation unit 320 can train a meta-model using training data including input image 110 and a low-resolution image. According to embodiments of the present disclosure, model adaptation unit 320 can train meta-model 230 such that the meta-model can perform image quality processing based on input image 110.

[0078] According to embodiments of this disclosure, the model adaptation unit 320 can input a low-resolution image included in the training data into the meta-model 230 through a forward propagation process to obtain a result image (which may be referred to as a predicted image) output by the meta-model 230.

[0079] According to embodiments of this disclosure, model adaptation unit 320 can compare the resulting image with the input image 110 included in the training data, and can adjust the parameters of meta-model 230 to minimize the difference between the two images. These parameters may include weights. Model adaptation unit 320 can calculate a loss, which may be the difference between the output image 120 and the input image 110. Model adaptation unit 320 can calculate the gradient of meta-model 230 by partially differentiating the loss using the parameters of meta-model 230. The gradient may have values ​​used to adjust the parameters of the neural network in the direction of minimizing the loss (or learning error) of the neural network according to the gradient descent algorithm.

[0080] According to embodiments of this disclosure, the gradient can be represented as Equation 1.

[0081] [Equation 1]

[0082]

[0083] In Equation 1 above, g can represent the gradient of the meta-model, J can represent the loss of the meta-model, and w can represent the weights of the meta-model. The gradient can be the partial derivative of the weights of the meta-model with respect to the loss. For example, the direction of the gradient can determine the direction in which the weights are moved to minimize the loss, and the magnitude of the gradient can determine the degree of weight update.

[0084] According to embodiments of this disclosure, the model adaptation unit 320 can repeatedly update parameters until the loss is minimized or reduced below a threshold. The model adaptation unit 320 can optimize the parameters of the meta-model. According to embodiments of this disclosure, the parameters of the meta-model updated by the model adaptation unit 320 can be represented by Equation 2 below.

[0085] [Equation 2]

[0086]

[0087] In equation 2, wi+1 w can represent the update weights of the metamodel. i The current weights of the meta-model are indicated, and `lr` can represent the learning rate of the meta-model. The gradient of the metamodel can be represented (corresponding to Equation 1). The weights of the metamodel can be updated by reflecting the gradient of the current weights of the metamodel.

[0088] According to embodiments of this disclosure, meta-model 230 may include a convolutional neural network (CNN) comprising multiple convolutional layers. A CNN may use multiple convolutional layers to perform convolution operations. Each of the multiple convolutional layers in a CNN may include one or more filters, each filter may have weights. Such filters may be referred to as weighted filters.

[0089] According to embodiments of this disclosure, model adaptation unit 320 can update the weights of the metamodel with computed gradients. The gradients can be used to update the weights of filters used in each convolutional layer of the metamodel 230.

[0090] According to embodiments of this disclosure, the model adaptation unit 320 can compute a number of gradients corresponding to the number of filters used in each convolutional layer. For example, the gradients may include multiple gradients corresponding to the multiple filters used in each convolutional layer. Each of the multiple gradients can be used to update the corresponding weights of the multiple filters.

[0091] According to embodiments of this disclosure, model adaptation unit 320 can remove at least one gradient from multiple gradients by applying a gradient mask with gradient pruning information to multiple gradients using gradient pruning unit 322. At least one gradient among the multiple gradients can be removed using the gradient mask, and the remaining gradients can be retained without being removed. The gradient mask may include information for masking, removing, or pruning multiple gradients.

[0092] According to embodiments of this disclosure, the gradient mask may be a binary mask having binary values ​​indicating whether to remove gradients used to update the weights of the metamodel. For example, the gradient mask may have vector values ​​"0" corresponding to gradients to be removed and vector values ​​"1" corresponding to gradients to be preserved. The gradient mask may be stored in an external database or internal memory of the image processing apparatus 100.

[0093] According to embodiments of this disclosure, model adaptation unit 320 can train metamodel 230 by removing some gradients from multiple gradients based on a gradient mask and updating the parameters of metamodel 230 with the remaining gradients. According to embodiments of this disclosure, model adaptation unit 320 can omit the process of updating the parameters of metamodel 230 corresponding to the removed gradients. The parameters of metamodel 230 can be updated based on the remaining gradients not removed based on the gradient mask. In this disclosure, gradient pruning can omit or remove a portion of the backpropagation process; therefore, gradient pruning can also be referred to as backpropagation pruning.

[0094] The weights updated by the model adaptation unit 320 according to embodiments of this disclosure can be represented by the following Equation 3.

[0095] [Equation 3]

[0096]

[0097] In equation 3, w (l) The values ​​can represent the weights corresponding to each filter in the l-th convolutional layer, where lr can represent the learning rate, and m... (l) It can represent the gradient mask of the l-th convolutional layer, g (l) This can represent the gradient of the l-th convolutional layer. The gradient mask m (l) This can include information used to remove the gradients used to update each filter.

[0098] According to embodiments of this disclosure, the gradient mask m (l) The vector value can be multiplied by multiple gradients g. (l) And m (l) g (l) It can be represented based on gradient mask m (l) The remaining gradient after removal. The weights w of the filter in the l-th convolutional layer. (l) Can be based on or not based on gradient mask m (l) The remaining gradient m that was removed (l) g (l) Updated.

[0099] According to embodiments of this disclosure, multiple gradients g (l) It can be represented as , where k x k can represent the width of the gradient data (or the number of columns in the gradient matrix). y It can represent the height of the gradient data (or the number of rows in the gradient matrix), c out Multiple gradients g can be represented (l) The number of gradient masks (or the number of gradient filters). According to embodiments of this disclosure, the gradient mask m (l) The size (or the number of rows in the gradient mask matrix) can be expressed as C.out .

[0100] The filter used for the l-th convolutional layer can be represented as follows: , where c out The number of output channels (or feature map channels) can be represented. For convenience, the number of input channels is shown as one ("1"), but the embodiments are not limited thereto.

[0101] According to embodiments of this disclosure, the gradient mask m (l) Size C out It can be equal to the gradient g (l) Quantity C out .

[0102] According to embodiments of this disclosure, model adaptation unit 320 can remove at least one gradient used to update at least one of a plurality of filters using a gradient mask, and can retain the remaining gradients used to update the remaining filters. Model adaptation unit 320 can train a meta-model by updating the remaining filters based on the remaining gradients.

[0103] According to embodiments of this disclosure, a gradient mask may include information for maintaining gradients that contribute significantly to improving the high-resolution performance of the meta-model. For example, gradients with larger gradient values ​​may contribute significantly to improving high-resolution performance. For example, gradients with larger gradient values ​​may have a greater degree of weight update, thus allowing for faster optimization of the meta-model weights. For example, a gradient mask may have information for maintaining gradients with larger gradient values ​​and removing gradients with smaller gradient values. A gradient mask may include information for removing gradients corresponding to the n minimum values ​​among the values ​​obtained by quantizing multiple gradients (where n is a natural number). In some embodiments, for example, a gradient mask may include information for removing gradients corresponding to values ​​among the values ​​obtained by quantizing multiple gradients that are equal to or less than a threshold. For example, the values ​​of multiple gradients can be obtained by summing the data present in the gradient data matrix of each gradient filter. Multiple gradients may correspond to multiple gradient filters. Reference is made below. Figures 17 to 18b An example describing the detailed process of generating a gradient mask.

[0104] According to embodiments of this disclosure, the image processing apparatus 100 can use gradient pruning to train a meta-model at high speed. By performing training using gradient pruning, the image processing apparatus 100 can reduce the computational load and memory usage during backpropagation. The image processing apparatus 100 can use gradient pruning to obtain an adaptive meta-model of the input image 110.

[0105] According to embodiments of this disclosure, the image quality processor 330 can use a meta-model trained by the model adaptation unit 320. The image quality processor 330 can use an updated meta-model to process the image quality of the input image 110. The image quality processor 330 can obtain a high-resolution image by processing the image quality of the input image 110 using a meta-model suitable for the input image 110.

[0106] According to embodiments of this disclosure, the image processing apparatus 100 can train a neural network model based on integer operations. Compared to FP operations, the image processing apparatus 100 can reduce computational load and minimize quantization error by performing training based on INT operations. See below for further details. Figures 10 to 13 A detailed example of this situation.

[0107] Although the image quality determiner 310, model adaptation unit 320, gradient pruning unit 322, and image quality processor 330 are in Figure 3 While shown as individual elements or components, embodiments are not limited thereto. For example, in some embodiments, one or more of the image quality determiner 310, model adaptation unit 320, gradient pruning unit 322, and image quality processor 330 may be combined, for example, implemented by a single processor. In this case, one or more of the image quality determiner 310, model adaptation unit 320, gradient pruning unit 322, and image quality processor 330 may be implemented as a dedicated processor, or may be implemented using a combination of software and general-purpose processors such as application processors (APs), central processing units (CPUs), or graphics processing units (GPUs). A dedicated processor may include memory for implementing embodiments of this disclosure or a memory processing unit for using external memory. As another example, the image quality determiner 310, model adaptation unit 320, gradient pruning unit 322, and image quality processor 330 may be configured as multiple processors.

[0108] Figure 4 This is a diagram used to explain the operation performed by an image processing apparatus according to an embodiment of the present disclosure to determine the image quality of an input image. Figure 4 The operation can be used Figure 3 The image quality determiner 310 is used to perform this.

[0109] Reference Figure 4 According to an embodiment of the present disclosure, the image processing apparatus 100 can determine the image quality of the input image 110 and generate a low-resolution image 250 based on the input image 110.

[0110] According to embodiments of the present disclosure, the image processing apparatus 100 may include a kernel prediction network 410, which may be a neural network trained to evaluate the image quality of an input image 110. According to embodiments of the present disclosure, the image processing apparatus 100 may use the kernel prediction network 410 to evaluate the image quality of the input image 110. The kernel prediction network 410 may use a neural network trained to analyze or evaluate the image quality of the input image 110. For example, the neural network may be a neural network trained to evaluate image or video quality using image quality assessment (IQA) techniques and / or video quality assessment (VQA) techniques.

[0111] According to embodiments of this disclosure, kernel prediction network 410 can output a predicted kernel 420. For example, kernel prediction network 410 can analyze an image and determine the degree of degradation that occurs when the image undergoes processes such as processing, compression, storage, transmission, and restoration. For example, the predicted kernel 420 may have information about at least one of the compressed image quality, blurred image quality, resolution, or noise of the input image 110.

[0112] According to embodiments of this disclosure, the image processing apparatus 100 can obtain a low-resolution image 250 based on the image quality of the input image 110 obtained by analyzing the input image 110. For example, the image processing apparatus 100 can obtain the low-resolution image 250 by applying a predicted kernel 420 to the input image 110 using a kernel prediction network 410. For example, the image processing apparatus 100 can obtain the low-resolution image 250 by performing processing to add at least one of compressed image quality, blurred image quality, reduced resolution, or increased noise to the input image 110.

[0113] According to embodiments of this disclosure, the image processing apparatus 100 can generate training data including an input image 110 and a low-resolution image 250. Figure 4 In the example shown, because the image received from the image processing device 100 via the kernel prediction network 410 can be converted from Full High Definition (HD) (FHD) to Ultra High Definition (UHD), the image processing device 100 can determine that the resolution has been reduced by a factor of two. The image processing device 100 can obtain the input image 110 and generate a low-resolution image 250 by degrading the input image 110 according to the degree of degradation of the input image 110.

[0114] Figure 5 This is a diagram used to explain the operations performed by an image processing apparatus according to embodiments of the present disclosure to adapt a metamodel corresponding to an input image. It can be used... Figure 3 The model adaptation unit 320 is used to perform Figure 5 The operation.

[0115] Reference Figure 5 The image processing device 100 can obtain a neural network model 510. The neural network model 510 can correspond to... Figure 2 Metamodel 230.

[0116] According to an embodiment of the present disclosure, the image processing apparatus 100 can train a neural network model 510 based on training data including an input image 110 and a low-resolution image 250 having image quality corresponding to the input image 110.

[0117] The neural network model 510 according to embodiments of the present disclosure may be a convolutional neural network (CNN) with multiple convolutional layers. Each of the multiple convolutional layers may include one or more filters, each filter may have weight values. For example, the image processing apparatus 100 may output a resulting image 520 by performing a convolution operation between a low-resolution image 250 and filters used in the convolutional layers of the neural network model 510. For example, the convolutional layers may include at least one convolutional layer, such as a first convolutional layer 610, a second convolutional layer 620, and a third convolutional layer 630.

[0118] A low-resolution image 250 can be input into a first convolutional layer 610. The 5x5x5 marking on the first convolutional layer 610 indicates that five filters, each with a size of 5x5, are used to perform convolution on the low-resolution image 250 as an input image. For example, since five filters can be used in the first convolutional layer 610, five feature maps 615 can be generated as a result of the convolution using these five filters. The feature maps 615 output from the first convolutional layer 610 can be input into a second convolutional layer 620. The second convolutional layer 620 can correspond to the first convolutional layer. The feature maps output from the second convolutional layer 620 can be input into a third convolutional layer 630. The third convolutional layer 630 can use a 5x5 filter to generate a result image 520, which can be a single output data point.

[0119] According to embodiments of this disclosure, the neural network model 510 may further include activation layers. For example, the activation layers may include sigmoid functions, Tanh functions, rectified linear unit (ReLU) functions, etc., to provide nonlinear characteristics to each feature map.

[0120] The image processing device 100 can calculate a gradient 640 to minimize a loss 530, such as the difference between the resulting image 520 and the input image 110, via a backpropagation process. For example, the direction of gradient 640 can represent the direction of movement of filter weights used in the convolutional layer to minimize the loss 530, and the magnitude of gradient 640 can represent the degree of update of the filter weights. The image processing device 100 can update the parameters of the neural network model 510 in the direction of decreasing gradient of the calculated loss 530 using gradient descent, and can repeat the update until the loss 530 becomes minimum.

[0121] Gradients can include multiple gradients corresponding to the multiple filters used in each convolutional layer. For example, gradient 640 can include multiple gradients, such as... Figure 5 and Figure 7 The first gradient 641, second gradient 642, third gradient 643, fourth gradient 644, and fifth gradient 645 are shown. Multiple gradients (e.g., the first to fifth gradients 641, 642, 643, 644, and 645) can each correspond to one of the five filters used in the first convolutional layer 610. For example, the five filters can be multiple filters, such as... Figure 7 The first filter 731, the second filter 732, the third filter 733, the fourth filter 734, and the fifth filter 735 are shown.

[0122] Reference Figure 6 and Figure 7 The convolution operation process in the convolutional layer according to embodiments of the present disclosure is further described. Figure 6 This is a view used to describe the convolution operations performed in the convolutional layers of a metamodel according to embodiments of the present disclosure. Figure 7 It is a view used to describe the relationship between filters and gradients in a convolutional layer of a metamodel according to embodiments of the present disclosure.

[0123] A single feature map 750 can be generated using multiplication and addition between the parameters of a 5x5 filter 730 used in the first convolutional layer 610 and the sample values ​​in the input data 705 corresponding to these parameters. Since five filters can be used in the first convolutional layer 610, five feature maps can be generated by using the convolution operation process of these five filters.

[0124] like Figure 6 As shown, input data 705 may include samples I1 to I49 (also referred to as pixels) of input data 705, and filter 730 may have filter kernels including weights F1 to F25. Additionally, feature map 750 may include samples M1 to M9. For example, feature map 750 may correspond to feature map 615.

[0125] In the convolution operation, the sample values ​​of samples I1, I2, I3, I4, I5, I8, I9, I10, I11, I12, I15, I16, I17, I18, I19, I22, I23, I24, I25, I26, I29, I30, I31, I32, and I33 of the input data 705 can be multiplied by the weights F1 to F25 of the filter 730, and the combined (e.g., summed) value of the multiplication operation results can be assigned as the value of sample M1 of the feature map 750. The stride of the convolution operation can be one ("1"), but the embodiments are not limited to this.

[0126] When filter 730 moves according to the stride until it reaches the last sample of input data 705, a convolution operation can be performed between the sample values ​​in input data 705 and the samples of filter 730, thus obtaining a feature map 750 of a specific size. For example, a 5x5 feature map 750 can be used. The loss function can be obtained based on the difference between feature map 750 and the resulting low-resolution image 250.

[0127] According to an embodiment, the weight values ​​of the neural network model 510, such as the weights of the filters 730 used in the convolutional layers of the neural network model 510 (e.g., the weights F1 to F25 of the filters 730), can be optimized by training the neural network model 510.

[0128] The convolutional layers included in neural network model 510 can be referenced above. Figure 5 The convolution operation described above is used for processing, but the above references... Figure 5 The convolution operation described is merely an example. Implementations are not limited to this.

[0129] Reference Figure 7Multiple gradients (e.g., first gradient to fifth gradient 641, 642, 643, 644, and 645) used to update multiple filters (e.g., first to fifth filters 731, 732, 733, 734, and 735) can be computed in the first convolutional layer 610. For example, the multiple gradients (e.g., first gradient to fifth gradient 641, 642, 643, 644, and 645) can correspond to loss functions of the weights of the multiple filters (e.g., first to fifth filters 731, 732, 733, 734, and 735), respectively. For example, the first gradient 641 can be the gradient of the loss function with respect to the weights of the first filter 731. For example, the second gradient 642 can be the gradient of the loss function with respect to the weights of the second filter 732. For example, the third gradient 643 can be the gradient of the loss function with respect to the weights of the third filter 733. For example, the fourth gradient 644 can be the gradient of the loss function with respect to the weights of the fourth filter 734. For example, the fifth gradient 645 could be the gradient of the loss function with respect to the weights of the fifth filter 735.

[0130] According to embodiments of this disclosure, multiple gradients (e.g., first to fifth gradients 641, 642, 643, 644, and 645) can be represented as follows: Multiple filters (e.g., filters 731, 732, 733, 734, and 735, from the first to the fifth filter) can be represented as follows: The individual sizes of multiple gradients (e.g., first to fifth gradients 641, 642, 643, 644, and 645) can correspond to the sizes of multiple filters (e.g., first to fifth filters 731, 732, 733, 734, and 735). For example, each gradient or each filter can have a size. The number of gradients (e.g., first to fifth gradients 641, 642, 643, 644, and 645) can be equal to the number of filters (e.g., first to fifth filters 731, 732, 733, 734, and 735). For example, the number of gradients can be represented as c. out The number of filters can be expressed as c out For example, since five filters of size 5x5 can be used in the first convolutional layer 610, five gradients of size 5x5 can be computed for each gradient used to update the five filters.

[0131] The image processing apparatus 100 can apply a gradient mask 650 with gradient pruning information to the calculated gradient 640. The gradient mask 650 may include gradient pruning information for pruning multiple gradients. The gradient mask 650 in... Figure 5 and Figure 7The gradient mask 650 is shown as a column vector, but the embodiments are not limited thereto. The gradient mask 650 may have vector values ​​"0" corresponding to the gradients 642, 644, and 645 to be removed, and may have vector values ​​"1" corresponding to the gradients 641 and 643 to be retained. The gradients to be removed 642, 644, and 645 may be gradients that contribute less to improving the performance of the neural network model 510, while the gradients to be retained 641 and 643 may be gradients that contribute more to improving the performance of the neural network model 510.

[0132] The size (or number of rows) of the gradient mask 650 can be equal to the number of filters in the neural network model 510, and can also be equal to the number of gradients 640. For example, the number of gradient masks 650 can be represented as c. out For example, gradient mask 650 may include five vector values ​​regarding whether to remove multiple gradients (e.g., first gradient to fifth gradient 641, 642, 643, 644, and 645) corresponding to multiple filters (e.g., first filters to fifth filters 731, 732, 733, 734, and 735), respectively.

[0133] Based on gradient mask 650, image processing apparatus 100 can remove second gradient 642, fourth gradient 644, and fifth gradient 645 from multiple gradients (e.g., first to fifth gradients 641, 642, 643, 644, and 645), while retaining first gradient 641 and third gradient 643 from the multiple gradients (e.g., first to fifth gradients 641, 642, 643, 644, and 645). For example, second gradient 642, fourth gradient 644, and fifth gradient 645 from the multiple gradients (e.g., first to fifth gradients 641, 642, 642, 644, 644, and 645) can be removed, and first gradient 641 and third gradient 643 from the multiple gradients (e.g., first to fifth gradients 641, 642, 643, 644, and 645) can be retained. First gradient 641 and third gradient 643 can be included in pruned gradient 660.

[0134] The image processing apparatus 100 can partially update the weights of the neural network model 510 based on the pruned gradient 660. For example, the image processing apparatus 100 can update the first filter 731 and the third filter 733, but may not update the second filter 732, the fourth filter 734, and the fifth filter 735. Because the update process of the second filter 732, the fourth filter 734, and the fifth filter 735 can be omitted, the backpropagation process can be simplified.

[0135] The image processing apparatus 100 can obtain a first filter 831 updated based on a first gradient 641, and can obtain a third filter 833 updated based on a third gradient 643. The first filter 831 and the third filter 833 can be repeatedly updated iteratively, and can be updated, for example, until the weights of the first filter 831 and the weights of the third filter 833 are optimized.

[0136] Refer again Figure 5 The image processing apparatus 100 can retain gradients that contribute to improving the high-resolution performance of the neural network model 510, and can repeatedly update the weights of the filters corresponding to the retained gradients, thus training the neural network model 510. According to embodiments of this disclosure, the image processing apparatus 100 can use gradient pruning to quickly train the neural network model 510 adapted to the input image 110, and can reduce the computational cost during the backpropagation process of updating weights by using gradients, thereby reducing memory usage.

[0137] The convolutional layers of neural network model 510 are not limited to Figure 5 The diagram shows a first convolutional layer 610, a second convolutional layer 620, and a third convolutional layer 630. For example, a neural network model 510 may include L convolutional layers. Each of the first convolutional layer 610, the second convolutional layer 620, and the third convolutional layer 630 may be referred to as the l-th convolutional layer (where L and l are natural numbers). Multiple filters may refer to filters within the l-th convolutional layer.

[0138] In the above example, gradient descent is described as an algorithm for training neural network model 510, but the embodiments are not limited thereto. Image processing apparatus 100 can use any training algorithm to train neural network model 510. Image processing apparatus 100 may selectively apply, for example, hyperparameters (e.g., learning rate, batch size, and termination condition) and optimization algorithms (e.g., stochastic gradient descent (SGD), Adam, and Adampsec) depending on system constraints such as memory, processor, operators, and power.

[0139] Figure 8 This is a diagram used to explain the results of gradient pruning according to embodiments of the present disclosure.

[0140] Reference Figure 8 In 801, J can represent the loss function, A can represent the activation, and G can represent the gradient. The activation can be an intermediate product of calculating the gradient by operating according to the chain rule of backpropagation. The loss function, activation, and gradient can each be represented as a convolution tensor, and the convolution tensor can refer to a high-dimensional matrix that has been extended to apply the convolution operation.

[0141] In 802, J pruneA can represent the pruning loss function. prune It can represent pruning activation, G prune The gradient of the pruning can be represented. W can represent the weights updated based on the pruning gradient. Because the image processing device 100 can omit the gradient removed by the gradient mask, the processing of performing operations on a portion of the activations used to compute the gradient can be omitted. Therefore, the amount of storage required to store the activations as intermediate products can be reduced. Because the image processing device 100 can omit the gradient removed by the gradient mask, the processing of performing operations on a portion of the loss function used to compute the gradient can be omitted. Therefore, the amount of computation required to compute the loss function can be reduced.

[0142] Figure 9 This is a flowchart of an operation method of an image processing apparatus 100 according to an embodiment of the present disclosure.

[0143] In operation 910, the image processing apparatus 100 according to an embodiment of the present disclosure can obtain a meta-model. For example, the image processing apparatus 100 can obtain a meta-model corresponding to the image quality of the input image 110. However, the embodiments are not limited thereto. The meta-model can be related to... Figure 2 Metamodel 230 or Figure 5 The neural network model 510 corresponds to this.

[0144] In operation 920, the image processing apparatus 100 according to an embodiment of the present disclosure can use training data corresponding to the input image 110 to compute multiple gradients of the meta-model.

[0145] The image processing apparatus 100 according to embodiments of the present disclosure can train a meta-model using training data including an input image 110 and a low-resolution image degraded from the input image 110. The meta-model can perform convolution operations between the low-resolution image and filters used in the convolutional layers of the meta-model, thereby outputting a feature map of the resulting image. Each convolutional layer may have one or more filters, and each filter may have weights.

[0146] Image processing device 100 can obtain a result image output by meta-model 230 by inputting a low-resolution image included in the training data into meta-model 230. Image processing device 100 can calculate a loss by comparing the result image with the input image 110, which can be the difference between the result image and the input image 110. Image processing device 100 can calculate the gradient of the meta-model to minimize the loss. This gradient can be represented according to Equation 1 above. The gradient can include multiple gradients obtained by partial differentiation of the loss function with each weight of the neural network.

[0147] The image processing device 100 can compute multiple gradients corresponding to multiple filters within each convolutional layer. These gradients can be used to update the corresponding weights of the multiple filters used in each convolutional layer. The image processing device 100 can optimize the weights of the multiple filters used in the convolutional layers of the meta-model.

[0148] The image processing apparatus 100 can calculate the number of gradients corresponding to the number of filters used in the convolutional layer. The image processing apparatus 100 can calculate multiple gradients capable of updating multiple filters separately. According to embodiments of this disclosure, the number of filters in the l-th convolutional layer can be equal to the number of gradients in the l-th convolutional layer.

[0149] According to embodiments of this disclosure, the image processing apparatus 100 may perform gradient pruning in operation 930, and then update the corresponding weights of multiple filters based on the remaining gradients.

[0150] In operation 930, the image processing apparatus 100 according to an embodiment of the present disclosure can remove at least one gradient from a plurality of gradients by applying a gradient mask having gradient pruning information to a plurality of gradients. The image processing apparatus 100 according to an embodiment of the present disclosure can also retain the remaining gradients among the plurality of gradients without removing them.

[0151] According to embodiments of this disclosure, a gradient mask may include gradient pruning information for masking, removing, or pruning multiple gradients. For example, the gradient pruning information may include information about whether to remove or retain any one of the multiple filters within each convolutional layer.

[0152] The image processing apparatus 100 according to an embodiment of the present disclosure can remove at least one gradient corresponding to at least one filter among a plurality of filters using a gradient mask, and can retain the residual gradients corresponding to the remaining filters among the plurality of filters. Therefore, the image processing apparatus 100 can update the weights of the remaining filters corresponding to the residual gradients without updating the weights of the at least one filter corresponding to at least one removed gradient.

[0153] According to embodiments of this disclosure, gradient mask 650 may have vector values ​​for removing gradients that contribute less to improving high-resolution performance from among a plurality of filters in the meta-model, and vector values ​​for retaining gradients that contribute more to improving high-resolution performance from among the plurality of filters. For example, the gradient mask may have vector values ​​"0" corresponding to gradients to be removed from the plurality of gradients, and may have vector values ​​"1" corresponding to gradients to be retained. For example, gradients with larger gradient values ​​may contribute more to improving high-resolution performance.

[0154] According to embodiments of this disclosure, the size of the gradient mask of the l-th convolutional layer can be equal to the number of gradients used to update the multiple filters within the l-th convolutional layer, respectively.

[0155] In operation 940, the image processing apparatus 100 according to an embodiment of the present disclosure can train a meta-model by updating the weights of the meta-model based on the residual gradient among multiple gradients.

[0156] According to embodiments of this disclosure, the image processing apparatus 100 can obtain the pruned gradient through a gradient mask. The pruned gradient may include the residual gradient in addition to the gradient removed by the gradient mask. The image processing apparatus 100 can update the weights of the residual filters corresponding to the residual gradients maintained by the gradient mask. The image processing apparatus 100 can train a meta-model based on the residual gradients by updating the filter weights. This has already been explained above with reference to Equation 3.

[0157] In operation 950, the image processing apparatus 100 according to an embodiment of the present disclosure can obtain a quality-processed output image from an input image based on a trained meta-model.

[0158] Image processing apparatus 100 according to embodiments of the present disclosure can use an updated meta-model to process the image quality of input image 110. Image processing apparatus 100 can generate high-resolution output image 120 by applying a meta-model adapted to input image 110 to input image 110.

[0159] According to embodiments of this disclosure, the image processing apparatus 100 can continue the forward propagation process without alteration, but can reduce the amount of gradient data that needs to be calculated during the backpropagation process, and reduce the computational cost for updating weights. Because the image processing apparatus 100 can remove gradients that contribute little to improving the high-resolution performance of the meta-model, and can avoid updating the weights of the filters corresponding to the removed gradients, the computational cost in the backpropagation process can be reduced, and memory usage can also be reduced.

[0160] According to embodiments of this disclosure, the image processing apparatus 100 can retain gradients that help improve the high-resolution performance of the meta-model, and can repeatedly update the weights of the filters corresponding to the retained gradients, thus enabling rapid image quality processing of the input image 110.

[0161] Figure 10 This is a block diagram of the internal structure of an image processing apparatus 100 according to an embodiment of the present disclosure.

[0162] Reference Figure 10The image processing apparatus 100 according to embodiments of the present disclosure may include an image quality determiner 310, a model adaptation unit 320-1, and an image quality processor 330. According to embodiments of the present disclosure, the model adaptation unit 320-1 may include a gradient pruning unit 322 and a quantizer 324. Figure 10 The image processing device 100 shown can be similar to Figure 3 The image processing apparatus 100 shown may include a quantizer 324 in addition to the model adaptation unit 320-1. The quantizer 324 may be implemented as at least one processor. The quantizer 324 may operate according to at least one instruction stored in memory.

[0163] According to embodiments of this disclosure, because the image processing apparatus 100 can train a meta-model in real time within its own space, it can utilize a large amount of computing and computational resources. For example, the training process of the meta-model may involve repeating the backpropagation process hundreds to tens of thousands of times. Because the image processing apparatus 100 may have limited computing resources and internal memory compared to a cloud server, it can train neural network models based on INT operations.

[0164] According to embodiments of this disclosure, the image processing apparatus 100 can perform quantization processing to convert FP data into INT data in order to train a neural network model based on INT operations. According to embodiments of this disclosure, INT data can refer to data that is not below or after a decimal point, and can be represented as a binary number, such as an 8-bit integer, a 16-bit integer, or a 32-bit integer. According to embodiments of this disclosure, FP data can refer to floating-point numbers with values ​​below or after a decimal point, and can be represented as binary numbers, such as a 16-bit integer or a 32-bit integer. According to embodiments of this disclosure, FP data can be converted into INT data using quantization processing and can be expressed accordingly. For example, when one FP data is converted into 8-bit INT data, the one FP data can be represented as two INT data, such as a data value and a data scale.

[0165] According to embodiments of this disclosure, model adaptation unit 320-1 can calculate gradients through forward and backward propagation processes, and the calculated gradients can be FP data. Quantizer 324 can obtain INT data via quantization processing of the FP gradients. Model adaptation unit 320-1 can perform convolution operations and train a meta-model using the INT data obtained by quantizer 324. Therefore, compared to training a meta-model using FP data, image processing apparatus 100 can reduce computational load by using INT data to train a meta-model associated with image processing apparatus 100. According to embodiments, the meta-model trained with INT data can be referred to as an INT neural network model. Data pieces of data in the meta-model trained with INT data can be represented as INT data instead of FP data.

[0166] According to embodiments of this disclosure, the memory of the image processing apparatus 100 can store INT weights (e.g., ... Figure 11 (As shown). The image processing apparatus 100 can use quantizer 324 to quantize the computed FP gradient into INT gradient. The image processing apparatus 100 can update the quantized INT gradient to the INT weights stored in memory. The image processing apparatus 100 can use the updated INT weights to perform a convolution operation for the next iteration. The image processing apparatus 100 according to embodiments of the present disclosure can use the INT data obtained using quantizer 324 to train a meta-model. Because the image processing apparatus 100 according to embodiments of the present disclosure can use INT weights to train the meta-model, the computational load can be reduced.

[0167] In some embodiments of the present disclosure, the memory of the image processing apparatus 100 may store FP weights (e.g., as shown in the figure). Figure 12 (As shown). The image processing apparatus 100 may not use quantizer 324 to quantize the computed FP gradient, and may update the FP weights with the FP gradient. The image processing apparatus 100 may quantize the updated FP weights into INT weights. The image processing apparatus 100 may use the quantized INT weights to perform a convolution operation for the next iteration. The image processing apparatus 100 according to embodiments of the present disclosure may use the INT data obtained by quantizer 324 to train a meta-model.

[0168] The quantizer 324 according to embodiments of this disclosure can perform scaling on the gradient as a preprocessing step before updating the gradient to the weights. For example, the quantizer 324 can scale the range of the gradient so that the range of the gradient is similar to the range of the weights. Scaling can refer to an operation that adjusts the range of data.

[0169] According to embodiments of this disclosure, when the INT weights are stored in the memory of the image processing apparatus 100, the image processing apparatus 100 can perform scaling relative to the INT gradient as a preprocessing step (e.g., as...) to update the INT gradients to the INT weights. Figure 11 (As shown). According to embodiments of this disclosure, since the INT weights can be stored in the memory of the image processing apparatus 100, quantization processing of converting FP gradients into INT gradients may be involved. In this case, the accuracy of the INT gradients converted from FP gradients may be reduced due to quantization errors. Therefore, when the gradient range is scaled to correspond to the weight range, the distribution distortion of the gradient data may increase. According to embodiments of this disclosure, the image processing apparatus 100 may use gradients with increased data distribution distortion to update weights, which may lead to distribution distortion of the weight data. When a meta-model is trained based on weights with data distribution distortion, data distribution distortion and quantization errors may increase.

[0170] Conversely, according to embodiments of this disclosure, when the FP weights are stored in the memory of the image processing apparatus 100, the image processing apparatus 100 can perform scaling relative to the FP gradient as a preprocessing step for updating the FP gradient to the FP weights (e.g., as...). Figure 12 (As shown). According to embodiments of this disclosure, since the FP weights can be stored in the memory of the image processing apparatus 100, quantization processing of the FP gradient can be omitted. In this case, there may be no quantization error, and the accuracy of the INT gradient can be kept relatively high. Therefore, when the gradient range is scaled to correspond to the weight range, the distribution distortion of the gradient data can be minimized. According to embodiments of this disclosure, the image processing apparatus 100 can update the weights using the gradient with minimized data distribution distortion and perform quantization relative to the updated weights to train the meta-model. Therefore, the data distribution distortion and quantization error of the image processing apparatus 100 can be minimized.

[0171] According to embodiments of this disclosure, the operation method of the quantizer 324 when the INT weights are stored in the memory of the image processing apparatus 100 may differ from the operation method of the quantizer 324 when the FP weights are stored in the memory of the image processing apparatus 100. Referring below to... Figure 11 and Figure 12 An example describing this operation method.

[0172] Figure 11 This is a view used to explain the quantization operation of the image processing apparatus 100 according to an embodiment of the present disclosure.

[0173] Reference Figure 11 According to an embodiment of the present disclosure, the image processing apparatus 100 may use the model adaptation unit 320-1 to perform quantization operations.

[0174] According to embodiments of this disclosure, the memory 1100 of the image processing apparatus 100 can store INT weights 1104. For example, the INT weights 1104 can be INT data comprising eight bits (e.g., 8-bit INT data). The range of the INT weights 1104 can be [-1, 1], and its precision can be 0.008. However, the number of bits, range, and precision of the INT weights 1104 are not limited to these specifications. Figure 11 Examples.

[0175] During operation S1110, the image processing device 100 can quantize the FP gradient 1101 calculated using the image processing device 100 into the INT gradient 1102.

[0176] The FP gradient 1101 can be FP data including 32 bits (e.g., 32-bit FP data). The range of the FP gradient 1101 can be [-0.002, 0.002], and its precision can be 9e-13. However, the number of bits, range, and precision of the FP gradient 1101 are not limited to these parameters. Figure 11 Examples.

[0177] The INT gradient 1102 can be 8-bit INT data. The range of the INT gradient 1102 can be [-0.002, 0.002], and its precision can be 1e-5. However, the number of bits, range, and precision of the INT gradient 1102 are not limited to... Figure 11 Examples.

[0178] Since the INT gradient 1102 is quantized from the FP gradient 1101, the INT gradient 1102 may have lower precision than the FP gradient 1101. The range of the INT gradient 1102 can be the same as the range of the FP gradient 1101.

[0179] In operation S1120, the image processing apparatus 100 may perform scaling on the INT gradient 1102 as a preprocessing step to update the INT gradient 1102 to the INT weights 1104. For example, the image processing apparatus 100 may adjust the range of the INT gradient 1102 so that the range of the INT gradient 1102 is the same as the range of the INT weights 1104. Scaling can refer to an operation that adjusts the range of data. For example, scaling may include a min-max normalization method that scales the minimum and maximum ranges of data to zero ("0") and one ("1"), a normalization method that performs scaling so that all values ​​are represented between negative one ("-1") and one ("1"), and a normalization method that normalizes the data to a normal distribution with a mean of zero ("0") and a variance of one ("1").

[0180] The image processing apparatus 100 can obtain the scaled INT gradient 1103 by scaling the range of the INT gradient 1102. For example, in Figure 11 In this context, scaling is illustrated as a normalization method that performs scaling such that all values ​​are represented between negative one ("-1") and one ("1"). However, the embodiments are not limited to this.

[0181] The scaled INT gradient 1103 can be INT data comprising eight bits. The range of the scaled INT gradient 1103 can be [-1, 1], similar to the range of the INT gradient 1104, and its precision can be 0.008. However, the number of bits, range, and precision of the INT gradient 1102 are not limited to... Figure 11 Examples.

[0182] According to embodiments of this disclosure, during the scaling process to make the range of INT gradient 1102 the same as the range of INT weight 1104, the accuracy of INT gradient 1102 may decrease when the difference between the two ranges is large. For example, the accuracy of scaled INT gradient 1103 may be significantly lower than that of unscaled INT gradient. Therefore, the data distribution of INT gradient may be distorted.

[0183] In operation S1130, the image processing device 100 can update the INT gradient 1103 to the INT weights 1104 stored in the memory 1100. The image processing device 100 can obtain the updated INT weights 1105 and can perform a convolution operation with respect to the next iteration (e.g., i+1) with the updated INT weights 1105.

[0184] In operation S1140, the image processing apparatus 100 can perform convolution operations for forward propagation using updated INT weights 1105. The image processing apparatus 100 can train a meta-model using training data corresponding to the input image using the updated INT weights 1105. The image processing apparatus 100 can obtain the meta-model based on INT operations. Therefore, the computational cost of repeated training of the meta-model by the image processing apparatus 100 can be reduced.

[0185] However, as described in the above-mentioned reference operation S1120, when the difference between the range of INT gradient 1102 and the range of INT weight 1104 is large, the distribution of INT gradient 1102 may be distorted during the scaling of the range of INT gradient 1102 according to the range of INT weight 1104. When the distribution of INT gradient 1102 is distorted, the distribution of INT weight 1104 may also be distorted. Because INT gradient 1102 can be quantized from FP gradient 1101 and can have lower precision than FP gradient 1101, the amount of precision reduction can increase with scaling, and the quantization error can increase.

[0186] Reference Figure 12 Examples of methods for resolving data distribution distortion and quantization error are described in the image processing apparatus 100 according to embodiments of the present disclosure, which can train a meta-model based on INT data and operate it.

[0187] Figure 12 This is a view used to explain the quantization operation of the image processing apparatus 100 according to an embodiment of the present disclosure.

[0188] Reference Figure 12 According to an embodiment of the present disclosure, the image processing apparatus 100 may use the model adaptation unit 320-1 to perform quantization operations.

[0189] According to embodiments of this disclosure, the memory 1200 of the image processing apparatus 100 can store FP weights 1203. For example, FP weights 1203 can be FP data comprising 32 bits. The range of FP weights 1203 can be [-1, 1], and its precision can be 4e-10. However, the number of bits, range, and precision of FP weights 1203 are not limited to... Figure 12 Examples.

[0190] In operation S1210, as a preprocessing step to update the FP weights 1203 from the FP gradient 1201 calculated by the image processing device 100, the image processing device 100 may perform scaling on the FP gradient 1201.

[0191] The FP gradient 1201 can be 32 bits of FP data. The range of the FP gradient 1201 can be [-0.002, 0.002], and its precision can be 9e-13. However, the number of bits, range, and precision of the FP gradient 1201 are not limited to... Figure 12 Examples.

[0192] The image processing apparatus 100 can adjust the range of the FP gradient 1201 so that the range of the FP gradient 1201 is the same as the range of the FP weights 1203. Scaling can refer to an operation that adjusts the range of data. For example, scaling can include a min-max normalization method that scales the minimum and maximum ranges of data to zero ("0") and one ("1"), a normalization method that performs scaling so that all values ​​are represented between negative one ("-1") and ("1"), and a normalization method that normalizes the data to a normal distribution with a mean of 0 and a variance of 1 ("1").

[0193] The image processing apparatus 100 can obtain a scaled FP gradient 1202 by scaling the range of the FP gradient 1201. For example, in Figure 12In this context, scaling is illustrated as a normalization method that performs scaling such that all values ​​are represented between negative one ("-1") and one ("1"). However, the embodiments are not limited to this.

[0194] The scaled FP gradient 1202 can be FP data comprising 32 bits. The range of the scaled FP gradient 1202 can be [-1, 1], similar to the range of the FP gradient 1203, and its precision can be 4e-10. However, the number of bits, range, and precision of the FP gradient 1201 are not limited to... Figure 12 Examples.

[0195] According to embodiments of this disclosure, during the scaling process to make the range of FP gradient 1201 the same as the range of FP weight 1203, the data distribution distortion of FP gradient 1201 can be minimized because the scaled FP gradient 1202 has high precision. For example, the precision of the scaled FP gradient 1202 can be higher than that of the FP weight 1203. Figure 11 The accuracy of the INT gradient 1103 was achieved through quantization and scaling.

[0196] During operation S1220, the image processing device 100 can update the FP gradient 1202 to the FP weights 1203 stored in the memory 1200. The image processing device 100 can then obtain the updated FP weights 1204.

[0197] During operation S1230, the image processing device 100 can quantize the updated FP weights 1204 into INT weights 1205.

[0198] The updated FP weight 1204 can include 32 bits of FP data, but the number of bits in the updated FP weight 1204 is not limited to this.

[0199] INT weight 1205 can be INT data including eight bits, but the number of bits in INT weight 1205 is not limited to this.

[0200] In operation S1240, the image processing apparatus 100 can use INT weights 1205 to perform convolution operations on the forward propagation process with respect to the next iteration (e.g., i+1). The image processing apparatus 100 can use INT weights 1205 to train a meta-model using training data corresponding to the input image. The image processing apparatus 100 can obtain the meta-model based on INT operations. Therefore, the computational cost of repeated training of the meta-model by the image processing apparatus 100 can be reduced.

[0201] According to embodiments of this disclosure, since the FP weights 1203 are stored in the memory of the image processing apparatus 100, the quantization processing of the FP gradients 1201 can be omitted. In this case, there may be no quantization error, and the accuracy of the FP gradients 1201 can be maintained at a relatively high level. For example, the accuracy of the FP gradients 1201 can be higher than... Figure 11 The accuracy of the INT gradient 1103 was achieved through quantization and scaling.

[0202] Therefore, when scaling is performed such that the range of FP gradient 1201 corresponds to the range of FP weight 1203, the data distribution distortion of FP gradient 1201 can be minimized. According to embodiments of this disclosure, image processing apparatus 100 can update FP weight 1203 using FP gradient 1201 with minimized data distribution distortion, and can perform quantization relative to the updated FP weight 1204 to train a meta-model. Thus, data distribution distortion and quantization error of image processing apparatus 100 can be minimized.

[0203] The image processing apparatus 100 according to embodiments of the present disclosure can reduce quantization errors while training a neural network model based on INT operations. Therefore, the image processing apparatus 100 can reduce the computational load used to train the neural network model and also prevent performance degradation of the AI ​​model.

[0204] The image processing apparatus 100 according to embodiments of the present disclosure can not only perform inference of a neural network model based on INT operations through the above-described quantization processing, but also perform training of a neural network model based on INT operations.

[0205] Figure 13 This is a flowchart of an operation method of an image processing apparatus 100 according to an embodiment of the present disclosure. Figure 13 The operation method of the image processing device 100 and according to Figure 12 The operation of the image processing apparatus 100 in the disclosed embodiments corresponds to that of the image processing apparatus 100.

[0206] In operation 1310, the image processing apparatus 100 according to embodiments of the present disclosure can obtain a meta-model. In some embodiments, operation 1310 can be combined with... Figure 9 The operation corresponds to 910.

[0207] In operation 1320, the image processing apparatus 100 according to embodiments of the present disclosure can use training data corresponding to the input image 110 to compute multiple gradients of the meta-model. In some embodiments, operation 1320 can be combined with... Figure 9 The operation corresponds to 920.

[0208] According to embodiments of this disclosure, the gradient 640 calculated by the image processing apparatus 100 may be or may include FP data. The image processing apparatus 100 may perform quantization processing based on the calculated FP gradient according to operations 1330 to 1360, and then train a meta-model using INT data. The image processing apparatus 100 may obtain the meta-model based on INT operations.

[0209] In operation 1330, the image processing apparatus 100 according to the embodiment can remove at least one FP gradient from multiple FP gradients by applying a gradient mask with gradient pruning information to multiple FP gradients. In some embodiments, operation 1330 can be combined with... Figure 9 The operation corresponds to 930.

[0210] In operation 1340, according to embodiments of the present disclosure, the image processing apparatus 100 can scale a plurality of FP gradients such that the range of the plurality of FP gradients corresponds to the weight range of the meta-model. In some embodiments, the plurality of FP gradients can be scaled with... Figure 12 The FP gradient 1201 corresponds to this.

[0211] As a preprocessing step to update the FP weights 1203 from the FP gradient 1201 calculated by the image processing apparatus 100, the image processing apparatus 100 can perform scaling on the FP gradient 1201. The image processing apparatus 100 can adjust the range of the FP gradient 1201 so that its range is similar to that of the FP weights 1203. Scaling can refer to an operation that adjusts the range of data. The image processing apparatus 100 can obtain a scaled FP gradient 1202 by scaling the range of the FP gradient 1201.

[0212] According to embodiments of this disclosure, during the scaling process to make the range of FP gradient 1201 the same as the range of FP weight 1203, the data distribution distortion of FP gradient 1201 can be minimized because the accuracy of the scaled FP gradient 1202 can be high. For example, the accuracy of the scaled FP gradient 1202 can be higher than that of the FP weight 1203. Figure 11 The accuracy of the INT gradient 1103 was achieved through quantization and scaling.

[0213] According to embodiments of this disclosure, the image processing apparatus 100 can be based on... Figure 9 Operations 930 and 950 perform gradient pruning, after which the remaining gradients can be scaled, the weights updated, the updated weights quantized, and the meta-model trained based on the quantized weights. In this case, multiple gradients can be the pruning gradients.

[0214] According to embodiments of this disclosure, the image processing apparatus 100 can scale according to... Figure 9The operation computes multiple gradients, updates the weights of multiple filters, quantizes the updated weights, and trains a meta-model based on the quantized weights.

[0215] In operation 1350, according to embodiments of the present disclosure, the image processing apparatus 100 may update some of the FP weights based on a plurality of scaled FP gradients. In some embodiments, the FP weights may be... Figure 12 The FP weights 1203 correspond to this. The image processing device 100 can update the FP weights 1203 stored in the memory 1200 with the scaled FP gradient 1202. The image processing device 100 can obtain the updated FP weights 1204.

[0216] In operation 1360, according to embodiments of this disclosure, the image processing apparatus 100 can quantize some of the updated FP weights into INT weights. Some of the updated FP weights may be weights updated based on the remaining gradients after gradient pruning. FP weights can be... Figure 12 The FP weight 1204 corresponds to the INT weight, and the INT weight can correspond to the INT weight 1205.

[0217] In operation 1370, according to embodiments of the present disclosure, the image processing apparatus 100 can train a meta-model by performing convolution operations based on INT weights. INT weights can be weights quantized from weights updated according to the residual gradients after gradient pruning. For example, the image processing apparatus 100 can use INT weights 1205 to perform convolution operations with respect to the forward propagation process of the next iteration (e.g., i+1). The image processing apparatus 100 can obtain the meta-model based on INT operations. Therefore, the computational cost of repeated training of the meta-model by the image processing apparatus 100 can be reduced.

[0218] The image processing device 100 can obtain a quality-processed output image from an input image based on a trained meta-model.

[0219] Because the image processing apparatus 100 according to the embodiments of this disclosure can train the meta-model based on INT operations, the computational load and resource usage for training the meta-model can be reduced.

[0220] Because the image processing apparatus 100 according to embodiments of the present disclosure can update the FP weights based on the FP gradient and then perform quantization processing, data distribution distortion and quantization error can be reduced. Therefore, the image processing apparatus 100 can reduce the computational cost for training the meta-model and also prevent performance degradation of the meta-model.

[0221] Figure 14 This is a block diagram of the internal structure of a server 1000 according to an embodiment of the present disclosure.

[0222] Reference Figure 14 According to embodiments of the present disclosure, server 1000 may include processor 1001 and memory 1002.

[0223] Memory 1002 may store one or more instructions. Memory 1002 may store at least one program executable by processor 1001. According to embodiments of this disclosure, memory 1002 may include a database storing training data for training meta-models of image processing apparatus 100. According to embodiments of this disclosure, at least one neural network model and / or predefined operating rules may be stored in memory 1002.

[0224] The memory 1002 may include at least one type of storage medium selected from flash memory, hard disk, multimedia card micro, card memory (e.g., Secure Digital (SD) or Extreme Digital (XD) memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), programmable ROM (PROM), magnetic storage, magnetic disk, and optical disk.

[0225] Server 1000 may include at least one processor 1001. Processor 1001 can control the overall operation of server 1000. Processor 1001 can control server 1000 to compress training datasets by executing one or more programs stored in memory 1002.

[0226] According to embodiments of this disclosure, processor 1001 can execute one or more instructions to obtain multiple low-resolution training images 1620 and 2620 with image quality degraded to various types of poor image quality based on training image 201 or 1610. Processor 1001 can train a neural network model by applying training image 201 or 1610 and the multiple low-resolution training images 1620 and 2620 to the neural network model. Processor 1001 can compute multiple gradients of the trained neural network model by applying training image 201 or 1610 and the multiple low-resolution training images 1620 and 2620 to the trained neural network model. Processor 1001 can generate gradient masks 203, 650, and 1860 for removing at least one gradient from the multiple computed gradients.

[0227] According to embodiments of this disclosure, processor 1001 can execute one or more instructions to obtain a plurality of modified neural networks 1630 and 2630 corresponding to a plurality of low-resolution training images 1620 and 2620, respectively. Processor 1001 can calculate a plurality of test losses 1690 and 2690 for the plurality of modified neural networks 1630 and 2630 by applying a test image 1650 to each of the plurality of modified neural networks 1630 and 2630. Processor 1001 can update the weights of the neural network models such that the sum of the plurality of test losses 1690 and 2690 for the plurality of modified neural networks 1630 and 2630 can be minimized.

[0228] According to embodiments of this disclosure, processor 1001 can execute one or more instructions to obtain a first low-resolution training image 1620 with image quality degraded to a first type of degraded image quality and a second low-resolution training image 2620 with image quality degraded to a second type of degraded image quality based on training image 201 or 1610. Processor 1001 can obtain a first neural network 1670 applying training image 201 or 1610 and the first low-resolution training image 1620 to a neural network model, and a second neural network 2670 applying training image 201 or 1610 and the second low-resolution training image 2620 to the neural network model. Processor 1001 can calculate a first test loss 1690 applying test image 1650 to the first neural network 1670, and calculate a second test loss 2690 applying test image 1650 to the second neural network 2670. Processor 1001 can update the weights of the neural network model based on the first test loss 1690 and the second test loss 2690.

[0229] According to embodiments of the present disclosure, processor 1001 may execute one or more instructions to generate gradient masks 203, 650, or 1860, each gradient mask having a first value for removing at least one gradient from a plurality of gradients and a second value for preserving the remaining gradients (e.g., one or more remaining gradients) among the plurality of gradients.

[0230] According to embodiments of the present disclosure, processor 1001 may execute one or more instructions to generate gradient masks 203, 650, or 1860, each gradient mask having a vector value "0" relative to the gradient corresponding to the n minimum values ​​of a plurality of gradients, each value being obtained by quantizing the plurality of gradients, and having a vector value "1" relative to the remaining gradients.

[0231] According to embodiments of the present disclosure, processor 1001 may execute one or more instructions to quantize multiple gradients of each gradient filter to generate gradient masks 203, 650, or 1860, each gradient mask having a vector value "0" for gradients having values ​​less than or equal to a threshold among the values ​​of the multiple gradients obtained by quantization, and a vector value "1" for gradients having values ​​exceeding the threshold.

[0232] According to embodiments of the present disclosure, processor 1001 may execute one or more instructions to sum data segments in the gradient data matrix of each of a plurality of gradient filters corresponding to a plurality of gradients, thereby quantizing the plurality of gradients.

[0233] According to embodiments of this disclosure, for a neural network model including at least one convolutional layer, the size of the gradient mask 203, 650, or 1860 of the first convolutional layer can be equal to the number of gradients used to update each of the plurality of filters within the first convolutional layer.

[0234] According to embodiments of this disclosure, the size of the gradient mask 203, 650, or 1860 of the first convolutional layer can be equal to the number of filters in the first convolutional layer.

[0235] Figure 15 This is a flowchart of a method for training a meta-model executed by server 1000 according to an embodiment of the present disclosure. Figure 16 This is a diagram used to explain the process of training a meta-model by server 1000 according to an embodiment of the present disclosure.

[0236] Reference Figure 15 and Figure 16 In operation 1510, server 1000 according to embodiments of the present disclosure can set parameters of initial neural network 1600. For example, initial neural network 1600 may be a neural network that has not yet been trained with a large training dataset. For example, initial neural network 1600 may be a neural network that has not yet undergone meta-learning. In some embodiments, initial neural network 1600 may be... Figure 2 The initial metamodel 210 or can be with Figure 2 This corresponds to the initial meta-model 210. The server 1000 can set the initial parameters of the initial neural network 1600 to be trained. The server 1000 can generate a meta-learning neural network by training the initial neural network 1600 with the initial parameters.

[0237] In operation 1520, the server 1000 according to embodiments of the present disclosure can train a neural network using training images for multiple kernels to obtain multiple modified neural networks. The multiple kernels may include image quality degradation kernels. For example, image quality degradation kernels may include kernels that process compression degradation relative to the training images, kernels that process blur degradation, kernels that process downsampling, and kernels that process noise addition. The server 1000 can apply each of the multiple kernels to the training images to generate multiple low-resolution training images with image quality degraded to various types of image degradation. The server 1000 can use the training images and the multiple low-resolution training images to train the neural network. The server 1000 can obtain multiple modified neural networks by training the neural network using the training images and the multiple low-resolution training images. Each of the multiple modified neural networks may be a neural network with high-resolution performance for each of the multiple kernels.

[0238] For example, refer to Figure 16 In operation 1521, server 1000 can train neural network 1630 using training image 1610 for first kernel (k1) 1615, and can obtain a first modified neural network 1670 with high-resolution performance for first kernel (k1) 1615. Server 1000 can generate a first low-resolution training image 1620 by applying first kernel (k1) 1615 to training image 1610. Server 1000 can perform at least one of compression degradation, blur degradation, downsampling, or noise addition on training image 1610 to generate first low-resolution training image 1620 with degraded image quality. Server 1000 can train neural network 1630 by applying training image 1610 and first low-resolution training image 1620 to neural network 1630. Server 1000 can calculate a first loss (Loss_k1) 1645, which can be the difference between output image 1640 generated by inputting first low-resolution training image 1620 into neural network 1630 and training image 1610. Server 1000 can update the parameters of neural network 1630 until the first loss 1645 is minimized. Server 1000 can then obtain neural network 1670 with the first modified parameters.

[0239] For example, server 1000 can generate a second low-resolution training image 2620 with degraded image quality by applying a second kernel (k2) 2615 to training image 1610. Server 1000 can use training image 1610 and the second low-resolution training image 2620 to train neural network 2630. Server 1000 can obtain a second modified neural network 2670 with high-resolution performance for the second kernel (k2) 2615. The second modified neural network 2670 can be a neural network with updated parameters based on the second kernel (k2) 2615 in operation 1522.

[0240] According to embodiments of this disclosure, the first core 1615 and the second core 2615 can be different types of image quality degradation kernels. The parameters of the first modified neural network 1670 and the second modified neural network 2670 can be different from each other.

[0241] In this disclosure, the corresponding operations of training neural network 1630 with training image 1610 for first core 1615 and training neural network 2630 with training image 1610 for second core 2615 can be referred to as internal loop.

[0242] In operation 1530, the server 1000 according to an embodiment of the present disclosure can use test images to evaluate the performance of multiple modified neural networks for multiple cores. The server 1000 can compute multiple losses from the multiple modified neural networks. The losses computed by the multiple modified neural networks can be referred to as test losses. The server 1000 can apply each of the multiple cores to the test images to generate multiple low-resolution test images with image quality degraded to various types of image degradation. The server 1000 can use the test images and the multiple low-resolution test images to train each of the multiple modified neural networks. The server 1000 can use the test images and the multiple low-resolution test images to evaluate the performance of the multiple modified neural networks. The server 1000 can compute the test loss for each of the multiple modified neural networks based on the performance results.

[0243] For example, refer to Figure 16In operation 1531, server 1000 can evaluate the performance of the first modified neural network 1670 using test image 1650 for the first core (k1) 1615. Server 1000 can generate a first low-resolution test image 1660 by applying the first core (k1) 1615 to test image 1650. Server 1000 can perform at least one of compression degradation, blur degradation, downsampling, or noise addition on test image 1650 to generate the first low-resolution test image 1660 with degraded image quality. Server 1000 can evaluate the performance of the first modified neural network 1670 by applying test image 1650 and the first low-resolution test image 1660 to the first modified neural network 1670. Server 1000 can calculate a first test loss (Test Loss_k1) 1690, which can be the difference between the output image 1680 generated by inputting the first low-resolution test image 1660 into the first modified neural network 1670 and the test image 1650. The first test loss 1690 can be the performance evaluation data for the first modified neural network 1670.

[0244] For example, in operation 1532, server 1000 can evaluate the performance of the second modified neural network 2670 using test image 1650 for the second core (k2) 2615. Server 1000 can generate a second low-resolution test image 2660 by applying the second core (k2) 2615 to test image 1650. Server 1000 can evaluate the performance of the second modified neural network 2670 by applying test image 1650 and the second low-resolution test image 2660 to the second modified neural network 2670. Server 1000 can calculate a second test loss (Test Loss_k2) 2690, which can be the difference between the output image 2680 generated by inputting the second low-resolution test image 2660 into the second modified neural network 2670 and the test image 1650. The second test loss 2690 can be performance evaluation data of the second modified neural network 2670.

[0245] According to embodiments of this disclosure, the first test loss 1690 of the first modified neural network 1670 and the second test loss 2690 of the second modified neural network 2670 may be different from each other.

[0246] In operation 1540, server 1000 according to embodiments of the present disclosure can sum the test losses. Server 1000 can sum multiple test losses calculated from multiple modified neural networks. For example, server 1000 can sum a first test loss 1690 calculated from a first modified neural network 1670 and a second test loss 2690 calculated from a second modified neural network 2670.

[0247] In operation 1550, according to embodiments of the present disclosure, server 1000 can generate a trained neural network by updating the parameters of an initial neural network 1600 based on multiple test losses. Server 1000 can update the parameters of the initial neural network 1600 until the sum of the multiple test losses is minimized. Server 1000 can obtain a neural network with updated parameters from the initial neural network 1600. Server 1000 can obtain a neural network trained based on the initial neural network 1600, such as a meta-learning neural network. According to embodiments of the present disclosure, the meta-learning neural network can be... Figure 2 This corresponds to the meta-model 220. The meta-model 220 may be the result of training multiple modified neural networks in the initial neural network 1600. In this disclosure, this may be referred to as the external loop.

[0248] According to embodiments of this disclosure, the meta-learning neural network can achieve better performance in generating high-resolution images compared to the initial neural network 1600. The meta-learning neural network can adapt to the actual input images in the image processing device 100 at a speed greater than that of the initial neural network 1600.

[0249] Figure 17 This is a flowchart of a method for obtaining a gradient mask performed by server 1000 according to an embodiment of the present disclosure. Figure 18a This is a diagram used to explain the process of calculating gradients by a trained neural network executed by server 1000 according to an embodiment of the present disclosure. Figure 18b This is a diagram used to explain the process of obtaining a gradient mask performed by server 1000 according to an embodiment of the present disclosure.

[0250] Reference Figure 17 In operation 1710, server 1000 can apply training images and at least one low-resolution training image to the trained meta-model to compute multiple gradients of the trained meta-model.

[0251] Reference Figure 18a The trained neural network model 1830 can be compared with... Figure 15 and Figure 16 The initial neural network 1600 corresponds to the neural network trained in this example. In some embodiments, the trained neural network model 1830 can be associated with... Figure 2The meta-model 220 corresponds to this. Server 1000 can compute gradient 1840 based on the trained neural network model 1830 by simulating a general meta-learning process. For example, server 1000 can train the gradient 1840 of the trained neural network model 1830 by applying training image 1810 and multiple low-resolution training images to the trained neural network model 1830. Training image 1810 can be compared with... Figure 16 The training image 1610 corresponds to multiple low-resolution training images. Figure 16 Multiple low-resolution training images 1620 and 2620 correspond to each other, and multiple kernels can be used with... Figure 16 The multiple cores 1615 and 2615 correspond to each other. The output image 1820 can be compared with... Figure 16 The output image corresponds to 1640.

[0252] Because server 1000 can use a pre-trained neural network to compute gradients, which can be intermediate products of the backpropagation process, and can generate gradient masks, the weights of the trained neural network model 1830 do not need to be updated. Server 1000 can obtain gradient 1840 and gradient mask 1860 with fewer iterations relative to the trained meta-model 1830.

[0253] Because the trained neural network model 1830 can include multiple convolutional layers, the server 1000 can compute multiple gradients 1840, the number of which can correspond to the number of filters within each of the multiple convolutional layers. For example, the first gradient 1841 can be the gradient of the loss function with respect to the weights of the first filter, and can represent information used to update the weights of the first filter. The second gradient 1842, the third gradient 1843, the fourth gradient 1844, and the fifth gradient 1845 can correspond to the second, third, fourth, and fifth filters, respectively. The multiple gradients 1840 can also be referred to as multiple gradient filters or multiple gradient maps.

[0254] Refer again Figure 17 In operation 1720, server 1000 can identify gradients that contribute less to the output of the meta-model from multiple gradients.

[0255] According to embodiments of this disclosure, the contribution of the gradient to the output of the meta-model can increase with the gradient value. For example, a gradient with a larger gradient value can result in a larger degree of weight update, thus enabling rapid optimization of the meta-model's weights. The degree of contribution to the meta-model's output can refer to the degree of improvement in the meta-model's high-resolution performance.

[0256] According to embodiments of this disclosure, server 1000 can quantize multiple gradients 1840 of each gradient filter to select gradients that can significantly contribute to the meta-model output. According to embodiments of this disclosure, server 1000 can identify n gradients (where n is a natural number) with minimum values ​​among the multiple quantized gradients 1840. Server 1000 can generate a gradient mask 1860 by setting vector values ​​for removing the n gradients with minimum values. According to embodiments of this disclosure, server 1000 can identify gradients corresponding to values ​​below a threshold from the multiple quantized gradients 1840. Server 1000 can generate a gradient mask 1860 by setting vector values ​​for removing gradients corresponding to values ​​below a threshold.

[0257] For example, refer to Figure 18b Server 1000 can quantize multiple gradients 1840 for each gradient filter. For each gradient filter, the multiple gradients 1840 can have gradient matrix data with k rows and k columns. Server 1000 can obtain the quantized gradients by summing the data in the gradient matrix data of each gradient filter using a normalization process. The quantized gradients can be represented as matrix 1850. For example, the sum of the 5x5 gradient data corresponding to the first gradient 1841 can be located in the first row 1851 of matrix 1850. The second gradient 1842 can correspond to the second row 1852, the third gradient 1843 can correspond to the third row 1853, the fourth gradient 1844 can correspond to the fourth row 1854, and the fifth gradient 1845 can correspond to the fifth row 1855. The number of rows in matrix 1850 can be 5, which can be equal to the number of gradients. According to embodiments of this disclosure, the method of quantizing multiple gradients 1840 is not limited to the above example.

[0258] For example, server 1000 can determine that in matrix 1850 corresponding to the quantization gradient, the second row 1852, the fourth row 1854, and the fifth row 1855 correspond to three minimum values. In some embodiments, for example, server 1000 can determine that in matrix 1850 corresponding to the quantization gradient, the values ​​of the second row 1852, the fourth row 1854, and the fifth row 1855 are less than or equal to a threshold (e.g., the value 7).

[0259] In operation 1730, server 1000 can train a gradient mask by setting vector values ​​to remove the identified gradients.

[0260] According to embodiments of this disclosure, gradient mask 1860 may include information for preserving gradients that significantly contribute to the meta-model output. For example, gradient mask 1860 may include information for preserving gradients with larger gradient values ​​and removing gradients with smaller gradient values. Server 1000 may generate gradient mask 1860 having vector values ​​"0" for removing at least one of a plurality of gradients 1840 and vector values ​​"1" for preserving the remainder of the plurality of gradients 1840. The gradient corresponding to vector value "0" may have a smaller contribution to the output of the neural network. The gradient corresponding to vector value "1" may have a larger contribution to the output of the neural network.

[0261] For example, server 1000 can set gradient mask 1860 to have vector values ​​"0" at the second row 1852, fourth row 1854, and fifth row 1855 of the matrix 1850 corresponding to the quantized gradient. Server 1000 can also set gradient mask 1860 to have vector values ​​"1" at the first row 1851 and third row 1853 of the matrix 1850 corresponding to the quantized gradient.

[0262] According to embodiments of this disclosure, gradient mask 1860 can be used in the process of training the meta-model of image processing apparatus 100.

[0263] In the embodiments described above in this disclosure, the gradient mask is shown to be obtained via a channel-by-channel method. However, the embodiments are not limited thereto. For example, the gradient mask can be obtained via a layer-by-layer method. For example, a gradient mask for removing gradients can be generated based on the convolutional layers of a neural network using a layer-by-layer method. For example, the gradient mask may include information for updating some convolutional layers without updating others.

[0264] Figure 19 This is a flowchart of a method for training a meta-model executed by server 1000 according to an embodiment of this disclosure.

[0265] In operation 1910, server 1000 according to an embodiment of the present disclosure can obtain multiple low-resolution training images with image quality that has been degraded into various types of degraded image quality based on training images.

[0266] According to embodiments of this disclosure, a first low-resolution training image can have degraded image quality by performing at least one of compression degradation, blur degradation, downsampling, or noise addition on the training image.

[0267] In operation 1920, server 1000 according to an embodiment of the present disclosure can apply training images and multiple low-resolution training images to a meta-model to obtain multiple modified neural networks corresponding to the multiple low-resolution training images respectively.

[0268] According to embodiments of the present disclosure, server 1000 can obtain, based on training images, a first low-resolution training image with image quality degraded to a first type of degraded image quality and a second low-resolution training image with image quality degraded to a second type of degraded image quality.

[0269] According to embodiments of this disclosure, server 1000 can apply training images and a first low-resolution training image to a meta-model to obtain a first modified neural network. Server 1000 can also apply training images and a second low-resolution training image to a meta-model to obtain a second modified neural network.

[0270] In operation 1930, server 1000 according to an embodiment of the present disclosure can train a meta-model by updating the weights of the meta-model based on the test loss of each of a plurality of modified neural networks.

[0271] According to embodiments of the present disclosure, a server 1000 can calculate a test loss for each of a plurality of modified neural networks by applying a test image to each of the modified neural networks. For example, according to an embodiment, the server 1000 can calculate a first test loss for a first modified neural network by applying a test image to a first modified neural network. For example, according to an embodiment, the server 1000 can calculate a second test loss for a second modified neural network by applying a test image to a second modified neural network.

[0272] According to embodiments of this disclosure, server 1000 can update the weights of the meta-model such that the sum of the corresponding test losses of multiple modified neural networks is minimized. For example, server 1000 can update the weights of the meta-model based on a first test loss and a second test loss.

[0273] In operation 1940, server 1000 according to embodiments of the present disclosure may apply training images and at least one low-resolution training image to a trained meta-model to compute multiple gradients of the trained meta-model. In some embodiments, operation 1940 may be combined with... Figure 17 The operation corresponds to 1710.

[0274] In operation 1950, server 1000 according to an embodiment of the present disclosure can generate a gradient mask for removing at least one gradient from a plurality of computed gradients.

[0275] According to embodiments of the present disclosure, server 1000 can identify gradients that contribute less to the output of the meta-model from a plurality of gradients.

[0276] According to embodiments of this disclosure, server 1000 can generate a gradient mask having a first value for removing at least one gradient from a plurality of gradients and a second value for retaining the remainder of the plurality of gradients.

[0277] The gradient mask according to embodiments of this disclosure can have vector values ​​for preserving gradients that significantly contribute to the output of the meta-model. As the gradient value increases, the contribution of the gradient to the output of the meta-model can increase.

[0278] According to embodiments of this disclosure, server 1000 can provide a meta-model that can be rapidly trained based on real data in image processing device 100 by pre-training a meta-model for various image quality degradation scenarios. Server 1000 can retain gradients used to update the weights of filters that help improve the high-resolution performance of the meta-model, and only repeatedly update the weights of filters corresponding to the retained gradients, thus enabling rapid training of the meta-model.

[0279] Figure 20 This is a block diagram of an image processing apparatus according to an embodiment of the present disclosure.

[0280] Reference Figure 20 The image processing apparatus 100 according to embodiments of the present disclosure may include a processor 101 and a memory 102.

[0281] The image processing device 100 may be fixed or portable, and may be, but is not limited to, a digital television capable of receiving digital broadcasts. The image processing device 100 may include at least one of the following: desktop PC, smartphone, tablet PC, mobile phone, video phone, e-book reader, laptop PC, netbook computer, digital camera, personal digital assistant (PDA), portable multimedia player (PMP), camcorder, navigation wearable device, smartwatch, home network system, security system, or medical device.

[0282] The image processing apparatus 100 can be implemented not only as a flat panel display device, but also as a curved surface display device with curvature or a flexible display device with adjustable curvature. Examples of the output resolution of the image processing apparatus 100 can include any of a variety of resolutions, such as HD, Full HD, Ultra HD, or a resolution higher or clearer than Ultra HD.

[0283] Memory 102 may store one or more instructions. Memory 102 may store at least one program executed by processor 101. At least one neural network and / or predefined operating rules or AI model may be stored in memory 102. Memory 102 may store data input to or output by image processing device 100.

[0284] The memory 102 may include at least one type of storage medium selected from flash memory, hard disk, multimedia card micro, card-type memory (e.g., SD or XD memory), RAM, SRAM, ROM, EEPROM, PROM, magnetic storage, magnetic disk and optical disk.

[0285] According to embodiments of this disclosure, memory 102 may store gradient masks. Gradient masks may represent information for removing or preserving gradients based on their contribution to the output of the trained meta-model.

[0286] According to embodiments of this disclosure, memory 102 may store FP weights 1203.

[0287] Processor 101 may include one or more processors. Processor 101 can control the overall operation of image processing apparatus 100. Processor 101 can control image processing apparatus 100 to perform functions by executing one or more programs stored in memory 102.

[0288] According to embodiments of this disclosure, processor 101 may be configured to execute one or more instructions to obtain a neural network model for image quality processing of input image 110. Processor 101 is also configured to compute multiple gradients by applying training data corresponding to the input image 110 to the neural network model and performing partial differentiation on the weights of the neural network model relative to the loss of the neural network model. Processor 101 may also be configured to remove at least one gradient from the multiple gradients by applying a gradient mask 203, 650, or 1860 with gradient pruning information to the multiple gradients. Processor 101 is further configured to train the neural network model by updating the weights of the neural network model based on the remaining gradients among the multiple gradients. Processor 101 is also configured to obtain a quality-processed output image 120 from the input image 110 based on the trained neural network model.

[0289] According to embodiments of this disclosure, gradient mask 203, 650, or 1860 may include information for removing at least one gradient from a plurality of gradients and information for maintaining the remaining gradients other than at least one gradient.

[0290] According to embodiments of this disclosure, processor 101 is further configured to execute one or more instructions to compute multiple gradients corresponding to each of a plurality of filters within at least one convolutional layer of the neural network model for updating the weights of the plurality of filters. Processor 101 is also configured to remove at least one gradient corresponding to at least one of the plurality of filters based on gradient masks 203, 650, or 1860. Processor 101 may also be configured to update the weights of remaining filters based on remaining gradients corresponding to remaining filters among the plurality of filters.

[0291] According to embodiments of this disclosure, gradient mask 203, 650, or 1860 may include information for removing gradients corresponding to the n minimum values ​​among those obtained by quantizing a plurality of gradients.

[0292] According to embodiments of this disclosure, gradient mask 203, 650, or 1860 may include information for removing gradients having values ​​less than or equal to a threshold from values ​​obtained by quantizing a plurality of gradients.

[0293] According to embodiments of this disclosure, for a neural network model including at least one convolutional layer, the size of the gradient mask 203, 650, or 1860 of the first convolutional layer can be equal to the number of gradients used to update each of the plurality of filters within the first convolutional layer. According to embodiments of this disclosure, processor 101 can be configured to execute one or more instructions to adjust the range of a plurality of gradients corresponding to FP data, such that the range of the plurality of gradients corresponds to the range of weights of the neural network model. Processor 101 can also be configured to update the FP weights 1203 with the plurality of gradients having adjusted ranges. Processor 101 can also be configured to convert the updated FP weights 1204 into INT weights 1205 by quantizing the updated FP weights 1204. Processor 101 can also be configured to perform convolution operations on the neural network model based on the INT weights 1205.

[0294] According to embodiments of this disclosure, processor 101 may also be configured to execute one or more instructions to remove at least one FP gradient from a plurality of FP gradients by applying a gradient mask to the plurality of FP gradients. Processor 101 may also be configured to update some FP weights based on the remaining FP gradients among the plurality of FP gradients. Processor 101 may also be configured to convert some of the updated FP weights into INT weights.

[0295] According to embodiments of this disclosure, processor 101 may also be configured to execute one or more instructions to obtain image quality of input image 110. Processor 101 may also be configured to obtain a low-resolution image based on the image quality value of input image 110 by performing at least one of compression degradation, blur degradation, resolution adjustment, or noise addition on input image 110.

[0296] Figure 21 This is a flowchart of the operation method of the image processing apparatus 100 and the server 1000 according to embodiments of the present disclosure.

[0297] Reference Figure 21 In operation 2110, server 1000 according to embodiments of the present disclosure can train a meta-model. Server 1000 can obtain multiple low-resolution training images based on training images, these multiple low-resolution training images having image quality degraded to various types of image degradation. Server 1000 can apply the training images and the multiple low-resolution training images to the meta-model to obtain multiple modified neural networks, each corresponding to one of the multiple low-resolution training images. Server 1000 can train the meta-model by updating the weights of the meta-model based on the test loss of each of the multiple modified neural networks. (Refer to above) Figure 15 and Figure 16 An example like this is described.

[0298] In operation 2120, server 1000 according to embodiments of the present disclosure can compute multiple gradients of the metamodel by applying training images to the metamodel. Server 1000 can generate a gradient mask for removing some of the multiple gradients.

[0299] In operation 2130, the server 1000 according to embodiments of the present disclosure can provide a meta-model and gradient mask to the image processing apparatus 100. A meta-model trained using specific training images can be paired with a gradient mask corresponding to that meta-model. The server 1000 can provide multiple meta-models and multiple corresponding gradient masks. The server 1000 can provide the meta-model and gradient mask to the image processing apparatus 100 by communicating with it using a communication interface or an offline upload process. In operation 2140, the image processing apparatus 100 according to embodiments of the present disclosure can download the meta-model and gradient mask from the server 1000. The image processing apparatus 100 can download the meta-model and gradient mask by communicating with it using a communication interface or an offline upload process.

[0300] According to embodiments of this disclosure, an image processing apparatus 100 that has obtained a meta-model and gradient mask can, according to... Figure 9Operation 910. According to embodiments of this disclosure, the image processing apparatus 100, having obtained the meta-model and gradient mask, can... Figure 13 Operation 1310.

[0301] Machine-readable storage media can be provided as non-transitory storage media. A non-transitory storage medium can be a tangible device, and this simply means that it does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is stored semi-permanently in the storage medium and cases where data is temporarily stored. For example, a non-transitory recording medium may include a buffer for temporarily storing data.

[0302] According to embodiments of this disclosure, the methods of various disclosed embodiments can be provided by being included in a computer program product. The computer program product, which may be a commodity, can be traded between a seller and a buyer. The computer program product is distributed in the form of a device-readable storage medium (e.g., an optical disc read-only memory (CD-ROM)), or can be distributed directly and online (e.g., downloaded or uploaded) between two user devices (e.g., smartphones) through an app store. In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) can be stored at least temporarily in a device-readable storage medium, such as the memory of a manufacturer's server, an app store's server, or a relay server, or can be temporarily generated.

Claims

1. An image processing apparatus (100), comprising: The memory (102) is configured to store one or more instructions; as well as At least one processor (101) is configured to execute one or more instructions stored in memory (102); and The at least one processor (101) is configured to execute one or more instructions to: A neural network model is obtained for performing image quality processing on the input image (110); Multiple gradients are computed by applying training data corresponding to the input image (110) to the neural network model and by performing partial differentiation on the weights of the neural network model relative to the loss of the neural network model. At least one gradient is removed from multiple gradients by applying a gradient mask (203, 650, 1860) that includes gradient pruning information to multiple gradients. The neural network model is trained by updating the weights of the neural network model based on one or more residual gradients among multiple gradients. as well as Using a trained neural network model, a quality-processed output image (120) is obtained based on the input image (110).

2. The image processing apparatus (100) according to claim 1, wherein, The gradient mask (203, 650, 1860) includes information for removing at least one gradient from a plurality of gradients and information for retaining one or more remaining gradients from the plurality of gradients.

3. The image processing apparatus (100) according to claim 1 or 2, wherein, The at least one processor (101) is configured to execute one or more instructions to: For each of the multiple filters included in at least one convolutional layer in a neural network model, multiple gradients are computed to update the weights of the multiple filters. Based on the gradient mask (203, 650, 1860), remove at least one gradient corresponding to at least one filter among a plurality of filters; and Based on the residual gradients corresponding to the residual filters, update the weights of one or more residual filters among a plurality of filters.

4. The image processing apparatus (100) according to any one of claims 1 to 3, wherein, The gradient mask (203, 650, 1860) includes information for removing gradients corresponding to the n minimum values ​​among a plurality of values ​​obtained by quantizing a plurality of gradients, where n is a natural number.

5. The image processing apparatus (100) according to any one of claims 1 to 4, wherein, The gradient mask (203, 650, 1860) includes information for removing gradients that correspond to values ​​less than or equal to a threshold among a plurality of values ​​obtained by quantizing a plurality of gradients.

6. The image processing apparatus (100) according to any one of claims 1 to 5, wherein, For a neural network model that includes at least one convolutional layer, the size of the gradient mask (203, 650, 1860) of the first convolutional layer is equal to the number of gradients used to update each of the multiple filters within the first convolutional layer.

7. The image processing apparatus (100) according to any one of claims 1 to 6, wherein, The memory (102) is also configured to store floating-point (FP) weights, and The at least one processor (101) is configured to execute one or more instructions to: Multiple gradients are obtained by adjusting the range of multiple gradients corresponding to the FP data, such that the range of multiple gradients corresponds to the range of weights of the neural network model (S1210). The adjusted gradients are then updated to the FP weights (1203) (S1220). The FP weights are converted into integer (INT) weights (1205) by quantizing the updated FP weights (1204) (S1230); and The neural network model is convolutionally operated on based on the INT weights (1205) (S1240).

8. The image processing apparatus (100) according to any one of claims 1 to 7, wherein, The at least one processor (101) is configured to execute one or more instructions to: At least one FP gradient among multiple FP gradients is removed by applying a gradient mask to multiple FP gradients (1330). Based on the remaining FP gradients among multiple FP gradients, update some FP weights (1350) in the FP weights; and Convert some of the updated FP weights to INT weights (1360).

9. The image processing apparatus (100) according to any one of claims 1 to 8, wherein, The at least one processor (101) is configured to execute one or more instructions to: Determine the image quality value of the input image (110); as well as Based on the image quality value of the input image (110), a low-resolution image is obtained by performing at least one of compression degradation, blur degradation, resolution adjustment, or noise addition on the input image (110).

10. A method for training a neural network model, the method comprising: Multiple low-resolution training images (1620, 2620) are obtained based on the training image (201, 1610), and the multiple low-resolution training images have image quality that has degraded to various types of degraded image quality. The neural network model was trained by applying training images (201, 1610) and multiple low-resolution training images (1620, 2620) to the neural network model. Multiple gradients of the trained neural network model are computed by applying the training image (201, 1610) and multiple low-resolution training images (1620, 2620) to the trained neural network model; and Generate a gradient mask (203, 650, 1860) for removing at least one gradient from a plurality of gradients.

11. The method according to claim 10, wherein, The training of the neural network model includes: Obtain multiple modified neural networks (1630, 2630) corresponding to multiple low-resolution training images (1620, 2620). By applying the test image (1650) to each of the multiple modified neural networks (1630, 2630), multiple test losses (1690, 2690) for the multiple modified neural networks (1630, 2630) are calculated; and Update the weights of the neural network model so that the sum of the multiple test losses (1690, 2690) of the multiple modified neural networks (1630, 2630) is minimized.

12. The method according to claim 10 or 11, wherein, The acquisition of the multiple low-resolution training images (1620, 2620) includes: Based on the training images (201, 1610), a first low-resolution training image (1620) with image quality degraded to the first type of degraded image quality and a second low-resolution training image (2620) with image quality degraded to the second type of degraded image quality are obtained. The training of the neural network model includes: A first neural network (1670) is obtained by applying the training image (201, 1610) and the first low-resolution training image (1620) to the neural network model, and a second neural network (2670) is obtained by applying the training image (201, 1610) and the second low-resolution training image (2620) to the neural network model. A first test loss (1690) is calculated by applying the test image (1650) to a first neural network (1670), and a second test loss (2690) is calculated by applying the test image (1650) to a second neural network (2670); and The weights of the neural network model are updated based on the first test loss (1690) and the second test loss (2690).

13. The method according to any one of claims 10 to 12, wherein, The generation of the gradient mask (203, 650, 1860) includes generating gradient masks (203, 650, 1860), each gradient mask having a first value for removing at least one gradient from a plurality of gradients and a second value for retaining one or more remaining gradients from the plurality of gradients.

14. The method according to any one of claims 10 to 13, wherein, The generation of the gradient mask (203, 650, 1860) includes generating a gradient mask (203, 650, 1860), each gradient mask having a vector value "0" associated with the gradient corresponding to the n minimum values ​​of a plurality of gradients, wherein the values ​​are obtained by quantizing the plurality of gradients, and having a vector value "1" associated with one or more residual gradients, wherein n is a natural number.

15. The method of claim 13 or 14, further comprising quantizing the multiple gradients by summing data segments in the gradient data matrix of each of the multiple gradient filters corresponding to the multiple gradients.

Images