Model training method, image processing method, and apparatus

Abstract

A model training method, an image processing method, and an apparatus. The model training method comprises: inputting a first image into a first model to obtain a positive sample; performing blur processing on the first image to obtain a plurality of second images; inputting the plurality of second images into a second model to obtain a plurality of negative samples, the second model being obtained on the basis of a historical model of a third model to be trained; inputting the first image into the third model to obtain an anchor sample; and training the third model on the basis of the positive sample, the anchor sample, and the plurality of negative samples. In the present application, on the one hand, the overhead of display storage for generating negative samples is reduced; on the other hand, the generated negative samples impose stronger lower-bound constraints on a model. The performance of a trained student model can be improved, and the effect of using the trained student model to perform image processing can be improved.

Description

A model training method, an image processing method and device

[0001] The present application claims priority to the Chinese patent application No. 202411263183.6, filed on September 9, 2024, entitled "A model training method, an image processing method and device", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence, in particular to a model training method, an image processing method and device. BACKGROUND

[0003] Image restoration is a basic problem in the field of image processing and computer vision, which aims to restore high-quality images from degraded low-quality images. Deep learning-based image processing models have achieved excellent results, but at the same time, they require high computational resources and running memory, making it difficult to implement end-side deployment.

[0004] Neural network distillation, as a model compression method, not only reduces the demand for computational resources and running memory, but also significantly improves the performance of student models. It has been widely used in high-level semantic visual tasks and has achieved excellent results.

[0005] In image processing tasks, directly applying the neural network distillation method for high-level visual tasks often has poor results, and the number of neural network distillation methods designed for image processing tasks is limited and the performance improvement is limited. SUMMARY

[0006] The present application provides a model training method, an image processing method and device, which can improve the performance of the model and the effect of image processing.

[0007] In a first aspect, a model training method is provided, comprising: inputting a first image into a first model to obtain a positive sample; performing blur processing on the first image to obtain a plurality of second images; inputting the plurality of second images into a second model to obtain a plurality of negative samples, the second model being obtained according to a history model of a third model to be trained; inputting the first image into the third model to obtain an anchor sample; and training the third model according to the positive sample, the anchor sample and the plurality of negative samples.

[0008] Based on the scheme provided in the embodiments of the present application, by inputting the plurality of second images obtained by blurring the first image into the second model, a plurality of negative samples are obtained. On the one hand, only one model needs to be loaded to generate a plurality of negative samples, thereby reducing the display storage overhead required for generating negative samples. On the other hand, the plurality of negative samples and the anchor sample have the same content and small quality gap, thereby strongly constraining the lower bound of the model. The method provided in the embodiments of the present application can improve the performance of the student model obtained by training and the effect of image processing using the student model obtained by training.

[0009] In some possible implementation manners, training the third model according to the positive sample, the anchor sample and the plurality of negative samples includes: inputting the positive sample, the anchor sample and the plurality of negative samples into a feature encoder; determining a difference between the positive sample and the anchor sample, and a difference between the positive sample and the plurality of negative samples according to the feature encoder; and updating parameters of the third model according to the difference between the positive sample and the anchor sample, and the difference between the anchor sample and the plurality of negative samples.

[0010] Based on the scheme provided in the embodiments of the present application, the parameters of the third model are updated according to the differences between the positive sample and the anchor sample and between the plurality of negative samples and the anchor sample, which can enable the third model to learn the first model and improve the performance of the third model.

[0011] In some possible implementation manners, the feature encoder is an encoder in a vector quantization generative adversarial network (VQGAN).

[0012] Based on the scheme provided in the embodiments of the present application, the encoder in the VQGAN that retains more image detail information is used for encoding, the encoder in the VQGAN is more suitable for the case where the positive sample, the negative sample and the anchor sample have the same content but different quality, and can obtain a more fine-grained difference between the modeling samples, which helps to improve the performance of the third model.

[0013] In some possible implementation manners, the blurring includes: performing size-random Gaussian blurring on the first image to obtain the plurality of second images.

[0014] Based on the scheme provided in the embodiments of the present application, the first image is slightly perturbed through size-random Gaussian blurring. On the one hand, a plurality of second images can be obtained, and then a plurality of negative samples can be obtained, thereby enhancing the lower bound constraint on the model. On the other hand, the second images obtained through other blurring methods may have poor image quality, which may cause the performance of the third model to decrease.

[0015] In some possible implementation manners, an update step of the second model is S, S is a positive integer; and the second model at the current update time is determined according to the second model at the last update time and the third model at the current update time.

[0016] Based on the scheme provided in the embodiments of the present application, by updating the second model once every S times of updating the third model, the influence of the parameters of the student model on the second model can be delayed, and the generation of negative samples by the second model that are closer to the positive samples than the anchor samples, which leads to the performance decline of the third model, can be avoided.

[0017] In some possible implementation ways, the parameters θ' of the second model at the current time of updating satisfy: n θ' = ωθ n + (1-ω)θ n + Δθ s , wherein θ n represents the parameters of the second model at the last time of updating, θ s represents the parameters of the third model at the current time of updating, and ω≠0.

[0018] In some possible implementation ways, ω satisfies ω=0.1; and / or, T represents the number of times of updating the third model, T satisfies T=100000n, S satisfies S=S'+1000, wherein S' corresponds to updating the third model for the T-Sth to (T-1)th time, S corresponds to updating the third model for the Tth to (T+S-1)th time, T is a positive integer, and n is a non-negative integer.

[0019] Based on the scheme provided in the embodiments of the present application, by increasing the update step of the second model once every 100000 times of updating the third model, the influence of the parameters of the student model on the second model can be delayed, and the generation of negative samples by the second model that are closer to the positive samples than the anchor samples, which leads to the performance decline of the third model, can be avoided.

[0020] In a second aspect, a method for image processing is provided, including: determining a to-be-processed image; inputting the to-be-processed image into an image processing model to obtain a processed image, the image processing model being trained according to a plurality of negative samples, wherein the plurality of negative samples are obtained by inputting a plurality of second images obtained by performing blur processing on a first image into a second model, the second model being obtained according to a third model, and the third model being a historical model of the image processing model.

[0021] In some possible implementation ways, the blur processing includes random size Gaussian blur on the first image.

[0022] In some possible implementation ways, the image processing model is trained according to the difference between a positive sample and an anchor sample, and according to the difference between the anchor sample and the plurality of negative samples, the positive sample being obtained by inputting the first image into a first model, and the anchor sample being obtained by inputting the first image into the third model.

[0023] In some possible implementation manners, the difference between the positive sample and the anchor sample, and the difference between the anchor sample and the plurality of negative samples are obtained according to a feature encoder, and the feature encoder is an encoder in a vector quantization generative adversarial network (VQGAN).

[0024] In some possible implementation manners, the update step of the second model is S, S is a positive integer; and the second model at the current update is determined according to the second model at the last update and the third model at the current update.

[0025] In some possible implementation manners, the parameter θ' of the second model at the current update satisfies θ' = ωθ + (1-ω)θ, where θ represents the parameter of the second model at the last update, θ represents the parameter of the third model at the current update, and ω≠0. n n = ωθ n +(1-ω)θ s , where θ n represents the parameter of the second model at the last update, θ s represents the parameter of the third model at the current update, and ω≠0.

[0026] In some possible implementation manners, ω satisfies ω = 0.1; and / or, T represents the number of times of updating the third model, T satisfies T = 100000n, S satisfies S = S' + 1000, where S' corresponds to updating the third model for the T-Sth to (T-1)th time, S corresponds to updating the third model for the Tth to (T+S-1)th time, T is a positive integer, and n is a non-negative integer.

[0027] In some possible implementation manners, the image processing model is trained according to the method in the first aspect or any possible implementation manner of the first aspect.

[0028] In a third aspect, a device for model training is provided, including a processor configured to: input a first image into a first model to obtain a positive sample; perform blur processing on the first image to obtain a plurality of second images; input the plurality of second images into a second model to obtain a plurality of negative samples, the second model being obtained according to a historical model of a third model to be trained; input the first image into the third model to obtain an anchor sample; and train the third model according to the positive sample, the anchor sample and the plurality of negative samples.

[0029] In some possible implementation manners, the processor is specifically configured to: input the positive sample, the anchor sample and the plurality of negative samples into a feature encoder; determine the difference between the positive sample and the anchor sample, and the difference between the positive sample and the plurality of negative samples according to the feature encoder; and update the parameter of the third model according to the difference between the positive sample and the anchor sample, and according to the difference between the anchor sample and the plurality of negative samples.

[0030] In some possible implementation manners, the feature encoder is an encoder in a vector quantization generative adversarial network (VQGAN).​

[0031] In some possible implementations, the processor is specifically used to: perform a Gaussian blur on the first image with randomized dimensions to obtain multiple second images.

[0032] In some possible implementations, the update step size of the second model is S, where S is a positive integer; the second model at the current update is determined based on the second model at the previous update and the third model at the current update.

[0033] In some possible implementations, the parameters θ′ of the second model at the current update n Satisfy: θ′ n =ωθ n +(1-ω)θ s , where θ n θ represents the parameters of the second model during the last update. s This represents the parameters of the third model in the current update, ω≠0.

[0034] In some possible implementations, ω satisfies ω=0.1; and / or, T represents the number of times the third model is updated, and when T satisfies T=100000n, S satisfies: S=S′+1000, where S′ corresponds to the third model being updated from the TSth to the T-1th time, and S corresponds to the third model being updated from the Tth to the T+S-1th time, where T is a positive integer and n is a non-negative integer.

[0035] In some possible implementations, the device is a chip.

[0036] Fourthly, an image processing apparatus is provided, including a processor for: determining an image to be processed; inputting the image to be processed into an image processing model to obtain a processed image, wherein the image processing model is trained based on multiple negative samples, wherein the multiple negative samples are obtained by inputting multiple second images obtained after blurring a first image into a second model, the second model is obtained based on a third model, and the third model is a historical model of the image processing model.

[0037] In some possible implementations, the blurring process includes applying a Gaussian blur of the first image with randomized dimensions.

[0038] In some possible implementations, the image processing model is trained based on the difference between positive samples and anchor samples, and on the difference between anchor samples and multiple negative samples. Positive samples are obtained by inputting the first image into the first model, and anchor samples are obtained by inputting the first image into the third model.

[0039] In some possible implementations, the difference between positive samples and anchor samples, as well as the difference between anchor samples and multiple negative samples, is obtained from a feature encoder, which is the encoder in the Vector Quantization Generative Adversarial Network (VQGAN).

[0040] In some possible implementations, the update step size of the second model is S, where S is a positive integer; the second model at the current update is determined based on the second model at the previous update and the third model at the current update.

[0041] In some possible implementations, the parameters θ′ of the second model at the current update n Satisfy: θ′ n =ωθ n +(1-ω)θ s , where θ n θ represents the parameters of the second model during the last update. s This represents the parameters of the third model in the current update, ω≠0.

[0042] In some possible implementations, ω satisfies ω=0.1; and / or, T represents the number of times the third model is updated, and when T satisfies T=100000n, S satisfies: S=S′+1000, where S′ corresponds to the third model being updated from the TSth to the T-1th time, and S corresponds to the third model being updated from the Tth to the T+S-1th time, where T is a positive integer and n is a non-negative integer.

[0043] In some possible implementations, the image processing model is trained according to the method in the first aspect or any possible implementation of the first aspect described above.

[0044] Fifthly, an apparatus for model training is provided, comprising: a processor for executing computer instructions stored in a memory, such that the apparatus performs the method of the first aspect or any possible implementation thereof, or the method of the second aspect or any possible implementation thereof.

[0045] In some possible implementations, the processor can be a general-purpose processor, which can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, integrated circuit, etc.; when implemented in software, the processor can be a general-purpose processor that reads software code stored in memory. This memory can be integrated into the processor or located outside the processor and exist independently.

[0046] In some possible implementations, the device also includes a memory.

[0047] In some possible implementation manners, the apparatus further includes a communication interface coupled with the processor, the communication interface configured to input and / or output information.

[0048] In some possible implementation manners, the apparatus is a chip.

[0049] In a sixth aspect, a computer program product is provided. When a computer program in the computer program product is executed by a computing device (for example, the computing device can be a processor), a method in the first aspect or any possible implementation manner of the first aspect, or a method in the second aspect or any possible implementation manner of the second aspect is implemented.

[0050] In a seventh aspect, a chip is provided. The chip obtains an instruction and executes the instruction to implement a method in the first aspect or any possible implementation manner of the first aspect, or a method in the second aspect or any possible implementation manner of the second aspect; or the chip includes a processor in the first aspect or any possible implementation manner of the first aspect, or a processor in the second aspect or any possible implementation manner of the second aspect.

[0051] In an eighth aspect, a computer readable storage medium is provided. The storage medium stores a computer program or an instruction. When the computer program or the instruction is executed by a computing device (for example, the computing device can be a processor), a method in the first aspect or any possible implementation manner of the first aspect, or a method in the second aspect or any possible implementation manner of the second aspect is implemented.

[0052] As an example, the computer readable storage includes, but is not limited to, one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), Flash memory, electrically EPROM (EEPROM), and hard drive.

[0053] In some possible implementation manners, the storage medium can be a non-volatile storage medium. BRIEF DESCRIPTION OF DRAWINGS

[0054] FIG. 1 is a schematic diagram of a training stage and an inference stage of a model.

[0055] FIG. 2 is a schematic diagram of a system architecture 200 suitable for embodiments of the present application.

[0056] FIG. 3 is a schematic diagram of a model training method according to an embodiment of the present application.

[0057] FIG. 4 is a schematic diagram of a model training flow according to an embodiment of the present application.

[0058] FIG. 5 is a schematic diagram of a result of image processing according to an embodiment of the present application.

[0059] FIG. 6 is a schematic diagram of a model training apparatus 600 according to an embodiment of the present application.

[0060] FIG. 7 is a schematic diagram of an image processing apparatus 700 according to an embodiment of the present application.

[0061] FIG. 8 is a schematic diagram of a chip system 800 according to an embodiment of the present application. DETAILED DESCRIPTION

[0062] The technical solutions in the present application will be described below with reference to the accompanying drawings.

[0063] Before introducing the solutions of the present application, the following points are explained.

[0064] (1) The present application will present various aspects, embodiments or features around a system including a plurality of devices, components, modules, etc. It should be understood and appreciated that each system can include additional devices, components, modules, etc., and / or can not include all of the devices, components, modules, etc. discussed in connection with the accompanying drawings. Furthermore, combinations of the solutions can also be used.

[0065] (2) In various embodiments of the present application, the terms and / or descriptions of different embodiments are consistent and can be mutually referred to if there is no special description and logical conflict. The technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0066] (3) The statements "in some possible implementations" and the like appearing in different places in the specification are not necessarily all referring to the same implementation, but mean "one or more but not all implementations", unless otherwise specifically emphasized. The terms "comprise", "include", "have" and their variants mean "including but not limited to", unless otherwise specifically emphasized.

[0067] (4) The business scenarios described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation to the technical solutions provided by the embodiments of the present application. It is known to those skilled in the art that, with the evolution of the self-distillation architecture / image processing architecture and the appearance of new business scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0068] (5) In the present application, “first”, “second”, etc. are only convenient for description, used to distinguish objects, and do not limit the scope of the embodiments of the present application. They are not used to describe the order or sequence of features. It should be understood that the objects thus described can be interchanged under appropriate circumstances, so as to be able to describe solutions other than the embodiments of the present application.

[0069] (6) In the present application, the words “exemplary”, “for example”, etc. are used to represent examples, illustrations or descriptions. Any embodiment or design scheme described as “exemplary” in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the use of the word “exemplary” is intended to present the concept in a specific way. In the embodiments of the present application, “of”, “corresponding” and “corresponding” can be used interchangeably at times, and it should be pointed out that when their differences are not emphasized, the meanings they express are consistent.

[0070] (7) “At least one” or “at least one” in the present application means one or more. “Multiple” means two or more. “And / or” describes the relationship between the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the following cases: A exists alone, A and B exist together, B exists alone, where A and B can be singular or plural. In the textual description of the present application, the character “ / ” generally represents that the associated objects before and after are in an “or” relationship; in the formulas of the present application, the character “ / ” represents that the associated objects before and after are in a “division” relationship. “Including at least one of A, B and C” can mean: including A; including B; including C; including A and B; including A and C; including B and C; including A, B and C.

[0071] (8) The arrows or blocks shown by dashed lines in the schematic diagrams in the drawing part of the present application specification represent optional steps or optional modules.

[0072] In order to better describe the solutions of the embodiments of the present application, the terms that may be involved in the embodiments of the present application will be described first.

[0073] (1) Machine learning (machine learning):

[0074] Machine learning is a method to realize artificial intelligent (AI), and the goal of machine learning is to design and analyze some algorithms (i.e., models) that can be automatically "learned" by a computer. The machine learning model is a kind of algorithm that automatically analyzes rules from data and uses the rules to predict unknown data. Machine learning models include a variety of models, and according to whether the model needs to rely on the label corresponding to the training data during training, the machine learning model can be divided into: unsupervised learning model; supervised learning model.

[0075] (2) Deep learning:

[0076] Deep learning is a new technical field generated in the process of machine learning research. In the machine learning method, almost all the features need to be determined by industry experts, and then the features are encoded. The typical structure of the current deep learning model is a deep neural network. The neural network is a mathematical model or a computational model that simulates the structure and function of a biological neural network (animal central nervous system, especially the brain). The neural network performs calculations by a large number of neuron connections. A neural network can include multiple neural network layers with different functions, each layer including parameters and calculation rules. According to the difference of calculation formula or function, different layers in the neural network have different names, for example: the layer performing convolution calculation is called a convolution layer, which is often used for feature extraction of input signals (e.g., images). A neural network can also be composed of multiple sub-neural networks. Different structures of neural networks can be suitable for different scenarios (e.g., classification, identification) or provide different effects when used in the same scenario. The structure of the neural network is different and specifically includes one or more of the following: the number of network layers in the neural network is different, the order of each network layer is different, the weight, parameter or calculation formula in each network layer is different. There are many different neural networks with high accuracy in the industry for identification or classification and other application scenarios. Some neural networks can be trained by specific data sets and used alone or combined with other neural networks (or other functional modules) to complete a task.

[0077] In other words, the deep learning model is actually a machine learning model with a complex structure of neural networks. According to whether the label corresponding to the training data is needed during the training of the deep learning model, the deep learning model can also be divided into a supervised learning model and an unsupervised learning model, which will not be described here. Classical deep learning models include convolutional neural networks (CNN), recurrent neural networks (RNN), recursive neural networks (RNN), Transformers, and the like.

[0078] (3) Supervised learning model:

[0079] The supervised learning model is a model obtained after determining the parameters of the initial AI model according to the data in the given training data set and the labels corresponding to the data in the training data set. The process of determining the parameters of the initial AI model using the data in the training data set and the labels corresponding to the data is also called supervised learning (or supervised training). The labels of the data in the training data set are usually manually annotated and used to identify the correct answer of the data on a specific task. Typical supervised learning models include support vector machines, neural network models, logistic regression models, decision trees, naive Bayes models, Gaussian discriminant models, and the like. Supervised learning models are usually used for classification or regression.

[0080] (4) Unsupervised learning model:

[0081] The unsupervised learning model is a model obtained after determining the parameters of the initial AI model according to the unlabeled data in the given training data set. The process of determining the parameters of the initial AI model using the unlabeled training data is also called unsupervised learning (or unsupervised training). Through unsupervised learning, the model can discover meaningful information and associations in the data and then make predictions on the results of the data. There are many kinds of unsupervised learning models, and the commonly used ones include clustering models, principal component analysis (PCA), anomaly detection models, autoencoders, generative adversarial networks (GAN), and the like.

[0082] (5) Self-supervised learning method:

[0083] Self-supervised learning methods can be seen as a special kind of unsupervised learning method, which uses pretext tasks to extract supervision information from large amounts of unsupervised data, and trains the network through the constructed supervision information, so as to learn representations valuable for downstream tasks.

[0084] (6) Contrastive learning method:

[0085] Contrastive learning is a self-supervised learning method that learns general features of a dataset by having the model learn which data points are similar or different without labels. Contrastive learning is not limited to a specific model or a specific type of model, but is a method of representation learning using unlabeled data. Contrastive learning learns the feature representation of samples by comparing them with positive and negative samples in the feature space. The generation or selection of negative samples is one of the main difficulties of contrastive learning.

[0086] Contrastive learning compares samples with samples similar in semantics (positive samples) and samples dissimilar in semantics (negative samples), hoping to design model structures and contrastive losses to make representations of similar semantics closer in representation space and representations of dissimilar semantics farther apart, achieving a clustering-like effect.

[0087] (7) Neural network distillation:

[0088] Neural network distillation, also known as model distillation, knowledge transfer or knowledge distillation, is a technique for compressing models, aiming to transfer the knowledge of a large, complex model (usually called the teacher model) to a smaller, simpler, more efficient model (usually called the student model). This method is often used to optimize the inference speed and resource consumption of the model while maintaining the prediction performance of the model as much as possible.

[0089] · The following briefly introduces some related content of neural network distillation:

[0090] ① Teacher model: a large model that has been trained, usually with high accuracy and complex structure. The purpose of the teacher model is to provide rich knowledge for the student model to learn.

[0091] ② Student model: a simpler model with fewer parameters, the purpose is to learn the behavior of the teacher model to achieve similar performance, the student model has faster inference speed and lower resource consumption compared to the teacher model.

[0092] ③ Distillation Process: In the distillation process, the student model is trained to mimic the behavior of the teacher model. Distillation can be achieved by minimizing the difference between the output of the student model and the output of the teacher model.

[0093] ④ Soft Targets: Also known as the output of the teacher model (such as probability distribution in classification tasks), the output of the teacher model is usually used as the target for training the student model, which can provide more rich information than hard labels (such as class labels).

[0094] ⑤ Temperature: In the distillation process, the temperature parameter is used to control the "softness" of the soft targets. Higher temperature will produce more "soft" probability distribution, which helps the student model learn smoother decision boundaries.

[0095] ⑥ Distillation Loss: Distillation loss usually includes two parts, one is the difference between the output of the student model and the output of the teacher model, the other is the difference between the output of the student model and the real label. The weight of the two parts is balanced by a hyperparameter (such as α).

[0096] The steps of neural network distillation are as follows:

[0097] Train the teacher model: First, a well-trained teacher model is needed, which performs well on the target task.

[0098] Prepare the student model: Define a simpler structure of the student model, which will learn and imitate the behavior of the teacher model.

[0099] Distillation training: Use the output of the teacher model as the target to train the student model. This usually involves calculating the difference between the output of the student model and the output of the teacher model, and minimizing the difference.

[0100] Fine-tuning: In some cases, the student model may still need additional fine-tuning on the real label to further improve the performance of the student model.

[0101] Figure 1 is a schematic diagram of the training and inference stages of the model.

[0102] As shown in FIG. 1, in the training stage of the student model, first, a training set for the deep learning model needs to be constructed based on the target, the training set includes multiple training data (for example, the training set includes multiple images), each training data is input into the pre-trained teacher model, and the target data can be obtained, which can represent the target of training the student model using the training data. For example, for a deep learning model (i.e., a student model) to be trained for an image processing task, the training set can include multiple images (i.e., training data) of different resolutions or image qualities.

[0103] When training the deep learning model, the training data can be input into the deep learning model after parameter initialization in batches, and the deep learning model performs calculation (i.e., inference) on the training data to obtain the prediction result for the training data. The prediction result obtained by inference and the target data corresponding to the training data are used as data for calculating the loss according to the loss function. The loss function is a function used to calculate the gap (i.e., loss value loss) between the prediction result of the model for the training data and the label of the training data in the model training stage. The loss function can be implemented by different mathematical functions, and the expressions of commonly used loss functions are: mean square error loss function, logarithmic loss function, least squares method, etc.

[0104] The loss value calculated based on the loss function can be used to update the parameters of the deep learning model, and the gradient descent method is commonly used for parameter updating. The training of the model is a repeated iterative process, and different training data is inferred and the loss value is calculated in each iteration. The goal of multiple iterations is to continuously update the parameters of the deep learning model to find the parameter configuration that makes the loss value of the loss function the lowest or tends to be stable.

[0105] It should be understood that the loss function is a function that maps a random event or its related random variable to a non-negative real number to represent the "risk" or "loss" of the random event. In applications, the loss function is usually associated with optimization problems as a learning criterion, that is, the loss function is minimized to solve and evaluate the model. For example, in machine learning, the loss function is used for parameteric estimation of the model, and the loss value obtained based on the loss function can be used to describe the difference between the predicted value and the actual value of the model. Common loss functions include mean square error loss function, support vector machine (SVM) hinge loss function, cross-entropy loss function, etc.

[0106] As shown in FIG. 1, the trained deep learning model can be used for inference of input data. In the inference stage, data of an actual application scenario is usually used as input data (for example, a low-resolution image can be used as input data), and the inference of the trained deep learning model can obtain an inference result (for example, a high-resolution image is obtained by the inference of the trained deep learning model on a low-resolution image). The inference stage is the actual application of the trained deep learning model, and can quickly use the AI capability to solve specific technical problems.

[0107] The above only takes the training of the most typical deep learning model as an example for description. The training of other types of models has slight differences, but the principle is similar. Most of them are to infer the training data, and to adjust the parameters in the model according to the inference result, so as to obtain a parameter combination that stabilizes the performance of the model.

[0108] FIG. 2 is a schematic diagram of a system architecture 200 suitable for embodiments of the present application.

[0109] As shown in FIG. 2, the data acquisition device 260 is configured to acquire training data and store the training data in the database 230. The training device 220 trains the target model / rule 201 based on the training data maintained in the database 230.

[0110] The following will describe in more details how the training device 220 obtains the target model / rule 201 based on the training data, which can be used to implement the image processing method provided by embodiments of the present application.

[0111] It should be noted that in actual applications, the training data maintained in the database 230 can come from the data acquisition device 260 or other devices. The training device 220 does not necessarily train the target model / rule 201 based on the training data maintained in the database 230, but can also obtain training data from the cloud or other places for model training. The above description should not be regarded as a limitation of the embodiments of the present application.

[0112] The target model / rule 201 trained by the training device 220 can be applied to different systems or devices, for example, to the execution device 210 shown in FIG. 2. The execution device 210 can be a terminal such as a mobile phone terminal, a tablet computer, a notebook computer, an augmented reality (AR) device / virtual reality (VR) device, a vehicle-mounted terminal, etc. The execution device 210 can also be a server or a cloud, etc. The execution device 210 is configured with an I / O interface 212 for data interaction with external devices. A user can input data to the I / O interface 212 through a client device 240. In addition, the input data can be input by the user, uploaded by the user through a shooting device, or come from a database, which is not limited in the present application.

[0113] The preprocessing module 213 is used for preprocessing the input data received by the I / O interface 212. In the embodiment of the present application, the preprocessing module 213 can be used to obtain the features of the image to be processed. During the preprocessing of the input data by the execution device 210 or during the execution of the calculation module 211 of the execution device 210, the execution device 210 can call the data, code, etc. in the data storage system 250 for corresponding processing, or store the data, instructions, etc. obtained by the corresponding processing in the data storage system 250. Finally, the I / O interface 212 returns the processing result to the client device 240, thereby providing the user.

[0114] It should be noted that the training device 220 can also perform different tasks for different targets. Specifically, the target model / rule 201 corresponding to different tasks can be generated based on different training data or based on the same training data and different target data. The corresponding target model / rule 201 can be used to achieve the above-mentioned target or complete the above-mentioned task, thereby providing the user with the required multi-task result.

[0115] It should be understood that FIG. 2 is only a schematic diagram of a system architecture, and the positional relationship between the devices, devices, modules, etc. shown in the figure does not constitute any limitation. For example, in FIG. 2, the data storage system 250 is an external memory relative to the execution device 210. In other cases, the data storage system 250 can also be placed in the execution device 210.

[0116] Image restoration is a basic problem in the field of image processing and computer vision, and the purpose is to restore high-quality images from degraded low-quality images. Deep learning-based image processing models have achieved excellent results, but at the same time, they have brought higher requirements for computing resources and running memory, making it difficult to implement end-side deployment.

[0117] As a model compression method, neural network distillation not only reduces the demand for computational resources and running memory, but also significantly improves the performance of student models. It has been widely applied in high-level semantic visual tasks and achieved excellent results.

[0118] High-level visual tasks and low-level visual tasks often have significant differences. For example:

[0119] ① Task characteristic difference: High-level visual tasks (such as image classification, object detection, etc.) and low-level image processing tasks (such as image super-resolution, image deblurring, image derain, image defogging, etc.) have significant differences in task characteristics. High-level visual tasks usually focus on semantic information of images, while low-level visual tasks focus on pixel-level details and quality recovery of images.

[0120] ② Data distribution difference: High-level visual tasks and low-level image processing tasks have different data distributions. For example, image classification datasets usually contain diverse classes and backgrounds, while image super-resolution datasets focus more on image texture and details.

[0121] ③ Loss function and evaluation index difference: The loss function (such as cross-entropy loss) and evaluation index (such as accuracy) commonly used in high-level visual tasks are different from the loss function (such as mean squared error, structural similarity index) and evaluation index (such as peak signal-to-noise ratio (PSNR), structural similarity (SSIM)) of low-level visual tasks.

[0122] PSNR and SSIM are two commonly used indicators for evaluating image quality, especially in image processing tasks such as image restoration, super-resolution, and denoising. They measure image quality from different perspectives, both in decibels (dB). PSNR is calculated based on mean squared error (MSE), which measures the error of image restoration relative to the original image. The higher the PSNR value, the closer the image quality is to the original image. SSIM is an index that measures the similarity between two images, considering brightness, contrast, and structure. The SSIM value ranges from -1 to 1, and the closer the SSIM value is to 1, the better the image quality and the more similar it is to the original image. PSNR focuses more on pixel-level errors, which is sensitive to global changes in images, but may not accurately reflect human perception of image quality. SSIM attempts to better reflect human perception of image quality by considering structural information and being more sensitive to local changes in images.

[0123] Directly applying neural networks for high-level visual tasks to low-level visual tasks may lead to inconsistent optimization goals.

[0124] ④Difficulty of neural network distillation: In high-level visual tasks, neural network distillation usually involves transferring semantic information from large and complex models to small models. However, in low-level visual tasks, neural network distillation may focus more on the texture and detail information of images, which is technically more challenging.

[0125] ⑤Risk of overfitting: Directly applying the model of high-level visual tasks to low-level visual tasks may lead to overfitting due to learning high-level semantic features that are not applicable to low-level visual tasks.

[0126] ⑥Computational resource constraints: Low-level visual tasks usually have more stringent constraints on computational resources, especially in real-time applications. The model of high-level visual tasks may be too complex to implement on resource-constrained devices.

[0127] Due to the differences between high-level visual tasks and low-level visual tasks, directly applying the neural network distillation method of high-level visual tasks in image processing tasks often does not work well, and the number of neural network distillation methods designed for image processing tasks is limited and the performance improvement is limited.

[0128] Therefore, the embodiments of the present application provide a model training method, an image processing method and device, which can improve the performance of the model and the effect of image processing.

[0129] The model training method, the image processing method and the device provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0130] FIG. 3 shows a schematic diagram of a model training method 1000 provided by an embodiment of the present application. As shown in FIG. 3, the method 1000 can include the following steps:

[0131] S1010, inputting the first image into the first model to obtain a positive sample.

[0132] In neural network distillation, positive samples and negative samples can provide upper and lower bound constraints for student models (the output of the student model is an anchor sample).

[0133] Positive samples refer to samples that match the target to be learned by the model. Negative samples refer to samples that do not match the target to be learned by the model. In a classification problem, if a sample belongs to the target class, it is considered a positive sample, and if a sample does not belong to the target class, it is considered a negative sample. In the image super-resolution problem, images with higher resolution (e.g., the output of the teacher model) can be considered as positive samples, and images with lower resolution can be considered as negative samples.

[0134] For example, in a face recognition task, if the face in a picture is the target person, then this picture is a positive sample, and other pictures that do not contain the face of the target person (for example, a picture whose face is not the target person, or a picture that only contains fruits, or a picture that only contains animals, etc.) are negative samples; in an image super-resolution task, if the resolution of a picture is high, then this picture can be used as a positive sample, and other pictures with lower resolution can be used as negative samples.

[0135] In the model training method 1000 provided in the embodiments of the present application, the first model can be a teacher model, the second model can be a negative sample generation model, and the third model can be a student model. Since the teacher model has good performance, after the first image used for training is processed by the teacher model and the student model, the output of the teacher model can be used as a positive sample, and the output of the student model can be used as an anchor sample.

[0136] S1020, performing blur processing on the first image to obtain a plurality of second images.

[0137] In computer vision and image processing, blur processing is a common operation. Some common blur techniques and their application scenarios are introduced as follows:

[0138] ① Mean blur

[0139] - Principle: Replace the value of each pixel with the average value of its neighborhood pixels.

[0140] - Application: Used for simple image smoothing and noise removal.

[0141] ② Gaussian blur

[0142] - Principle: Use a Gaussian function (normal distribution curve) as a weight window to perform weighted averaging on the pixel and its neighborhood.

[0143] - Application: Used for more natural image smoothing and reduction of high-frequency noise.

[0144] ③ Median blur

[0145] - Principle: Replace the value of each pixel with the median value of its neighborhood pixels.

[0146] - Application: Particularly suitable for removing salt and pepper noise while maintaining clear edges.

[0147] ④ Bilinear filter

[0148] - Principle: Smooths the image while maintaining sharp edges. It takes into account the intensity differences and spatial distances between pixels.

[0149] - Application: Used in scenarios where edge information needs to be preserved, such as skin smoothing.

[0150] ⑤ Motion blur

[0151] - Principle: Simulates the blurring effect caused by camera or object movement, usually achieved by applying linear blur to the image.

[0152] Applications: Used to simulate motion effects, or to estimate and compensate for motion blur in image restoration.

[0153] ⑥ Defocus blur

[0154] - Principle: Simulates the blurring effect caused by an object not being on the focal plane.

[0155] Applications: Used for image compositing and visual effects enhancement.

[0156] ⑦ Radial blur

[0157] - Principle: Apply blur radially outward from the center of the image.

[0158] - Application: Used to create visual effects of rotation or scaling.

[0159] ⑧ Dynamic blur

[0160] - Principle: Dynamically adjust the blur level based on the image content.

[0161] Applications: Used for video processing and real-time image processing to simulate different visual effects.

[0162] By blurring the first image, multiple second images can be obtained. For example, by using various different blurring techniques to blur the first image, multiple second images can be obtained; or, by using a certain blurring technique to blur the first image, but with different degrees of blurring, multiple second images can also be obtained.

[0163] S1030, input multiple second images into the second model to obtain multiple negative samples. The second model is obtained based on the historical model of the third model to be trained.

[0164] By inputting multiple second images into a second model, multiple negative samples can be obtained. Compared to loading multiple negative sample generation models to generate multiple negative samples, this reduces the overhead of display storage.

[0165] S1040, input the first image into the third model to obtain an anchor sample.

[0166] Positive samples are objects that need to be focused on during the model training process, as they directly reflect the features and goals that the model needs to learn. Negative samples are also important during the training process, as they help the model distinguish between targets and non-targets, thereby improving the model's generalization ability and accuracy. Positive samples and negative samples are interdependent and interact during the model training process. They together form the model's training data set, which guides the model's learning.

[0167] Anchor samples often appear in contrast learning, metric learning, and certain deep learning scenarios, and are relative to other samples such as positive samples and negative samples. In contrast learning, anchor samples are the core samples that algorithms focus on, used for comparison and contrast with other samples.

[0168] In contrast learning, the similarity between anchor samples and positive samples should be as high as possible, while the similarity between anchor samples and negative samples should be as low as possible. In this way, algorithms can learn useful representations through comparison and contrast, so that similar samples are closer in the representation space, and dissimilar samples are farther apart.

[0169] In certain deep learning frameworks, anchor samples are also used as benchmarks or reference points to evaluate the relationship or difference between other samples and anchor samples.

[0170] S1050, training the third model according to the positive sample, the anchor sample, and the plurality of negative samples.

[0171] In the process of neural network distillation, the student model and the teacher model process the first image and generate high-resolution images, and the student model and the teacher model share part of the model parameters. The student model learns the behavior of the teacher model through the rich knowledge constructed by the contrast loss in multiple iterations. During the optimization process of the student model, the student model before training completion can be referred to as the historical model of the student model. For example, a certain student model is trained through 5 iterations, and the models obtained in the first four iterations are the historical models of the student model, and the model obtained in the fifth iteration is the student model. The student model meets the requirements of the model in the model training and can be used for image processing.

[0172] In adjacent iteration periods, the historical model and the current student model have similar parameter distributions, and thus can have similar image quality of image restoration results. By taking the historical model with similar parameter distribution to the current student model as the model for generating negative samples, and inputting the plurality of negative samples subjected to blurring into the model, a plurality of negative samples with similar image quality to the anchor sample can be generated by the same model. Compared with selecting an image other than the first image as a negative sample, by inputting the second image subjected to blurring on the first image into the historical model of the student model, the generated negative sample has similar image quality and consistent content to the anchor sample, and the negative sample provides stronger lower bound constraint on the model.

[0173] The teacher model can be a pre-trained model. Before training the student model, the pre-trained teacher model is loaded. It can be understood that the training sample set used for pre-training the teacher model can be the same or different from the sample set used for neural network distillation, which is not limited in the embodiments of the present application.

[0174] Based on the scheme provided in the embodiments of the present application, by inputting a plurality of second images subjected to blurring on the first image into the second model, a plurality of negative samples are obtained. On the one hand, only one model needs to be loaded to generate a plurality of negative samples, reducing the display storage overhead required for generating negative samples. On the other hand, the plurality of negative samples and the anchor sample have the same content and small quality difference, and provide stronger lower bound constraint on the model. The method provided in the embodiments of the present application can improve the performance of the student model obtained by training and the effect of image processing using the student model obtained by training.

[0175] In some possible implementation manners, the step S1050 can include:

[0176] S1050a, inputting the positive sample, the anchor sample and the plurality of negative samples into a feature encoder.

[0177] S1050b, determining the difference between the positive sample and the anchor sample, and the difference between the positive sample and the plurality of negative samples according to the feature encoder.

[0178] S1050c, updating the parameters of the third model according to the difference between the positive sample and the anchor sample, and the difference between the anchor sample and the plurality of negative samples.

[0179] The feature encoder can extract features of the input positive sample, the plurality of negative samples and the anchor sample, and perform contrastive distillation through an image encoding space. In the image encoding space, the anchor sample is “pulled” to a position closer to the positive sample, and is “pushed” to a position farther away from the plurality of negative samples. Through the effect of the positive sample and the plurality of negative samples on the anchor sample, the parameters of the student model can be updated, and the training of the student model can be realized.

[0180] Based on the scheme provided in the embodiments of the present application, the parameters of the third model are updated according to the differences between the positive sample and the anchor sample, and the plurality of negative samples and the anchor sample, so that the third model can learn the first model, and the performance of the third model can be improved.

[0181] Common encoders used for distillation include the network structure of visual geometry group (VGG), such as VGG16 and VGG19. The structure of VGG is simple and has strong applicability, and is suitable for transfer learning.

[0182] However, the semantic feature space of VGG has weak ability to model sample differences. In particular, for the case where the positive sample or the negative sample has consistent content with the anchor sample, but the quality is different, the semantic feature modeling sample difference has weak ability.

[0183] In some possible implementation manners, the feature encoder in step S1050b is an encoder in a vector quantized generative adversarial network (VQGAN).

[0184] VQGAN is a model combining vector quantization and generative adversarial network, which is used to generate high-quality images. In VQGAN, the encoder is a key component of the entire model, and its main function is to compress the input image into a low-dimensional latent space representation. The image encoding method of VQGAN retains more image detail information, which helps to model sample differences in a more fine-grained manner. The following are some core features and working principles of the encoder in VQGAN:

[0185] 1. Image compression: The encoder receives the input image and converts it into a lower-dimensional latent vector. This process can be seen as compressing the image information into a more compact representation for subsequent processing and generation.

[0186] 2. Feature extraction: The encoder is usually composed of multiple layers of convolutional networks, which can extract useful features from the input image. These features capture important information of the image, such as shape, texture, color, etc.

[0187] 3. Vector quantization: In VQGAN, the output of the encoder will pass through a vector quantization layer. The role of this layer is to discretize the continuous latent vector and map it to the nearest vector in a pre-defined codebook. The codebook is a finite size vector set, and each vector represents a point in the latent space.

[0188] 4. Information loss: Since the vector quantization process is irreversible, the encoder introduces some information loss when compressing the image information into latent vectors. This loss helps the model learn a more abstract and robust image representation.

[0189] 5. Collaboration with the generator: The output of the encoder (i.e., the quantized latent vector) is fed into the generator, which is tasked with reconstructing a high-quality image from this latent vector. The encoder and generator collaborate during training, optimizing to generate increasingly realistic images.

[0190] 6. Training stability: VQGAN introduces a regularization mechanism through vector quantization, which helps improve the stability of model training. Compared to traditional generative adversarial networks, VQGAN can generally generate higher-quality images and has a more stable training process.

[0191] 7. Wide range of applications: The encoder in VQGAN can be used not only for image generation tasks but also for image compression, image-to-image translation, image inpainting, and other image processing tasks.

[0192] In summary, the encoder in VQGAN is a key component for achieving high-quality image generation. It provides a rich latent space representation for the generator through effective image compression and feature extraction, enabling the entire model to generate realistic images.

[0193] Based on the scheme provided in the embodiments of the present application, the encoder in the VQGAN that retains more image detail information is used for encoding. The encoder in the VQGAN is more suitable for the case where the content of the positive and negative samples and the anchor sample is consistent, but the quality is different, can obtain a more fine-grained difference of the modeling sample, and helps to improve the performance of the third model.

[0194] In some possible implementation ways, the blurring processing includes: performing size-random Gaussian blurring on the first image to obtain a plurality of second images.

[0195] The above introduces a variety of common blurring techniques. However, the result obtained by using the blurring techniques such as mean blurring and median blurring to blur the first image may make the first image more blurred, the quality of the generated negative sample worse, and more detail information lost, which is contrary to the original intention of blurring processing: achieving a one-to-many correspondence relationship from one first image to multiple negative samples through blurring processing.

[0196] In the method 1000, a plurality of negative samples can be obtained by performing a size-random Gaussian blur on the first image. For example, the size of the Gaussian blur processing performed on the first image can be a random size such as 2*2, 3*3, or 5*5. The Gaussian blur can achieve a one-to-many correspondence from the first image to the plurality of negative samples by slightly disturbing the first image while avoiding losing more detailed information.

[0197] Based on the scheme provided in the embodiments of the present application, the first image is slightly disturbed by the size-random Gaussian blur. On the one hand, a plurality of second images can be obtained, and then a plurality of negative samples can be obtained, thereby enhancing the lower bound constraint on the model. On the other hand, the second image obtained by other blur processing methods can result in poor image quality, which can cause the performance of the third model to decrease.

[0198] In some possible implementation manners, the update step of the second model is S, S is a positive integer; and the second model in the current update is determined according to the second model in the last update and the third model in the current update.

[0199] In the method 1000, the student model learns the behavior of the teacher model through multiple iterations (updating parameters). In the method 1000, the second model is obtained according to the historical model of the third model to be trained. In order to delay the influence of the parameters of the student model on the second model, and avoid that the negative samples generated by the second model are closer to the positive samples than the anchor samples, the second model for generating the negative samples is not updated every time the student model updates the parameters. Instead, the second model is updated once every S times of updating the parameters of the student model. Moreover, the second model in the current update is determined according to the second model in the last update and the third model in the current update.

[0200] For example, when T represents the number of times of updating the third model, T=74 and S=25, the student model has been updated 74 times; when the student model completes the 50th update, the second model at this time is updated according to the parameters of the student model at this time and the parameters of the second model when the student model is updated for the 25th time. In the 26th to 49th and 51st to 74th updates of the student model, the parameters of the second model remain unchanged.

[0201] Based on the scheme provided in the embodiments of the present application, the second model is updated once every S times of updating the third model, which can delay the influence of the parameters of the student model on the second model, avoid that the negative samples generated by the second model are closer to the positive samples than the anchor samples, and cause the performance of the third model to decrease.

[0202] In some possible implementation manners, the parameters θ' of the second model in the current update satisfy: n satisfy: θ' n ​= ωθ n + (1-ω)θ s

[0203] wherein θ n denotes the parameter of the second model at the last update, θ s denotes the parameter of the third model at the current update, and ω≠0.

[0204] In some possible implementations, ω satisfies ω=0.1; and / or, T denotes the number of times of updating the third model, T satisfies T=100000n, S satisfies S=S'+1000, wherein S' corresponds to updating the third model for the T-Sth time to the T-1th time, S corresponds to updating the third model for the Tth time to the T+S-1th time, T is a positive integer, and n is a non-negative integer.

[0205] For example, the updating step of the second model is increased by 1000 every 100000 times of updating the student model. The initial updating step of the second model is 1000.

[0206] As the updating proceeds, the difference between the models at each update is smaller in the later stage of the updating than in the initial stage of the updating. Due to fluctuations, the negative samples generated in the later stage can be closer to the positive samples than the anchor samples.

[0207] In the process of updating the parameter of the second model based on the parameter of the student model, the updating step of the second model can be variable. For example, as the number of times of updating the student model increases, the updating step of the second model can be lengthened, and the influence of the parameter of the student model on the second model can be delayed, so as to avoid the negative samples generated by the second model being closer to the positive samples than the anchor samples. If the negative samples are closer to the positive samples than the anchor samples, the negative samples will push the anchor samples away from the positive samples, causing degradation of the performance of the model.

[0208] In the process of updating the parameter of the second model based on the parameter of the student model, ω can also be variable. For example, ω can increase as T increases.

[0209] For example, the parameter of the second model can be updated based on the parameter of the student model in an exponential moving average (EMA) manner. The EMA can help reduce the influence of short-term fluctuations of the parameter of the student model on the second model by giving a higher weight to the parameter of the second model after the last update.

[0210] Based on the scheme provided in the embodiments of the present application, the updating step of the second model is increased once every 100000 times of updating the third model, which can delay the influence of the parameter of the student model on the second model, avoid the negative samples generated by the second model being closer to the positive samples than the anchor samples, and cause degradation of the performance of the third model.

[0211] FIG. 4 is a schematic diagram of a flow 2000 of model training according to an embodiment of the present application. The flow 2000 can be used to implement the method 1000 described above.

[0212] As shown in (a) of FIG. 4, a low-quality image is input into a teacher model, a student model, and a dynamic negative sample generator, to generate a positive sample, an anchor sample, and a plurality of negative samples, respectively.

[0213] The parameters of the teacher model are fixed and unchangeable; the parameters of the student model are updated with iteration; the dynamic negative sample generator (DNSG) includes a cascaded blur module and a dynamic negative sample generator model (DNSGM). The blur module is used for Gaussian blur processing of the first image with random size, to generate N second images. The N second images are spliced along the batch dimension and input into the dynamic negative sample generator model, to generate N negative samples. The parameters of the dynamic negative sample generator model are updated according to the parameters of the student model every certain period of time. The teacher model can be a possible implementation of the first model, the student model can be a possible implementation of the third model, and the dynamic negative sample generator model can be a possible implementation of the second model.

[0214] As shown in (b) of FIG. 4, the positive sample, the anchor sample, and the plurality of negative samples are input into a feature encoder, and the output of the feature encoder is compared and regularized in an image coding space, to achieve distillation. For example, represents the output of the feature encoder input with the positive sample, represents the output of the feature encoder input with the anchor sample output by the student model updated for the T-th time, represents the output of the feature encoder input with the negative sample output by the dynamic negative sample generator model at the current time of updating, represents the output of the feature encoder input with the anchor sample output by the student model updated for the T-S-th time, represents the output of the feature encoder input with the negative sample output by the dynamic negative sample generator model at the last time of updating, represents the output of the feature encoder input with the anchor sample output by the student model updated for the T-2S-th time, and represents the output of the feature encoder input with the negative sample output by the dynamic negative sample generator model at the last two times of updating.

[0215] For example, when S = 1000 and T = 3000, represents the output of inputting the positive sample into the feature encoder, represents the output of inputting the anchor sample output by the student model updated for the 3000th time into the feature encoder without padding, represents the output of inputting the negative sample output by the dynamic negative sample generation model updated for the 3rd time into the feature encoder without padding, the 3rd update of the dynamic negative sample generation model is determined according to the parameters of the dynamic negative sample generation model updated for the 2nd time and the parameters of the student model updated for the 3000th time; represents the output of inputting the anchor sample output by the student model updated for the 2000th time into the feature encoder with cross-line padding, represents the output of inputting the negative sample output by the dynamic negative sample generation model updated for the 2nd time into the feature encoder with cross-line padding, the 2nd update of the dynamic negative sample generation model is determined according to the parameters of the dynamic negative sample generation model updated for the 1st time and the parameters of the student model updated for the 2000th time; represents the output of inputting the anchor sample output by the student model updated for the 1000th time into the feature encoder with point padding, represents the output of inputting the negative sample output by the dynamic negative sample generation model updated for the 1st time into the feature encoder with point padding, the 1st update of the dynamic negative sample generation model is determined according to the initial parameters of the dynamic negative sample generation model and the parameters of the student model updated for the 1000th time.

[0216] The plurality of second images are input into the dynamic negative sample generation model, and a plurality of negative samples can be obtained. Compared with using only low-quality images as negative samples, the plurality of negative samples are more; compared with using other samples in the same batch as negative samples, the content of the plurality of negative samples is consistent with that of the anchor sample. The plurality of negative samples can obtain a better constraint effect on the anchor sample. The plurality of negative samples respectively "push" the anchor sample in the image encoding space. As the number of student model updates increases, the anchor sample generated after the student model is updated is closer to the positive sample than the anchor sample generated before the student model is updated; as the number of dynamic negative sample generation model updates increases, the negative sample generated after the dynamic negative sample generation model is updated is closer to the positive sample than the negative sample generated before the dynamic negative sample generation model is updated.

[0217] The method and process of model training provided in the embodiments of the present application are described above in combination with FIG. 3 and FIG. 4. The student model obtained based on the method 1000 or the process 2000 can be used for image processing tasks. For example, image super-resolution, image defogging, image deblurring, image derain, and the like. The technical solutions of the method 1000 or the process 2000 do not need to be modified for different tasks.

[0218] An example method of image processing includes: determining an image to be processed; inputting the image to be processed into an image processing model to obtain a processed image, the image processing model being trained according to a plurality of negative samples, wherein the plurality of negative samples are obtained by inputting a plurality of second images obtained by performing blur processing on a first image into a second model, the second model being obtained according to a third model, the third model being a historical model of the image processing model.

[0219] In some possible implementation manners, the blur processing includes performing size-random Gaussian blur on the first image.

[0220] In some possible implementation manners, the image processing model is trained according to a difference between a positive sample and an anchor sample, and a difference between the anchor sample and the plurality of negative samples, the positive sample being obtained by inputting the first image into a first model, and the anchor sample being obtained by inputting the first image into the third model.

[0221] In some possible implementation manners, the difference between the positive sample and the anchor sample, and the difference between the anchor sample and the plurality of negative samples are obtained according to a feature encoder, the feature encoder being an encoder in a vector quantization generative adversarial network (VQGAN).

[0222] In some possible implementation manners, an update step of the second model is S, S being a positive integer; and the second model in a current update is determined according to the second model in a previous update and the third model in the current update.

[0223] In some possible implementation manners, a parameter θ' of the second model in the current update satisfies θ' = ωθ + (1-ω)θ', wherein θ represents the parameter of the second model in the previous update, θ' represents the parameter of the third model in the current update, and ω≠0. n satisfies θ' = ωθ + (1-ω)θ', wherein θ represents the parameter of the second model in the previous update, θ' represents the parameter of the third model in the current update, and ω≠0. n = ωθ n +(1-ω)θ s , wherein θ n represents the parameter of the second model in the previous update, θ s represents the parameter of the third model in the current update, and ω≠0.

[0224] In some possible implementation manners, ω satisfies ω = 0.1; and / or, T represents a number of times of updating the third model, T satisfies T = 100000n, S satisfies S = S' + 1000, wherein S' corresponds to updating the third model for the T-Sth time to the (T-1)th time, S corresponds to updating the third model for the Tth time to the (T+S-1)th time, T is a positive integer, and n is a non-negative integer.

[0225] The low-quality picture is input into the student model trained by the method 1000 or the flow 2000, and a high-quality picture can be output.

[0226] The student model corresponding to Tables 1 to 5 and FIG. 5 below is trained according to the method 1000 and the following conditions: using the pre-trained Encoder in the VQGAN method as the feature encoder, and selecting 5 different scale feature layers in the image coding space for contrastive distillation; the initial S = 1000, and S increases by 1000 every 100,000 times of training of the student model; θ' S n = ωθ n +(1-ω)θ s , wherein θ' S n represents the parameters of the DNSGM after this update, θ n represents the parameters of the DNSGM before this update (i.e., the parameters of the DNSGM after the last update), θ s represents the parameters of the student model at the time of this update, and ω satisfies ω = 0.1.

[0227] The results of image processing by the student model trained by the above method are shown in Tables 1 to 5 and FIG. 5 (Tables 1 to 5 are the results of 16 times down-sampling processing).

[0228] Table 1

[0229] As shown in Table 1, Table 1 shows the performance comparison of the Transformer structure on public datasets.

[0230] The state of the art (SOTA) refers to the best performance that can be achieved by current technology or methods in a certain field or task. In the fields of computer vision, machine learning and other technologies, SOTA performance represents the most advanced research results and the most efficient solutions in the field.

[0231] On the SwinIR model, the method 1000 achieves SOTA performance on multiple public datasets at three different scales (see the bolded part in Table 1). On the Set14 dataset of *2, the method 1000 improves the PSNR by 0.21 dB compared to scratch, and improves the PSNR by 0.1 dB compared to the SOTA method of DUKD (see the italic part in Table 1).

[0232] Table 2

[0233] As shown in Table 2, Table 2 shows the performance comparison of the CNN structure on public datasets.

[0234] On RCAN model, the method 1000 achieves SOTA performance on multiple public datasets at three different scales (see Table 2, bolded part). On the Urban100 dataset of *2, the method 1000 improves the PSNR by 0.19 dB compared with scratch, and improves the PSNR by 0.05 dB compared with the SOTA method of DUKD (see Table 2, italic part).

[0235] Table 3

[0236] As shown in Table 3, Table 3 shows the performance comparison of heterogeneous distillation on public datasets.

[0237] The method 1000 is applicable to heterogeneous distillation, where the types and structures of the teacher model and the student model can be different. Taking SwinIR as the teacher model and RCAN as the student model as an example, the method 1000 can still significantly improve the performance of the student model. On the Manga109 dataset of *3, the method 1000 improves the PNSR by 0.23 dB compared with scratch (see Table 3, italic part).

[0238] FIG. 5 is a schematic diagram of a result of image processing provided by an embodiment of the present application.

[0239] As shown in FIG. 5(a), on the Urban100 dataset, the method 1000 can more accurately and clearly restore the texture details of the image compared with other methods. The student model trained by the method 1000 has fewer artifacts (diagonal lines in the picture in FIG. 5(a)) in the processed picture, and the horizontal and vertical lines are clear.

[0240] FIG. 5(b) is a schematic diagram of another result of image processing provided by an embodiment of the present application.

[0241] As shown in FIG. 5(b), on the Urban100 dataset, the method 1000 can more accurately and clearly restore the texture details of the image compared with other methods. The student model trained by the method 1000 has more accurate zebra lines in the processed picture (the zebra lines of the pictures processed by different models in FIG. 5(a) have different degrees of blur, and the area ratio of the blurred part of the zebra lines in the picture processed by the method 1000 is smaller).

[0242] Table 4

[0243] Table 4 shows the performance comparison of VQGAN and DNSG ablation.

[0244] Ablation Study is an experimental method used to determine the contribution of different components in a model to the final performance. By gradually removing or modifying certain parts of the model, researchers can assess the impact of these parts on the model's performance.

[0245] As shown in Table 4, on Urban100, VQGAN improves PSNR by 0.05dB compared to logits method (see bolded part in Table 4). Combined with DNSG, it further improves by 0.04dB (see italic part in Table 4).

[0246] Table 5

[0247] Table 5 shows the performance comparison of the student model obtained by method 1000 and other method ablation.

[0248] As shown in Table 5, as the feature encoder, VQGAN improves the PSNR of our DNSG contrastive model, model contrastive learning for image restoration (MCLIR), negative samples generator (NSG) by 0.07dB (see bolded part in Table 5). As the negative sample generator, DNSG improves the PSNR of VQGAN as the feature encoder compared to VGG by 0.05dB (see italic part in Table 5).

[0249] FIG. 6 is a schematic diagram of a device 600 for model training or image processing according to an embodiment of the present application. As shown in FIG. 6, the device 600 can be a model training device with a model training function or an image processing device with an image processing function, or a component (e.g., a unit, a module, a chip, or a chip system) configured in the model training device or the image processing device. The device 600 includes a processing unit 620, and optionally, a transceiver unit 610. The transceiver unit 610 can be configured to perform a transceiving function. The transceiver unit 610 can also be referred to as a communication interface or a communication unit. The processing unit 620 can be configured to process a to-be-processed image or a first image.

[0250] Optionally, the device 600 can further include a storage unit, which can be configured to store instructions and / or data. The processing unit 620 can read the instructions and / or data in the storage unit, so that the device implements the foregoing method embodiments.

[0251] Exemplarily, the apparatus 600 can be a model training device with a model training function, or a model training apparatus applied to a model training device, or matched with the model training device, capable of implementing the method executed by the model training device, such as a chip, a chip system or a circuit. For details, refer to the related description of the chip system shown in FIG. 8.

[0252] Exemplarily, the apparatus 600 can be an image processing device with an image processing function, or an image processing apparatus applied to an image processing device, or matched with the image processing device, capable of implementing the method executed by the image processing device, such as a chip, a chip system or a circuit. For details, refer to the related description of the chip system shown in FIG. 8.

[0253] As a design, the apparatus 600 can be configured to execute the steps or processes executed by the method embodiments of FIG. 3. The processing unit 620 is configured to execute the processing-related operations in the method embodiments of FIG. 3 (for example, the above steps S1010-S1050). The transceiver unit 610 can be configured to receive images, and the processing unit 620 can take the images received by the transceiver unit 610 as the first images; or the processing unit 620 can take one of the images received by the transceiver unit 610 as the first image.

[0254] As a design, the transceiver unit 610 can also be configured to receive images to be processed, and the processing unit 620 can be configured to input the images to be processed into the image processing model to obtain processed images.

[0255] It should be understood that the specific processes of the units in executing the above corresponding steps have been described in detail in the above method embodiments, and thus will not be described here again for the sake of brevity.

[0256] It should also be understood that the apparatus 600 here is embodied in the form of functional units. The term “unit” here can refer to an application specific integrated circuit (ASIC), an electronic circuit, a processor (for example, a shared processor, a dedicated processor or a group processor, etc.) and a memory for executing one or more software or firmware programs, a combination logic circuit and / or other suitable components supporting the described functions.

[0257] The apparatus 600 of each of the above solutions can have functions to implement corresponding steps in the above method 1000. The functions can be implemented by hardware, or implemented by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions; for example, the transceiver unit can be replaced by a transceiver (for example, the transmitting unit in the transceiver unit can be replaced by a transmitter, and the receiving unit in the transceiver unit can be replaced by a receiver), and other units such as the processing unit / image processing unit can be replaced by a processor, which respectively performs the transceiving operation and related processing operation in each method embodiment.

[0258] In addition, the transceiver unit 610 described above can also be a transceiver circuit (for example, can include a receiving circuit and a transmitting circuit), and the processing unit can be a processing circuit.

[0259] It should be noted that the apparatus in FIG. 6 can be the model training device / image processing device in the foregoing embodiments, or can be a chip or a chip system, for example, a system on chip (SoC). The transceiver unit can be an input / output circuit, a communication interface; and the processing unit can be a processor or a microprocessor integrated on the chip or an integrated circuit. In this regard, no limitation is made.

[0260] FIG. 7 is a schematic diagram of an apparatus 700 for model training or image processing according to an embodiment of the present application. As shown in FIG. 7, the apparatus 700 includes a processor 710. Optionally, the apparatus 700 further includes a communication interface 730, which is configured to receive and / or send signals or data, and the processor 710 is configured to process the received signals or data. For example, the processor 710 is configured to process the signals or data received and / or sent by the communication interface 730.

[0261] Optionally, the apparatus 700 further includes a memory 720, and the processor 710 is coupled to the memory 720. The memory 720 is configured to store programs or instructions and / or data of model training or image processing, and the processor 710 is configured to execute the programs or instructions of model training or image processing stored in the memory 720, or read the data stored in the memory 720, to perform the methods in the above method embodiments. For example, the processor 710 can obtain the data or algorithm stored in the memory 720 to perform the above model training or image processing.

[0262] Optionally, the processor 710 is one or more.

[0263] Optionally, the memory 720 is one or more.

[0264] Optionally, the memory 720 and the processor 710 are integrated together, or are separately arranged.

[0265] As an example, the processor 710 can have the functions of the processing unit 620 shown in FIG. 6, the memory 720 can have the functions of a storage unit, and the communication interface 730 can have the functions of the transceiver unit 610 shown in FIG. 6.

[0266] As an example, the apparatus 700 can be an image processing device, or an image processing apparatus, such as a chip, a chip system, or a circuit, applied to or matched with the image processing device, and capable of implementing a data processing method performed by the image processing device. For details, refer to the related description of the chip system shown in FIG. 8.

[0267] As an example, the apparatus 700 can be a model training device, or a model training apparatus, such as a chip, a chip system, or a circuit, applied to or matched with the model training device, and capable of implementing a model training method performed by the model training device. For details, refer to the related description of the chip system shown in FIG. 8.

[0268] As a design, the apparatus 700 is configured to perform the steps or processes in the method embodiments of FIG. 3, the communication interface 730 is configured to perform the transceiving-related operations in the method embodiments, and the processor 710 is configured to perform the processing-related operations in the method embodiments of FIG. 3 (for example, determining a to-be-processed image; inputting the to-be-processed image into an image processing model to obtain a processed image).

[0269] It should be understood that the processor mentioned in the embodiments of the present application can be a device or a part of circuit for processing function in the following devices: a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), ASICs, field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as hardware code processing performed by the processor, or executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 720, and the processor 710 reads the information in the memory 720, and combines the hardware to complete the functions required by the units included in the electronic device, or executes the data processing method of the method embodiments of the present application.

[0270] The communication interface uses a transceiving device such as, but not limited to, a transceiver, to enable communication between the apparatus 700 and other devices or communication networks. For example, the to-be-processed data can be obtained through the communication interface.

[0271] The bus can include a pathway for transmitting information between various components of the apparatus 700 (for example, the memory 720, the processor 710, the communication interface 730).

[0272] Next, the chip system in the image processing device / model training device is described in conjunction with FIG. 8.

[0273] FIG. 8 is a schematic diagram of a chip system 800 provided in an embodiment of the present application. The chip system 800 (or also referred to as a processing system) includes a logic circuit 810 and an input / output interface 820.

[0274] The logic circuit 810 can be a processing circuit in the chip system 800. The logic circuit 810 can be coupled to a storage unit, and invoke instructions in the storage unit, so that the chip system 800 can implement the methods and functions of the embodiments of the present application. The input / output interface 820 can be an input / output circuit in the chip system 800, and output information processed by the chip system 800, or input to-be-processed data or signaling information into the chip system 800 for processing.

[0275] For example, if the image processing device is installed with the chip system 800, the logic circuit 810 is coupled to the input / output interface 820, and the input / output interface 820 can input the input information to the logic circuit 810 for processing.

[0276] The embodiments of the present application provide a computer readable storage medium, which stores computer instructions for implementing the method executed by the image processing device in each of the above method embodiments.

[0277] For example, the computer program is executed by a computer, so that the computer can implement the method executed by the image processing device / model training device in each of the above method embodiments.

[0278] The embodiments of the present application provide a computer program product, which includes instructions executed by a computer to implement the method executed by the image processing device / model training device in each of the above method embodiments.

[0279] The explanations and beneficial effects of the related contents in any of the above provided devices can refer to the corresponding method embodiments provided above, and will not be described herein again.

[0280] Those skilled in the art can clearly understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0281] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0282] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the above-described device embodiments are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0283] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0284] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically, or two or more units can be integrated into one unit.

[0285] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the technical solutions that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0286] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for model training, the method comprising: The method comprises: inputting a first image into a first model to obtain a positive sample; performing blur processing on the first image to obtain a plurality of second images; inputting the plurality of second images into a second model to obtain a plurality of negative samples, the second model being obtained according to a historical model of a third model to be trained; inputting the first image into the third model to obtain an anchor sample; training the third model according to the positive sample, the anchor sample and the plurality of negative samples.

2. The method of claim 1, wherein, The method for training the third model according to the positive sample, the anchor sample and the plurality of negative samples comprises: inputting the positive sample, the anchor sample and the plurality of negative samples into a feature encoder; determining a difference between the positive sample and the anchor sample, and a difference between the positive sample and the plurality of negative samples according to the feature encoder; updating parameters of the third model according to the difference between the positive sample and the anchor sample, and the difference between the anchor sample and the plurality of negative samples.

3. The method of claim 2, wherein: the feature encoder is an encoder in a vector quantization generative adversarial network (VQGAN).

4. The method according to any one of claims 1 to 3, characterized in that, The blur processing comprises: performing random-size Gaussian blur on the first image to obtain the plurality of second images.

5. The method of any one of claims 1 to 4, wherein: an update step of the second model is S, S being a positive integer; the second model at a current update is determined according to the second model at a previous update and the third model at the current update.

6. The method of claim 5, wherein, parameters θ' of the second model at the current update n satisfies: θ' = ωθ + (1 - ω)θ n = ωθ n +(1-ω)θ s where θ n denotes the parameters of the second model at the last update, θ s denotes the parameters of the third model at the current update, ω≠0.

7. The method of claim 6, wherein: ω satisfies ω = 0.1; and / or T represents a number of times of updating the third model, T satisfies T = 100000n, S satisfies S = S' + 1000, wherein S' corresponds to updating the third model for the T-Sth to (T-1)th time, S corresponds to updating the third model for the Tth to (T+S-1)th time, T is a positive integer, and n is a non-negative integer.

8. A method of image processing, characterized by, The method comprises: determining a to-be-processed image; inputting the to-be-processed image into an image processing model to obtain a processed image, the image processing model being trained according to a plurality of negative samples, wherein the plurality of negative samples are obtained by inputting a plurality of second images obtained by performing blur processing on a first image into a second model, the second model being obtained according to a third model, and the third model being a historical model of the image processing model.

9. The method of claim 8, wherein: the blur processing comprises performing random-size Gaussian blur on the first image.

10. The method of claim 8 or 9, wherein: the image processing model is trained according to a difference between a positive sample and an anchor sample, and a difference between the anchor sample and the plurality of negative samples, the positive sample being obtained by inputting the first image into a first model, and the anchor sample being obtained by inputting the first image into the third model.

11. The method of claim 10, wherein: The difference between the positive sample and the anchor sample, and the difference between the anchor sample and the plurality of negative samples are obtained according to a feature encoder, which is an encoder in a vector quantization generative adversarial network (VQGAN).

12. An apparatus for model training, the apparatus comprising: The processor is configured to: input the first image into a first model to obtain a positive sample; blur process the first image to obtain a plurality of second images; input the plurality of second images into a second model to obtain a plurality of negative samples, the second model being obtained according to a history model of a third model to be trained; input the first image into the third model to obtain an anchor sample; train the third model according to the positive sample, the anchor sample and the plurality of negative samples.

13. The apparatus of claim 12, wherein, The device is a chip.

14. A computer program product, characterised in that, When the computer program in the computer program product is executed by a computing device, the method of any one of claims 1-7, or the method of any one of claims 8-11 is implemented.

15. A computer-readable storage medium, characterized in that, The storage medium stores a computer program or instructions, and when the computer program or instructions are executed by a computing device, the method of any one of claims 1-7, or the method of any one of claims 8-11 is implemented.

Images