A model training method, image enhancement method and device

Abstract

The embodiment of the present application discloses a model training method, image enhancement method and device, which can be applied to the image processing field in the field of artificial intelligence, and can be specifically used for super-resolution reconstruction tasks. The method reconstructs a set of loss functions for the model composed of a generative network, a teacher network, and a student network, which is used to improve the training effect of the generative network and the student network. In terms of improving the training effect of the generative network, based on the characteristics of the super-resolution reconstruction task (i.e., the super-resolution image has all the information of the low-resolution image and contains more detailed information), a loss function for training the generative network is constructed for the super-resolution reconstruction task, which improves the training effect of the model on the super-resolution reconstruction task; in terms of improving the training effect of the student network, since the knowledge distillation without data is very difficult, in order to reduce the difficulty of distillation, the student network is trained by progressive distillation until the entire student network is trained, which reduces the difficulty of distillation.

Description

Technical Field

[0001] The present application relates to the field of machine learning, and in particular to a model training method, image enhancement method and device. Background Art

[0002] In order to apply neural networks to small mobile devices with limited computing resources (such as mobile phones), they need to be compressed and accelerated. Knowledge distillation is a model compression technique that distills the feature representation "knowledge" learned by a complex and powerful network and transfers it to a network with a small number of parameters and weak learning ability. Knowledge distillation can transfer the knowledge of one network to another network by first training a teacher network with training data, and then using the output of this teacher network to train a student network.

[0003] Since neural network compression usually requires training data, in actual applications, due to restrictions such as privacy policies and laws, training data is often not available. Currently, there is a lot of work dedicated to developing data-free model compression techniques. These data-free model compression techniques are generally composed of a model framework consisting of a generator network, a teacher network, and a student network, and the student network is trained using distillation technology. The training process is as follows: the generator network receives random noise and then generates images, which are simultaneously input into the teacher network and the student network. Then the student network is trained using distillation technology so that the output of the student network is similar to the output of the teacher network, and the generator network is trained so that the distribution of the images generated by the generator network is similar to the distribution of the data set.

[0004] Current data compression-free techniques mainly focus on classification and segmentation tasks. Summary of the invention

[0005] The embodiments of the present application provide a model training method, an image enhancement method and a device. The model includes a generative network, a teacher network and a student network. A new loss function (i.e., a first loss function) for training the generative network is proposed for super-resolution reconstruction tasks, thereby improving the training effect of the model on super-resolution reconstruction tasks.

[0006] Based on this, the embodiments of the present application provide the following technical solutions:

[0007] In the first aspect, the embodiment of the present application first provides a model training method, which can be used in the field of artificial intelligence. The method includes: first, the training device uses the constructed first loss function and the second loss function to train the generative network in the model, wherein the first loss function is used to characterize the difference between the first image and the second image, the first image is an image generated by the generative network according to the input random initialization noise, the second image is an image obtained by downsampling the third image, and the third image is an image obtained by super-resolution reconstruction of the input first image by the teacher network. It should be noted that in the embodiment of the present application, the teacher network is a pre-trained neural network, and its model parameters remain unchanged, while the model parameters of the generative network during the first training can be randomly initialized. After the training device performs an iterative training on the generative network, it will obtain the generative network after this round of training. After that, the training device further uses the third loss function to train the student network in the model, wherein the third loss function is used to characterize the difference between the fourth image and the fifth image, the fifth image is an image obtained by super-resolution reconstruction of the input sixth image by the teacher network, the sixth image is an image generated by the trained generative network according to the input random initialization noise, and the fourth image is an image obtained by super-resolution reconstruction of the input sixth image by the student network. It should also be noted that in the embodiment of the present application, the model parameters of the teacher network remain unchanged, while the model parameters of the generator network are the generator network obtained by the above training, and the model parameters of the student network during the first training can also be randomly initialized. Finally, the training device repeats the above process of alternating training of the generator network and the student network until the iteration termination condition (which can be called the first iteration termination condition) is reached.

[0008] In the above-mentioned embodiment of the present application, a model training method is provided, which includes a generative network, a teacher network and a student network. For a specific super-resolution reconstruction task, a new loss function (i.e., a first loss function) for training the generative network is proposed, which improves the training effect of the model on the super-resolution reconstruction task.

[0009] In a possible implementation of the first aspect, the training of the student network can be performed in layers. Specifically, the training device uses a first loss function and a second loss function to train the generating network. The first loss function is used to characterize the difference between the first image and the second image. The first image is an image generated by the generating network according to the input random initialization noise, the second image is an image obtained by downsampling the third image, and the third image is an image obtained by super-resolution reconstruction of the input first image by the teacher network. The second loss function is related to the third loss function. After the training device uses the first loss function and the second loss function to train the generating network for a round, it will first use the third loss function to train only the first sublayer (which can be called Bx) and the second neural network layer (which can be called T) of the student network. In the embodiment of the present application, the subnetwork composed of the first sublayer Bx and the second neural network layer T of the student network can be called the student network S1. In this round of training, the model parameters of the first sublayer Bx and the initialized model parameters of the second neural network layer T are all randomly initialized values. The training device repeats the above steps until the first iteration termination condition is reached. After the first iteration termination condition is reached, it is considered that the training of the student network S1 is completed. After the training device has gone through the above steps, it obtains the trained student network S1, that is, the training device can obtain the model parameter value of the first sublayer after training (which can be called the first model parameter value) and the model parameter value of the second neural network layer after training (which can be called the second model parameter value). After that, the training device continues to train the generated network using the first loss function and the second loss function, and uses the third loss function to train the trained first sublayer Bx, the second sublayer (which can be called By) and the trained second neural network layer T. In this embodiment of the application, the subnetwork composed of the first sublayer Bx, the second sublayer By and the second neural network layer T of the student network can be called the student network S2. In this round of training, the initial values ​​of the model parameters of the first sublayer Bx and the model parameters of the second neural network layer T are the first model parameter values ​​and the second model parameter values ​​obtained by the above training, respectively. Since the second sublayer By was not trained in the previous round, the initialized model parameters of the second sublayer By are randomly initialized values. Finally, the training device repeatedly executes the process of training the generated network using the first loss function and the second loss function and training the trained first sublayer, the second sublayer and the trained second neural network layer using the third loss function until the second iteration termination condition is reached. In the embodiment of the present application, since the student network is only divided into two stages for training, namely, the student network S1 and the student network S2, when the training of the student network S2 is completed, it means that the training of the entire student network is completed, so that the values ​​of the model parameters of the entire student network can be obtained.

[0010] In the above implementation of the present application, the student network is divided into two parts, one part is the first neural network layer, and the other part is the second neural network layer. In the process of progressive training, the first sublayer and the second neural network layer of the first neural network layer are first trained (at this time, the model parameters of the second sublayer of the first neural network layer are kept unchanged). After training this part, the model parameter values ​​of the first sublayer and the second neural network layer of the first neural network layer (obtained after training) are used as the initialization values ​​of the model parameters of the first sublayer and the second neural network layer of the first neural network layer of the entire student network (the values ​​of the model parameters of the second sublayer are randomly initialized values), and the first neural network layer and the second neural network layer are continued to be trained, thereby obtaining the final trained student network. This progressive training process reduces the difficulty of training the student network.

[0011] In a possible implementation manner of the first aspect, reaching the second iteration termination condition includes: reaching a preset training round, or the third loss function converges, or the third loss function reaches a preset threshold.

[0012] In the above-mentioned implementation manner of the present application, several methods for determining the second iteration termination condition are described, which are flexible and widely applicable.

[0013] In a possible implementation manner of the first aspect, reaching the first iteration termination condition includes: reaching a preset training round, or the third loss function converges, or the third loss function reaches a preset threshold.

[0014] In the above-mentioned implementation manner of the present application, several methods for determining the termination conditions of the first iteration are also described, which are flexible and widely applicable.

[0015] In a possible implementation of the first aspect, a plurality of randomly initialized noises are generally input into the generative network together during training, so that a plurality of first images can be obtained. For example, when the number of randomly initialized noises is n (n≥2), the number of first images and second images obtained is n. In this case, the first loss function is used to characterize the average difference or mean square error between the n first images and the n second images.

[0016] In the above-mentioned implementation mode of the present application, it is specifically explained how the first loss function is represented when there are multiple first images and second images, which covers a variety of practical situations and is feasible.

[0017] In a possible implementation of the first aspect, when the randomly initialized noise is m (m≥2), the fourth image and the fifth image obtained are both m. In this case, the third loss function is used to characterize the average difference or mean square error between the m fourth images and the m fifth images.

[0018] In the above-mentioned implementation mode of the present application, it is specifically explained how the third loss function is represented when there are multiple fourth images and fifth images, which covers a variety of practical situations and is feasible.

[0019] In a possible implementation of the first aspect, the training device can also deploy the trained student network on the target device. For example, it can be deployed on edge devices with limited computing resources such as mobile phones, smart wearable devices (such as smart bracelets, smart watches, etc.). This is because the network size of the pre-trained teacher network is generally large and is not suitable for deployment on edge devices with limited computing power, while the network size of the student network is generally small and suitable for deployment.

[0020] In the above implementation of the present application, the student network obtained after training can be deployed on the target device for practical application. Generally, since the network size of the student network is smaller than that of the teacher network, the inference speed of the target device can be improved and the user experience can be enhanced.

[0021] The second aspect of the embodiment of the present application further provides an execution device, the method comprising: first, the execution device can obtain an input image to be super-resolution reconstructed, and the input image can be called a target image. Afterwards, the execution device performs super-resolution reconstruction on the target image through a trained student network, thereby obtaining a reconstructed enhanced image. It should be noted that in the embodiment of the present application, the trained student network is the student network trained in the above training stage.

[0022] In the above-mentioned implementation mode of the present application, it is explained that the student network obtained after training is deployed on the target device to perform super-resolution reconstruction tasks. Generally, since the network size of the student network is smaller than that of the teacher network, the trained student network can improve the inference speed of the target device and enhance the user experience.

[0023] A third aspect of the embodiments of the present application provides a training device, which has the function of implementing the method of the first aspect or any possible implementation of the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0024] In a fourth aspect, an embodiment of the present application provides an execution device, which has the function of implementing the method of the second aspect or any possible implementation of the second aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0025] The fifth aspect of the embodiment of the present application provides a training device, which may include a memory, a processor and a bus system, wherein the memory is used to store programs, and the processor is used to call the programs stored in the memory to execute the method of the first aspect of the embodiment of the present application or any possible implementation method of the first aspect.

[0026] The sixth aspect of the embodiments of the present application provides an execution device, which may include a memory, a processor, and a bus system, wherein the memory is used to store programs, and the processor is used to call the programs stored in the memory to execute the method of the second aspect of the embodiments of the present application or any possible implementation method of the second aspect.

[0027] In a sixth aspect, the present application provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on a computer, the computer can execute the method of the first aspect or any possible implementation of the first aspect, or the computer can execute the method of the second aspect or any possible implementation of the second aspect.

[0028] The seventh aspect of the embodiment of the present application provides a computer program, which, when running on a computer, enables the computer to execute the method of the first aspect or any possible implementation of the first aspect, or enables the computer to execute the method of the second aspect or any possible implementation of the second aspect.

[0029] The eighth aspect of the embodiment of the present application provides a chip, which includes at least one processor and at least one interface circuit, the interface circuit is coupled to the processor, the at least one interface circuit is used to perform a transceiver function, and send instructions to at least one processor, at least one processor is used to run a computer program or instruction, which has the function of implementing the method as described in the first aspect or any possible implementation of the first aspect, or, it has the function of implementing the method as described in the second aspect or any possible implementation of the second aspect, the function can be implemented by hardware, can be implemented by software, can also be implemented by a combination of hardware and software, the hardware or software includes one or more modules corresponding to the above functions. In addition, the interface circuit is used to communicate with other modules outside the chip. For example, the interface circuit can send the student network obtained after training on the chip to various edge devices (such as mobile phones, smart watches, smart glasses, etc.) to perform super-resolution image reconstruction tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic diagram of the structure of the artificial intelligence main framework provided in the embodiment of the present application;

[0031] Figure 2 A schematic diagram of a model framework provided for an embodiment of the present application;

[0032] Figure 3 A schematic diagram of the overall framework of the image enhancement system provided in an embodiment of the present application;

[0033] Figure 4 A flowchart of a training method for a model provided in an embodiment of the present application;

[0034] Figure 5 Another schematic diagram of a flow chart of a training method for a model provided in an embodiment of the present application;

[0035] Figure 6 A schematic diagram of the structural decomposition of the student network provided in an embodiment of the present application;

[0036] Figure 7 A schematic diagram of performing progressive training on a student network provided in an embodiment of the present application;

[0037] Figure 8 Another schematic diagram of performing progressive training on a student network provided in an embodiment of the present application;

[0038] Fig. 9 A system architecture diagram of an application provided in an embodiment of the present application;

[0039] Fig.10 A schematic diagram of a flow chart of an image enhancement method provided in an embodiment of the present application;

[0040] Fig.11 A schematic diagram of an application scenario provided by an embodiment of the present application;

[0041] Fig.12 A schematic diagram of a training device provided in an embodiment of the present application;

[0042] Fig.13 A schematic diagram of an execution device provided in an embodiment of the present application;

[0043] Fig.14 Another schematic diagram of the training device provided in the embodiment of the present application;

[0044] Fig.15 Another schematic diagram of an execution device provided in an embodiment of the present application;

[0045] Fig.16 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The embodiments of the present application provide a model training method, an image enhancement method and a device. The model includes a generative network, a teacher network and a student network. A new loss function (i.e., a first loss function) for training the generative network is proposed for super-resolution reconstruction tasks, thereby improving the training effect of the model on super-resolution reconstruction tasks.

[0047] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0048] The embodiments of the present application involve a lot of relevant knowledge about neural networks. In order to better understand the solutions of the embodiments of the present application, the relevant terms and concepts that may be involved in the embodiments of the present application are introduced below. It should be understood that the explanation of the relevant concepts may be limited due to the specific circumstances of the embodiments of the present application, but it does not mean that the present application is limited to the specific circumstances. There may be differences in the specific circumstances of different embodiments, which are not specifically limited here.

[0049] (1) Neural Network

[0050] A neural network can be composed of neural units. Specifically, it can be understood as a neural network with an input layer, a hidden layer, and an output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. Among them, a neural network with many hidden layers is called a deep neural network (DNN). The work of each layer in the neural network can be expressed mathematically. To describe, from a physical level, the work of each layer in the neural network can be understood as completing the transformation from input space to output space (i.e., from the row space to the column space of the matrix) through five operations on the input space (the set of input vectors). These five operations include: 1. Dimension increase / reduction; 2. Enlargement / reduction; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are represented by Completed, operation 4 is completed by "+b", and operation 5 is implemented by "a()". The word "space" is used here because the classified object is not a single thing, but a class of things. Space refers to the collection of all individuals of this class of things, where W is the weight matrix of each layer of the neural network, and each value in the matrix represents the weight value of a neuron in the layer. The matrix W determines the spatial transformation from the input space to the output space described above, that is, the W of each layer of the neural network controls how to transform the space. The purpose of training a neural network is to eventually obtain the weight matrices of all layers of the trained neural network. Therefore, the training process of a neural network is essentially about learning how to control spatial transformation, or more specifically, learning the weight matrix.

[0051] (2) Loss function

[0052] In the process of training a neural network, because we hope that the output of the neural network is as close as possible to the value we really want to predict, we can compare the current network's predicted value with the target value we really want, and then update the weight matrix of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the neural network). For example, if the network's predicted value is high, adjust the weight matrix to make it predict lower, and keep adjusting until the neural network can predict the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the neural network becomes a process of minimizing this loss as much as possible.

[0053] (3) Back propagation algorithm

[0054] In the training process of the neural network, the error back propagation (BP) algorithm can be used to correct the size of the parameters in the initial neural network model, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the initial neural network model are updated by back propagating the error loss information, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, which aims to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0055] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0056] First, the overall workflow of the AI ​​system is described in Figure 1 , Figure 1 The figure shows a structural diagram of the main framework of artificial intelligence. The following is an explanation of the above artificial intelligence theme framework from the two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has undergone a condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the industrial ecology process of the system.

[0057] (1) Infrastructure

[0058] The infrastructure provides computing power support for the AI ​​system, enables communication with the outside world, and supports it through the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips (CPU, NPU, GPU, ASIC, FPGA and other hardware acceleration chips); the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0059] (2) Data

[0060] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.

[0061] (3) Data processing

[0062] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

[0063] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0064] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0065] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

[0066] (4) General ability

[0067] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0068] (5) Smart products and industry applications

[0069] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical applications. Its application areas mainly include: smart terminals, smart manufacturing, smart transportation, smart homes, smart medical care, smart security, autonomous driving, smart cities, etc.

[0070] The embodiments of the present application can be applied to the optimization design of a model framework composed of a generating network, a teacher network, and a student network, and can be specifically applied to the optimization design of the loss function of the model framework. The model framework whose loss function is optimized by the present application can be specifically applied to the computer vision field in the field of artificial intelligence, and can be specifically applied to the super-resolution reconstruction field in the field of image processing. Specifically, combined with Figure 1 Generally speaking, the data in the data set acquired by the infrastructure in the embodiments of the present application can be a plurality of image data acquired through sensors such as surveillance cameras and mobile phone camera modules (also referred to as training samples or training data, and a plurality of training data constitute a training set) or image data obtained based on the acquired video data. As long as the training set meets the function of iteratively training the neural network, the specific embodiments of the present application do not limit the data type in the training set.

[0071] In order to facilitate understanding of this solution, firstly, the principle of the model training method provided in the embodiment of the present application and the model framework involved are introduced. Figure 2 , Figure 2The model framework 200 provided in the embodiment of the present application (in the following embodiments, the model framework may also be referred to as the model) is composed of a generation network 201, a teacher network 202, and a student network 203. In the training stage, the generation network 201 is used to generate an image from randomly initialized noise. The randomly initialized noise can be recorded as z, and the image generated by the generation network 201 can be recorded as G(z). The generated image G(z) is further input to the teacher network 202. The teacher network 202 is a pre-trained neural network. The teacher network 202 can perform super-resolution reconstruction on the input image G(z), and then iteratively train the generation network 201 according to the first loss function and the second loss function constructed in the embodiment of the present application. It should be noted that during the training of the generation network 201, the model parameters of the student network 203 are kept constant. The student network 203 remains unchanged, and can also be called a fixed student network 203 (the teacher network 202 has been pre-trained, and its model parameters remain unchanged); the image generated by the randomly initialized noise of the generative network 201 after training can be recorded as G(z)′, and then the image G(z)′ generated by the trained generative network 201 is input into the student network 203, and the student network 203 can perform super-resolution reconstruction on the input image G(z)′, and then iteratively train the student network 203 according to the third loss function constructed according to the embodiment of the present application. It should also be noted that during the training of the student network 203, the model parameters of the generative network 201 remain unchanged, that is, the generative network 201 is fixed; finally, in the process of alternating training of the generative network 201 and the student network 203, the iterative termination condition of the training is reached, and at this time, it is considered that the training of the model framework 200 is completed.

[0072] It should be noted that the embodiments of the present application do not limit the network types of the generating network, the teacher network, and the student network. For example, they can be convolutional neural networks (CNN) or recurrent neural networks (RNN), and the specific selection is made by the user according to the target task to be performed.

[0073] Next, combine Figure 3 For an introduction to the overall framework of the image enhancement system provided in the embodiment of the present application, please refer to Figure 3 , Figure 3 A system architecture diagram of a task processing system provided in an embodiment of the present application, in Figure 3In the embodiment, the image enhancement system 300 includes an execution device 210, a training device 220, a database 230, a client device 240, a data storage system 250 and a data acquisition device 260 (in the embodiment of the present application, the data acquisition device 260 can be used to generate randomly initialized noise), the execution device 210 includes a computing module 211 and an input / output (I / O) interface 212, the computing module 211 includes a model framework 200, for example, the model framework 200 can be the above-mentioned Figure 2 Corresponding to the model framework 200 described in the embodiment, in the embodiment of the present application, the model framework 200 includes a generation network 201, a teacher network 202 and a student network 203.

[0074] Among them, in the training stage, the data acquisition device 260 can be used to generate randomly initialized noise. The training device 220 generates the initial image to be processed by the generator network 201 in the model framework 200 based on the randomly initialized noise generated by the data acquisition device 260. The generator network 201 in the model framework 200 is trained using the first loss function and the second loss function constructed in the embodiment of the present application, and the student network 203 in the model framework 200 is trained using the third loss function constructed in the embodiment of the present application. Finally, in the process of alternating training of the generator network 201 and the student network 203, the iterative termination condition of the training is reached, and at this time, the training of the model framework 200 is considered to be completed. The student network 203 in the trained model framework 200 can be applied to different systems or devices (i.e., the execution device 210). For example, it can be applied to terminal devices such as mobile phones, tablets, and laptops with limited computing resources, and can also be applied to wearable devices such as smart watches, smart bracelets, and smart glasses. It can also be applied to smart cars, networked cars, robots and other intelligent mobile bodies, and can also be applied to monitoring systems (such as cameras), networked cameras (IP cameras, IPC) in security systems, augmented reality (AR), virtual reality (VR), identity recognition devices (such as attendance machines, punch cards, etc.), smart speakers and other edge devices. Specifically, the student network obtained by the above training is not limited to devices with limited computing resources, but can also be applied to cloud servers, computing platforms and other other devices on which neural networks can be deployed. This application does not limit this.

[0075] In the inference phase, the execution device 210 can call the data, code, etc. in the data storage system 250, or store the data, instructions, etc. in the data storage system 250. The data storage system 250 can be placed in the execution device 210, or the data storage system 250 can be an external memory relative to the execution device 210. The computing module 211 performs super-resolution reconstruction on each input real image data through the model framework 200 obtained after training.

[0076] exist Figure 3 In the embodiment, the execution device 210 is configured with an I / O interface 212 for data exchange with external devices, and a "user" can input data to the I / O interface 212 through a client device 240. For example, the client device 240 can be a camera device of a monitoring system, and the image captured by the camera device is input as input data to the computing module 211 of the execution device 210, and the computing module 211 performs super-resolution reconstruction on the input image to obtain an enhanced image, and then outputs the enhanced image to the camera device, or directly displays it on the display interface of the execution device 210 (if any), or stores it in the storage module of the execution device 210 for subsequent use. The specific embodiment of the present application does not limit how to process the obtained enhanced image.

[0077] In addition, in some embodiments of the present application, the client device 240 can also be integrated into the execution device 210. For example, when the execution device 210 is a mobile phone, the target image to be processed can be directly obtained through the mobile phone (such as an image captured by the camera of the mobile phone, or an image obtained based on a video captured by the camera of the mobile phone, etc.) or received from other devices (such as another mobile phone). The computing module 211 in the mobile phone then performs super-resolution reconstruction on the target image to obtain an image enhancement result (i.e., an enhanced image), and directly presents the enhanced image on the display interface of the mobile phone, or stores it. The product form of the execution device 210 and the client device 240 is not limited here.

[0078] It is worth noting that Figure 3 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 3 In the embodiment, the data storage system 250 is an external memory relative to the execution device 210. In other cases, the data storage system 250 may also be placed in the execution device 210; Figure 3 In the embodiment, the client device 240 is an external device relative to the execution device 210. In other cases, the client device 240 may also be integrated into the execution device 210.

[0079] In some embodiments of the present application, for example Figure 3In the embodiment, the training device 220 and the execution device 210 are distributed and independent devices, but Figure 3 This is only a schematic diagram of the architecture of the image enhancement system provided by an embodiment of the present invention. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. In other embodiments of the present application, the training device 220 and the execution device 210 can be integrated into the same device. In addition, Figure 3 The examples in are not intended to limit the number of each type of device, for example, the database 230 may communicate with multiple client devices 240 .

[0080] It should also be noted that the training of the model framework 200 described in the embodiment of the present application can be implemented on the cloud side. For example, the training device 220 on the cloud side (the training device 220 can be set on one or more servers or virtual machines) can obtain a training set, and train the model framework 200 according to multiple groups of training data (or called training samples) in the training set to obtain a trained model framework 200. After that, the trained model framework 200 is sent to the execution device 210 for application, or the trained student network 203 in the model framework 200 is sent to the execution device 210 for application. For example, the target task can be an image enhancement task, and the student network 203 in the trained model framework 200 can be sent to the execution device 210 for image super-resolution reconstruction. For example, Figure 3 As described in the corresponding system architecture, the model framework 200 is trained by the training device 220, and the student network 203 in the trained model framework 200 is then sent to the execution device 210 for use; the training of the model framework 200 described in the above embodiment can also be implemented on the terminal side, that is, the training device 220 can be located on the terminal side, for example, the training set can be obtained by the terminal device (such as a mobile phone, a smart watch, etc.), a wheeled mobile device (such as an autonomous driving vehicle, an assisted driving vehicle, etc.), etc., and the model framework 200 is trained according to the multiple groups of training data in the training set to obtain the trained model framework 200. The student network 203 in the trained model framework 200 can be used directly on the terminal device, or it can be sent by the terminal device to other devices for use. The specific embodiment of the present application does not limit on which device (cloud side or terminal side) the model framework 200 is trained or applied.

[0081] In combination with the above description, the specific implementation process of the training phase and the application phase of the image enhancement method provided in the embodiment of the present application will be explained below.

[0082] 1. Training Phase

[0083] In the embodiment of the present application, the training phase describes the above Figure 3The training device 220 in the training set performs a training operation on the model framework 200 using the training data in the training set. For details, please refer to Figure 4 , Figure 4 A flow chart of a training method for a model provided in an embodiment of the present application, wherein the model includes a generating network, a teacher network, and a student network, and the training method for the model may specifically include the following steps:

[0084] 401. The training device trains the generative network using a first loss function and a second loss function, wherein the first loss function is used to characterize the difference between a first image and a second image, wherein the first image is an image generated by the generative network according to input random initialization noise, the second image is an image obtained by downsampling the third image, and the third image is an image obtained by super-resolution reconstruction of the input first image by the teacher network, and the second loss function is related to the third loss function.

[0085] First, the training device trains the generative network in the model using the constructed first loss function and second loss function, wherein the first loss function is used to characterize the difference between the first image and the second image, the first image is an image generated by the generative network according to the input random initialization noise, the second image is an image obtained by downsampling the third image, and the third image is an image obtained by super-resolution reconstruction of the input first image by the teacher network. It should be noted that in the embodiment of the present application, the teacher network is a pre-trained neural network, and its model parameters remain unchanged, while the model parameters of the generative network during the first training can be randomly initialized.

[0086] For ease of understanding, the following Figure 2 The model framework 200 in the figure explains the physical meaning of the first loss function. First, the training device obtains randomly initialized noise z, which is input into the generator network 201 (the generator network can be represented as G), and the first image G(z) is obtained. The first image G(z) is then input into the teacher network 202 (the teacher network can be represented as ), the teacher network 202 performs super-resolution reconstruction on the input first image G(z) to obtain a third image Then, the third image Downsampling (the downsampling operation can be expressed as ), get the second image Based on this, the representation form of the first loss function L1 can be obtained. The first loss function L1 can be expressed as shown in the following formula (1):

[0087]

[0088] It should be noted that formula (1) represents the case where the first image G(z) involved in the first loss function L1 is a single image. In practical applications, multiple noises z are generally input into the generative network together during training, thereby obtaining multiple first images. For example, when the randomly initialized noise z is n (n≥2), the obtained first images and second images are both n. In this case, the first loss function L1 is used to characterize the average difference or mean square difference between the n first images and the n second images. Taking the average difference as an example, the first loss function L1 can be expressed as shown in the following formula (2):

[0089]

[0090] The embodiment of the present application is a first loss function constructed based on the basic characteristics of the super-resolution reconstruction task, because a teacher network is pre-trained using the super-resolution dataset. Then, to the teacher network Input a low-resolution image I from the dataset L , we can get the result of super-resolution reconstruction I S ,Right now Considering that low-resolution images in super-resolution datasets are generally obtained by downsampling high-resolution images, if the teacher network is well trained, then I S Yinghe I L The corresponding high-resolution image I H Stay consistent, S The downsampled I SL It should also be consistent with I L Be consistent, that is:

[0091]

[0092] In the data-free super-resolution reconstruction task, it is expected that the image generated by the generative network in the model can obey the distribution of the training set. Then for the generative network, the input random noise z, the image generated is G(z), which should be consistent with the super-resolution result. Downsampling Be consistent, that is:

[0093]

[0094] Therefore, according to the characteristics, the embodiment of the present application constructs a first loss function (i.e., as shown in the above formula (1) or formula (2)) to constrain the training of the generated network.

[0095] It should be noted that in the embodiment of the present application, in addition to the above-mentioned first loss function of the constrained generative network training process, the embodiment of the present application also provides a second loss function for jointly constraining the training of the generative network with the first loss function. The second loss function (which can be represented by L2) is related to the third loss function (which can be represented by L3). For example, the second loss function L2 can be a function that performs negative logarithm processing on the third loss function L3, wherein the second loss function can also be called an adversarial loss function, and the third loss function is a loss function used to constrain the training of the student network, which will be specifically described in the following step 402.

[0096] It should be noted that, in some embodiments of the present application, the negative logarithmic processing of the third loss function L3 to obtain the second loss function L2 can be obtained by the following formula (3):

[0097] L2=-log L3 (3)

[0098] In addition, during the iterative training of the model framework, the value of the third loss function L3 gradually decreases. Therefore, in some embodiments of the present application, in order to reduce the error of the second loss function L2, the third loss function L3 is subjected to negative logarithmic processing to obtain the second loss function L2, which can also be obtained by the following formula (4):

[0099] L2=-log(L3+p) (4)

[0100] Wherein, p is a preset positive number, for example, p=1.

[0101] It should also be noted that, in the embodiment of the present application, the training device uses the first loss function and the second loss function to train the generated network, which is an iterative training process. For example, it may be to reach a preset training round (e.g., stop training after 100 training rounds), or other training termination conditions (e.g., any one of the first loss function and the second loss function converges, any one of the first loss function and the second loss function reaches a preset threshold, the value after the sum of the first loss function and the second loss function reaches a preset threshold, etc.), which is not limited here.

[0102] 402. The training device trains the student network using a third loss function, where the third loss function is used to characterize the difference between the fourth image and the fifth image, where the fifth image is an image obtained by super-resolution reconstruction of the sixth image by the teacher network, the sixth image is an image generated by the trained generation network according to the input random initialization noise, and the fourth image is an image obtained by super-resolution reconstruction of the sixth image by the student network.

[0103] After the training device performs one iteration of training on the generative network, it will obtain the generative network after this round of training. After that, the training device further uses the third loss function to train the student network in the model, wherein the third loss function is used to characterize the difference between the fourth image and the fifth image, the fifth image is the image obtained by super-resolution reconstruction of the input sixth image by the teacher network, the sixth image is the image generated by the trained generative network according to the input random initialization noise, and the fourth image is the image obtained by super-resolution reconstruction of the input sixth image by the student network. It should also be noted that in the embodiment of the present application, the model parameters of the teacher network remain unchanged, while the model parameters of the generative network are the generative network obtained by the training in step 401 above, and the model parameters of the student network during the first training can also be randomly initialized.

[0104] For ease of understanding, we still combine Figure 2 The model framework 200 in the figure explains the physical meaning of the third loss function representation. First, the training device obtains randomly initialized noise z, and the noise z is input into the above-mentioned trained generation network 201 to obtain the sixth image G(z)′. In the embodiment of the present application, the difference between the sixth image G(z)′ and the first image G(z) is that the first image G(z) is the training data of the generation network 201, which is used to train the model parameters of the generation network 201. The sixth image G(z)′ can be regarded as an image obtained by the generation network 201 based on the input test data. After the generation network 201 completes the current round of iterative training based on the first image G(z), the image is generated by the randomly initialized noise z that is input again. Afterwards, the sixth image G(z)′ is input into the teacher network 202, and the teacher network 202 performs super-resolution reconstruction on the input sixth image G(z)′ to obtain the fifth image. In addition, the sixth image G(z)′ is not only input into the teacher network 202, but also input into the student network 203. The student network 203 performs super-resolution reconstruction on the input sixth image G(z)′ to obtain the fourth image S(G(z))′. Based on this, the representation form of the third loss function L3 can be obtained. The third loss function L3 can be expressed as shown in the following formula (5):

[0105]

[0106] It should also be noted that formula (5) represents the case where the fourth image involved in the third loss function L3 is an image. When the randomly initialized noise z is m (m≥2), the fourth image and the fifth image obtained are both m. In this case, the third loss function L3 is used to characterize the average difference or mean square error between the m fourth images and the m fifth images. The expression form is similar to the above formula (2) and will not be repeated here.

[0107] It should be noted that in the embodiment of the present application, the training device uses the third loss function to train the student network, which is also an iterative training process. For example, it may be to reach a preset training round (e.g., stop training after 200 training rounds), or other training termination conditions (e.g., the third loss function converges, the third loss function reaches a preset threshold, etc.), which are not specifically limited here.

[0108] 403. The training device repeatedly executes the above steps until the first iteration termination condition is reached.

[0109] The training device repeats the above process of alternating training of the generating network and the student network until the iteration termination condition (which may be referred to as the first iteration termination condition) is reached. It should be noted that each time the generating network is trained, the model parameters of the student network remain unchanged from the model parameter values ​​obtained in the previous round of training; similarly, each time the student network is trained, the model parameters of the generating network remain unchanged from the model parameter values ​​obtained in the previous round. During the entire alternating training process (the alternating training of the generating network and the student network is called a round of training), the values ​​of the model parameters of the teacher network remain unchanged (the model parameter values ​​obtained during pre-training).

[0110] It should be noted that, in some implementations of the present application, the first iteration termination condition may be determined in a variety of ways, including but not limited to:

[0111] (1) The third loss function reaches a preset threshold.

[0112] In an embodiment of the present application, a threshold value (e.g., 0.03) can be set in advance for the third loss function. During the iterative training of the entire model, after each training, it is determined whether the value of the third loss function obtained in the current round of training reaches the threshold value. If not, the training continues. If the preset threshold value is reached, the training is terminated. The values ​​of the model parameters of each network in the model determined by the current round of training are used as the model parameter values ​​of the final trained model.

[0113] (2) The third loss function begins to converge.

[0114] In an embodiment of the present application, during the iterative training of the entire model, if the difference between the value of the third loss function obtained in the current round of training and the value of the third loss function obtained in the previous round of training is within a preset range (e.g., 0.01), then the third loss function is considered to have converged, and the training can be terminated. Then, the values ​​of the model parameters of each network in the model determined by the current round of training are used as the model parameter values ​​of the final trained model.

[0115] (3) Training reaches the preset number of rounds.

[0116] In an embodiment of the present application, the iterative rounds of model training can be pre-configured (e.g., 1000 times). After each round of training, the values ​​of the model parameters of each network in the entire model corresponding to that round can be stored until the number of training iterations reaches a preset number. Afterwards, the models obtained in each round are verified using test data, and the set of model parameter values ​​with the best performance is selected as the final model parameter values ​​of the model, or the set of model parameters obtained from the last training is used as the final model parameter values ​​of the model.

[0117] Finally, in some embodiments of the present application, the student network obtained after training in the above steps 401 to 403 can be deployed on the target device, for example, it can be deployed on edge devices with limited computing resources such as mobile phones, smart wearable devices (such as smart bracelets, smart watches, etc.). This is because the network size of the pre-trained teacher network is generally large and is not suitable for deployment on edge devices with limited computing power, while the network size of the student network is generally small and suitable for deployment.

[0118] It should be noted that in the above embodiments of the present application, all neural network layers of the student network are trained together as a whole. Since knowledge distillation without data is very difficult, in order to reduce the difficulty of knowledge distillation, in some other embodiments of the present application, a layered training method (i.e., progressive distillation) can be used to train the student network. For details, please refer to Figure 5 , Figure 5 Another flow chart of the training method of the model provided in the embodiment of the present application, the model includes a generating network, a teacher network and a student network. In the method, the training of the student network is performed in layers. For ease of understanding, in the embodiment of the present application, the student network is divided into two parts as an example. Figure 6 For example, in Figure 6 In the above example, these two parts can be respectively called the first neural network layer (including multi-layer structures, such as Figure 6 ) and a second neural network layer (including a multi-layer structure), wherein the first neural network layer includes adjacent first sub-layers and second sub-layers, specifically, see Figure 6 As shown in the hierarchical diagram, the training method of the model may specifically include the following steps:

[0119] 501. The training device trains the generative network using a first loss function and a second loss function, wherein the first loss function is used to characterize the difference between a first image and a second image, wherein the first image is an image generated by the generative network according to input random initialization noise, the second image is an image obtained by downsampling the third image, and the third image is an image obtained by super-resolution reconstruction of the input first image by the teacher network, and the second loss function is related to the third loss function.

[0120] In the embodiment of the present application, step 501 is similar to step 401 in the above embodiment. Please refer to the above step 401 for details and will not be described in detail here.

[0121] 502. The training device trains the first sublayer and the second neural network layer of the student network using the third loss function, wherein the model parameters of the first sublayer and the model parameters of the second neural network layer are all randomly initialized values.

[0122] After the training device performs a round of training on the generating network using the first loss function and the second loss function, it will first use the third loss function to train only the first sublayer (which can be called Bx) and the second neural network layer (which can be called T) of the student network. In the embodiment of the present application, the subnetwork composed of the first sublayer Bx and the second neural network layer T of the student network can be called the student network S1. During this round of training, the model parameters of the first sublayer Bx and the initialized model parameters of the second neural network layer T are all randomly initialized values.

[0123] 503. The training device repeatedly executes the above steps until the first iteration termination condition is reached.

[0124] The training device repeatedly performs the above steps 501 and 502 until the first iteration termination condition is reached. After the first iteration termination condition is reached, the student network S1 is considered to be trained. In the embodiment of the present application, the first iteration termination condition is similar to the above first iteration termination condition. For details, please refer to the description in the above step 503, which will not be repeated here.

[0125] 504. The training device obtains a first model parameter value of the trained first sublayer and a second model parameter value of the trained second neural network layer.

[0126] After the training device goes through the above steps 501 and 503, it obtains the trained student network S1, that is, the training device can obtain the model parameter value of the first sub-layer after training (which can be called the first model parameter value) and the model parameter value of the second neural network layer after training (which can be called the second model parameter value).

[0127] 505. The training device trains the generated network using the first loss function and the second loss function, and trains the trained first sublayer, the second sublayer and the trained second neural network layer using the third loss function, wherein the model parameters of the trained first sublayer are the first model parameter values, the model parameters of the trained second neural network layer are the second model parameter values, and the model parameter values ​​of the second sublayer are random initialization values.

[0128] Afterwards, the training device continues to train the generated network using the first loss function and the second loss function, and uses the third loss function to train the trained first sublayer Bx, the second sublayer (which can be called By) and the trained second neural network layer T. In the embodiment of the present application, the subnetwork composed of the first sublayer Bx, the second sublayer By and the second neural network layer T of the student network can be called the student network S2. During this round of training, the initial values ​​of the model parameters of the first sublayer Bx and the model parameters of the second neural network layer T are the first model parameter values ​​and the second model parameter values ​​obtained by the above training, respectively. Since the second sublayer By was not trained in the previous round, the initialized model parameters of the second sublayer By are randomly initialized values.

[0129] 506. The training device repeatedly executes the process of training the generated network using the first loss function and the second loss function and training the trained first sublayer, the second sublayer and the trained second neural network layer using the third loss function until the second iteration termination condition is reached.

[0130] Finally, the training device repeatedly executes the process of training the generated network using the first loss function and the second loss function and training the trained first sublayer, the second sublayer and the trained second neural network layer using the third loss function until the second iteration termination condition is reached. In the embodiment of the present application, since the student network is only divided into two stages for training, namely, the student network S1 and the student network S2, when the training of the student network S2 is completed, it means that the training of the entire student network is completed, so that the values ​​of the model parameters of the entire student network can be obtained.

[0131] It should be noted that in some embodiments of the present application, similar to the first iteration termination condition, the second iteration termination condition can also be judged in a variety of ways, including but not limited to: the third loss function reaches a preset threshold, the third loss function begins to converge, the training reaches a preset round, etc., which is similar to the above-mentioned judgment method for the first iteration termination condition. For details, please refer to the above-mentioned description of the first iteration termination condition, which will not be repeated here.

[0132] It should also be noted that in Figure 5In the corresponding embodiment, the student network is divided into two parts, one part is the first neural network layer, and the other part is the second neural network layer. In the process of progressive training, the first sublayer and the second neural network layer of the first neural network layer are first trained (at this time, the model parameters of the second sublayer of the first neural network layer are kept unchanged). After training this part, the model parameter values ​​of the first sublayer and the second neural network layer of the first neural network layer (obtained after training) are used as the initialization values ​​of the model parameters of the first sublayer and the second neural network layer of the first neural network layer of the entire student network (the values ​​of the model parameters of the second sublayer are randomly initialized values), and the first neural network layer and the second neural network layer are continued to be trained to obtain the final trained student network. In other embodiments of the present application, the first neural network layer of the student network to be trained can also be divided into more sublayers. After each sublayer is trained, another sublayer is added, and the previous sublayer is added to continue training until all sublayers are trained.

[0133] Combined with the above Figure 5 Corresponding embodiments, in the embodiments of the present application, such as Figure 7 As shown, the student network to be trained can still be divided into two parts, the first part is the first neural network layer, and the second part is the second neural network layer. The difference is that the first neural network layer can be further divided into p parts (which can be called B1, B2, ..., Bp respectively), p ≥ 2, so that it is divided into p steps for training. In the kth step, the student network to be trained is composed of the first sublayer B1, the second sublayer B2, ..., the kth sublayer Bk and the second neural network layer T of the first neural network layer. Therefore, the student network to be trained in the kth step can be called student network k, and the first sublayer B1, the second sublayer B2, ..., the k-1th sublayer Bk-1 and the second neural network layer T of the first neural network layer in the student network k are initialized by the values ​​of the model parameters trained in the k-1th step, and in each step, the training strategy of the student network to be trained is similar.

[0134] The following is based on Figure 7 The steps shown are divided to describe the training process of the student network to be trained:

[0135] 1) Initialize the generated network, at this time k=1.

[0136] 2) Initialize the student network Sk to be trained. Specifically, when k=1, all model parameters in the student network S1 are randomly initialized, and the remaining student networks S2, S3, ..., Sp, etc. must load the model parameter values ​​obtained by training in the previous step.

[0137] 3) Randomly initialize the noise z, input the noise z into the generative network to obtain the first image G(z), and then input the first image G(z) into the teacher network to obtain the third image For the third image Downsample to get the second image The model parameters of the generated network are updated (i.e., the generated network is trained) using the first loss function L1 constructed by the above formula (1) or (2) and the second loss function L2 constructed by the above formula (3) or (4) (need to be combined with the third loss function L3 of formula (5)).

[0138] 4) Repeat step 3) multiple times until the first preset iteration condition is reached.

[0139] 5) Randomly initialize noise z, input the noise z into the trained generative network to obtain the sixth image G(z)′, and then input the sixth image G(z)′ into the teacher network to obtain the fifth image In addition, the sixth image G(z)′ is input into the student network Sk to obtain the fourth image S(G(z))′, and the model parameters of the student network Sk are updated using the third loss function L3 shown in the above formula (5) (i.e., the student network Sl is trained).

[0140] 6) Repeat step 5) multiple times until the second preset iteration condition is reached.

[0141] 7) Repeat steps 3) to 6) multiple times until the student network Sk training is completed.

[0142] 8) Determine whether k is equal to p. If Sk=Sp, it indicates that the last student network has also been trained, and the training is stopped at this time; if Sk≠Sp, start from step 2), assign k=k+1, and continue training Sk until the next round of student network Sk training is completed.

[0143] In the above implementation of the present application, step 2) and step 8) are the core steps for progressively training the student network.

[0144] It should be noted that in the above embodiments of the present application, the student network is divided into two parts, one part is the first neural network layer, and the other part is the second neural network layer. In fact, the second neural network layer is the tail of the student network. Considering that most of the intermediate layers of the super-resolution network have jump connections, therefore, in some embodiments of the present application, the student network can also be divided according to a new division method, for example, Figure 8As shown, the student network is divided into three parts, namely the head part (i.e., the head part, referred to as H), the middle part (i.e., the body part, referred to as B) and the tail part (i.e., the tail part, referred to as T), and the body part can be divided according to the sub-layer division method of the first neural network layer mentioned above. For example, the body part is divided into p sub-parts (which can be called B1, B2, ..., Bp respectively), p ≥ 2, and thus divided into p steps for training. In the kth step, the student network to be trained is composed of the head part, the first sub-part B1 of the body part, the second sub-part B2, ..., the kth sub-part Bk and the tail part. Similarly, the student network to be trained in the kth step can be called student network k, and the head part, the first sub-part B1 of the body part, the second sub-part B2, ..., the k-1th sub-part Bk-1 and the tail part in the student network k are initialized by the values ​​of the model parameters trained in the k-1th step, and in each step, the training strategy of the student network to be trained is similar. The specific training process is the same as the above steps 1) to 8), please refer to the above steps 1) to 8) for details, and will not be repeated here.

[0145] In the above-mentioned implementation of the present application, in terms of improving the training effect of the generative network, based on the basic characteristics of the super-resolution reconstruction task (i.e., the super-resolution image has all the information of the low-resolution image and contains more detailed information), a new loss function (i.e., the first loss function) for training the generative network is proposed for the super-resolution reconstruction task, which improves the training effect of the model on the super-resolution reconstruction task. In terms of improving the training effect of the student network, since knowledge distillation without data is very difficult, in order to reduce the difficulty of knowledge distillation, the present application gradually trains the student network (training a part of the student network each time), and gradually expands to train the entire student network, thereby reducing the difficulty of distillation.

[0146] Generally speaking, the network size of the teacher network (such as the number of model parameters, the number of network layers, etc.) is relatively large, while the network size of the student network is generally smaller. Therefore, the student network obtained by the training method of the model described above in this application can be applied to terminal devices such as mobile phones, tablets, and laptops with limited computing resources, and can also be applied to wearable devices such as smart watches, smart bracelets, and smart glasses. It can also be applied to smart cars, connected cars, robots and other intelligent mobile bodies, and can also be applied to monitoring systems (such as cameras), networked cameras (IP cameras, IPC) in security systems, augmented reality (AR), virtual reality (VR), identity recognition devices (such as attendance machines, punch cards, etc.), smart speakers and other edge devices.

[0147] Specifically, the student network obtained after the above training is not limited to be used only on devices with limited computing resources, but can also be used on cloud servers, computing platforms and any other devices on which neural networks can be deployed. This application does not make any specific limitations on this.

[0148] It should also be noted that in addition to being deployed on the target device for super-resolution reconstruction tasks, the trained student network can also be used as a teacher network in other model training processes and used to train other neural networks. This application does not specifically limit the application of the trained student network.

[0149] A typical application scenario of the model training method provided in the embodiment of the present application is: a friendly business or customer has a demand for model compression in super-resolution reconstruction tasks, but the training data cannot be transmitted due to personal privacy or legal terms, and can only be given to a pre-trained model (i.e., the teacher network). In this case, the model training method provided in the embodiment of the present application can be used to compress the super-resolution model, that is, based on the teacher network provided by the friendly business or customer, a trained lightweight super-resolution model (i.e., a trained student network) is obtained in combination with the generated network. For details, please refer to Fig. 9 , Fig. 9 A system architecture diagram of an application provided in an embodiment of the present application, such as Fig. 9 As shown, in actual application, first, the structure of the lightweight super-resolution model is determined according to the pre-trained super-resolution model (i.e., the teacher network) provided by a friendly business or customer and the target performance (i.e., the network structure of the student network is determined according to the teacher network), and then the training architecture of the entire model is composed of the pre-trained super-resolution model provided by the friendly business or customer, the randomly initialized lightweight super-resolution model and the randomly initialized generation network, and the trained lightweight super-resolution model (i.e., the trained student network) is obtained using the model training method provided in the embodiment of the present application.

[0150] 2. Reasoning Stage

[0151] In the embodiment of the present application, the training phase describes the above Figure 3 The execution device 210 uses the trained student network 203 to perform super-resolution reconstruction on the input target image. For details, please refer to Fig.10 , Fig.10 A schematic diagram of a flow chart of an image enhancement method provided in an embodiment of the present application may specifically include the following steps:

[0152] 1001. An execution device acquires a target image.

[0153] First, the execution device may also obtain an input image to be subjected to super-resolution reconstruction, and the input image may be referred to as a target image.

[0154] 1002. The execution device performs super-resolution reconstruction on the target image through the trained student network to obtain an enhanced image.

[0155] Afterwards, the execution device performs super-resolution reconstruction on the target image through the trained student network to obtain a reconstructed enhanced image. It should be noted that in the embodiment of the present application, the trained student network is the student network trained in the above-mentioned training stage. The specific training process of the student network please refer to the above-mentioned training stage, which will not be repeated here.

[0156] It should be noted that the trained student network obtained through training in the embodiment of the present application can be applied to image super-resolution reconstruction tasks in various occasions. Multiple application scenarios of multiple products will be introduced below.

[0157] (1) Camera image restoration

[0158] Camera photo restoration is a very important technology and has great use value in processing scenarios such as mobile phone imaging effects. At present, the main method of camera image restoration is to use a neural network model with a large network volume (such as a teacher network), which consumes a lot of computing resources and has a slow restoration speed. The student network trained by this application is deployed with less computing and storage overhead and fast running speed, and real-time super-resolution reconstruction can be achieved for the reconstructed image with extremely small computing / storage overhead.

[0159] (2) Mobile phone photography optimization

[0160] The student network trained in this application can be used for camera optimization of terminals (such as mobile phones, smart watches, personal computers, etc.), such as Fig.11 As shown, taking a mobile phone as an example, when a user uses a mobile phone to take a photo, automatically capturing faces, animals and other targets can help the mobile phone to automatically focus and beautify. If the mobile phone is far away from the photographed object, the image taken by the mobile phone may not be clear. Therefore, the student network trained in this application can be applied to the mobile phone. The optimized image quality is not only clear, but also due to the small network size, the optimization speed is faster, which can bring better user experience to users and improve the quality of mobile phone products.

[0161] It should be noted that the trained student network described in this application can be applied not only to the application scenarios described above, but also to various sub-fields in the field of artificial intelligence. For example, the method and the trained student network can be used in the super-resolution reconstruction tasks of images / videos in various application scenarios such as electronic screen display, autonomous driving, and video surveillance. As long as the fields and devices that can use neural networks can be used, the trained student network provided by the embodiments of this application can be applied, and no further examples are given here.

[0162] In order to have a more intuitive understanding of the beneficial effects brought by the embodiments of the present application, the following is a further comparison of the technical effects brought by the embodiments of the present application. The present application tests the effectiveness of the method provided by the present application on a very deep network for super-resolution (VDSR) and an enhanced deep residual networks for single image super-resolution (EDSR). The specific test process is as follows: given a trained VDSR (or EDSR) model and data set (only for comparison effect, not required for actual application of the present application), the number of channels of the original model (i.e., the teacher network) is halved to obtain the target network structure (i.e., the student network). Then, a training framework is formed based on a randomly initialized generating network, a pre-trained teacher network, and a randomly initialized student network, and the student network is trained using the model training method provided in the present application, and finally a trained student network is obtained. When comparing performance, the present application compares the difference between a student network trained by the present application and a student network trained with a data set. When making comparisons, we look at the performance of the network on public data sets in the industry, such as Set5, Set14, B100 and Urban100. The evaluation indicators are peak signal-to-noise ratio (PSNR) and structural similarity (SSIM). The higher the values ​​of both, the better the performance of the network.

[0163] The beneficial effects brought by the present application are shown in Table 1 and Table 2. In Table 1 and Table 2, Teacher refers to the performance obtained by directly training the teacher network with the data set; Student refers to the performance obtained by directly training the student network with the data set; Bicubic refers to not using the student network, but directly using the bicubic interpolation technology to upsample the low-resolution image to obtain a super-resolution image; Noise refers to directly inputting noise into the teacher network and the student network, and using the noise to train the student network. The performance obtained; Ours refers to the performance obtained by training the student network using the model training method provided in the embodiment of the present application. It can be seen from Table 1 and Table 2 that the student network trained by the model training method provided in the embodiment of the present application can obtain performance similar to that of the student network trained with the data set (PSNR is only 0.16dB lower on Set5 X2), which is significantly better than the super-resolution results obtained by bicubic interpolation and the super-resolution results obtained by noise.

[0164] Table 1. Performance comparison of this application and other existing methods on VDSR

[0165]

[0166] Table 2. Performance comparison of this application and other existing methods on EDSR

[0167]

[0168] On the basis of the above embodiments, in order to better implement the above solutions of the embodiments of the present application, the following also provides related devices for implementing the above solutions. Fig.12 , Fig.12 A schematic diagram of a training device provided in an embodiment of the present application, the training device 220 may specifically include: a first training module 1201, a second training module 1202, and an iteration trigger module 1203, wherein the first training module 1201 is used to train the generative network using a first loss function and a second loss function, the first loss function is used to characterize the difference between the first image and the second image, the first image is an image generated by the generative network according to input random initialization noise, the second image is an image obtained by downsampling the third image, the third image is an image obtained by super-resolution reconstruction of the input first image by the teacher network, and the second loss function The number is related to the third loss function; a second training module 1202 is used to train the student network using the third loss function, the third loss function is used to characterize the difference between the fourth image and the fifth image, the fifth image is an image obtained by super-resolution reconstruction of the input sixth image by the teacher network, the sixth image is an image generated by the trained generation network according to the input random initialization noise, and the fourth image is an image obtained by super-resolution reconstruction of the input sixth image by the student network; an iteration triggering module 1203 is used to trigger the first training module 1201 and the second training module 1202 to repeat their respective steps until the first iteration termination condition is reached.

[0169] In the above-mentioned embodiment of the present application, a model training method is provided, which includes a generative network, a teacher network and a student network. For a specific super-resolution reconstruction task, a new loss function (i.e., a first loss function) for training the generative network is proposed, which improves the training effect of the model on the super-resolution reconstruction task.

[0170] In a possible design, the student network may include a first neural network layer and a second neural network layer, the first neural network layer includes adjacent first sublayers and second sublayers, and the second training module 1202 is specifically used to: train the first sublayer and the second neural network layer using the third loss function, wherein the model parameters of the first sublayer and the model parameters of the second neural network layer are both randomly initialized values;

[0171] In this case, the training device 220 also includes an acquisition module 1204, which is used to obtain the first model parameter value of the trained first sublayer and the second model parameter value of the trained second neural network layer; in addition, the training device 220 also includes a third training module 1205, which is used to train the generated network using the first loss function and the second loss function, and to train the trained first sublayer, the second sublayer and the trained second neural network layer using the third loss function, wherein the model parameter of the trained first sublayer is the first model parameter value, the model parameter of the trained second neural network layer is the second model parameter value, and the model parameter value of the second sublayer is a random initialization value; the third training module 1205 is also used to repeatedly execute the process of training the generated network using the first loss function and the second loss function and training the trained first sublayer, the second sublayer and the trained second neural network layer using the third loss function until the second iteration termination condition is reached.

[0172] In the above implementation of the present application, the student network is divided into two parts, one part is the first neural network layer, and the other part is the second neural network layer. In the process of progressive training, the first sublayer and the second neural network layer of the first neural network layer are first trained (at this time, the model parameters of the second sublayer of the first neural network layer are kept unchanged). After training this part, the model parameter values ​​of the first sublayer and the second neural network layer of the first neural network layer (obtained after training) are used as the initialization values ​​of the model parameters of the first sublayer and the second neural network layer of the first neural network layer of the entire student network (the values ​​of the model parameters of the second sublayer are randomly initialized values), and the first neural network layer and the second neural network layer are continued to be trained, thereby obtaining the final trained student network. This progressive training process reduces the difficulty of training the student network.

[0173] In a possible design, reaching the second iteration termination condition includes: reaching a preset training round, or, the third loss function converges, or, the third loss function reaches a preset threshold.

[0174] In the above-mentioned implementation manner of the present application, several methods for determining the second iteration termination condition are described, which are flexible and widely applicable.

[0175] In a possible design, reaching the first iteration termination condition includes: reaching a preset training round, or, the third loss function converges, or, the third loss function reaches a preset threshold.

[0176] In the above-mentioned implementation manner of the present application, several methods for determining the termination conditions of the first iteration are also described, which are flexible and widely applicable.

[0177] In one possible design, multiple randomly initialized noises are generally input into the generative network together during training, so that multiple first images can be obtained. For example, when the randomly initialized noises are n (n≥2), the obtained first images and second images are both n. In this case, the first loss function is used to characterize the average difference or mean square error between the n first images and the n second images.

[0178] In the above-mentioned implementation mode of the present application, it is specifically explained how the first loss function is represented when there are multiple first images and second images, which covers a variety of practical situations and is feasible.

[0179] In a possible design, when the randomly initialized noise is m (m≥2), the fourth image and the fifth image obtained are both m. In this case, the third loss function is used to characterize the average difference or mean square error between the m fourth images and the m fifth images.

[0180] In the above-mentioned implementation mode of the present application, it is specifically explained how the third loss function is represented when there are multiple fourth images and fifth images, which covers a variety of practical situations and is feasible.

[0181] In a possible design, the iteration trigger module 1203 is also used to deploy the trained student network on the target device. For example, it can be deployed on edge devices with limited computing resources such as mobile phones, smart wearable devices (such as smart bracelets, smart watches, etc.). This is because the network size of the pre-trained teacher network is generally large and is not suitable for deployment on edge devices with limited computing power, while the network size of the student network is generally small and suitable for deployment.

[0182] In the above implementation of the present application, the student network obtained after training can be deployed on the target device for practical application. Generally, since the network size of the student network is smaller than that of the teacher network, the inference speed of the target device can be improved and the user experience can be enhanced.

[0183] It should be noted that the information interaction, execution process, etc. between the modules / units in the training device 220 are the same as those in the present application. Figure 4 or Figure 5 The corresponding method embodiments are based on the same concept. For specific contents, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0184] The present application also provides an execution device, see Fig.13 , Fig.13A schematic diagram of an execution device provided for an embodiment of the present application, the execution device may specifically include: an acquisition module 1301 and a reconstruction module 1302, wherein the acquisition module 1301 is used to acquire a target image; the reconstruction module 1302 is used to perform super-resolution reconstruction on the target image through a trained student network to obtain an enhanced image. It should be noted that in the embodiment of the present application, the trained student network is the student network trained in the above-mentioned training stage. For the specific training process of the student network, please refer to the above-mentioned training stage.

[0185] In the above-mentioned implementation mode of the present application, it is explained that the student network obtained after training is deployed on the target device to perform super-resolution reconstruction tasks. Generally, since the network size of the student network is smaller than that of the teacher network, the trained student network can improve the inference speed of the target device and enhance the user experience.

[0186] It should be noted that the information interaction and execution process between the modules / units in the execution device are the same as those in this application. Fig.10 The corresponding method embodiments are based on the same concept. For specific contents, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0187] Next, another training device provided by the embodiment of the present application is introduced. Fig.14 , Fig.14 A schematic diagram of a structure of a training device provided in an embodiment of the present application, wherein the training device may be deployed with Fig.12 The training device 220 described in the corresponding embodiment is used to implement Fig.12 The functions of the training device 220 in the corresponding embodiment are specifically implemented by one or more servers. The training device may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1422 and memory 1432, and one or more storage media 1430 (such as one or more mass storage devices) storing application programs 1442 or data 1444. Among them, the memory 1432 and the storage medium 1430 can be short-term storage or permanent storage. The program stored in the storage medium 1430 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the training device. Furthermore, the central processing unit 1422 can be configured to communicate with the storage medium 1430 to execute a series of instruction operations in the storage medium 1430 on the training device.

[0188] The training device may also include one or more power supplies 1426, one or more wired or wireless network interfaces 1450, one or more input and output interfaces 1458, and / or, one or more operating systems 1441, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0189] In the embodiment of the present application, the central processor 1422 is used to execute Figure 4 or Figure 5 The training device in the corresponding embodiment executes the training method of the model. For example, the central processor 1422 can be used to: first, train the generative network in the model using the constructed first loss function and the second loss function, wherein the first loss function is used to characterize the difference between the first image and the second image, the first image is an image generated by the generative network according to the input random initialization noise, the second image is an image obtained by downsampling the third image, and the third image is an image obtained by super-resolution reconstruction of the input first image by the teacher network. It should be noted that in the embodiment of the present application, the teacher network is a pre-trained neural network, and its model parameters remain unchanged, while the model parameters of the generative network during the first training can be randomly initialized. After an iterative training of the generative network, the generative network after this round of training will be obtained, and then the student network in the model is further trained using the third loss function, wherein the third loss function is used to characterize the difference between the fourth image and the fifth image, the fifth image is an image obtained by super-resolution reconstruction of the input sixth image by the teacher network, the sixth image is an image generated by the trained generative network according to the input random initialization noise, and the fourth image is an image obtained by super-resolution reconstruction of the input sixth image by the student network. It should also be noted that in the embodiment of the present application, the model parameters of the teacher network remain unchanged, while the model parameters of the generator network are the generator network obtained by the above training, and the model parameters of the student network during the first training can also be randomly initialized. Finally, the above process of alternating training of the generator network and the student network is repeated until the iteration termination condition (which can be called the first iteration termination condition) is reached.

[0190] It should be noted that the specific manner in which the CPU 1422 performs the above steps is different from that in the present application. Figure 4 or Figure 5 The corresponding method embodiments are based on the same concept, and the technical effects they bring are also the same as those of the above-mentioned embodiments of the present application. For specific contents, please refer to the description in the method embodiments shown above in the present application, and will not be repeated here.

[0191] Next, an execution device provided by an embodiment of the present application is introduced. Fig.15, Fig.15 A schematic diagram of the structure of the execution device provided in the embodiment of the present application. The execution device can be specifically manifested as various terminal devices, such as virtual reality VR devices, mobile phones, tablets, laptops, smart wearable devices, monitoring data processing devices or radar data processing devices, etc., which are not limited here. Among them, the execution device can be deployed with Fig.13 The execution device described in the corresponding embodiment is used to implement Fig.13 The function of the execution device in the corresponding embodiment. Specifically, the execution device includes: a receiver 1501, a transmitter 1502, a processor 1503 and a memory 1504 (wherein the number of the processor 1503 in the execution device can be one or more, Fig.15 In the example of FIG. 1501 , the processor 1503 may include an application processor 15031 and a communication processor 15032. In some embodiments of the present application, the receiver 1501, the transmitter 1502, the processor 1503 and the memory 1504 may be connected via a bus or other means.

[0192] The memory 1504 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1503. A portion of the memory 1504 may also include a non-volatile random access memory (NVRAM). The memory 1504 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0193] The processor 1503 controls the operation of the execution device. In a specific application, the various components of the execution device are coupled together through a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus, and a status signal bus, etc. However, for the sake of clarity, various buses are referred to as bus systems in the figure.

[0194] This application Figure 8The method disclosed in the corresponding embodiment can be applied to the processor 1503, or implemented by the processor 1503. The processor 1503 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 1503 or the instructions in the form of software. The above-mentioned processor 1503 can be a general-purpose processor, a digital signal processor (digital signal processing, DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (ASIC), a field programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The processor 1503 can implement or execute the present application Figure 8 The methods, steps and logic block diagrams disclosed in the corresponding embodiments. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor, or a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1504, and the processor 1503 reads the information in the memory 1504, and completes the steps of the above method in combination with its hardware.

[0195] The receiver 1501 can be used to receive input digital or character information and generate signal input related to the relevant settings and function control of the execution device. The transmitter 1502 can be used to output digital or character information through the first interface; the transmitter 1502 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 1502 can also include a display device such as a display screen.

[0196] In one embodiment of the present application, the processor 1503 is used to perform super-resolution reconstruction on the input target image through the trained student network to obtain a corresponding enhanced image. The trained student network can be a network trained by the present application. Figure 4 or Figure 5 The corresponding training method is obtained. For specific contents, please refer to the description in the method embodiment shown in the above-mentioned application, which will not be repeated here.

[0197] The present application also provides a computer-readable storage medium in which a program for signal processing is stored. When the program is run on a computer, the computer executes the above-mentioned Figure 4 or Figure 5 The steps performed by the training device described in the embodiment shown, or the computer is made to perform the above Fig.10 The illustrated embodiments describe the steps performed by the execution device.

[0198] The training device and execution device provided in the embodiment of the present application may be specifically a chip, and the chip includes: a processing unit and a communication unit. The processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin or a circuit. The processing unit may execute the computer execution instructions stored in the storage unit, so that the chip in the training device executes the above Figure 4 or Figure 5 The steps performed by the training device described in the embodiment shown, or the chip in the execution device performs the above Fig.10 The illustrated embodiments describe the steps performed by the execution device.

[0199] Optionally, the storage unit is a storage unit within the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0200] For details, please refer to Fig.16 , Fig.16 A schematic diagram of the structure of a chip provided in an embodiment of the present application, the chip can be expressed as a neural network processor NPU 200, NPU 200 is mounted on the host CPU (Host CPU) as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 2003, which is controlled by the controller 2004 to extract matrix data from the memory and perform multiplication operations.

[0201] In some implementations, the operation circuit 2003 includes multiple processing units (process engines, PEs) inside. In some implementations, the operation circuit 2003 is a two-dimensional systolic array. The operation circuit 2003 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 2003 is a general-purpose matrix processor.

[0202] For example, suppose there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 2002 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 2001 and performs matrix operation with matrix B. The partial result or final result of the matrix is ​​stored in the accumulator 2008.

[0203] The unified memory 2006 is used to store input data and output data. The weight data is directly transferred to the weight memory 2002 through the direct memory access controller (DMAC) 2005. The input data is also transferred to the unified memory 2006 through the DMAC.

[0204] The bus interface unit 2010 (BIU for short) is used for the interaction between the AXI bus and the DMAC and the instruction fetch buffer (IFB) 2009 .

[0205] The bus interface unit 2010 is used for the instruction fetch memory 2009 to obtain instructions from the external memory, and is also used for the storage unit access controller 2005 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0206] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 2006 or to transfer weight data to the weight memory 2002 or to transfer input data to the input memory 2001.

[0207] The vector calculation unit 2007 includes multiple operation processing units, which further process the output of the operation circuit when necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as Batch Normalization, pixel-level summation, upsampling of feature planes, etc.

[0208] In some implementations, the vector calculation unit 2007 can store the processed output vector to the unified memory 2006. For example, the vector calculation unit 2007 can apply a linear function and / or a nonlinear function to the output of the operation circuit 2003, such as linear interpolation of the feature plane extracted by the convolution layer, and then, for example, a vector of accumulated values ​​to generate an activation value. In some implementations, the vector calculation unit 2007 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 2003, for example, for use in a subsequent layer in a neural network.

[0209] An instruction fetch buffer 2009 connected to the controller 2004, for storing instructions used by the controller 2004;

[0210] The unified memory 2006, the input memory 2001, the weight memory 2002 and the instruction fetch memory 2009 are all on-chip memories. The external memory is private to the NPU hardware architecture.

[0211] The processor mentioned in any of the above places may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the program of the above-mentioned first aspect method.

[0212] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0213] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0214] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0215] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

Claims

1. A model training method, applied to super-resolution reconstruction tasks, characterized in that: The model includes a generator network, a teacher network and a student network. The generator network is used to generate images from randomly initialized noise. The teacher network is a pre-trained neural network. The method includes: The generative network is trained using a first loss function and a second loss function, wherein the first loss function is used to characterize the difference between a first image and a second image, the first image is an image generated by the generative network according to input random initialization noise, the second image is an image obtained by downsampling a third image, the third image is an image obtained by super-resolution reconstruction of the first image input by the teacher network, and the second loss function is related to the third loss function; The student network is trained using the third loss function, wherein the third loss function is used to characterize the difference between the fourth image and the fifth image, wherein the fifth image is an image obtained by super-resolution reconstruction of the input sixth image by the teacher network, the sixth image is an image generated by the trained generation network according to input random initialization noise, and the fourth image is an image obtained by super-resolution reconstruction of the input sixth image by the student network; Repeat the above steps until the first iteration termination condition is reached.

2. The method according to claim 1, characterized in that The student network includes a first neural network layer and a second neural network layer, the first neural network layer includes adjacent first sublayers and second sublayers, and the training of the student network using the third loss function includes: Using the third loss function to train the first sublayer and the second neural network layer, wherein the model parameters of the first sublayer and the model parameters of the second neural network layer are both randomly initialized values; After repeatedly executing the above steps until the first iteration termination condition is reached, the method further includes: Obtaining a first model parameter value of the trained first sublayer and a second model parameter value of the trained second neural network layer; The generating network is trained using the first loss function and the second loss function, and the trained first sublayer, the second sublayer and the trained second neural network layer are trained using the third loss function, wherein the model parameters of the trained first sublayer are the first model parameter values, the model parameters of the trained second neural network layer are the second model parameter values, and the model parameter values ​​of the second sublayer are random initialization values; Repeat the process of training the generating network using the first loss function and the second loss function and training the trained first sublayer, the second sublayer and the trained second neural network layer using the third loss function until the second iteration termination condition is reached.

3. The method according to claim 2, characterized in that The second iteration termination condition is reached including: A preset training round is reached, or the third loss function converges, or the third loss function reaches a preset threshold.

4. The method according to any one of claims 1 to 3, characterized in that The first iteration termination condition is reached including: A preset training round is reached, or the third loss function converges, or the third loss function reaches a preset threshold.

5. The method according to any one of claims 1 to 3, characterized in that: The number of the first images is n, the number of the second images is n, and n≥2. The first loss function is used to characterize the difference between the first image and the second image and includes: The first loss function is used to characterize the average difference or mean square error between the n first images and the n second images.

6. The method according to any one of claims 1 to 3, characterized in that The number of the fourth images is m, the number of the fifth images is m, m≥2, and the third loss function is used to characterize the difference between the fourth image and the fifth image, including: The third loss function is used to characterize the average difference or mean square error between the m fourth images and the m fifth images.

7. The method according to any one of claims 1 to 3, characterized in that The method further comprises: Deploy the trained student network on the target device.

8. An image enhancement method, characterized in that: include: Get the target image; The target image is super-resolved and reconstructed by a trained student network to obtain an enhanced image. The trained student network is a student network trained by any one of claims 1 to 7.

9. A training device, characterized in that: The model includes a generator network, a teacher network and a student network. The generator network is used to generate images from randomly initialized noise. The teacher network is a pre-trained neural network. The training device includes: A first training module is used to train the generative network using a first loss function and a second loss function, wherein the first loss function is used to characterize the difference between a first image and a second image, the first image is an image generated by the generative network according to input random initialization noise, the second image is an image obtained by downsampling a third image, the third image is an image obtained by super-resolution reconstruction of the input first image by the teacher network, and the second loss function is related to the third loss function; a second training module, for training the student network using the third loss function, wherein the third loss function is used to characterize the difference between the fourth image and the fifth image, the fifth image being an image obtained by super-resolution reconstruction of the input sixth image by the teacher network, the sixth image being an image generated by the trained generation network according to input random initialization noise, and the fourth image being an image obtained by super-resolution reconstruction of the input sixth image by the student network; The iteration triggering module is used to trigger the first training module and the second training module to repeatedly execute their respective steps until a first iteration termination condition is reached.

10. The training device according to claim 9, characterized in that The student network includes a first neural network layer and a second neural network layer, the first neural network layer includes adjacent first sublayers and second sublayers, and the second training module is specifically used for: Using the third loss function to train the first sublayer and the second neural network layer, wherein the model parameters of the first sublayer and the model parameters of the second neural network layer are both randomly initialized values; The training device also includes an acquisition module for acquiring a first model parameter value of the first sublayer after training and a second model parameter value of the second neural network layer after training; The training device also includes a third training module, which is used to train the generating network using the first loss function and the second loss function, and to train the trained first sublayer, the second sublayer and the trained second neural network layer using the third loss function, wherein the model parameters of the trained first sublayer are the first model parameter values, the model parameters of the trained second neural network layer are the second model parameter values, and the model parameter values ​​of the second sublayer are random initialization values; The third training module is also used to repeatedly execute the process of training the generating network using the first loss function and the second loss function and training the trained first sublayer, the second sublayer and the trained second neural network layer using the third loss function until the second iteration termination condition is reached.

11. The training device according to claim 10, characterized in that The second iteration termination condition is reached including: A preset training round is reached, or the third loss function converges, or the third loss function reaches a preset threshold.

12. The training device according to any one of claims 9 to 11, characterized in that The first iteration termination condition is reached including: A preset training round is reached, or the third loss function converges, or the third loss function reaches a preset threshold.

13. The training device according to any one of claims 9 to 11, characterized in that The number of the first images is n, the number of the second images is n, and n≥2. The first loss function is used to characterize the difference between the first image and the second image and includes: The first loss function is used to characterize the average difference or mean square error between the n first images and the n second images.

14. The training device according to any one of claims 9 to 11, characterized in that The number of the fourth images is m, the number of the fifth images is m, m≥2, and the third loss function is used to characterize the difference between the fourth image and the fifth image, including: The third loss function is used to characterize the average difference or mean square error between the m fourth images and the m fifth images.

15. The training device according to any one of claims 9 to 11, characterized in that The iteration trigger module is further used for: Deploy the trained student network on the target device.

16. An execution device, characterized in that: include: An acquisition module, used for acquiring a target image; A reconstruction module is used to perform super-resolution reconstruction on the target image through a trained student network to obtain an enhanced image, wherein the trained student network is a student network obtained after training by the training device of any one of claims 9 to 15.

17. A training device, comprising a processor and a memory, wherein the processor is coupled to the memory, wherein: The memory is used to store programs; The processor is used to execute the program in the memory so that the training device performs the method as described in any one of claims 1-7.

18. An execution device, comprising a processor and a memory, wherein the processor is coupled to the memory, characterized in that: The memory is used to store programs; The processor is configured to execute the program in the memory so that the execution device executes the method according to claim 8.

19. A computer-readable storage medium comprising a program, which, when executed on a computer, causes the computer to execute the method according to any one of claims 1 to 8.

20. A computer program product comprising instructions, which, when executed on a computer, causes the computer to perform the method according to any one of claims 1 to 8.

21. A chip, comprising a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface to execute the method according to any one of claims 1 to 8.

Images