Neural network training method and related equipment for image processing

Abstract

The embodiment of the present application relates to the field of artificial intelligence, and provides a neural network training method and related equipment for image processing, the method comprising: constructing a super network corresponding to the processing task, the super network may include multiple sub-networks, and the multiple sub-networks share network parameters with the super network; performing sampling training on the super network; determining the target sub-network based on the network parameters of the trained super network, the target sub-network may be a sub-network in the super network or obtained by mutating the sub-network in the super network, and the performance of the target sub-network meets the preset requirements; performing quantization training on the target sub-network to obtain a quantized target sub-network, and during the quantization training process, the target sub-network and the super network share the quantizer used for quantization training. By using the target sub-network of the present application to share the quantizer with the super network, the distillation effect in the network structure search can be realized in the quantization training, and the image processing effect of the target sub-network finally deployed on the terminal device side can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence (AI), and in particular to a neural network training method and related equipment for image processing. Background Art

[0002] With the development of artificial intelligence technology, neural networks are increasingly used in terminal devices. Since different application scenarios or different terminal devices have different requirements for neural networks, it is necessary to select the appropriate neural network according to the application scenario or terminal device.

[0003] The model quantization algorithm is a method of compressing a neural network, that is, the parameters (such as weights) of the neural network represented by floating point numbers (high precision) are approximated by low-precision integers. After quantization to low precision, the optimization technology on the mobile platform can be applied to accelerate the calculation process of the neural network, and the model capacity of the original neural network will also be reduced after quantization, so that it can better adapt to terminal devices with limited storage space. The relevant model quantization algorithm usually directly quantizes the trained small model using quantization aware training (QAT) or post training quantization (PTQ). The network structure of the small model will not change during the quantization process. Since the parameters after quantization have certain errors, the model performance of the quantized small model will be lost to a certain extent. Taking the small model used in the image processing scenario as an example, if the model performance of the quantized small model is lost to a certain extent, it will cause the terminal device to run this small model for image processing. The image processing effect is not good. Summary of the invention

[0004] The present application provides a neural network training method and related equipment for image processing. In the process of quantizing the sub-network, the sub-network and the super-network share a quantizer, so that the distillation effect in the network structure search is realized in the quantization training. This can solve the problem of performance loss in the sub-network during the quantization process and improve the image processing effect of the target sub-network finally deployed on the terminal device side.

[0005] In a first aspect, the present application provides a neural network training method for image processing, the method comprising: constructing a super network corresponding to a processing task, the super network may include multiple sub-networks, and the multiple sub-networks share network parameters with the super network; performing sampling training on the super network, the training samples of the super network include image samples; based on the network parameters of the trained super network, determining a target sub-network, the target sub-network is a sub-network in the super network or is obtained by mutating a sub-network in the super network, and the performance of the target sub-network meets preset requirements; performing quantization training on the target sub-network to obtain a quantized target sub-network, the target sub-network and the super network share a quantizer used for quantization training. By adopting the above technical scheme, the super network is sampled and trained, and a sub-network whose performance meets the preset requirements is selected as the target sub-network. The target sub-network can be a sub-network in the super network, or a sub-network obtained based on the mutation of the sub-network in the super network. Therefore, it is possible to obtain a sub-network with better performance from the super network covering a limited number of sub-networks as the target sub-network for subsequent quantization training without causing the problem of too large a search space. In the process of quantizing the target sub-network, the target sub-network and the super network share a quantizer, so that the distillation effect in the network structure search is realized in the quantization training, which can solve the problem of performance loss of the target sub-network during the quantization process, and improve the image processing effect of the target sub-network finally deployed on the terminal device side. For example, if the processing task is image denoising, the image denoising effect of the target sub-network finally deployed on the terminal device side can be improved.

[0006] In a possible implementation, constructing a super network corresponding to a processing task includes: establishing a search space based on a preset neural network, the search space indicating the range of variation of the network structure of the preset neural network, and the preset neural network includes a block corresponding to the processing task, and the block consists of a single neural network layer or multiple neural network layers; constructing a super network based on the search space. Using the above technical solution, a super network is constructed by means of a search space, and the search space defines the range of variation of the network structure based on the preset neural network, so that the super network can cover multiple sub-networks.

[0007] In a possible implementation, a super network is trained based on n sampling training stages in sequence, the n sampling training stages are respectively defined as the i-th sampling training stage, i=1, 2, ..., n, n is a positive integer greater than 1, and the previous sampling training stage of the i-th sampling training stage is defined as the i-1-th sampling training stage. The super network is sampled and trained, including: in the i-1-th sampling training stage, sampling the super network based on a preset sampling rule to obtain the i-1-th sub-network; obtaining a data set for network training, training the i-1-th sub-network using the data set, and updating the network parameters of the i-1-th sub-network after training to the super network; in the i-th sampling training stage, sampling the super network whose network parameters have been updated in the i-1 sampling training stage based on the preset sampling rule to obtain the i-th sub-network; training the i-th sub-network using the data set, and updating the network parameters of the i-1-th sub-network after training to the super network whose network parameters have been updated in the i-1-th sampling training stage. By adopting the above technical solution, a sub-network is obtained by sampling the super-network, the sub-network is trained, and then the network parameters of the trained sub-network are updated to the super-network. The above sub-network sampling, training and updating process are repeated to realize the training of the super-network. Compared with the training method of the super-network, by training the sub-network, the sub-network and the super-network share network parameters, which can improve the training efficiency and performance of the super-network.

[0008] In a possible implementation, the search space uses blocks as the search dimension, and the preset sampling rules include: performing block sampling in the search space based on the single path one shot (SPOS) method to determine one or more target blocks; determining the number of channels of each target block; determining the depth of each target block; and constructing a subnetwork based on each target block, the number of channels of each target block, and the depth. Using the above technical solution, block sampling is first performed in the search space to determine the target blocks for constructing the subnetwork, and then the number of channels and depth of each target block are determined, thereby determining a unique subnetwork, and realizing the sampling of the subnetwork from the supernetwork.

[0009] In a possible implementation, a target subnetwork is determined based on the network parameters of a trained supernetwork, including: sampling the supernetwork to obtain m subnetworks, and constructing a first population based on the m subnetworks, where m is a positive integer greater than 1; performing performance evaluation on each subnetwork in the first population based on a preset validation set and the network parameters of the trained supernetwork; selecting the top k subnetworks in performance from the first population, where k is less than m; mutating the k subnetworks with a preset mutation probability until mk subnetworks are obtained, and the mk subnetworks all satisfy the preset subnetwork structure constraints; constructing a second population based on the k subnetworks and the mk subnetworks, and performing performance evaluation on each subnetwork in the second population; selecting a subnetwork from the second population whose performance evaluation result meets the preset requirements as the target subnetwork. By adopting the above technical solution, an excellent sub-network is searched in the search space through evolutionary learning (sub-network mutation), so that a sub-network with better performance can be obtained as the target sub-network from the search space covering a limited number of sub-networks for subsequent quantitative training, without causing the problem of too large a search space, and the performance of the target sub-network finally deployed on the terminal device side can be improved.

[0010] In a possible implementation, the target subnetwork is determined based on z population iterations, where z is a positive integer greater than 1, and the subnetwork with the best performance evaluation result is selected from the second population as the target subnetwork, including: if the current population iteration number is less than z, repeat the steps of selecting the subnetworks with the top k performance rankings from the currently constructed population, mutating the k subnetworks, and constructing a new population based on the k subnetworks and the mk subnetworks obtained by mutation; if the current population iteration number is equal to z, select the subnetwork whose performance evaluation result meets the preset requirements from the currently constructed population as the target subnetwork. By adopting the above technical solution, the number of population iterations can be set according to the actual available network training time. In the case where the available network training time is long, a relatively large number of population iterations can be set. The larger the number of population iterations, the higher the possibility of searching for an excellent subnetwork in the search space.

[0011] In a possible implementation, the target subnetwork is trained for quantization, including: inputting a random data into the target subnetwork to generate a directed acyclic graph (DAG) for forward reasoning, the data type of the random data is the same as the data type of the data used to train the super network; performing operator fusion on the target subnetwork based on the DAG and the preset fusion strategy; adding a pseudo-quantization node to the target subnetwork after operator fusion, and the target subnetwork and the super network share the pseudo-quantization node; iteratively training the target subnetwork with the added pseudo-quantization node. By adopting the above technical solution, a fusion strategy can be set according to the actual processing task to perform operator fusion on the target subnetwork to obtain one or more target operators, and at the same time, a quantization-aware training method can be used to perform quantization training on the target subnetwork, and the target subnetwork and the super network share a quantizer (pseudo-quantization node), so as to realize the distillation effect in the network structure search in the quantization training, solve the problem of performance loss of the target subnetwork during the quantization process, and improve the performance of the target subnetwork finally deployed on the terminal device side.

[0012] In a possible implementation, adding pseudo-quantization nodes to the target sub-network after operator fusion includes: locating the activation and weight of each layer on the target sub-network after operator fusion; adding pseudo-quantization nodes to the activation and weight of each layer. By adopting the above technical solution, by adding pseudo-quantization nodes to the activation and weight of each layer of the target sub-network, the target sub-network is quantized, so that the target sub-network with the pseudo-quantization nodes can be iteratively trained later.

[0013] In a possible implementation, iterative training of a target subnetwork with added pseudo-quantized nodes includes multiple quantized training stages executed in sequence, and each of the multiple quantized training stages includes: sampling a supernetwork to obtain j subnetworks, the j subnetworks including a target subnetwork with added pseudo-quantized nodes and j-1 subnetworks with a larger network structure than the target subnetwork, and the j-1 subnetworks all contain the network structure of the target subnetwork; performing backward reasoning training on each of the j subnetworks to obtain a gradient corresponding to each subnetwork; and updating the gradient of the target subnetwork based on the gradient of each subnetwork in the j subnetworks. The above technical solution is adopted to update the gradient of the target subnetwork by sampling multiple subnetworks (the multiple subnetworks include the target subnetwork) from the search space and jointly performing backward reasoning training, so as to realize the distillation effect in the network structure search in the quantized training, and complete pruning at the same time as the quantized training, and adopt a method of using multiple subnetworks to accompany the training of the target subnetwork, which can further improve the performance of the target subnetwork finally deployed in the terminal device.

[0014] In one possible implementation, the quantization training of the target subnetwork also includes: tracking the weight of each layer on the target subnetwork; when it is identified that the target subnetwork has dynamically jittering weights, the dynamically jittering weights are locked. The above technical solution is adopted to track the weight of each layer on the target subnetwork, identify the dynamically jittering weights, and for the dynamically jittering weights, it is considered that the weights are no longer converging and are only jittering meaninglessly. By locking the dynamically jittering weights, for example, assigning a fixed value to the dynamically jittering weights, the dynamically jittering weights are locked, which can accelerate the convergence speed of the target subnetwork and improve the network training efficiency.

[0015] In one possible implementation, the weight of each layer on the target subnetwork is tracked, including: determining the shadow weight corresponding to the weight of each layer on the target subnetwork based on an exponential moving average (EMA) algorithm; and identifying the weight with dynamic jitter based on the shadow weight. By adopting the above technical solution, the accuracy of weight jitter identification can be improved by identifying the weight with dynamic jitter through the EMA algorithm.

[0016] In a possible implementation, locking the weight of dynamic jitter includes: determining a relatively stable state value of the weight of dynamic jitter, and assigning a value to the weight of dynamic jitter based on the relatively stable state value. With the above technical solution, for the weight of dynamic jitter, a relatively stable state value is assigned to the weight of dynamic jitter. The relatively stable state value can be a value that appears more frequently when the weight is jittering, or a value calculated based on a value that appears more frequently, such as a value calculated based on a weighted average method. This not only achieves locking the weight of dynamic jitter, but also makes the weight assignment accurate.

[0017] In a possible implementation, the neural network training method applied to image processing further includes: performing image processing on the image to be processed based on the quantized target subnetwork, the image processing including at least one of denoising, demosaicing, and automatic white balance, and the image samples including at least one of a plurality of sample pairs of noisy images and non-noisy images, a plurality of sample pairs of mosaic images and non-mosaic images, and a plurality of sample pairs of images not subjected to white balance processing and images subjected to white balance processing. By adopting the above technical solution, after obtaining the quantized target subnetwork, the target subnetwork can be used to perform image processing on the image to be processed, the image processing including but not limited to denoising, demosaicing, automatic white balance, etc., the image processing function of the target subnetwork is related to the image samples used in the training process, for example, the image samples include a plurality of sample pairs of noisy images and non-noisy images, and the target subnetwork may have an image denoising function.

[0018] In a second aspect, an embodiment of the present application provides a neural network training device for image processing, the neural network training device comprising: a construction module for constructing a super network corresponding to a processing task, the super network comprising multiple sub-networks, and the multiple sub-networks share network parameters with the super network; a training module for sampling and training the super network, the training samples of the super network comprising image samples; a determination module for determining a target sub-network based on the network parameters of the trained super network, the target sub-network being a sub-network in the super network or being obtained by mutating a sub-network in the super network, and the performance of the target sub-network meets preset requirements; a quantization module for performing quantization training on the target sub-network to obtain a quantized target sub-network, the target sub-network and the super network sharing a quantizer used for quantization training.

[0019] In a third aspect, an embodiment of the present application provides a server, comprising a memory and a processor, wherein the memory is used to store computer-readable instructions; the processor is used to read the computer-readable instructions and implement the method provided by the first aspect and its optional implementation method.

[0020] In a fourth aspect, an embodiment of the present application provides a terminal device, which is deployed with a target subnetwork obtained by the method provided based on the first aspect and its optional implementation method, and the target subnetwork can be used for image processing, and the image processing includes at least one of noise reduction processing, de-mosaic processing, and automatic white balance processing.

[0021] In a fifth aspect, an embodiment of the present application provides a computer storage medium storing computer-readable instructions, and the computer-readable instructions, when executed by a processor, implement the method provided by the first aspect and its optional implementation.

[0022] In a sixth aspect, an embodiment of the present application provides a computer program product, which includes computer-readable instructions. When the computer-readable instructions are executed by a processor, the method provided by the first aspect and its optional implementation method is implemented.

[0023] In a seventh aspect, an embodiment of the present application provides a neural network deployment system, comprising a terminal device and a server, wherein the server is used to execute the method provided in the first aspect and its optional implementation, and to deploy a quantized target subnetwork in the terminal device.

[0024] The technical effects obtained by the above-mentioned second aspect, third aspect, fourth aspect, fifth aspect, sixth aspect and seventh aspect are similar to the technical effects obtained by the corresponding technical means in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0026] Figure 1 A schematic diagram of an artificial intelligence main framework provided for an embodiment of the present application;

[0027] Figure 2 A schematic diagram of an artificial intelligence system provided in an embodiment of the present application;

[0028] Figure 3 A schematic diagram of a neural network training system provided in an embodiment of the present application;

[0029] Figure 4 A flowchart of a neural network training method for image processing provided in an embodiment of the present application;

[0030] Figure 5 A schematic diagram of a neural network training device provided in an embodiment of the present application sampling a super network to obtain a sub-network;

[0031] Figure 6 A schematic diagram of population update provided in an embodiment of the present application;

[0032] Figure 7 A schematic diagram of a neural network training device provided in an embodiment of the present application sampling a super network to obtain a sub-network during quantization training;

[0033] Figure 8 A schematic diagram of a neural network training device provided in an embodiment of the present application for identifying weights of dynamic jitter;

[0034] Fig. 9 Another schematic diagram of a flow chart of a neural network training method for image processing provided in an embodiment of the present application;

[0035] Fig.10 A schematic diagram of the structure of the execution device provided in the embodiment of the present application;

[0036] Fig.11 Another structural schematic diagram of an execution device provided in an embodiment of the present application;

[0037] FIG12( a) is a schematic diagram of a scene of image denoising provided in an embodiment of the present application;

[0038] FIG12( b ) is a schematic diagram of a scene of image de-mosaicing provided in an embodiment of the present application;

[0039] FIG12( c ) is a schematic diagram of a scene of automatic white balance of an image provided by an embodiment of the present application;

[0040] Fig.13 A schematic diagram of the structure of a neural network training device provided in an embodiment of the present application;

[0041] Fig.14 A schematic diagram of the structure of a neural network training device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The following will describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0043] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, words such as "exemplary", "or", "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "or", "for example" is intended to present related concepts in a concrete way.

[0044] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by technicians in the technical field of the present application. The terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. It should be understood that, unless otherwise specified in the present application, " / " means or. For example, A / B can represent A or B. "And / or" in the present application is only a kind of association relationship describing associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. "At least one" means one or more. "Multiple" means two or more than two. For example, at least one of a, b or c can represent: a, b, c, a and b, a and c, b and c, a, b and c seven situations. It should be understood that the order of the steps shown in the flowchart of this article can be changed, and some can also be omitted.

[0045] To facilitate understanding of the embodiments of the present application, first Figure 1 Describe the overall workflow of the artificial intelligence system. Figure 1 The figure shows a schematic diagram of an artificial intelligence main framework, which describes the overall workflow of the artificial intelligence system and is applicable to the needs of the general artificial intelligence field.

[0046] The following will explain the above artificial intelligence theme framework from two dimensions: "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis).

[0047] The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data undergoes a condensed process of "data-information-knowledge-wisdom".

[0048] The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the system's industrial ecological process.

[0049] (1) Infrastructure

[0050] The infrastructure provides computing power support for the AI ​​system, enables communication with the outside world, and supports it through the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips (CPU, NPU, GPU, ASIC, FPGA and other hardware acceleration chips); the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system on the basic platform for calculation.

[0051] (2) Data

[0052] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data includes graphics, images, voice, text, IoT data of traditional devices, business data of existing systems, and perception data such as force, displacement, liquid level, temperature, and humidity.

[0053] (3) Data processing

[0054] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

[0055] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0056] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0057] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

[0058] (4) General ability

[0059] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0060] (5) Smart products and industry applications

[0061] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart manufacturing, smart transportation, smart home, smart medical care, smart security, autonomous driving, safe cities, smart terminals, etc.

[0062] The embodiments of the present application can be applied to the optimization design of representation learning models. The representation learning models optimized by the present application can be applied to various sub-fields in the field of artificial intelligence, such as speech processing, computer vision, text processing, natural language processing, etc. Figure 1 It is explained that the data in the dataset acquired by the infrastructure can be different types of data obtained through sensors such as sound collection equipment and cameras, such as audio data, image data, text data or video data, etc. The data in the dataset is converted into a representation vector using the representation learning model optimized by this application. At the same time, the representation vector contains as much data information as possible that is useful for subsequent tasks, making it easier to extract useful information when subsequently building a classifier or other prediction tasks.

[0063] The embodiments of the present application can be applied to Figure 2 The artificial intelligence system shown in Figure 1 is Figure 2 As shown, the artificial intelligence system includes a terminal device and a server, and the terminal device and the server are connected via a communication link.

[0064] The terminal device can be any computing device, such as a personal computer, a computer workstation, a smart phone, a tablet computer, a smart camera, a virtual reality (VR) device, a smart car or other type of cellular phone, a media consumption device, a wearable device, a set-top box, a game console, etc. Based on the advantages of neural networks, Figure 2 The terminal device shown deploys a neural network, and the terminal device deployed with the neural network can be applied to a variety of scenarios. As an example, the terminal device can be used to perform image processing on pictures taken by users, and the image processing may include denoising, de-mosaicing, white balance, etc.

[0065] In addition, different types of terminal devices have different requirements for neural networks. For example, different terminal devices have different computing delays for the same neural network due to differences in underlying hardware. The difference in type can refer to different types of terminal devices, such as smartphones and tablets, or different models of the same type of terminal devices. Therefore, a suitable neural network can be trained for a specified terminal device based on the type of terminal device, application scenario, and other possible restrictions.

[0066] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and concepts of the neural networks that may be involved in the embodiments of the present application are first introduced below.

[0067] (1) Neural Network

[0068] A neural network can be composed of neural units. A neural unit can be a s The output of the operation unit can be:

[0069]

[0070] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s For x s The weight of the neural unit, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0071] (2) Deep Neural Networks

[0072] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple hidden layers. According to the position of different layers, the neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer.

[0073] Although DNN looks complicated, the work of each layer can be represented by the following linear relationship expression: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After processing, the output vector Since DNN has many layers, the coefficient W and the offset vector The number of these parameters is also relatively large. The definitions of these parameters in DNN are as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as

[0074] It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by many layers of vector W).

[0075] (3) Convolutional Neural Network

[0076] Convolutional neural network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network contains a feature extractor consisting of a convolution layer and a subsampling layer, which can be regarded as a filter. Convolutional layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolutional layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layers. A convolutional layer usually contains several feature planes, each of which can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way to extract image information is independent of position. The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connection between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0077] (4) Loss Function

[0078] In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0079] (5) Hypernetwork

[0080] A hypernetwork can be understood as including at least one layer of space, each layer of space containing multiple substructures, each substructure including at least one layer of neurons; two substructures in the multiple substructures can contain the same layer of neurons, that is, two substructures in the multiple substructures can contain the same structure. In the embodiment of the present application, a hypernetwork can also be called a neural network.

[0081] (6) Subnetwork

[0082] The subnetwork can be understood as: selecting at least one substructure from each layer of the hypernetwork space, and then connecting the selected substructures in sequence according to the order of the spatial layers to form a neural network.

[0083] Among them, the neural network in the embodiments of the present application mainly refers to a deep neural network.

[0084] The following introduces the system architecture provided by the embodiments of the present application.

[0085] See attached Figure 3 The embodiment of the present application provides a neural network training system 200. The neural network training system 200 includes an execution device 210, a neural network training apparatus 220, a database 230, a user device 240, a data storage system 250 and a data acquisition device 260. The execution device 210 includes a computing module 211 and an input / output (I / O) device interface 212.

[0086] In the training phase, the data acquisition device 260 can be used to collect data sets (i.e., training sets and validation sets) for network training, and store the data sets in the database 230. The data sets may include samples from multiple source domains (which may be referred to as training data or sample data), and the sample data in the data sets may be image data, video data, text data, etc., as long as the data sets meet the function of iteratively training the super network of the embodiment of the present application, which is specifically related to the target task to be performed and is not limited here.

[0087] The neural network training device 220 can be used to iteratively train the super network based on the data set maintained in the database 230, and obtain the target sub-network 201 that can be deployed in the execution device 220 by performing model miniaturization processing (such as pruning, quantization, distillation, etc.) on the trained super network. The neural network training device 220 can be Figure 2 The following will describe in more detail how the neural network training device 220 trains the target sub-network 201 that can be deployed on the terminal device based on the sample data of the data set. Figure 4-8 Related description in .

[0088] The hypernetwork in the embodiment of the present application may specifically be a deep learning model, such as a deep neural network. It should be noted that in actual applications, the training data maintained in the database 230 may not all come from the data acquisition device 260, but may also be received from other devices. It should also be noted that the neural network training device 220 may not necessarily train the hypernetwork based entirely on the training data maintained by the database 230, but may also obtain training data from the cloud or other places for hypernetwork training. The above description should not be used as a limitation on the embodiments of the present application.

[0089] The target sub-network 201 obtained by the neural network training device 220 can be applied in different systems or devices. Figure 3 The execution device 220 shown in FIG. 2 may be Figure 2 The terminal devices shown are, for example, mobile terminals, tablet computers, laptop computers, augmented reality (AR) / virtual reality (VR), vehicle terminals, televisions, Internet of Things devices, etc. The execution device 210 interacts with external devices via the I / O device interface 212. The user can input data to the I / O device interface 212 via the user device 240. The input data may include, but is not limited to, image data and video data to be processed input by the user device 240 in the embodiment of the present application.

[0090] In the inference phase, the execution device 210 can call the data, code, etc. in the data storage system 250, or store the data, instructions, etc. in the data storage system 250. The data storage system 250 can be built into the execution device 210, or can be used as an external memory of the execution device 210. The computing module 211 can process the image data to be processed received by the I / O device interface 212 through the target subnetwork 201. The I / O device interface 212 can also return the processing results to the user device 240 (for example, the denoised image data), so as to provide them to the user.

[0091] It should be noted that the attached Figure 3 It is only a schematic diagram of a system architecture provided in an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure is only for illustrative purposes and does not constitute any limitation.

[0092] Depend on Figure 3 As can be seen from the description in, the embodiment of the present application includes a training phase and a reasoning phase, and the training phase and the reasoning phase are described below respectively.

[0093] 1. Training Phase

[0094] In the embodiment of the present application, the training phase is as described above. Figure 3 The neural network training device 220 uses the sample data in the training set to train the super network and performs model miniaturization processing on the trained super network to obtain the target sub-network 201 that can be deployed in the execution device 210.

[0095] like Figure 4 As shown, a neural network training method for image processing provided in an embodiment of the present application is provided. The method can be applied to a neural network training device 220. The method includes the following steps:

[0096] Step S41: construct a super network corresponding to the processing task.

[0097] In some embodiments, the processing task may indicate a task of a super network. For example, if the processing task is image denoising, a super network for image denoising may be constructed. The super network may include multiple sub networks, and the multiple sub networks share network parameters with the super network.

[0098] In some embodiments, the construction of a super network corresponding to the processing task can be achieved in the following manner: a search space is established based on a preset neural network, the search space indicates the range of changes in the network structure of the preset neural network, and the preset neural network contains a block corresponding to the processing task, and the Block can be composed of a single neural network layer or multiple neural network layers; a super network is constructed based on the search space. Since the search space indicates the range of changes in the network structure of the preset neural network, the search space can cover multiple sub-networks, that is, the super network constructed based on the search space can also cover multiple sub-networks. The preset neural network can be an existing neural network architecture, which is not limited in the embodiments of the present application.

[0099] For example, if the task of the hypernetwork is image denoising, the preset neural network must include a block that can perform image denoising, so that the hypernetwork completed by subsequent training has the ability to perform image denoising. For another example, if the task of the hypernetwork is to de-mosaic an image, the preset neural network must include a block that can perform image de-mosaicing, so that the hypernetwork completed by subsequent training has the ability to perform image de-mosaicing.

[0100] In some embodiments, the neural network training device may be deployed with an automatic machine learning (AutoML) system. The neural network training device deployed with the AutoML system may modify the preset neural network according to the processing task specified by the user, and add a block corresponding to the processing task, so that a super network corresponding to the processing task can be constructed later.

[0101] In some embodiments, the neural network training device can train a super network that can be used to perform the corresponding task based on different types of training data for different tasks, and then perform model miniaturization processing on the super network to obtain a target sub network that can be deployed in the execution device. Since the target sub network inherits the network parameters (such as weights, activation values, etc.) of the super network, the target sub network can also be used to perform the corresponding task. For example, if you want to obtain a target sub network for image denoising, the training set obtained by the neural network training device may include multiple groups of sample pairs of noisy images and non-noisy images. If you want to obtain a sub network for de-mosaicing, the training set obtained by the neural network training device may include multiple groups of sample pairs of mosaic images and non-mosaic images.

[0102] Step S42: Perform sampling training on the hypernetwork.

[0103] In some embodiments, sampling and training a supernetwork may refer to sampling and training a supernetwork using image samples. The sampling and training of a supernetwork may include: sampling a supernetwork to obtain a subnetwork, training the subnetwork, and then updating the network parameters of the trained subnetwork to the supernetwork. Repeating the above sampling, training, and updating processes can achieve training of the supernetwork. The neural network training device may sample a supernetwork to obtain a subnetwork based on a preset sampling rule. The preset sampling rule may be pre-set and stored in the neural network training device. The preset sampling rule may be set according to actual needs, and the embodiments of the present application are not limited to this.

[0104] In some embodiments, the super network can be trained based on n sampling training stages in sequence, the n sampling training stages are respectively defined as the i-th sampling training stage, i=1, 2, ..., n, n is a positive integer greater than 1, and the sampling training stage before the i-th sampling training stage is defined as the i-1-th sampling training stage. In the i-1-th sampling training stage, the neural network training device can sample the super network based on a preset sampling rule to obtain the i-1-th sub-network, train the i-1-th sub-network using a training set, and update the network parameters of the trained i-1-th sub-network to the super network; in the i-th sampling training stage, the neural network training device samples the super network whose network parameters have been updated in the i-1 sampling training stage based on a preset sampling rule to obtain the i-th sub-network, train the i-th sub-network using a training set, and update the network parameters of the trained i-th sub-network to the super network whose network parameters have been updated in the i-1-th sampling training stage.

[0105] For example, in the first sampling training stage, the neural network training device samples the super network for the first time based on the preset sampling rules to obtain the first sub-network, and then uses the training data of the training set to train the first sub-network until the first sub-network converges, thus completing the training of the first sub-network. The network parameters of the trained first sub-network can be updated to the super network, that is, training the first sub-network is equivalent to training the super network once. Figure 5 As shown, the neural network training device trains the super network N s Perform the first sampling and get the first sub-network N 1 , the first sub-network N after training 1 The network parameters can be updated to the super network N s .

[0106] In the second sampling training stage, the neural network training device samples the super network (super network that has been trained once) for the second time to obtain the second sub network, and then trains the second sub network using the training data of the training set until the second sub network converges, and the training of the second sub network is completed. The network parameters of the trained second sub network can be updated to the super network, that is, training the second sub network is equivalent to training the super network that has been trained once again.

[0107] In the i-th sampling training stage, the neural network training device samples the super network (the super network that has been trained i-1 times) for the i-th time to obtain the i-th sub-network, and then trains the i-th sub-network using the training data of the training set until the i-th sub-network converges, thus completing the training of the i-th sub-network. The network parameters of the trained i-th sub-network can be updated to the super network, that is, training the i-th sub-network is equivalent to training the super network that has been trained i-1 times again.

[0108] In some embodiments, the preset sampling rules may include three dimensions: Block sampling dimension, width (number of channels) sampling dimension and depth sampling dimension. The process of sampling a super network to obtain a sub network may include: determining one or more Blocks, and then determining the width or depth of the Blocks. There is no restriction on the order of determining the depth or width of the Blocks. For example, the neural network training device may implement sampling of the super network once to obtain a sub network in the following manner: a1. Sampling the super network with Block as the dimension, for example, a single path one-step method may be used. a1. Determine the number of channels of each target block. For example, for each layer of the target block, the channels can be sampled from front to back in steps of 32, 64, or 96, i.e., the 1st to 32nd channels are sampled, or the 1st to 64th channels are sampled, or the 1st to 96th channels are sampled; a2. Determine the depth of each target block. Each target block can include multiple layers or multiple stages. The sampling can be performed in sequence starting from the first layer, such as sampling the first layer, or sampling the first and second layers, or sampling the first, second, and third layers. The sampling can also be performed in sequence starting from the first stage, such as sampling the first stage, or sampling the first and second stages, or sampling the first, second, and third stages; a3. Construct a subnetwork based on each target block and the width and depth of each target block.

[0109] In some embodiments, the larger the search space, the more sub-networks are covered, and a sub-network more suitable for the terminal device may be searched out; however, the larger the search space, the greater the difficulty of the search, and it will also lead to a longer search time, more resource consumption, and difficulty in responding quickly. Therefore, the neural network training device can construct a search space according to the sampling rules of the sub-network. Assuming that the Block sampling dimension includes k1 methods, the width sampling dimension includes k2 methods, and the depth sampling dimension includes k3 methods, the search space can be set to cover k1*k2*k3 sub-networks, and k1, k2, and k3 are all positive integers.

[0110] S43, determining a target sub-network based on the trained network parameters of the super-network.

[0111] In some embodiments, the target subnetwork may refer to a subnetwork whose performance meets preset requirements. For example, the preset requirements may refer to a subnetwork with the best performance evaluation result among multiple subnetworks to be performance evaluated. Subsequently, by performing quantization training on the target subnetwork, a subnetwork that can be deployed on a terminal device and has better performance (quantized target subnetwork) is obtained.

[0112] The target subnetwork can be a subnetwork within the search space, or a subnetwork obtained based on the mutation of a subnetwork within the search space. This allows searching for a subnetwork with better performance as the target subnetwork from a search space covering a limited number of subnetworks without causing the problem of an overly large search space.

[0113] In some embodiments, the neural network training apparatus may determine the target sub-network by:

[0114] b1. Set the evolutionary learning parameters of the subnetwork. The evolutionary learning parameters may include population size, subnetwork structure constraints, maximum number of iterations, validation set, and mutation probability. The population size indicates the number of subnetworks contained in the population, the subnetwork structure constraints indicate the constraints on the network structure of the mutated subnetwork, the maximum number of iterations indicates the number of population updates, the validation set contains sample data for performance evaluation of the subnetwork, and the mutation probability indicates the probability of subnetwork mutation;

[0115] b2. Assuming that the population size is m (m is a positive integer greater than 1), sample m subnetworks from the search space and add the m subnetworks to the population. The sampling method can be random sampling or other sampling methods, which is not limited in the present embodiment;

[0116] b3. Evaluate the performance of each subnetwork in the population based on the validation set and the network parameters of the trained supernetwork. Since each subnetwork shares network parameters with the supernetwork, the network structure of each subnetwork inherits the network parameters of the corresponding network structure in the supernetwork, that is, the network parameters of each subnetwork can be determined based on the network parameters of the trained supernetwork, and each subnetwork is verified by the sample data in the validation set. Then, the performance of each subnetwork can be evaluated based on the processing results of each subnetwork on each sample data in the validation set. For example, taking the peak signal-to-noise ratio (PSNR) of the subnetwork obtained based on the processing results of each subnetwork on each sample data in the validation set, the best performance evaluation result is the maximum PSNR as an example;

[0117] b4. Select the top k subnetworks with the highest PSNR value from the population (PSNR is sorted from high to low), where k is less than m, to update the population based on the k subnetworks;

[0118] b5. Mutate the k subnetworks with a mutation probability until the number of subnetworks satisfying the subnetwork structure constraint is equal to m, and form a new population with the m subnetworks, that is, the updated population includes the original k subnetworks and mk mutated subnetworks, and perform performance evaluation on each subnetwork in the new population; the mk mutated subnetworks may all be obtained based on the mutation of the k subnetworks (that is, they may be understood as having undergone one mutation), or some may be obtained based on the mutation of the k subnetworks, and some may be obtained based on the mutation of the mutated subnetworks again with a mutation probability, which may be understood as having undergone two, three or more mutations, and the embodiments of the present application are not limited to this;

[0119] like Figure 6 As shown in the figure, the top k subnetworks with the highest PSNR values ​​can be selected from the population Po, and the remaining mk subnetworks are removed from the population Po. The k subnetworks are mutated with the mutation probability to obtain multiple mutated subnetworks. Then, mk subnetworks that meet the subnetwork structure constraints are selected from the multiple mutated subnetworks and added to the population Po, thus achieving an update of the population Po.

[0120] b6. Repeat steps b3, b4 and b5 based on the maximum number of iterations. Assume that the maximum number of iterations is 5 times, that is, repeat steps b3, b4 and b5 5 times to update the population. The maximum number of iterations can be set according to the actual available training time, which is not limited in the present embodiment of the application;

[0121] b7. When the current population iteration number is equal to the maximum iteration number, the sub-network with the largest PSNR value is selected from the population as the target sub-network, that is, after the iteration is completed, the sub-network with the largest PSNR value is selected from the finally updated population as the target sub-network.

[0122] S44, performing quantization training on the target sub-network to obtain a quantized target sub-network, and the target sub-network and the super-network share a quantizer used for quantization training.

[0123] In some embodiments, by performing quantization training on the target subnetwork, the size of the target subnetwork can be reduced, so that the target subnetwork deployed on the terminal device side can achieve low latency and low power consumption of the subnetwork operation to the greatest extent while ensuring processing performance. The neural network training device can use quantization aware training (QAT), post-training quantization (PTQ), etc. to perform quantization training on the target subnetwork, and the target subnetwork and the super network use the same quantization parameters for quantization training, that is, the target subnetwork and the super network share a quantizer for iterative training, which can further improve the performance of the target subnetwork. For example, if the target subnetwork is used for image denoising, the target subnetwork and the super network share a quantizer, which can further improve the image denoising performance of the target subnetwork.

[0124] In some embodiments, the neural network training device can add a quantizer to the target sub-network in the following manner: c1. Input random data to the target sub-network to generate a directed acyclic graph (DAG) for forward reasoning. acyclic graph, DAG), the type of input data is the same as the type of training data, for example, if the training data is image data, the input data is also image data, and if the training data is text data, the input data is also text data; c2. Operator fusion is performed on the target sub-network based on DAG and a preset fusion strategy. For example, the preset fusion strategy can define the target operator that needs to be fused, and the target operator can be set according to the task of the target sub-network. A dominator tree can be generated based on DAG first, and then the operator fusion is performed according to the dominator tree to obtain the target operator; c3. A quantizer is added to the target sub-network after operator fusion. For each layer of the target sub-network, pseudo-quantization nodes can be added for the activation of each layer and the weight of the layer respectively. The pseudo-quantization node can refer to a node used for pseudo-quantization operation, thereby realizing the quantization of the weight of each layer and the quantization of the activation of each layer. If the target sub-network has a dynamic convolution part, for the dynamic convolution part, pseudo-quantization nodes can be added for the tensors that actually participate in the forward reasoning process, that is, the pseudo-quantization node is a linear mapping of the tensors that actually participate in the forward reasoning process. A pseudo-quantization node can be considered equivalent to a quantizer. Taking the pseudo-quantization operation on a floating-point number x as an example, the pseudo-quantization operation can be understood as quantizing the floating-point number x and then dequantizing it. A pseudo-quantization node can be defined by a scaling factor and an output bit width. The scaling factor defines the step size of the quantizer, and the output bit width defines the range of the quantizer output. The scaling factor and output bit width of the pseudo-quantization node can be set according to the inference performance available on the terminal device side. For example, for 8-bit quantization, the scaling factor defines the floating-point length f corresponding to the fixed point 0 to 255. lenThe output bit width defines the upper and lower limits of the floating point. The length of the interval between the upper and lower limits is the floating point length f. len .

[0125] It can be understood that since the target sub-network and the super-network share a quantizer, that is, after adding a pseudo-quantization node to the target sub-network, a portion of the super-network having the same network structure as the target sub-network is also added with a pseudo-quantization node.

[0126] In some embodiments, for the target subnetwork after adding pseudo-quantization nodes, the neural network training device can perform iterative training on the target subnetwork in the following manner. The number of iterative training can be set according to the actual available training time, which is not limited in the embodiments of the present application: d1. Sample multiple subnetworks from the search space, assuming j (j is a positive integer greater than 1) subnetworks, j subnetworks are the target subnetworks with pseudo-quantization nodes added and j-1 subnetworks larger than the target subnetwork. A subnetwork larger than the target subnetwork may refer to a subnetwork that contains the network structure of the target subnetwork, but is different from the target subnetwork. Since the target subnetwork shares a quantizer with the supernetwork, the parts of these j-1 subnetworks larger than the target subnetwork that have the same network structure as the target subnetwork are also added with pseudo-quantization nodes. For example, Figure 7 As shown, a subnetwork 202 ( Figure 7 The diagram shows two subnetworks 202 that are larger than the target subnetwork 201. Compared with the target subnetwork 201, the subnetworks 202 also include other additional network structures st1 and st2. d2. Perform backward reasoning training on each of the j subnetworks based on the data set to obtain the gradient corresponding to each subnetwork. d3. Update the gradient of the target subnetwork based on the gradient of each of the j subnetworks. The specific gradient update algorithm can be selected according to actual needs. The embodiment of the present application does not limit this. For example, a weighted operation can be performed on the gradients of the j subnetworks to obtain a first gradient, and then the first gradient is updated to the gradient of the target subnetwork. Since the super network and the target subnetwork share network parameters, updating the gradient of the target subnetwork is equivalent to updating the gradient of the super network. d4. Repeat steps d1 to d3. The number of repetitions is the number of iterative training of the target subnetwork.

[0127] The above-mentioned iterative training samples multiple sub-networks from the search space and conducts backward reasoning training together, so as to realize the distillation effect in the network structure search in the quantitative training, and completes pruning at the same time as the quantitative training, which can further improve the performance of the target sub-network finally deployed on the terminal device.

[0128] In some embodiments, the gradient update of the target subnetwork is a gradient descent update, and the gradient update formula is as follows:

[0129]

[0130] in, is the updated gradient of the target sub-network, is the gradient before the target sub-network is updated, η is the learning rate, is the partial derivative, α is the learning factor, For loss, is the gradient of the network structure of other additional parts.

[0131] In some embodiments, the neural network training device can also perform iterative training using only the supernetwork and the target subnetwork. Specifically, the number of iterative training can be set first, and then the target subnetwork can be iteratively trained in the following manner: e1. Based on the data set, backward reasoning training is performed on the target subnetwork and the supernetwork respectively to obtain the gradients corresponding to each of them; e2. The target subnetwork is gradient updated based on the gradient of the target subnetwork and the gradient of the supernetwork. The specific gradient update algorithm can be selected according to actual needs, and the embodiment of the present application is not limited to this. For example, the gradient of the target subnetwork can be updated in a weighted manner; e3. Steps e1 to e2 are repeated, and the number of repetitions is the number of iterative training of the target subnetwork.

[0132] In some embodiments of the present application, in order to make the quantization training of the target sub-network converge better and accelerate the convergence speed, the weight of the dynamic jitter in the target sub-network can be identified during the quantization training of the target sub-network, and the weight of the dynamic jitter is locked according to the weight stable state. The weight of the dynamic jitter indicates that the weight no longer converges and only jitters meaninglessly. By locking the value of the weight, the convergence speed of the target sub-network can be accelerated.

[0133] For example, the neural network training device can realize the identification of dynamically jittering weights and weight locking in the following ways: f1. Construct a weight monitor. The weight monitor can be constructed based on an existing function or algorithm to track each weight in the target subnetwork, which is not limited in the present embodiment of the application; f2. Based on the weight monitor, monitor each weight in the target subnetwork and identify the weights in dynamic jitter. For example, the shadow weights corresponding to each weight can be calculated based on the exponential moving average (EMA) algorithm. Dynamic jitter can be identified based on the shadow weights, which can improve the recognition accuracy; f3. For the weights in dynamic jitter, determine the relatively stable state value of the weight, set the value of the weight to the relatively stable state value, and lock the weight. The relatively stable state value can refer to a value that occurs more frequently, or it can be a value calculated based on a value that occurs more frequently, such as a value calculated based on a weighted average method. For example, if the weight of a certain dynamic jitter fluctuates between 0.2 and 0.3, the relatively stable state value is 0.2 or 0.3. The weight value of the dynamic jitter can be set to 0.2 or 0.3 to lock the weight.

[0134] like Figure 8 As shown in Figure 2, as the target sub-network is iteratively trained, the shadow weight change curve of the first weight of the target sub-network obtained based on the EMA algorithm is as follows: Figure 8 As shown by the curve L1 in Figure 8 It can be seen that after 25 iterations of training, the shadow weight of the first weight has been jittering near the quantization threshold and no longer converges. The first weight can be identified as a dynamically jittering weight. After monitoring that the jitter is maintained for a preset number of iterations, the first weight can be locked according to the relatively stable state value of the first weight. The preset number of iterations can be set according to actual needs, and this application does not limit this. For example, the preset number of iterations is 20 times. When it is determined that the shadow weight of the first weight has been jittering near the quantization threshold from the 25th iteration to the 44th iteration, the first weight is locked according to the relatively stable state value of the first weight, that is, starting from the 45th iteration of training, the first weight is a fixed value.

[0135] In the above training phase, the neural network training device obtains a target sub-network that can be deployed on the terminal device. The following describes the process of model application, that is, the process of executing the device to perform reasoning based on the target sub-network.

[0136] 2. Reasoning Stage

[0137] In the embodiments of this application, please refer to Fig. 9 , Fig. 9Another flowchart of a neural network training method for image processing provided in an embodiment of the present application. The neural network training method provided in an embodiment of the present application may include the following steps:

[0138] Step S91, the execution device obtains data to be processed.

[0139] In the embodiment of the present application, the data to be processed may be of the same type as the sample data of the data set. For example, the training phase uses image data for training, and the data to be processed acquired by the execution device is also image data, such as a frame of image.

[0140] Step S92: The execution device inputs the data to be processed into the target sub-network for processing.

[0141] Taking the target subnetwork for image denoising as an example, the data to be processed can be the data (matrix data) output by the image sensor, and the execution device can convert the matrix data into a tensor and input it into the target subnetwork. The data output by the target subnetwork is matrix data.

[0142] If the training phase is performed using RGB image data, the target subnetwork may include three output channels (R output channel, G output channel, and B output channel), and the data output by the three output channels are all matrix data.

[0143] Step S93: The execution device generates a processing result of the data to be processed based on the output of the target sub-network.

[0144] Taking image denoising by the target sub-network as an example, the execution device can generate a denoised image based on the output (matrix data) of the target sub-network.

[0145] Please also read Fig.10 , Fig.10 A schematic diagram of the structure of an execution device provided in one embodiment of the present application.

[0146] The execution device 100 includes an acquisition module 101 , an input module 102 and a generation module 103 .

[0147] The acquisition module 101 is used to acquire data to be processed.

[0148] The input module 102 is used to input the data to be processed into the target sub-network for processing.

[0149] The generating module 103 is used to generate the processing result of the data to be processed based on the output of the target sub-network.

[0150] It should be noted that the information interaction and execution process between the modules / units in the execution device are the same as those in this application. Fig. 9The method embodiments are based on the same concept, and the specific contents can be found in the description of the method embodiments shown in the previous description of this application, which will not be repeated here.

[0151] The present application also provides an execution device. Fig.11 , Fig.11 A schematic diagram of a structure of an execution device provided in an embodiment of the present application, wherein the execution device 110 may be deployed with Fig. 9 The target subnetwork described in the corresponding embodiment is used to implement Fig. 9 The execution device 110 includes a receiver 111, a transmitter 112, a processor 113 and a memory 114 (the number of the processor 113 in the execution device 110 can be one or more, Fig.11 In the example of FIG. 1 , the processor 113 may include an application processor 1131 and a communication processor 1132. In some embodiments of the present application, the receiver 111, the transmitter 112, the processor 113 and the memory 114 may be connected via a bus or other means.

[0152] The memory 114 may include a read-only memory and a random access memory, and provides instructions and data to the processor 113. A portion of the memory 114 may also include a non-volatile random access memory (NVRAM). The memory 114 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0153] The processor 113 controls the operation of the execution device. In a specific application, the various components of the execution device are coupled together through a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus, and a status signal bus, etc. However, for the sake of clarity, various buses are referred to as bus systems in the figure.

[0154] The above embodiments of the present application Fig. 9The disclosed method can be applied to the processor 113, or implemented by the processor 113. The processor 113 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 113. The above processor 113 can be a general processor, a digital signal processor (digital signal processing, DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (application specific integrated circuit, ASIC), a field programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The processor 113 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 114, and the processor 113 reads the information in the memory 114 and completes the steps of the above method in combination with its hardware.

[0155] The receiver 111 can be used to receive input digital or character information, and generate signal input related to the relevant settings and function control of the execution device. The receiver 111 can also include a camera module. The transmitter 112 can be used to output digital or character information through the first interface; the transmitter 112 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 12 can also include a display device such as a display screen.

[0156] In one embodiment of the present application, the application processor 1131 is used to execute Fig. 9 The corresponding embodiment executes the function of the device. It should be noted that the application processor 1131 executes Fig. 9 For the specific implementation methods and beneficial effects of the functions of the execution device in the corresponding embodiments, please refer to Fig. 9 The descriptions in the corresponding method embodiments will not be repeated here one by one.

[0157] The following uses the target subnetwork provided in the present application for processing an image as an example to introduce an example of an application scenario of the target subnetwork in the embodiment of the present application. Figure 12(a) is a schematic diagram of a scene for denoising an image provided in the embodiment of the present application. Figure 12(b) is a schematic diagram of a scene for de-mosaicing an image provided in the embodiment of the present application. Figure 12(c) is a schematic diagram of a scene for automatically white balancing an image provided in the embodiment of the present application.

[0158] As shown in Figure 12(a), a shooting application 122a and a camera system 123a are deployed in the client 121a. The shooting application 122a can be a shooting application that comes with the client 121a when it leaves the factory or a third-party shooting application downloaded through the network. A neural network training service 125a is deployed in the cloud server 124a. The neural network training service 125a is used to obtain the target subnetwork 126a provided in the embodiment of the present application based on data set training and quantization. The data set includes multiple groups of sample pairs of noisy images and non-noisy images. The client 121a can communicate with the cloud server 124a, so that the target subnetwork 126a obtained by the cloud server 124a can be deployed in the client 121a.

[0159] Specifically, when the user uses the client 121a to take a photo, the client 121a can start the shooting application 122a in response to the user operation. After the client 121a starts the shooting application 122a, the shooting application 122a can shoot an image in response to the user's shooting operation. For example, the user clicks the "shoot" icon in the shooting application 122a to initiate a shooting request for the currently displayed preview image, and then the camera system 123a can generate a shot image based on the preview image.

[0160] After generating the captured image, the camera system 123a may call the target subnetwork 126a to perform denoising on the captured image to obtain the denoised captured image (hereinafter referred to as the first captured image). After obtaining the first captured image, the camera system 123a may return the first captured image to the capturing application 122a, so that the user can determine whether to save the first captured image. The capturing application 122a may respond to the user's save operation and save the first captured image to the memory of the electronic device 10, or not save it. For example, the user clicks the "Save" icon in the capturing application 122a, and the capturing application 122a saves the first captured image to the read-only memory of the electronic device 10. The user clicks the "Don't Save" icon in the capturing application 122a, and the capturing application 122a deletes the first captured image, that is, the first captured image will not be saved to the read-only memory of the electronic device 10.

[0161] As shown in Figure 12(b), a photo album application 132a is deployed in the client 131a. The photo album application 132a can be a photo album application that comes with the client 131a when it leaves the factory or a third-party photo album application downloaded through the network. A neural network training service 134a is deployed in the cloud server 133a. The neural network training service 134a is used to obtain the target subnetwork 135a provided in the embodiment of the present application based on data set training and quantization. The data set includes multiple groups of sample pairs with mosaic images and without mosaic images. The client 131a can communicate with the cloud server 133a, so that the target subnetwork 135a obtained by the cloud server 133a can be deployed in the client 131a.

[0162] Specifically, when the user wants to de-mosaic the first image in the album application 132a (the first image contains mosaics), the client 131a can start the album application 132a in response to the user operation. After the client 131a starts the album application 132a, the album application 132a can select the first image in response to the user's shooting operation. For example, the user clicks on the first image previewed in the album application 132a, and then clicks on the "Edit" icon, and multiple image editing function icons pop up (assuming that multiple image editing function icons include de-mosaic icons). The user then clicks on the "Mosaic" icon, and the album application 132a can call the target subnetwork 135a to de-mosaic the first image to obtain an image after de-mosaicing (hereinafter referred to as the second image). The album application 132a can save the second image to the memory of the electronic device 10 in response to the user's save operation, or not save it.

[0163] If the second image is saved, the second image can be saved in the form of overwriting the original first image (i.e., after saving the second image to the album application 132a, the first image no longer exists in the album application 132a), or it can be saved in the form of a new image (i.e., after saving the second image to the album application 132a, the first image still exists in the album application 132a), and the embodiment of the present application is not limited to this.

[0164] As shown in FIG. 12( c), a shooting application 142a and a camera system 143a are deployed in the client 141a. The shooting application 142a can be a shooting application that comes with the client 141a when it leaves the factory or a third-party shooting application downloaded through the network. A neural network training service 145a is deployed in the cloud server 144a. The neural network training service 145a is used to obtain the target subnetwork 146a provided in the embodiment of the present application based on data set training and quantization. The data set includes multiple groups of sample pairs of images that have not been processed with white balance and images that have been processed with white balance. The client 141a can communicate with the cloud server 144a, so that the target subnetwork 146a obtained by the cloud server 144a can be deployed in the client 141a.

[0165] Specifically, when the user uses the client 141a to take a photo, the client 141a can start the shooting application 122a in response to the user operation. After the client 141a starts the shooting application 142a, the shooting application 142a can shoot an image in response to the user's shooting operation. For example, the user clicks the "shoot" icon in the shooting application 142a to initiate a shooting request for the currently displayed preview image, and then the camera system 143a can generate a shot image based on the preview image.

[0166] After generating the captured image, the camera system 143a may call the target subnetwork 146a to perform white balance processing on the captured image to obtain a captured image after white balance processing (hereinafter referred to as a second captured image). After obtaining the second captured image, the camera system 143a may return the second captured image to the capturing application 142a to facilitate the user to determine whether to save the second captured image. The capturing application 142a may respond to the user's saving operation and save the second captured image to the memory of the electronic device 10, or not save it.

[0167] In some embodiments, during the shooting preview stage, the target subnetwork 146a may also be called to perform white balance processing on the preview image, so that the preview image displayed by the shooting application 142a is also an image after white balance processing.

[0168] Please also read Fig.13 , Fig.13 A schematic diagram of the structure of a neural network training device provided in an embodiment of the present application.

[0169] The neural network training device 130 includes a construction module 131 , a training module 132 , a determination module 133 and a quantization module 134 .

[0170] The construction module 131 is used to construct a super network corresponding to the processing task, where the super network includes multiple sub-networks, and the multiple sub-networks share network parameters with the super network.

[0171] The training module 132 is used to perform sampling training on the super network.

[0172] The determination module 133 is used to determine the target sub-network based on the network parameters of the trained super-network, where the target sub-network is a sub-network in the super-network or is obtained by mutating a sub-network in the super-network, and the performance of the target sub-network meets the preset requirements.

[0173] The quantization module 134 is used to perform quantization training on the target sub-network to obtain a quantized target sub-network. The target sub-network and the super-network share a quantizer used for quantization training.

[0174] It should be noted that the information interaction and execution process between the modules / units in the neural network training device are the same as those in this application. Figures 4 to 8 The corresponding method embodiments are based on the same concept. For specific contents, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0175] Please also read Fig.14 , Fig.14 A schematic diagram of the structure of a neural network training device provided in an embodiment of the present application.

[0176] The neural network training device 140 may be equipped with Fig.13 The neural network training device 130 described in the corresponding embodiment is used to implement Fig.13 The function of the corresponding neural network training device or Figures 4 to 8 Corresponding method embodiments. Specifically, the neural network training device 140 is implemented by one or more servers. The neural network training device 140 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 142 (for example, one or more processors) and memory 143, and one or more storage media 146 (for example, one or more mass storage devices) storing application programs 144 or data 145. Among them, the memory 143 and the storage medium 146 can be short-term storage or permanent storage. The program stored in the storage medium 146 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the neural network training device. Furthermore, the central processor 142 can be configured to communicate with the storage medium 146 to execute a series of instruction operations in the storage medium 146 on the neural network training device 140.

[0177] The neural network training device 140 may also include one or more power supplies 147, one or more wired or wireless network interfaces 148, one or more input and output interfaces 149, and / or one or more operating systems 150, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0178] In one embodiment of the present application, the central processor 142 is used to implement Fig.13 The function of the neural network training device in the corresponding embodiment or Figures 4 to 8 Corresponding method embodiments. It should be noted that for the central processor 142 to execute Figures 4 to 8 The specific implementation methods and beneficial effects of the functions of the neural network training device in the corresponding embodiments can be referred to in Figure 11 to Figure 1 2 The descriptions in the corresponding method embodiments will not be repeated here one by one.

[0179] In another embodiment of the present application, the central processor 142 is used to implement Fig.13 The function of the neural network training device in the corresponding embodiment. It should be noted that the central processor 142 executes Fig.13 The specific implementation methods and beneficial effects of the functions of the neural network training device in the corresponding embodiments can be referred to in Fig.13 The descriptions in the corresponding method embodiments will not be repeated here one by one.

[0180] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores a program, which, when executed on a computer, enables the computer to execute the steps executed by the neural network training apparatus in the above embodiment, or to execute the steps executed by the above execution device.

[0181] The present application also provides a computer program product which, when executed on a computer, enables the computer to execute the steps executed by the neural network training apparatus described above, or to execute the steps executed by the execution device described above.

[0182] The present application also provides a neural network deployment system, including a terminal device and a server, wherein the server is used to execute the steps executed by the neural network training device in the above embodiment, and the server is also used to deploy the quantized target sub-network in the terminal device.

Claims

1. A neural network training method for image processing, characterized in that: The method comprises: Constructing a super network corresponding to the processing task, the super network comprising a plurality of sub networks, the plurality of sub networks sharing network parameters with the super network; Performing sampling training on the super network, wherein the training samples of the super network include image samples; Based on a preset validation set and the network parameters of the trained super-network, each sub-network in the population is respectively evaluated for performance, and a sub-network whose performance evaluation result meets preset requirements is selected from the population as a target sub-network, wherein the population includes a sub-network obtained by sampling the super-network and a sub-network obtained by mutating the sampled sub-network, and a pseudo-quantization node is added to the target sub-network, and the pseudo-quantization node is a node used for performing a pseudo-quantization operation; Performing quantization training on the target sub-network to obtain a quantized target sub-network, wherein the target sub-network shares the pseudo-quantization node with the super-network; The quantization training of the target sub-network includes multiple quantization training stages, and each quantization training stage includes: Sampling the super network to obtain j sub-networks, wherein the j sub-networks include a target sub-network to which the pseudo-quantization nodes are added and j-1 sub-networks having a larger network structure than the target sub-network, and the j-1 sub-networks all include the network structure of the target sub-network; Performing backward inference training on each of the j subnetworks to obtain a gradient corresponding to each subnetwork; Update the gradient of the target subnetwork based on the gradient of each subnetwork in the j subnetworks; The target sub-network adds the pseudo quantization node, including: Inputting a random data into the target subnetwork to generate a directed acyclic graph DAG for forward reasoning, wherein the data type of the random data is the same as the data type of the data used to train the supernetwork; Performing operator fusion on the target sub-network based on the DAG and a preset fusion strategy; Add the pseudo quantization node to the target sub-network after operator fusion.

2. The method according to claim 1, characterized in that The step of constructing a hypernetwork corresponding to the processing task includes: Establishing a search space based on a preset neural network, the search space indicating a variation range of a network structure of the preset neural network, and the preset neural network comprising a block corresponding to the processing task, the block consisting of a single neural network layer or a plurality of neural network layers; The super network is constructed based on the search space.

3. The method according to claim 2, characterized in that The super network is trained based on n sampling training stages in sequence, the n sampling training stages are respectively defined as i-th sampling training stages, i=1, 2, ..., n, n is a positive integer greater than 1, the previous sampling training stage of the i-th sampling training stage is defined as the i-1-th sampling training stage, and the sampling training of the super network includes: In the i-1th sampling training stage, the super network is sampled based on a preset sampling rule to obtain an i-1th sub-network; Acquire a data set for network training, train the i-1th sub-network using the data set, and update the network parameters of the trained i-1th sub-network to the super-network; In the i-th sampling training stage, sampling the super network whose network parameters have been updated in the i-1-th sampling training stage based on the preset sampling rule to obtain the i-th sub-network; The i-th sub-network is trained using the data set, and the network parameters of the trained i-th sub-network are updated to the super-network whose network parameters have been updated in the i-1-th sampling training stage.

4. The method according to claim 3, characterized in that The search space uses blocks as search dimensions, and the preset sampling rules include: Perform block sampling in the search space based on the single-path one-step method SPOS to determine one or more target blocks; Determine the number of channels for each target block; Determining the depth of each target block; A subnetwork is constructed based on each target block, the number of channels of each target block, and the depth.

5. The method according to claim 1, characterized in that The method of performing performance evaluation on each sub-network in the population based on the preset validation set and the network parameters of the trained super-network, and selecting a sub-network whose performance evaluation result meets the preset requirements from the population as a target sub-network, includes: Sampling the super network to obtain m sub-networks, and constructing a first population based on the m sub-networks, where m is a positive integer greater than 1; Performing performance evaluation on each sub-network in the first population based on a preset validation set and the trained network parameters of the super-network; Select the top k sub-networks in terms of performance from the first population, where k is less than m; Mutating the k sub-networks with a preset mutation probability until mk sub-networks are obtained, and the mk sub-networks all satisfy the preset sub-network structure constraints; constructing a second population based on the k sub-networks and the mk sub-networks, and performing performance evaluation on each sub-network in the second population; A subnetwork whose performance evaluation result meets the preset requirement is selected from the second population as the target subnetwork.

6. The method according to claim 5, characterized in that The target sub-network is determined based on z population iterations, where z is a positive integer greater than 1, and the sub-network with the best performance evaluation result is selected from the second population as the target sub-network, including: If the current population iteration number is less than z, repeat the steps of selecting the top k subnetworks in performance from the currently constructed population, mutating the k subnetworks, and constructing a new population based on the k subnetworks and the mutated mk subnetworks; If the current population iteration number is equal to z, a subnetwork whose performance evaluation result meets the preset requirement is selected from the currently constructed population as the target subnetwork.

7. The method according to claim 1, characterized in that The adding the pseudo quantization node to the target sub-network after operator fusion includes: Locate the activations and weights of each layer on the target subnetwork after operator fusion; The pseudo quantization nodes are added to the activation and weight of each layer respectively.

8. The method according to claim 7, characterized in that The method further comprises: Tracking the weights of each layer on the target subnetwork; When it is identified that the target sub-network has a dynamically jittered weight, the dynamically jittered weight is locked.

9. The method according to claim 8, characterized in that The tracking of the weight of each layer on the target subnetwork includes: Determine a shadow weight corresponding to the weight of each layer on the target subnetwork based on an exponential moving average (EMA) algorithm; Based on the shadow weights, weights with dynamic jitter are identified.

10. The method according to claim 8, characterized in that The step of locking the weight of the dynamic jitter includes: A relatively stable state value of the weight of the dynamic jitter is determined, and the weight of the dynamic jitter is assigned based on the relatively stable state value.

11. The method according to any one of claims 1 to 10, characterized in that The method further comprises: Based on the quantized target subnetwork, image processing is performed on the image to be processed, wherein the image processing includes at least one of noise reduction processing, demosaicing processing, and automatic white balance processing, and the image samples include at least one of a plurality of groups of sample pairs of noisy images and non-noise images, a plurality of groups of sample pairs of mosaic images and non-mosaic images, and a plurality of groups of sample pairs of images that have not been processed with white balance and images that have been processed with white balance.

12. A neural network training device for image processing, characterized in that: The device comprises: A construction module, used to construct a super network corresponding to the processing task, wherein the super network includes a plurality of sub networks, and the plurality of sub networks share network parameters with the super network; A training module, used for performing sampling training on the super network, wherein the training samples of the super network include image samples; A determination module, used to perform performance evaluation on each subnetwork in the population based on a preset validation set and the network parameters of the trained supernetwork, and select a subnetwork whose performance evaluation result meets preset requirements from the population as a target subnetwork, wherein the population includes a subnetwork obtained by sampling the supernetwork and a subnetwork obtained by mutating the sampled subnetwork, and the target subnetwork is added with a pseudo-quantization node, which is a node used for performing a pseudo-quantization operation; A quantization module, used for performing quantization training on the target sub-network to obtain a quantized target sub-network, wherein the target sub-network shares the pseudo-quantization node with the super-network; The quantization module performs quantization training on the target sub-network, including multiple quantization training stages, and each quantization training stage includes: Sampling the super network to obtain j sub-networks, wherein the j sub-networks include a target sub-network to which the pseudo-quantization nodes are added and j-1 sub-networks having a larger network structure than the target sub-network, and the j-1 sub-networks all include the network structure of the target sub-network; Performing backward inference training on each of the j subnetworks to obtain a gradient corresponding to each subnetwork; Update the gradient of the target subnetwork based on the gradient of each subnetwork in the j subnetworks; The target sub-network adds the pseudo quantization node, including: Inputting a random data into the target subnetwork to generate a directed acyclic graph DAG for forward reasoning, wherein the data type of the random data is the same as the data type of the data used to train the supernetwork; Performing operator fusion on the target sub-network based on the DAG and a preset fusion strategy; Add the pseudo quantization node to the target sub-network after operator fusion.

13. A server, characterized in that: It comprises a memory and a processor, wherein the memory is used to store computer-readable instructions; the processor is used to read the computer-readable instructions and implement the neural network training method for image processing as described in any one of claims 1 to 11.

14. A terminal device, characterized in that: The terminal device is deployed with a target sub-network obtained based on the neural network training method for image processing according to any one of claims 1 to 11, and the target sub-network is used for image processing, and the image processing includes at least one of noise reduction processing, de-mosaic processing, and automatic white balance processing.

15. A computer storage medium, characterized in that Computer-readable instructions are stored, and when the computer-readable instructions are executed by a processor, a neural network training method for image processing as claimed in any one of claims 1 to 11 is implemented.

16. A computer program product, characterized in that The computer program product includes computer-readable instructions, and when the computer-readable instructions are executed by a processor, the neural network training method for image processing described in any one of claims 1 to 11 is implemented.

17. A neural network deployment system, characterized in that: It comprises a terminal device and a server, wherein the server is used to execute the neural network training method applied to image processing as described in any one of claims 1 to 11, and to deploy the quantized target subnetwork in the terminal device.

Images