Image processing method, model compression method and related equipment

Abstract

The present application provides an image processing method, a model compression method and related equipment. The image processing method uses the first neural network obtained by modular distillation training to process the image, which can save the computing power of the mobile terminal and reduce the consumption of the mobile terminal. In some embodiments, the first neural network is obtained by combining quantization-aware training with modular distillation training, which can further reduce the consumption of the mobile terminal. The model compression method uses the first convolution module in the second neural network as the teacher model and the corresponding second convolution module in the first neural network as the student model to perform modular distillation training to update the parameters in the first neural network, which can significantly improve the computing efficiency and memory optimization of the GPU server. In some embodiments, quantization-aware training is performed on the first neural network in the modular distillation training architecture, which can further improve the computing efficiency and memory optimization of the GPU server.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to an image processing method, a model compression method and related equipment. Background Art

[0002] With the development of artificial intelligence (AI) and neural network (NN) technologies, neural network models have been widely used in AI applications in the fields of image processing, speech recognition, and natural language processing, making the number of AI applications using neural network models increase year by year.

[0003] The parameters of a neural network model are usually in the millions, tens of millions, or hundreds of millions. Therefore, the storage and computing power requirements of the terminal devices that run AI applications using neural network models are relatively high, which limits the use of neural network models on terminal devices. In order to reduce computing costs and power consumption, the artificial intelligence model can be compressed, and the compressed convolutional neural network model can be deployed to the mobile terminal. For example, when it is necessary to deploy a model for pixel-level image tasks such as optimizing image resolution on a terminal device such as a mobile phone, a device with a graphics processing unit (GPU) server will be used to compress the model and deploy the compressed model to the mobile terminal. Summary of the invention

[0004] The present application provides an image processing method, a model compression method and related devices. The image processing method can save the computing power of the mobile terminal and reduce the power consumption of the mobile terminal; the model compression method can significantly improve the computing efficiency and memory optimization of the GPU server.

[0005] It should be noted that the model compression method provided in this application is a general compression scheme for artificial intelligence models. The model compression method is highly adaptable, and can realize the distillation of submodules of the model for artificial intelligence models of different application types (optionally, realize the combination of distillation of submodules and quantization), greatly reducing the power consumption of the model caused by calculation on the mobile terminal. The image processing method provided in this application can be understood as an example of an application scenario applicable to the model compression method provided in this application, and does not limit the application scenarios to which the model compression method is applicable.

[0006] In a first aspect, an image processing method is provided, which is applied to an electronic device, the method comprising: acquiring an image; processing the image based on a first neural network; and outputting the image processed by the first neural network, wherein the first neural network is obtained by module-by-module distillation training, wherein the first neural network is obtained by module-by-module distillation training, comprising: taking each first convolution module of X first convolution modules included in the second neural network as a teacher model, taking each second convolution module of X second convolution modules included in the first neural network as a student model, and updating the parameters of the first neural network according to the first output result of each first convolution module and the second output result of each second convolution module, wherein the first output result is obtained by inputting input data to each first convolution module, and the second output result is obtained by inputting input data to each second convolution module, the X second convolution modules correspond one-to-one to the X first convolution modules, each first convolution module includes M convolution layers, each second convolution module includes N convolution layers, X>1, M>N≥1, and X, M and N are all integers.

[0007] In the embodiment of the present application, the first neural network is obtained by modular distillation training, and the parameter amount of each convolution module in the first neural network is reduced compared to the parameter amount of each convolution module in the pre-trained second neural network. Therefore, the parameter amount of the first neural network is also significantly reduced compared to the parameter amount of the second neural network, and the performance is significantly improved, thereby saving the computing power of the mobile terminal and reducing the power consumption of the mobile terminal.

[0008] In combination with the first aspect, in a possible embodiment, processing an image based on a first neural network includes: obtaining image data based on the image, the image data including data of multiple pixels in the image; inputting the image data into the first neural network; performing convolution calculations on the image data by X·N convolutional layers included in X second convolutional modules, wherein X·N represents the multiplication of X and N, a first calculation result of the j-th convolutional layer among the X·N convolutional layers is calculated by other non-convolutional layers to obtain a second calculation result, the second calculation result is input into the j+1-th convolutional layer and calculated by the j+1-th convolutional layer, 1≤j≤X·N-1 and j is an integer.

[0009] As an example, after the electronic device acquires an image, it obtains image data (hereinafter referred to as image data #1 for the convenience of explanation), and image data #1 includes data of multiple pixels in the image. For example, the image data #1 can be in the original (RAW) format or the red, green and blue (RGB) format. Image data #1 is input into the first neural network. Specifically, the input layer first parses and processes image data #1, and outputs a feature map that can be processed by the network layer (including convolution layer, pooling layer, activation layer, etc.). The X second convolution modules include X·N convolution layers, which perform convolution calculations on the vector. The jth convolution layer in the X·N convolution layers outputs the first calculation result; the jth calculation result is input into other non-convolution layers, and the second calculation result is output. The second calculation result is input into the j+1th convolution layer, where 1≤j≤X·N-1 and j is an integer. The calculation result output by the X·Nth convolution layer is input into the output layer, and the output layer parses and processes the calculation result, and outputs the image processed by the first neural network (hereinafter referred to as image data #2 for the convenience of explanation). It can be understood that after each convolution layer in the first neural network performs convolution calculation on the feature map input to the convolution layer, the calculation result is calculated by other non-convolution layers and then input into the next convolution layer, and finally the first neural network outputs the processed image. In a possible implementation, the image processed by the first neural network can be image data in RAW format or RGB format.

[0010] For example, in a mobile game scenario, the resolution of an image is optimized by the first neural network. Generally, the mobile terminal first separates the acquired frames of images into text data and image data #1, and then inputs the image data #1 into the neural network model. The first neural network calculates the image data #1 in the manner described in the above example and outputs the image data #2 with optimized resolution.

[0011] In combination with the first aspect, in a possible embodiment, after input data is input to each second convolution module, the parameters of each second convolution module are also updated through quantization-aware training, wherein the parameters of each second convolution module before the quantization-aware training are all floating-point values, and the parameters of each second convolution module after the quantization-aware training are all integer values.

[0012] In the above scheme, when the first neural network is obtained through modular distillation and quantization-aware training, the first neural network is further compressed compared to the second neural network, thereby further saving the computing power of the mobile terminal and reducing the power consumption of the mobile terminal.

[0013] In combination with the first aspect, in a possible embodiment, the parameters of the first neural network are updated according to the first output result of each first convolution module and the second output result of each second convolution module, including: the parameters in the i-th second convolution module in the first neural network are obtained by back propagation based on the i-th first loss function value; the i-th first loss function value is obtained according to the i-th first sub-loss function value and the i-th second sub-loss function value, wherein the i-th first sub-loss function value is obtained according to the i-th first output result of the i-th first convolution module in X first convolution modules and the i-th second output result of the i-th second convolution module in X second convolution modules, and the i-th second sub-loss function value is obtained according to the i-th second output result and the target data, 1≤i≤X and i is an integer.

[0014] In the above scheme, the first neural network has higher accuracy than the neural network obtained by traditional distillation training, thereby improving the effect of image processing.

[0015] It can be understood that compared to the traditional distillation training architecture in which the output results of a neural network (as a student model) are used to fit the output results of another neural network (as a teacher model), the output results of a second convolution module (as a student model) in the modular distillation training architecture can better fit the output results of a second convolution module (as a teacher model), thereby improving the accuracy of the final output results of the first neural network.

[0016] In combination with the first aspect, in a possible embodiment, the parameters of the second neural network are updated according to the first output result of each first convolution module and the second output result of each second convolution module.

[0017] In combination with the first aspect, in a possible embodiment, the parameters of the second neural network are updated according to the first output result of each first convolution module and the second output result of each second convolution module, including: the parameters of the second neural network are obtained by back propagation based on the i-th first loss function value, the i-th first loss function value is calculated based on the i-th first sub-loss function value and the i-th second sub-loss function value, wherein the i-th first sub-loss function value is obtained based on the i-th first output result of the i-th first convolution module among X first convolution modules and the i-th second output result of the i-th second convolution module among X second convolution modules, the i-th second sub-loss function value is obtained based on the i-th second output result and the target data, 1≤i≤X and i is an integer.

[0018] In combination with the first aspect, in a possible embodiment, the parameters of the second neural network and the parameters of the first neural network are also updated by the sum of the X first loss function values ​​corresponding to the X second convolution modules and the second loss function value, wherein the second neural network is used as a teacher model and the first neural network is used as a student model, the second loss function value is obtained based on the third output result of the second neural network and the fourth output result of the first neural network, and the third output result and the fourth output result are obtained by inputting input data to the second neural network and the first neural network respectively.

[0019] In the above scheme, the first neural network has higher accuracy than the neural network obtained by traditional distillation training, thereby improving the effect of image processing.

[0020] It can be understood that the parameters of the first neural network are updated through module-by-module back propagation and then updated through global back propagation. That is to say, the parameters of the first neural network are updated multiple times in a distillation training architecture, so that the accuracy of the first neural network finally obtained is higher than that of the neural network obtained by traditional distillation training.

[0021] In combination with the first aspect, in a possible embodiment, the second loss function value is obtained based on the third output result of the second neural network and the fourth output result of the first neural network, including: the second loss function value is calculated based on the third sub-loss function value and the fourth sub-loss function value, wherein the third sub-loss function value is obtained based on the third output result and the fourth output result, and the fourth sub-loss function value is obtained based on the fourth output result and the target data.

[0022] In a second aspect, a model compression method is provided, comprising: taking each of the X first convolution modules included in the second neural network as a teacher model, taking each of the X second convolution modules included in the first neural network as a student model, inputting input data for each first convolution module and each second convolution module respectively, the X second convolution modules correspond one-to-one to the X first convolution modules, each first convolution module includes M convolution layers, each second convolution module includes N convolution layers, X>1, M>N≥1 and X, M and N are all integers; updating the parameters of the first neural network according to the first output result of each first convolution module and the second output result of each second convolution module.

[0023] In the embodiment of the present application, on the one hand, by dividing the first neural network and the second neural network into a plurality of convolution modules respectively, the computational complexity of each second convolution module is much lower than the computational complexity of the entire first neural network, and the computational complexity of each first convolution module is also much lower than the computational complexity of the entire second neural network, thereby saving training time compared to the commonly used distillation training; on the other hand, the calculations of each second convolution module can be independent of each other so that each second convolution module can be calculated in parallel, and each first convolution module is similar thereto, which can also save time compared to the commonly used distillation training architecture in which the calculations of each convolution layer of the teacher network and the super-resolution network are interrelated. Therefore, the model compression method provided in the present application can significantly improve the training efficiency, thereby significantly improving the computing efficiency and memory optimization of the GPU server.

[0024] In combination with the second aspect, in a possible embodiment, quantization-aware training is performed on each second convolution module, wherein the parameters of each second convolution module before the quantization-aware training are all floating-point values, and the parameters of each second convolution module after the quantization-aware training are all integer values.

[0025] The above scheme performs quantization-aware training on each second convolution module in the distillation training architecture, so that the parameters of the first neural network trained based on the distillation training architecture are all integer values, and quantization and distillation are performed simultaneously to obtain the first neural network trained by distillation and quantization, further improving the computing efficiency and memory optimization of the GPU server.

[0026] In combination with the second aspect, in a possible embodiment, updating the parameters of the first neural network according to the first output result of each first convolution module and the second output result of each second convolution module includes: obtaining the i-th first sub-loss function value according to the i-th first output result of the i-th first convolution module in the X first convolution modules and the i-th second output result of the i-th second convolution module in the X second convolution modules; obtaining the i-th second sub-loss function value according to the i-th second output result and the target data; obtaining the i-th first loss function value according to the i-th first sub-loss function value and the i-th second sub-loss function value; performing back propagation based on the i-th first loss function value to update the parameters in the i-th second convolution module, 1≤i≤X and i is an integer.

[0027] The above scheme performs backpropagation on the first neural network based on the first loss function value in a modular manner to update the parameters of the first neural network, which can improve the convergence performance of the first neural network and the accuracy of the final output results, and make up for the accuracy loss that may be caused by quantization-aware training and distillation.

[0028] It can be understood that for multiple first convolution modules and multiple second convolution modules, the intermediate result supervision of each module is performed, that is, the output result of the corresponding second convolution module is supervised based on the output result of a certain first convolution module. Compared with the difference in the complexity of the network structure between the first neural network and the second neural network in the traditional distillation training architecture, the difference in the complexity of the network structure between the first convolution module and the corresponding second convolution module is obviously much smaller. Therefore, compared with the traditional distillation training architecture in which the output result of the first neural network is used to fit the output result of the second neural network, the output result of the second convolution module can not only better fit the output result of the first convolution module, thereby improving the accuracy of the output result of the second convolution module, and then improve the accuracy of the final output result of the first neural network; and the time required for fitting will also be shorter, so that the convergence performance of the first neural network is also improved during the modular distillation training process.

[0029] In combination with the second aspect, in a possible embodiment, the method further includes: updating the parameters of the second neural network according to the first output result of each first convolution module and the second output result of each second convolution module.

[0030] The above scheme performs backpropagation on the second neural network based on the first loss function value in a modular manner to update the parameters of the second neural network, which can improve the convergence performance of the first neural network and the accuracy of the final output result.

[0031] It can be understood that for a certain first convolution module and a corresponding second convolution module, the same loss function value is used to update parameters. Compared with the difference in the complexity of the network structure between the first neural network and the second neural network in the traditional distillation training architecture, the difference in the complexity of the network structure between the first convolution module and the corresponding second convolution module is obviously much smaller. Thus, making the accuracy of the second convolution module reach a certain threshold loss function value can also make the accuracy of the first convolution module reach or approach the threshold, the accuracy of the first convolution module is guaranteed, and thus the accuracy of the second neural network can also be guaranteed. In addition, since the parameters are updated in modules, compared with updating the parameters of the entire second neural network, the computational complexity is significantly reduced and multiple modules can be calculated in parallel, which can also improve the convergence performance.

[0032] In combination with the second aspect, in a possible embodiment, updating the parameters of the second neural network according to the first output result of each first convolution module and the second output result of each second convolution module includes: obtaining the i-th first sub-loss function value according to the i-th first output result of the i-th first convolution module in the X first convolution modules and the i-th second output result of the i-th second convolution module in the X second convolution modules; obtaining the i-th second sub-loss function value according to the i-th second output result and the target data; calculating the i-th first loss function value according to the i-th first sub-loss function value and the i-th second sub-loss function value; and performing back propagation based on the i-th first loss function value to update the parameters in the i-th first convolution module, where 1≤i≤X and i is an integer.

[0033] In combination with the second aspect, in a possible embodiment, the method further includes: taking the second neural network as a teacher model and the first neural network as a student model, inputting input data for the second neural network and the first neural network respectively; obtaining a second loss function value according to the third output result of the second neural network and the fourth output result of the first neural network; and updating the parameters of the second neural network and the parameters of the first neural network according to the sum of the X first loss function values ​​corresponding to the X second convolution modules and / or the second loss function value.

[0034] The above scheme, after performing back-propagation on the modules, also performs global back-propagation on the entire neural network, which can further improve the training efficiency and the accuracy of the first neural network and the second neural network obtained in the end, and make up for the accuracy loss that may be caused by quantization-aware training and distillation.

[0035] It can be understood that, since the parameters of the first neural network and the second neural network have been updated once based on the back propagation of the sub-modules before the global back propagation, the back propagation of the sub-modules can play an auxiliary role in the global back propagation, so that when the global back propagation is performed, the time for the first neural network and the second neural network to be trained to convergence is significantly reduced, thereby improving the training efficiency. In addition, the parameters of each second convolution module in the first neural network and each first convolution module in the second neural network can be updated multiple times in a distillation training architecture, which also improves the training efficiency and the accuracy of the first neural network and the second neural network obtained in the end.

[0036] In combination with the second aspect, in a possible embodiment, a second loss function value is obtained based on the third output result of the second neural network and the fourth output result of the first neural network, including: obtaining a third sub-loss function value based on the third output result and the fourth output result; obtaining a fourth sub-loss function value based on the fourth output result and the target data; and calculating the second loss function value based on the third sub-loss function value and the fourth sub-loss function value.

[0037] In a third aspect, the present application provides an electronic device, comprising one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, and the computer program code includes computer instructions. When the one or more processors execute the computer instructions, the electronic device executes the method described in the first aspect or the second aspect and any possible implementation method of the first aspect or the second aspect.

[0038] In a fourth aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device, and the chip system includes one or more processors, which are used to call computer instructions so that the electronic device executes the method described in the first aspect or the second aspect and any possible implementation method of the first aspect or the second aspect.

[0039] In a fifth aspect, a computer-readable storage medium is provided, in which instructions are stored, and when the computer-readable storage medium is run on a computer, the computer executes the optimization method of the image processing model of the first aspect or the second aspect mentioned above.

[0040] In a sixth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the method for optimizing the image processing model described in the first aspect or the second aspect.

[0041] The technical effects obtained by the third, fourth, fifth and sixth aspects are similar to the technical effects obtained by the corresponding technical means in the first or second aspects, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of an example of a traditional distillation training architecture;

[0043] Figure 2 A schematic diagram of an example of a model compression method provided in an embodiment of the present application;

[0044] Figure 3 A schematic diagram of an example of a modular distillation training architecture provided in an embodiment of the present application;

[0045] Figure 4 A schematic diagram of another example of a modular distillation training architecture provided in an embodiment of the present application;

[0046] Figure 5 A schematic diagram of another example of a modular distillation training architecture provided in an embodiment of the present application;

[0047] Figure 6A schematic diagram of another example of a modular distillation training architecture provided in an embodiment of the present application;

[0048] Figure 7 A schematic diagram of an example of an image processing method provided in an embodiment of the present application;

[0049] Figure 8 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0050] Fig. 9 A block diagram of a software system of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0052] It should be understood that the "multiple" mentioned in this application refers to two or more. In the description of this application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solution of this application, the words "first" and "second" are used to distinguish between the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not limit them to be different.

[0053] The model compression technology involved in this application mainly includes distillation and quantization. In the current model compression process, distillation and quantization are two separate processes. In order to more clearly illustrate the embodiments of this application, distillation and quantization are first introduced before introducing the embodiments of this application.

[0054] 1. Distillation training technology.

[0055] The distillation training involved in this application refers to pre-training a teacher network, and then in the training process of the super-resolution network, the teacher network is used as the teacher model and the super-resolution network is used as the student model, so that the teacher network guides the super-resolution network for training. Among them, the number of network layers included in the teacher network is significantly greater than the number of network layers included in the super-resolution network, and the number of parameters of the teacher network is significantly greater than the number of parameters of the super-resolution network; in other words, the network structure of the teacher network is significantly more complex than the network structure of the super-resolution network. Specifically, in the training process of the super-resolution network, the output result of the super-resolution network is no longer fitted to the target data, but the output result of the teacher network when the same input data is input. That is, the output result of the teacher network is used as a supervisory signal for the output result of the super-resolution network to supervise the training of the super-resolution network. The purpose of distillation training is to improve the fitting ability of the super-resolution network, that is, the learning ability. Since the number of parameters of the super-resolution network is relatively small, the learning ability is relatively weak. It is difficult to converge to the same degree as the teacher network by directly training the super-resolution network, that is, the error between the output result of the super-resolution network and the actual value is difficult to drop to the same degree as the error between the output result of the teacher network and the actual value. Therefore, during the training process, the output of the teacher network when inputting the same input data is used as the label of the input data (or supervisory signal), and the output of the super-resolution network is used to fit the output of the teacher network, and converge to a degree similar to that of the teacher network, which can improve the learning ability of the super-resolution network. The above process of using a larger network structure as a teacher model to guide the learning of the student model is called distillation training.

[0056] Figure 1 This is a schematic diagram of an example of a traditional distillation training architecture. Figure 1 As shown, the same input data is input to the teacher network and the super-resolution network, and the output results of the teacher network and the super-resolution network are output result #1 and output result #2 respectively. Output result #1 is used as a supervision signal to supervise the training of the super-resolution network. In addition, back propagation is performed based on the loss function of the distillation training architecture to update the parameters of the teacher network and the super-resolution network. In this architecture, the loss function of the distillation training architecture consists of two parts, one is the difference between output result #1 and output result #2, and the other is the loss function of training the super-resolution network to convergence.

[0057] During the distillation training process, for each set of training data, the input data is used not only to calculate the output result #2 of the super-resolution network, but also to synchronously calculate the output result #1 of the teacher network. And because the teacher network has a larger number of parameters, it takes longer to calculate the output result #1. Therefore, the time taken for distillation training is more than doubled that of directly training the super-resolution network.

[0058] 2. Quantification.

[0059] Quantization can convert floating-point numbers stored in the model layer into fixed-point numbers. Specifically, it converts floating-point values ​​into bit-type integer values, thereby increasing the speed of calculation and reducing the cost of calculation. For example, converting floating-point values ​​such as single-precision floating-point numbers (float) and double-precision floating-point numbers (double) into integer values ​​such as 16-bit integers (int16) and 8-bit integers (int8). A 32-bit floating-point number (float32) occupies 4 bytes, while an 8-bit integer (int8) occupies only 1 byte at most. Therefore, the advantage of quantization is not only that integer values ​​are several times faster than floating-point values ​​in memory reading, but also that fixed-point operations are easier to implement hardware acceleration than floating-point operations.

[0060] However, although quantization can improve model performance, it will cause a loss of model accuracy. For example, when converting a 32-bit floating point number (float32) to an 8-bit integer (int8), because the range of values ​​that a 32-bit floating point number (float32) can maintain is larger than that of an 8-bit integer (int8), a large number of values ​​cannot be represented by an 8-bit integer (int8) and can only be rounded to an 8-bit integer (int8) value.

[0061] Quantization includes post-training quantization (PTQ) and quantization-aware training (QAT). PTQ is a form of offline quantization that is separate from actual training. When the model design itself does not have too much redundancy, post-quantization will significantly lose image quality. QAT combines the training of the model itself with the quantization process. It is a form of online quantization that can better reduce the loss of quantized image quality. QAT adds an additional branch to the model training process for model quantization. Specifically, in the process of training the model itself, an additional quantization-aware training module is added. The quantization-aware training module assists in updating the model's own parameters, so that after the model training is completed, an int-type open neural network exchange (Onnx) model can be automatically generated. Compared with the form of training first and then quantizing in PTQ, QAT can skip the process of intermediate float model parameters and directly generate int-type model parameters.

[0062] As described in the background technology, when it is necessary to deploy a model for pixel-level image tasks such as optimizing image resolution on a mobile terminal such as a mobile phone, a GPU server will be used to compress the model and deploy the compressed model to the mobile terminal. For example, in a mobile game scenario, it is necessary to optimize the resolution of several frames of images. Generally, the mobile terminal first separates the several frames of images into text data and image data, and then inputs the image data into a neural network model, which outputs the image data with optimized resolution. The optimization of the resolution of an image by a neural network is a pixel-level image task, which is very demanding on the computing power and power consumption of the mobile terminal.

[0063] In view of this, the present application provides an image processing method and a model compression method.

[0064] According to the image processing method provided by the present application, the first neural network obtained by modular distillation training is used to process the image, and the image processed by the first neural network is output. Specifically, each of the X second convolution modules in the first neural network is a student model, and each of the X first convolution modules in the pre-trained second neural network is a teacher model; the X first convolution modules correspond one-to-one to the X second convolution modules, and the output results of the second convolution modules are used to fit the output results of the corresponding first convolution modules by inputting the same input data into the corresponding convolution modules in the two neural networks, so as to converge to a degree close to the corresponding first convolution module, thereby obtaining the first neural network after parameter update. Optionally, the first neural network is obtained by modular distillation and quantization-aware training, and the quantization-aware training is combined in the modular distillation training architecture.

[0065] It can be understood that the first neural network is obtained by modular distillation training, and the number of parameters of each convolution module in the first neural network is reduced compared to the number of parameters of each convolution module in the pre-trained second neural network. Therefore, the number of parameters of the first neural network is also significantly reduced compared to the number of parameters of the second neural network, thereby saving the computing power of the mobile terminal and reducing the power consumption of the mobile terminal. In the case of combining quantization-aware training with the modular distillation training architecture, the first neural network is further compressed compared to the second neural network, thereby further saving the computing power of the mobile terminal and reducing the power consumption of the mobile terminal.

[0066] The image processing method provided in the embodiment of the present application is applicable to a variety of electronic devices, such as mobile phones, tablet computers, cameras, smart wearable devices, etc., and the embodiment of the present application is not limited to this.

[0067] As mentioned above in the introduction to distillation training technology, the time consumption of the commonly used distillation training process will be more than doubled compared to directly training the super-resolution network.

[0068] According to the model compression method provided by the present application, the first neural network is obtained by modular distillation training. Specifically, each of the X second convolution modules in the first neural network is a student model, and each of the X first convolution modules in the pre-trained second neural network is a teacher model; the X first convolution modules correspond to the X second convolution modules one by one, and by inputting the same input data into the corresponding convolution modules in the two neural networks, the output result of the second convolution module is used to fit the output result of the corresponding first convolution module, so as to converge to a degree close to the corresponding first convolution module, thereby obtaining the first neural network after parameter update.

[0069] It can be understood that, on the one hand, by dividing the first neural network and the second neural network into multiple convolution modules respectively, the computational complexity of each second convolution module is much lower than the computational complexity of the entire first neural network, and the computational complexity of each first convolution module is also much lower than the computational complexity of the entire second neural network, thereby saving training time compared to the commonly used distillation training; on the other hand, the calculations of each second convolution module can be independent of each other so that each second convolution module can be calculated in parallel, and each first convolution module is similar thereto, compared to the commonly used distillation training architecture in which the calculations of each convolution layer of the teacher network and the super-resolution network are interrelated, which can also save time. Therefore, the present application can significantly improve the training efficiency, thereby significantly improving the computing efficiency and memory optimization of the GPU server.

[0070] It should be noted that the model compression method provided in this application is a general compression scheme for artificial intelligence models. The model compression method is highly adaptable, and can realize the distillation of submodules of the model for artificial intelligence models of different application types (optionally, realize the combination of distillation of submodules and quantization), greatly reducing the power consumption of the model caused by calculation on the mobile terminal. The image processing method provided in this application can be understood as an example of an application scenario applicable to the model compression method provided in this application, and does not limit the application scenarios to which the model compression method is applicable.

[0071] The following will be combined Figures 2 to 6 , the model compression method provided in the embodiment of the present application is introduced in detail.

[0072] Figure 2Schematic diagram of the model compression method 100 provided in the embodiment of the present application. It should be noted that the execution subject of the method 100 may be a hardware device with data information processing capabilities and / or the necessary software required to drive the hardware device to work. Optionally, the execution subject may include a workstation, a server, a computer, a user terminal and other intelligent devices. Among them, the user terminal includes but is not limited to a mobile phone, a computer, an intelligent voice interaction device, a smart home appliance, a car terminal, etc. Exemplarily, the method 100 can be executed by a GPU server of a computer.

[0073] S101, taking each of the X first convolution modules included in the second neural network as a teacher model, taking each of the X second convolution modules included in the first neural network as a student model, and inputting input data into each of the first convolution modules and each of the second convolution modules respectively.

[0074] Among them, the X second convolution modules correspond one-to-one to the X first convolution modules, each first convolution module includes M convolution layers, each second convolution module includes N convolution layers, X>1, M>N≥1, and X, M and N are all integers.

[0075] Figure 3 A schematic diagram of an example of a modular distillation training architecture provided in an embodiment of the present application. Figure 3 As shown, the second neural network includes 3 first convolution modules, and each first convolution module includes 4 convolution layers, and the first neural network includes 3 second convolution modules, and each second convolution module includes 2 convolution layers. The 3 first convolution modules correspond to the 3 second convolution modules one by one.

[0076] Among them, in addition to the convolution layer, the first neural network and the second neural network also include other layers, such as activation layer, fully connected layer, pooling layer, etc.

[0077] S102: Update the parameters of the first neural network according to the first output result of each first convolution module and the second output result of each second convolution module.

[0078] In which, taking the i-th first convolution module among the X first convolution modules as the teacher model and the i-th second convolution module among the X second convolution modules as the student model as an example, in S101, the same input data is input to the i-th first convolution module and the i-th second convolution module respectively to obtain the i-th first output result and the i-th second output result; taking the i-th first output result as the supervision signal, the i-th second output result of the i-th second convolution module is used to fit the i-th first output result of the i-th first convolution module, and converges to a degree close to the i-th first convolution module. In which, 1≤i≤X and i is an integer.

[0079] In the model compression method provided in the embodiment of the present application, on the one hand, by dividing the first neural network and the second neural network into multiple convolution modules respectively, the computational complexity of each second convolution module is much lower than the computational complexity of the entire first neural network, and the computational complexity of each first convolution module is also much lower than the computational complexity of the entire second neural network, thereby saving training time compared to the commonly used distillation training; on the other hand, the calculations of each second convolution module can be independent of each other so that each second convolution module can be calculated in parallel, and each first convolution module is similar thereto, compared to the commonly used distillation training architecture in which the calculations between each convolution layer of the teacher network and the super-resolution network are interrelated, which can also save time. Therefore, the model compression method provided in the present application can significantly improve the training efficiency, thereby significantly improving the computing efficiency and memory optimization of the GPU server.

[0080] Specifically, the fitting of the i-th second output result to the i-th first output result can be achieved through the back propagation of the modules.

[0081] Figure 4 Schematic diagram of another example of a modular distillation training architecture provided in an embodiment of the present application. Figure 4 As shown, back propagation is performed based on the i-th first loss function value to update the parameters of the i-th second convolution module. Optionally, back propagation is performed based on the i-th first loss function value to update the parameters of the i-th first convolution module.

[0082] Among them, the i-th first loss function value can be obtained in the following way. First, according to the i-th first output result and the i-th second output result, the i-th first sub-loss function value is obtained. For example, the difference between the i-th first output result and the i-th second output result is determined as the i-th first sub-loss function value. Then, according to the i-th second output result and the target data, the i-th second sub-loss function value is obtained. For example, the difference between the i-th second output result and the target data is determined as the i-th second sub-loss function value. Thus, according to the i-th first sub-loss function value and the i-th second sub-loss function value, the i-th first loss function value is obtained. For example, the sum of the i-th first sub-loss function value and the i-th second sub-loss function value is determined as the i-th first loss function value.

[0083] Understandably, Figure 1 In the traditional distillation training architecture shown in the figure, a global model distillation network structure is used, that is, the loss function value is passed up layer by layer from the last convolutional layer of the teacher network and the super-resolution network. This training method makes the network convergence performance very low, resulting in an increase in network training time.

[0084] And in Figure 4In the modular distillation training architecture shown, modular intermediate result supervision is performed for multiple first convolution modules and multiple second convolution modules, that is, the output result of a certain first convolution module is used to supervise the output result of the corresponding second convolution module. Compared with the difference in the complexity of the network structure between the first neural network and the second neural network in the traditional distillation training architecture, the difference in the complexity of the network structure between the first convolution module and the corresponding second convolution module is obviously much smaller. Therefore, compared with the traditional distillation training architecture in which the output result of the first neural network is used to fit the output result of the second neural network, not only can the output result of the second convolution module better fit the output result of the first convolution module, thereby improving the accuracy of the output result of the second convolution module, and then improving the accuracy of the final output result of the first neural network; and the time required for fitting will also be shorter, so that the convergence performance of the first neural network in the modular distillation training process is also improved.

[0085] Therefore, backpropagating the first neural network based on the first loss function value in a modular manner to update the parameters of the first neural network can improve the convergence performance of the first neural network and the accuracy of the final output result.

[0086] It is also understandable that Figure 1 In the traditional distillation training architecture shown in the figure, the same loss function value is used to update the parameters of the teacher network and the super-resolution network. Due to the large difference in the complexity of the network structures of the teacher network and the super-resolution network, it is difficult to achieve a certain accuracy requirement for the super-resolution network with a simpler network structure while also ensuring that the accuracy of the teacher network with a much more complex network structure is also high with the same loss function value. Therefore, in the traditional distillation training architecture, the accuracy of the teacher network is difficult to guarantee.

[0087] And in Figure 4 In the modular distillation training architecture shown, the same loss function value is used to update parameters for a first convolution module and a corresponding second convolution module. Compared with the difference in network structure complexity between the first neural network and the second neural network in the traditional distillation training architecture, the difference in network structure complexity between the first convolution module and the corresponding second convolution module is significantly smaller. Thus, making the accuracy of the second convolution module reach a certain threshold loss function value can also make the accuracy of the first convolution module reach or approach the threshold, the accuracy of the first convolution module is guaranteed, and thus the accuracy of the second neural network can also be guaranteed. In addition, since the parameters are updated in modules, compared with updating the parameters of the entire second neural network, the computational complexity is significantly reduced and multiple modules can be calculated in parallel, which can also improve the convergence performance.

[0088] Therefore, backpropagating the second neural network based on the first loss function value in a modular manner to update the parameters of the second neural network can improve the convergence performance of the first neural network and the accuracy of the final output result.

[0089] After performing the back propagation of the modules, optionally, global back propagation is also performed on the entire neural network to achieve the fitting of the i-th second output result to the i-th first output result.

[0090] Figure 5 A schematic diagram of another example of a modular distillation training architecture provided in an embodiment of the present application. Figure 5 As shown, according to the sum of the X first loss function values ​​and the second loss function value corresponding to the X second convolution modules, the parameters of the second neural network and the parameters of the first neural network are updated. Specifically, based on the sum of the X first loss function values ​​and / or the second loss function value, back propagation is performed. For example, in the case of back propagation based on the sum of the X first loss function values ​​and the second loss function value, the sum of the X first loss function values ​​and the second loss function value may be added to obtain a third loss function value, and back propagation is performed based on the third loss function value.

[0091] Among them, the second loss function value can be obtained in the following way. Take the second neural network as the teacher model, take the first neural network as the student model, input data for the second neural network and the first neural network respectively; according to the third output result of the second neural network and the fourth output result of the first neural network, obtain the second loss function value. Exemplarily, first obtain the third sub-loss function value according to the third output result and the fourth output result. For example, the difference between the third output result and the fourth output result is determined as the third sub-loss function value. Subsequently, the fourth sub-loss function value is obtained according to the fourth output result and the target data. For example, the difference between the fourth output result and the target data is determined as the fourth sub-loss function value. Thus, the second loss function value is calculated according to the third sub-loss function value and the fourth sub-loss function value. For example, the sum of the third sub-loss function value and the fourth sub-loss function value is determined as the second loss function value.

[0092] It is understandable that, as mentioned above, global back-propagation is performed in the traditional distillation training architecture, and the network convergence performance is very low, resulting in increased network training time.

[0093] And in Figure 5 In the modular distillation training architecture shown in FIG, since the parameters of the first neural network and the second neural network have been updated based on the modular back propagation before the global back propagation, the modular back propagation can assist the global back propagation, making the global back propagation in the traditional distillation training architecture more efficient. Figure 5 In the case of global back propagation in the modular distillation training architecture shown, the time for the first neural network and the second neural network to train to convergence is significantly reduced, which improves the training efficiency. In addition, multiple parameter updates can be performed on each second convolution module in the first neural network and each first convolution module in the second neural network in a distillation training architecture, which also improves the training efficiency and the accuracy of the first neural network and the second neural network obtained in the end.

[0094] Therefore, after performing back propagation on the modules, global back propagation is also performed on the entire neural network, which can further improve the training efficiency and the accuracy of the first neural network and the second neural network finally obtained.

[0095] Optionally, method 100 may further include: performing quantization-aware training on each second convolution module, wherein the parameters of each second convolution module before the quantization-aware training are all floating-point values, and the parameters of each second convolution module after the quantization-aware training are all integer values.

[0096] Figure 6 A schematic diagram of another example of a modular distillation training architecture provided in an embodiment of the present application. Figure 6 As shown, compared to the traditional distillation training architecture (e.g. Figure 1 A QAT module is added to the distillation training architecture shown in FIG. 1 . In the process of distillation training for each second convolution module, the QAT module performs QAT on each second convolution module, thereby converting the parameters in the first neural network into integer values. It can be understood that the i-th second output result is output by the i-th second convolution module after QAT.

[0097] The above scheme performs QAT on each second convolution module in the distillation training architecture, so that the parameters of the first neural network trained based on the distillation training architecture are all integer values, and quantization and distillation are performed simultaneously to obtain the first neural network trained by distillation and quantization, further improving the computing efficiency and memory optimization of the GPU server.

[0098] It is understandable that if Figure 1 In the traditional distillation network architecture shown in the figure, on the one hand, the global back propagation based on the loss function value is a one-way propagation process. This back propagation updates parameters in a way that will make the convergence of the neural network worse as the number of network layers increases. On the other hand, as mentioned above, although quantization can improve model performance, it will cause a loss of model accuracy. On the basis of the traditional distillation training architecture, after adding QAT, in order to minimize the accuracy loss caused by QAT, more data is needed as training data for training.

[0099] However, when training neural networks for other niche scenarios such as mobile gaming scenarios, due to the small amount of data in such niche scenarios, it is often difficult to achieve good training results, that is, the accuracy of the neural network finally trained is generally poor.

[0100] And in Figure 6 In the modular distillation training architecture shown, firstly, since each first convolution module and each second convolution module require significantly fewer network layers to be trained relative to the second neural network and the first neural network, respectively, the modular back propagation to update parameters will not significantly deteriorate the convergence of the first neural network and the second neural network. Secondly, on the basis of the modular distillation training architecture, after adding QAT, through the modular back propagation described above, optionally global back propagation is also performed for the entire neural network, which can ensure the accuracy of the final first neural network and the second neural network without requiring a large amount of data as training data for training. Thus, the error caused by distillation and QAT can be reduced.

[0101] therefore, Figure 6 The modular distillation training architecture shown can still ensure the accuracy of the first neural network and the second neural network when training neural networks for other niche scenarios such as mobile game scenarios.

[0102] It should be noted that there is no limitation on the categories of neural network models involved in this application, for example, they may include traditional machine learning models, deep learning models or reinforcement learning models.

[0103] It should be noted that there is no limitation on the application types of the neural network models involved in this application, for example, they may include speech models or image models.

[0104] As an example, the neural network involved in the present application is a speech model, and the speech model includes but is not limited to a speech recognition model, a speech synthesis model, etc. For example, when the neural network involved in the present application is a speech recognition model, the input data is a sample speech, the target data is a reference recognition text of the sample speech, the second neural network is a model pre-trained according to the sample speech and the reference recognition text of the sample speech, and the sample speech and the reference recognition text of the sample speech are used to perform distillation training on the first neural network.

[0105] As an example, the neural network involved in the present application is an image model, and the image model includes but is not limited to an action recognition model, a face recognition model, a text recognition model, etc. For example, when the neural network involved in the present application is an action recognition model, the input data is a sample image, and the target data is a reference recognition action of the sample image. For example, when the neural network involved in the present application is a face recognition model, the input data is a sample image, and the target data is a reference recognition face result of the sample image. Among them, the reference recognition face result includes but is not limited to face position, face width, face height, number of faces, etc. For example, when the neural network involved in the present application is a text recognition model, the input data is a sample image, and the target data is a reference recognition text of the sample image.

[0106] As another example, the present application relates to an image processing model of a neural network. For example, the image processing model is used to optimize the display effect of an image, the input data is sample image data, and the target data is reference image data whose display effect of the sample image data reaches a preset condition. Specifically, the image processing model is used to optimize the resolution of an image, the input data is sample image data, and the target data is reference image data whose resolution of the sample image data reaches a target threshold.

[0107] The following will be combined Figure 7 , the image processing method provided in the embodiments of the present application is introduced in detail. Figure 7 Schematic diagram of an image processing method 200 provided in an embodiment of the present application. Figure 7 As shown, the method 200 includes S201 to S203. The method 200 is executed by an electronic device.

[0108] S201, acquiring an image.

[0109] It is understandable that the electronic device can obtain the image processed by the first neural network model in a variety of ways, such as the electronic device can capture the image through an image capturing device (such as a camera); for example, the electronic device can download the image, etc.; for example, the electronic device can exchange information such as images with other electronic devices through some applications. This application does not limit this.

[0110] S202: Process the image based on a first neural network model.

[0111] The first neural network is obtained by modular distillation training. Each of the X first convolution modules included in the second neural network is used as a teacher model, and each of the X second convolution modules included in the first neural network is used as a student model. The parameters of the first neural network are updated according to the first output result of each first convolution module and the second output result of each second convolution module, wherein the first output result and the second output result are obtained by inputting input data to each first convolution module and each second convolution module respectively, the X second convolution modules correspond to the X first convolution modules one by one, each first convolution module includes M convolution layers, each second convolution module includes N convolution layers, X>1, M>N≥1, and X, M and N are all integers.

[0112] Optionally, the first neural network is obtained through modular distillation and quantization-aware training, and the quantization-aware training is combined in the modular distillation training architecture.

[0113] Specifically, the method of obtaining the first neural network model through modular distillation training can refer to the relevant description of the model compression method 100 above.

[0114] S203, output the image processed by the first neural network model.

[0115] As an example of method 200, after the electronic device acquires an image, it obtains image data (hereinafter referred to as image data #1 for the convenience of explanation), and image data #1 includes data of multiple pixels in the image. For example, the image data #1 can be in the original (RAW) format or the red, green and blue (RGB) format. Image data #1 is input into the first neural network. Specifically, the input layer first parses and processes the image data #1, and outputs a feature map that can be processed by the network layer (including convolution layer, pooling layer, activation layer, etc.). The X·N convolution layers included in the X second convolution modules perform convolution calculation on the vector, and the jth convolution layer of the X·N convolution layers outputs the first calculation result; the jth calculation result is input into other non-convolution layers, and the second calculation result is output, and the second calculation result is input into the j+1th convolution layer, 1≤j≤X·N-1 and j is an integer. The calculation result output by the X·Nth convolution layer is input to the output layer, and the output layer parses and processes the calculation result, and outputs the image processed by the first neural network (hereinafter referred to as image data #2 for the convenience of explanation). It can be understood that after each convolution layer in the first neural network performs convolution calculation on the feature map input to the convolution layer, the calculation result is calculated by other non-convolution layers and then input to the next convolution layer, and finally the first neural network outputs the processed image. In a possible implementation, the image processed by the first neural network can be image data in RAW format or RGB format.

[0116] For example, in a mobile game scenario, the resolution of an image is optimized by the first neural network. Generally, the mobile terminal first separates the acquired frames of images into text data and image data #1, and then inputs the image data #1 into the neural network model. The first neural network calculates the image data #1 in the manner described in the above example and outputs the image data #2 with optimized resolution.

[0117] In the image processing method provided in the embodiment of the present application, the first neural network is obtained by modular distillation training, and the parameter amount of each convolution module in the first neural network is reduced compared to the parameter amount of each convolution module in the pre-trained second neural network. Therefore, the parameter amount of the first neural network is also significantly reduced compared to the parameter amount of the second neural network, thereby saving the computing power of the mobile terminal and reducing the power consumption of the mobile terminal. Quantization-aware training is combined with the modular distillation training architecture, and the first neural network is further compressed compared to the second neural network, thereby further saving the computing power of the mobile terminal and reducing the power consumption of the mobile terminal.

[0118] Exemplarily, the model compression method provided in the present application can also be used to obtain a compressed neural network for face recognition, or a compressed network for semantic recognition, etc., which is not limited in the present application.

[0119] Please refer to Figure 8 , Figure 8 The hardware structure diagram of an electronic device 1000 provided in an embodiment of the present application is shown in FIG. Figure 8 The electronic device 1000 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0120] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 1000. In other embodiments of the present application, the electronic device 1000 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0121] The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc.

[0122] The controller may be the nerve center and command center of the electronic device 1000. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0123] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0124] In some embodiments, the processor 110 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0125] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to realize the shooting function of the electronic device 1000. The processor 110 and the display screen 194 communicate via the DSI interface to realize the display function of the electronic device 1000.

[0126] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 1000. In other embodiments of the present application, the electronic device 1000 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0127] The charging management module 140 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193 and the wireless communication module 160.

[0128] The wireless communication function of the electronic device 1000 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0129] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 1000 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0130] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 1000. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1.

[0131] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 1000. The wireless communication module 160 can be one or more devices integrating at least one communication processing module.

[0132] The electronic device 1000 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0133] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 1000 may include 1 or N display screens 194, where N is an integer greater than 1.

[0134] The electronic device 1000 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0135] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0136] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 1000 may include 1 or N cameras 193, where N is an integer greater than 1.

[0137] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 1000 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0138] Video codecs are used to compress or decompress digital videos. The electronic device 1000 may support one or more video codecs. Thus, the electronic device 1000 may play or record videos in various coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0139] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 1000 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0140] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 1000. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0141] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 1000 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created by the electronic device 1000 during use (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0142] The electronic device 1000 can implement audio functions, such as music playback, recording, etc., through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone interface 170D and the application processor. The button 190 includes a power button, a volume button, etc. The button 190 can be a mechanical button or a touch button. The electronic device 1000 can receive key input and generate key signal input related to the user settings and function control of the electronic device 1000. The motor 191 can generate a vibration prompt. The indicator 192 can be an indicator light, which can be used to indicate the charging status, the change of power, and can also be used to indicate messages, missed calls, notifications, etc. The SIM card interface 195 is used to connect a SIM card.

[0143] The sensor module 180 may include one or more sensors, which may be of the same type or of different types. Figure 8 The sensor module 180 shown is only an exemplary division method. There may be other division methods, which are not limited in this application.

[0144] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. When a touch operation is applied to the display screen 194, the electronic device detects the touch operation intensity according to the pressure sensor 180A. The electronic device can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions.

[0145] The gyro sensor 180B can be used to determine the motion posture of the electronic device. In some embodiments, the angular velocity of the electronic device around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for anti-shake shooting.

[0146] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device in all directions (generally three axes). When the electronic device is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0147] The distance sensor 180F is used to measure the distance. The electronic device can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0148] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device, which is different from the position of the display screen 194.

[0149] The air pressure sensor 180C is used to measure air pressure. The magnetic sensor 180D includes a Hall sensor. The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The electronic device uses a photodiode to detect infrared reflected light from nearby objects. The ambient light sensor 180L is used to sense the brightness of ambient light. The fingerprint sensor 180H is used to obtain fingerprints. The temperature sensor 180J is used to detect temperature. The bone conduction sensor 180M can obtain vibration signals.

[0150] Next, the software system of the electronic device 1000 will be described.

[0151] Exemplarily, the electronic device 1000 may be a mobile phone. The software system of the electronic device 1000 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software system of the electronic device 1000.

[0152] Fig. 9 This is a block diagram of a software system of an electronic device 1000 provided in an embodiment of the present application. Fig. 9 , the layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android runtime) and the system layer, the kernel layer and the hardware abstraction layer (HAL).

[0153] The application layer can include a series of application packages. Fig. 9 As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0154] The application framework layer provides application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Fig. 9 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, etc. The window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc. The content provider is used to store and obtain data and make these data accessible to applications. These data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc. The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build the display interface of the application, and the display interface can be composed of one or more views, such as a view for displaying a text message notification icon, a view for displaying text, and a view for displaying pictures. The phone manager is used to provide communication functions of the electronic device 1000, such as management of call status (including connecting, hanging up, etc.). The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, etc. The notification manager enables the application to display notification information in the status bar, which can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of an icon or scroll bar text, such as a notification of an application running in the background. The notification manager can also be a notification that appears on the screen in the form of a dialog window, such as a text message in the status bar, a reminder sound, an electronic device vibrating, an indicator light flashing, etc.

[0155] Android runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system. The core library consists of two parts: one is the function that the Java language needs to call, and the other is the Android core library. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0156] The system library can include multiple functional modules, such as: surface manager, media library, 3D graphics processing library (such as: OpenGL ES), 2D graphics engine (such as: SGL), etc. The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing, etc. The 2D graphics engine is a drawing engine for 2D drawing.

[0157] The Hardware Abstraction Layer (HAL) is an interface layer between the operating system kernel and the upper software, and its purpose is to abstract the hardware. The hardware abstraction layer is an abstract interface driven by the device kernel, which is used to implement the application programming interface that provides access to the underlying device to the higher-level Java API framework. HAL contains multiple library modules, such as camera, display, Bluetooth, audio, etc. Each library module implements an interface for a specific type of hardware component. When the system framework layer API requires access to the hardware of the portable device, the Android operating system will load the library module for the hardware component.

[0158] The kernel layer is the layer between hardware and software. The kernel layer contains at least device drivers such as camera driver, processor driver, display driver, audio driver, etc. The device driver is the interface between the I / O system and related hardware, used to drive the corresponding hardware devices.

[0159] It should be noted that the application provides Fig. 9 The software structure diagram of the electronic device shown is only an example and does not limit the specific module division in different layers of the Android operating system. For details, please refer to the introduction of the Android operating system software structure in conventional technology.

[0160] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0161] The above are optional embodiments provided for the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the technical scope disclosed in the present application shall be included in the protection scope of the present application.

Claims

1. An image processing method, applied to an electronic device, characterized in that: The method comprises: Get the image; Processing the image based on a first neural network; Outputting an image processed by the first neural network, where the first neural network is obtained through modular distillation training; The first neural network is obtained through modular distillation training, including: Each of the X first convolution modules included in the second neural network is used as a teacher model, and each of the X second convolution modules included in the first neural network is used as a student model, and the parameters of the first neural network are updated according to the first output result of each first convolution module and the second output result of each second convolution module, wherein the first output result is obtained by inputting input data to each first convolution module, and the second output result is obtained by inputting the input data to each second convolution module, the X second convolution modules correspond to the X first convolution modules one by one, each first convolution module includes M convolution layers, and each second convolution module includes N convolution layers, X>1, M>N≥1, and X, M and N are all integers; The parameters of the first neural network are updated according to the first output result of each first convolution module and the second output result of each second convolution module, including: The parameters in the i-th second convolution module in the first neural network are obtained by back propagation based on the i-th first loss function value; the i-th first loss function value is obtained based on the i-th first sub-loss function value and the i-th second sub-loss function value, wherein the i-th first sub-loss function value is obtained based on the i-th first output result of the i-th first convolution module among the X first convolution modules and the i-th second output result of the i-th second convolution module among the X second convolution modules, and the i-th second sub-loss function value is obtained based on the i-th second output result and target data, 1≤i≤X and i is an integer.

2. The method according to claim 1, characterized in that The processing of the image based on the first neural network includes: Obtain image data according to the image, wherein the image data includes data of a plurality of pixels in the image; Inputting the image data into the first neural network; The X·N convolutional layers included in the X second convolutional modules perform convolution calculations on the image data, wherein a first calculation result output by a j-th convolutional layer among the X·N convolutional layers is calculated by other non-convolutional layers to obtain a second calculation result, and the second calculation result is input into the j+1-th convolutional layer and calculated by the j+1-th convolutional layer, 1≤j≤X·N-1 and j is an integer.

3. The method according to claim 1 or 2, characterized in that After inputting the input data to each second convolution module, the parameters of each second convolution module are also updated through quantization-aware training, wherein the parameters of each second convolution module before the quantization-aware training are all floating-point values, and the parameters of each second convolution module after the quantization-aware training are all integer values.

4. The method according to claim 1 or 2, characterized in that: The parameters of the second neural network are updated according to the first output result of each of the first convolution modules and the second output result of each of the second convolution modules.

5. The method according to claim 4, characterized in that The parameters of the second neural network are updated according to the first output result of each of the first convolution modules and the second output result of each of the second convolution modules, including: The parameters of the second neural network are obtained by back propagation based on the i-th first loss function value, and the i-th first loss function value is calculated based on the i-th first sub-loss function value and the i-th second sub-loss function value, wherein the i-th first sub-loss function value is obtained based on the i-th first output result of the i-th first convolution module among the X first convolution modules and the i-th second output result of the i-th second convolution module among the X second convolution modules, and the i-th second sub-loss function value is obtained based on the i-th second output result and target data, 1≤i≤X and i is an integer.

6. The method according to claim 1, characterized in that The parameters of the second neural network and the parameters of the first neural network are also updated by the sum of the X first loss function values ​​corresponding to the X second convolution modules and the second loss function value, wherein the second neural network is used as a teacher model and the first neural network is used as a student model, the second loss function value is obtained according to the third output result of the second neural network and the fourth output result of the first neural network, and the third output result and the fourth output result are obtained by inputting the input data into the second neural network and the first neural network respectively.

7. The method according to claim 6, characterized in that The second loss function value is obtained according to the third output result of the second neural network and the fourth output result of the first neural network, including: The second loss function value is calculated based on the third sub-loss function value and the fourth sub-loss function value, wherein the third sub-loss function value is obtained based on the third output result and the fourth output result, and the fourth sub-loss function value is obtained based on the fourth output result and the target data.

8. A model compression method, characterized in that: include: Each of the X first convolution modules included in the second neural network is used as a teacher model, and each of the X second convolution modules included in the first neural network is used as a student model, input data is inputted to each of the first convolution modules and each of the second convolution modules respectively, the X second convolution modules correspond to the X first convolution modules one by one, each of the first convolution modules includes M convolution layers, each of the second convolution modules includes N convolution layers, X>1, M>N≥1, and X, M and N are all integers; Obtaining an i-th first sub-loss function value according to an i-th first output result of an i-th first convolution module among the X first convolution modules and an i-th second output result of an i-th second convolution module among the X second convolution modules; Obtaining an i-th second sub-loss function value according to the i-th second output result and target data; Obtaining an i-th first loss function value according to the i-th first sub-loss function value and the i-th second sub-loss function value; Back propagation is performed based on the i-th first loss function value to update the parameters in the i-th second convolution module, where 1≤i≤X and i is an integer.

9. The method according to claim 8, characterized in that The method further comprises: Quantization-aware training is performed on each of the second convolution modules, wherein the parameters of each of the second convolution modules before the quantization-aware training are all floating-point values, and the parameters of each of the second convolution modules after the quantization-aware training are all integer values.

10. The method according to claim 8 or 9, characterized in that The method further comprises: Update the parameters of the second neural network according to the first output result of each of the first convolution modules and the second output result of each of the second convolution modules.

11. The method according to claim 10, characterized in that Updating the parameters of the second neural network according to the first output result of each of the first convolution modules and the second output result of each of the second convolution modules includes: Obtaining an i-th first sub-loss function value according to an i-th first output result of an i-th first convolution module among the X first convolution modules and an i-th second output result of an i-th second convolution module among the X second convolution modules; Obtaining an i-th second sub-loss function value according to the i-th second output result and target data; Calculate an i-th first loss function value according to the i-th first sub-loss function value and the i-th second sub-loss function value; Back propagation is performed based on the i-th first loss function value to update parameters in the i-th first convolution module, where 1≤i≤X and i is an integer.

12. The method according to claim 8, characterized in that The method further comprises: Taking the second neural network as a teacher model and the first neural network as a student model, inputting the input data into the second neural network and the first neural network respectively; Obtaining a second loss function value according to a third output result of the second neural network and a fourth output result of the first neural network; Update the parameters of the second neural network and the parameters of the first neural network according to the sum of the X first loss function values ​​corresponding to the X second convolution modules and / or the second loss function value.

13. The method according to claim 12, characterized in that The obtaining a second loss function value according to the third output result of the second neural network and the fourth output result of the first neural network includes: Obtaining a third sub-loss function value according to the third output result and the fourth output result; Obtaining a fourth sub-loss function value according to the fourth output result and the target data; A second loss function value is calculated based on the third sub-loss function value and the fourth sub-loss function value.

14. An electronic device comprising a memory and one or more processors, characterized in that: The memory is used to store a computer program; the processor is used to call the computer program so that the electronic device executes the method according to any one of claims 1 to 7, or the electronic device executes the method according to any one of claims 8 to 13.

15. A computer storage medium, characterized in that include: Computer instructions; when the computer instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 7, or the electronic device executes the method according to any one of claims 8 to 13.

16. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device executes the method as described in any one of claims 1 to 7, or the electronic device executes the method as described in any one of claims 8 to 13.

Images