Image processing method, model compression method and related equipment

Abstract

The invention provides an image processing method, a model compression method and related equipment. According to the image processing method, the image is processed based on the first neural network, and the first neural network is obtained by performing redundant compression processing on the neural network with a more complex structure, so that the computing power of the mobile terminal can be saved, and the power consumption of the mobile terminal can be reduced. Optionally, the redundant compression processing is combined with quantization and distillation training, so that the performance of the first neural network is greatly optimized, and the power consumption of the mobile terminal is further reduced. According to the model compression method, whether redundancy exists or not is judged according to the output results before and after quantization of the current neural network, if yes, the redundancy is deleted, pruning is achieved without manual intervention, the operation efficiency and the memory optimization degree of a GPU server can be improved, and the labor cost can be saved. Optionally, whether the precision of the neural network after redundancy deletion is obviously reduced or not is evaluated, and if the precision is obviously reduced, the latest deleted network layer is recovered, so that the balance between the precision and the performance is kept.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to an image processing method, a model compression method and related equipment. Background Art

[0002] With the development of artificial intelligence (AI) and neural network (NN) technologies, neural network models have been widely used in AI applications in the fields of image processing, speech recognition, and natural language processing, making the number of AI applications using neural network models increase year by year.

[0003] The parameters of a neural network model are usually in the millions, tens of millions, or hundreds of millions. Therefore, the storage and computing power requirements of the terminal devices that run AI applications using neural network models are relatively high, which limits the use of neural network models on terminal devices. In order to reduce computing costs and power consumption, the artificial intelligence model can be compressed, and the compressed convolutional neural network model can be deployed to the mobile terminal. For example, when it is necessary to deploy a model for pixel-level image tasks such as optimizing image resolution on a terminal device such as a mobile phone, a device with a graphics processing unit (GPU) server will be used to compress the model and deploy the compressed model to the mobile terminal. Summary of the invention

[0004] The present application provides an image processing method, a model compression method and related devices. The image processing method can save the computing power of the mobile terminal and reduce the power consumption of the mobile terminal; the model compression method can significantly improve the computing efficiency and memory optimization of the GPU server, and save labor costs.

[0005] It should be noted that the model compression method provided in this application is a general compression scheme for artificial intelligence models. The model compression method is highly adaptable and can achieve adaptive model redundancy compression for artificial intelligence models of different application types, greatly reducing the power consumption of the model generated by calculation on the mobile terminal. The image processing method provided in this application can be understood as an example of an application scenario applicable to the model compression method provided in this application, and does not limit the application scenarios applicable to the model compression method.

[0006] In a first aspect, the present application provides an image processing method, applied to an electronic device, the method comprising:

[0007] Acquire an image; process the image based on a first neural network; output the image processed by the first neural network, wherein the first neural network is obtained by performing redundant compression processing on a first compressible network layer included in a second neural network, the first compressible network layer includes a plurality of first network layers of the same type and parameters, wherein the first neural network is obtained by performing redundant compression processing on the first compressible network layer included in the second neural network, including: the first neural network is obtained by training the second neural network by skipping the first first network layer among a plurality of first network layers when a similarity measure between a first output result and a second output result is less than or equal to a first threshold, wherein the first output result is obtained by inputting input data into the second neural network, the second output result is obtained by inputting input data into a third neural network, and the third neural network is obtained by quantizing the second neural network after training.

[0008] In the embodiment of the present application, the first neural network is obtained through redundant compression processing, so that compared with the neural network before redundant compression processing, the first neural network reduces redundant network layers, has a simpler network structure, and has improved performance, thereby saving computing power on the mobile terminal and reducing power consumption on the mobile terminal.

[0009] It can be understood that skipping the first first network layer among the multiple first network layers to train the second neural network means that after deleting the first first network layer, the remaining network layers in the second neural network are trained. In other words, the first first network layer is not used for training the second neural network layer.

[0010] Exemplarily, the parameters of the second neural network are all floating point values, and the parameters of the third neural network model are all integer values. It can be understood that the processing of post-training quantization can improve the computing speed and reduce the computing cost.

[0011] In combination with the first aspect, in a possible embodiment, processing the image based on the first neural network includes:

[0012] Image data is obtained according to the image, the image data including data of a plurality of pixels in the image; the image data is input into a first neural network; the image data is calculated by Z network layers in the first neural network, wherein the calculation result of the jth network layer among the Z network layers is input into the j+1th network layer and calculated by the j+1th network layer, 1≤j≤Z-1, Z>1 and j and Z are integers.

[0013] Among them, after the electronic device acquires the image, it obtains image data (hereinafter referred to as image data #1 for the convenience of explanation), and image data #1 includes data of multiple pixels in the image. For example, the image data #1 can be in the original (RAW) format or the red, green and blue (RGB) format. The image data #1 is input into the first neural network, and the image data is calculated by the Z network layers in the first neural network. Specifically, the input layer first parses and processes the image data #1, and outputs a feature map that can be processed by the network layer (including convolution layer, pooling layer, activation layer, etc.). The first network layer of the Z network layers calculates the vector, and the calculation result output by the first network layer is input into the second network layer for calculation... The jth network layer calculates the vector, and the calculation result output by the jth network layer is input into the j+1th network layer and calculated by the j+1th network layer, 1≤j≤Z-1, Z>1 and j and Z are integers. The calculation result output by the Zth network layer is input to the output layer, and the output layer parses and processes the calculation result to output the image processed by the first neural network (hereinafter referred to as image data #2 for convenience of explanation). In one possible implementation, the image processed by the first neural network can be image data in RAW format or RGB format.

[0014] For example, in a mobile game scenario, the resolution of an image is optimized by the first neural network. Generally, the mobile terminal first separates the acquired frames of images into text data and image data #1, and then inputs the image data #1 into the neural network model. The first neural network calculates the image data #1 in the manner described in the above example and outputs the image data #2 with optimized resolution.

[0015] In combination with the first aspect, in a possible embodiment, the first neural network is obtained by training the second neural network by skipping the first first network layer among multiple first network layers when the similarity measure between the first output result and the second output result is less than or equal to the first threshold, including:

[0016] The first neural network is obtained by training the second neural network by skipping the first first network layer based on a first loss function value when a similarity measure between the first output result and the second output result is less than or equal to a first threshold, wherein the first loss function value includes a similarity measure between the first output result and the second output result.

[0017] The above scheme, by applying the similarity measure of the output results of the second neural network before and after quantization to the loss function value used to train the second neural network and performing back propagation, can reduce the difference between the output results of the second neural network and the third neural network before and after quantization, thereby accelerating the convergence in the process of training the second neural network, and also helping to improve the accuracy of the first neural network, thereby improving the image processing effect.

[0018] In combination with the first aspect, in a possible embodiment, the multiple first network layers are Y first network layers, Y>1 and Y is an integer, the first first network layer includes first identification information, the first identification information is used to indicate deletion of the current first network layer, and the second first network layer to the Yth first network layer among the Y first network layers include second identification information, and the second identification information is used to indicate that the current network layer is deletable.

[0019] Exemplarily, the first identification information and the second identification information are realized by means of a dropout operation. Specifically, a dropout layer is configured in each of the multiple first network layers to control training parameters, or to mark whether to update the parameters of the layer, or to mark whether to train the layer.

[0020] Among them, when the parameter of the dropout layer is set to 1, the dropout layer is the second identification information, indicating that the current network layer configured with the dropout layer can be deleted. When the parameter of the dropout layer is set to 0, the dropout layer is the first identification information, indicating that the corresponding network layer parameters will not be updated, or the network will skip this layer for training, or the network layer will not affect the final training result of the network, or the network layer is deleted from the network. Therefore, in the above scheme, the 2nd first network layer to the Yth first network layer are all configured with a dropout layer with a parameter set to 1. The first first network layer is configured with a dropout layer with a parameter set to 0.

[0021] In combination with the first aspect, in a possible embodiment, the first neural network is obtained by performing redundant compression processing on a first compressible network layer included in the second neural network, including:

[0022] The first neural network is based on pre-maintained compressible information, and is obtained by performing redundant compression processing on the first compressible network layer included in the second neural network, wherein the compressible information is used to indicate that i network layers have been deleted, and the first first network layer is the i-th network layer deleted among the i network layers, 1≤i≤Y+i-1 and i is an integer; wherein the compressible information includes Y+i-1 sub-information, and each sub-information in the Y+i-1 sub-information is used to indicate the compressible state of the Y+i-1 network layer; wherein the first i-1 sub-information in the Y+i-1 sub-information is used to indicate that the corresponding i-1 network layer has been deleted, and the order in which the i-1 network layer is deleted, and the i+1th to Y+i-1th sub-information in the Y+i-1 sub-information is used to indicate that the corresponding Y-1 network layer has not been deleted.

[0023] The above scheme synchronously updates the compressible information according to the deletion of the network layer, so as to record the deletion of the network layer in all processes of the model compression method in real time and improve the accuracy of model redundancy compression. Therefore, it helps to improve the accuracy of the first neural network and further improve the effect of image processing.

[0024] In combination with the first aspect, in a possible embodiment, the difference between the first loss function value and the second loss function value is less than a second threshold, the first loss function is used to train the second neural network, and the second loss function value is used to train the first neural network.

[0025] The above scheme can determine whether deleting the first network layer makes the accuracy of the first neural network significantly worse than that of the second neural network by judging whether the difference between the first loss function value and the second loss function value is less than the second threshold. If the difference between the first loss function value and the second loss function value is less than the second threshold, it means that the accuracy of the first neural network is not significantly worse than that of the second neural network. Thus, it is possible to reduce the situation where the accuracy is significantly lost due to blindly deleting redundant network layers, achieve a balance between accuracy and performance, and thus ensure the effect of image processing.

[0026] In combination with the first aspect, in a possible embodiment, the first neural network is obtained by training the second neural network by skipping the first first network layer among multiple first network layers when the similarity measure between the first output result and the second output result is less than or equal to the first threshold, including:

[0027] The first neural network is obtained by performing distillation training on the second neural network by skipping the first first network layer among multiple first network layers when the similarity measure between the first output result and the second output result is less than or equal to the first threshold.

[0028] The above solution integrates model quantization, model distillation and model redundancy compression, thereby merging the three methods of model compression technology to achieve model compression at multiple levels, thereby greatly optimizing model performance, further saving mobile computing power and reducing mobile power consumption.

[0029] In combination with the first aspect, in a possible embodiment, the distillation training is modular distillation training, wherein each of the X first convolution modules included in the fourth neural network is a teacher model, and each of the X second convolution modules included in the second neural network is a student model, and the first neural network is obtained by updating the parameters of the second neural network according to the first output result of each first convolution module and the second output result of each second convolution module, wherein the first output result is obtained by inputting input data for each first convolution module, and the second output result is obtained by inputting input data for the convolution layer other than the first first network layer in each second convolution module, the X second convolution modules correspond one-to-one to the X first convolution modules, each first convolution module includes M convolution layers, each second convolution module includes N convolution layers, X>1, M>N≥1, and X, M and N are all integers.

[0030] In the above scheme, compared with the difference in network structure complexity between the second neural network and the fourth neural network in the traditional distillation training architecture, the difference in network structure complexity between the first convolution module and the corresponding second convolution module is obviously much smaller. Therefore, compared with the traditional distillation training architecture in which the output result of the second neural network is used to fit the output result of the fourth neural network, the output result of the second convolution module can not only better fit the output result of the first convolution module, thereby improving the accuracy of the output result of the second convolution module, but also improve the accuracy of the final output result of the first neural network obtained by updating the parameters of the second neural network. Therefore, the image processing effect can be further improved.

[0031] In a second aspect, the present application provides a model compression method, comprising:

[0032] Input data into a second neural network to obtain a first output result, wherein the second neural network includes a first compressible network layer, and the first compressible network layer includes multiple first network layers with consistent types and parameters; the second neural network is quantized after training to obtain a third neural network, the parameters of the second neural network are all floating-point values, and the parameters of the third neural network model are all integer values; input data into the third neural network to obtain a second output result; when the similarity measure between the first output result and the second output result is less than a first threshold, the first first network layer among the multiple first network layers is skipped to train the second neural network to obtain the first neural network.

[0033] The above scheme determines whether there is redundancy in the output results before and after quantization of the current neural network, and deletes a network layer when redundancy exists, so as to achieve adaptive and precise pruning, thereby achieving model redundancy compression. The above scheme does not need to compare the output results of the current neural network with those of another neural network to determine whether there is redundancy in the current neural network. Compared with the traditional method of analyzing model redundancy, it can improve the efficiency of analyzing model redundancy and deleting redundant network layers, thereby improving the efficiency of model compression. Compared with traditional structured pruning, the above scheme can achieve complete manual intervention and save labor costs. Therefore, by deleting redundant network structures and achieving compression of the model, the computing efficiency and memory optimization degree of the GPU server can be improved. In addition, through the processing of post-training quantization, the computing speed can be improved, the computing cost can be reduced, and the computing efficiency and memory optimization degree of the GPU server can be improved.

[0034] It can be understood that skipping the first first network layer among the multiple first network layers to train the second neural network means that after deleting the first first network layer, the remaining network layers in the second neural network are trained. In other words, the first first network layer is not used for training the second neural network layer.

[0035] In combination with the second aspect, in a possible embodiment, skipping the first first network layer among the plurality of first network layers to train the second neural network includes:

[0036] Based on the first loss function value, the first first network layer is skipped to train the second neural network, wherein the first loss function value includes a similarity measure between the first output result and the second output result.

[0037] The above scheme, by applying the similarity measure of the output results of the second neural network before and after quantization to the loss function value used to train the second neural network and performing back propagation, can reduce the difference between the output results of the second neural network and the third neural network before and after quantization, thereby accelerating the convergence in the process of training the second neural network.

[0038] It can be understood that the process of outputting the first output result by the second neural network and the second output result by the third neural network is differentiable, so the similarity measure of the two output results can be used for back propagation. In other words, taking the similarity measure of the two output results as the difference between the two output results as an example, the difference between the first output result and the second data result can be used as the loss function value for back propagation.

[0039] In combination with the second aspect, in a possible embodiment, the multiple first network layers are Y first network layers, Y>1 and Y is an integer, the first first network layer includes first identification information, the first identification information is used to indicate deletion of the current first network layer, and the second first network layer to the Yth first network layer among the Y first network layers all include second identification information, and the second identification information is used to indicate that the current network layer is deletable; skipping the first first network layer among the Y first network layers to train the second neural network includes: according to the first identification information included in the first first network layer, skipping the first first network layer to train the second neural network.

[0040] The above scheme, by adding identification information (including the first identification information or the second identification information) to the Y first network layers respectively, so as to skip the first network layer including the first identification information, and not skip the first network layer including the second identification information for training, can enable the second neural network layer to adaptively try to delete each network layer in the compressible network layer, that is, the redundant layers can be automatically deleted during the training process, which is more efficient than the traditional method of manually modifying the network structure during the model redundancy compression process.

[0041] Exemplarily, the first identification information and the second identification information are realized by means of a dropout operation. Specifically, a dropout layer is configured in each of the multiple first network layers to control training parameters, or to mark whether to update the parameters of the layer, or to mark whether to train the layer.

[0042] Among them, when the parameter of the dropout layer is set to 1, the dropout layer is the second identification information, indicating that the current network layer configured with the dropout layer can be deleted. The 2nd to the Yth first network layers are all configured with a dropout layer with a parameter set to 1. When the parameter of the dropout layer is set to 0, the dropout layer is the first identification information, indicating that the corresponding network layer parameters will not be updated, or the network will skip this layer for training, or the network layer will not affect the final training results of the network, or the network layer is deleted from the network. The first first network layer is configured with a dropout layer with a parameter set to 0.

[0043] In conjunction with the second aspect, in a possible embodiment, the method further includes:

[0044] According to the similarity measure between the first output result and the second output result being less than the first threshold, the second identification information in the first first network layer is updated to the first identification information.

[0045] Exemplarily, the conversion between the first identification information and the second identification information is achieved by means of a dropout operation. The parameter of the dropout layer configured in the first first network layer is changed from 1 to 0, thereby achieving the conversion from the second identification information to the first identification information.

[0046] The above scheme uses the elimination property of the dropout operation to enable the second neural network layer to adaptively try to delete each network layer in the compressible network layer, that is, it can automatically delete redundant layers during the training process, which is more efficient than manually modifying the network structure during the traditional model redundancy compression process.

[0047] In combination with the second aspect, in a possible embodiment, the method also includes: updating the pre-maintained compressible information, the updated compressible information is used to indicate that i network layers have been deleted, and the first first network layer is the i-th network layer deleted among the i network layers, 1≤i≤Y+i-1 and i is an integer; wherein the compressible information includes Y+i-1 sub-information, and each sub-information in the Y+i-1 sub-information is used to indicate the compressible state of the Y+i-1 network layers; wherein the first i-1 sub-information in the Y+i-1 sub-information is used to indicate that the corresponding i-1 network layers have been deleted, and the order in which the i-1 network layers are deleted, and the i+1th to Y+i-1th sub-information in the Y+i-1 sub-information is used to indicate that the corresponding Y-1 network layers have not been deleted.

[0048] The above scheme synchronously updates the compressible information according to the deletion of the network layer, so as to record the deletion of the network layer in all processes of the model compression method in real time and improve the accuracy of model redundant compression.

[0049] In combination with the second aspect, in a possible embodiment, the method also includes: saving a first loss function value for training the second neural network; obtaining a second loss function value for training the first neural network; and determining whether a difference between the first loss function value and the second loss function value is less than a second threshold.

[0050] The above scheme can determine whether deleting the first network layer makes the accuracy of the first neural network significantly worse than that of the second neural network by judging whether the difference between the first loss function value and the second loss function value is less than the second threshold. Thus, it is possible to reduce the situation where the accuracy is obviously lost due to blindly deleting redundant network layers, and achieve a balance between accuracy and performance. In addition, the above scheme adaptively attempts to delete each network layer in the compressible network layer, and analyzes whether there will be a significant loss of accuracy after deletion, so that the number of redundant layers can be automatically deleted and the deletion results can be judged during the training process, which is more efficient than the traditional method of manually modifying the network structure during the model redundancy compression process.

[0051] In combination with the second aspect, in a possible embodiment, the multiple first network layers are Y first network layers, and when the difference between the first loss function value and the second loss function value is less than a second threshold, the method further includes:

[0052] Input data into the first neural network to obtain a third output result, wherein the first neural network includes a second compressible network layer, the second compressible network layer includes Y-1 second network layers of the same type and parameters, and the Y-1 second network layers correspond one-to-one to the second to the Yth first network layers among the Y first network layers; the first neural network is quantized after training to obtain a fifth neural network, the parameters of the first neural network are all floating-point values, and the parameters of the fifth neural network model are all integer values; input data into the fifth neural network to obtain a fifth output result; when the similarity measure between the third output result and the fifth output result is less than a first threshold, the first second network layer among the Y-1 second network layers is skipped to train the first neural network to obtain a sixth neural network; or, when the similarity measure between the third output result and the fifth output result is greater than or equal to the first threshold, the first neural network is used as the final neural network, and the execution of the model compression method is stopped.

[0053] The above scheme automatically continues to compress model redundancy of the first neural network when the accuracy of the first neural network is not significantly worse than that of the second neural network. Compared with traditional structured pruning, it can achieve complete manual intervention and save labor costs.

[0054] In conjunction with the second aspect, in a possible embodiment, the method further includes:

[0055] The pre-maintained compressible information is updated, and the updated compressible information is used to indicate that i+1 network layers have been deleted, and the first second network layer is the i+1th network layer deleted in the i+1 network layers, 1≤i≤Y+i-1 and i is an integer; wherein the compressible information includes Y+i-1 sub-information, and each sub-information in the Y+i-1 sub-information is used to indicate the compressible state of the Y+i-1 network layers; wherein the first i sub-information in the Y+i-1 sub-information is used to indicate that the corresponding i network layers have been deleted, and the order in which the i network layers are deleted, and the i+2th to Y+i-1th sub-information in the Y+i-1 sub-information is used to indicate that the corresponding Y-2 network layers have not been deleted.

[0056] The above scheme synchronously updates the compressible information according to the deletion of the network layer, so as to record the deletion of the network layer in all processes of the model compression method in real time and improve the accuracy of model redundant compression.

[0057] In combination with the second aspect, in a possible embodiment, when the difference between the first loss function value and the second loss function value is greater than or equal to the second threshold, the method further includes:

[0058] Update the first identification information in the first first network layer to the second identification information, wherein the first identification information is used to indicate deletion of the current first network layer, and the second identification information is used to indicate that the current network layer is deletable; use the second neural network as the final neural network, and stop executing the model compression method.

[0059] The above scheme, when the accuracy of the first neural network is significantly worse than that of the second neural network, restores the network layer that has been deleted most recently, which can reduce the obvious loss of accuracy due to blindly deleting redundant network layers and achieve a balance between accuracy and performance.

[0060] In combination with the second aspect, in a possible embodiment, the first identification information in the first first network layer is updated to the second identification information, including: determining that the last deleted network layer among the network layers that have been deleted in the model compression method is the first first network layer based on the pre-maintained compressible information, the compressible information is used to indicate that i network layers have been deleted, and the first first network layer is the i-th network layer deleted among the i network layers; updating the first identification information in the i-th first network layer to the second identification information, i≥1 and i is an integer.

[0061] The above scheme, through the pre-maintained compressible information, can quickly find the network layer that has been deleted most recently, thereby improving the efficiency of redundant compression.

[0062] In combination with the second aspect, in a possible embodiment, when i>1, the method further includes:

[0063] The compressible information is updated, and the updated compressible information is used to indicate that the corresponding i-1 network layers have been deleted and the order in which the i-1 network layers are deleted.

[0064] The above scheme synchronously updates the compressible information according to the deletion of the network layer, so as to record the deletion of the network layer in all processes of the model compression method in real time and improve the accuracy of model redundancy compression.

[0065] In combination with the second aspect, in a possible embodiment, the second neural network is obtained by distillation training, and the first neural network is obtained by performing distillation training on the second neural network.

[0066] The above solution integrates model quantization, model distillation and model redundancy compression, thereby merging the three methods of model compression technology to achieve model compression at multiple levels, thereby greatly optimizing model performance.

[0067] In combination with the second aspect, in a possible embodiment, skipping the first first network layer among the multiple first network layers to train the second neural network to obtain the first neural network includes:

[0068] Each of the X first convolution modules included in the fourth neural network is used as a teacher model, and each of the X second convolution modules included in the second neural network is used as a student model, input data is input for each first convolution module, and input data is input for the convolution layer other than the first first network layer in each second convolution module, the X second convolution modules correspond to the X first convolution modules one by one, each first convolution module includes M convolution layers, each second convolution module includes N convolution layers, X>1, M>N≥1, and X, M and N are all integers; according to the first output result of each first convolution module and the second output result of each second convolution module, the parameters of the second neural network are updated to obtain the first neural network.

[0069] The above scheme, on the one hand, by dividing the second neural network and the fourth neural network into multiple convolution modules respectively, the computational complexity of each second convolution module is much lower than the computational complexity of the entire second neural network, and the computational complexity of each first convolution module is also much lower than the computational complexity of the entire fourth neural network, thereby saving training time compared to the commonly used distillation training; on the other hand, the calculations of each second convolution module can be independent of each other so that each second convolution module can be calculated in parallel, and each first convolution module is similar, compared to the commonly used distillation training architecture in which the calculations between the various convolution layers of the teacher network and the super-resolution network are interrelated, it can also save time. Therefore, it can significantly improve the training efficiency, thereby significantly improving the computing efficiency and memory optimization of the GPU server.

[0070] In a third aspect, the present application provides an electronic device, comprising one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, and the computer program code includes computer instructions. When the one or more processors execute the computer instructions, the electronic device executes the method described in the first aspect or the second aspect and any possible implementation method of the first aspect or the second aspect.

[0071] In a fourth aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device, and the chip system includes one or more processors, which are used to call computer instructions so that the electronic device executes the method described in the first aspect or the second aspect and any possible implementation method of the first aspect or the second aspect.

[0072] In a fifth aspect, a computer-readable storage medium is provided, in which instructions are stored, and when the computer-readable storage medium is run on a computer, the computer executes the optimization method of the image processing model of the first aspect or the second aspect mentioned above.

[0073] In a sixth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the method for optimizing the image processing model described in the first aspect or the second aspect.

[0074] The technical effects obtained by the third, fourth, fifth and sixth aspects are similar to the technical effects obtained by the corresponding technical means in the first or second aspects, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 A schematic diagram of an example of model redundancy provided in an embodiment of the present application;

[0076] Figure 2 A schematic diagram of an example of a method for analyzing model redundancy provided in an embodiment of the present application;

[0077] Figure 3 A schematic diagram of an example of a traditional distillation training architecture;

[0078] Figure 4A and Figure 4B A schematic flow chart of a model compression method provided in an embodiment of the present application;

[0079] Figure 5 A schematic diagram of an example of a model compression method provided in an embodiment of the present application;

[0080] Fig. 6A A schematic diagram of an example of a second neural network provided in an embodiment of the present application;

[0081] Figure 6B A schematic diagram of an example of compressible information corresponding to a second neural network provided in an embodiment of the present application;

[0082] Fig. 7A A schematic diagram of another example of a second neural network provided in an embodiment of the present application;

[0083] Figure 7B A schematic diagram of another example of compressible information corresponding to the second neural network provided in an embodiment of the present application;

[0084] Figure 8 A schematic diagram of an example of a modular distillation training architecture provided in an embodiment of the present application;

[0085] Fig. 9A schematic diagram of an example of an image processing method provided in an embodiment of the present application;

[0086] Fig.10 A schematic diagram of the hardware structure of an electronic device 1000 provided in an embodiment of the present application;

[0087] Fig.11 A block diagram of a software system of an electronic device 1000 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0088] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0089] It should be understood that the "multiple" mentioned in this application refers to two or more. In the description of this application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solution of this application, the words "first" and "second" are used to distinguish between the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not limit them to be different.

[0090] Model redundancy is a common problem in the design of neural network models. Model redundancy will reduce the computing efficiency of terminal devices and cause unnecessary power consumption overhead for terminal devices. Traditional model compression for model redundancy obtains the final compressed neural network model by manually modifying the network structure of the neural network model and trying to modify the network structure multiple times. In order to more clearly illustrate the embodiments of the present application, before introducing the embodiments of the present application, the relevant content of model redundancy involved in this application, the concept of compressible modules and several model compression technologies are first explained.

[0091] 1. Model redundancy.

[0092] Figure 1 A schematic diagram of an example of model redundancy provided in an embodiment of the present application. Given two neural network models, model #1f1(·) and model #2f2(·), for the same input data I, if the difference #1 between the output results O1 and O2 is less than the threshold #1, and o(f1(·))>o(f2(·)), then redundancy exists in model #1f1(·). Wherein, o(f1(·)) is the complexity of f1(·), and o(f2(·)) is the complexity of f2(·).

[0093] Because terminal devices have higher requirements for the performance and power consumption of neural network models, the analysis and removal of model redundancy is of great significance to terminal devices. There are many ways to analyze model redundancy, including Figure 1 The calculation method of the process shown also includes the method of analyzing model redundancy provided in the embodiment of the present application.

[0094] 2. The method of analyzing model redundancy provided by this application.

[0095] By comparing the output results of the model before and after quantization, it is determined whether there is redundancy in the model.

[0096] Figure 2 A schematic diagram of an example of a method for analyzing model redundancy provided in an embodiment of the present application. Figure 2 As shown, firstly, given an input data, the output result #1 of the model #3 which has not been quantized is obtained. Model #3 is quantized by the quantization model to obtain model #3'. The same input data is input into model #3' to obtain a new output result, namely output result #2. By comparing the two output results, if the similarity measure (similarity measure) between output result #1 and output result #2 is less than the threshold #2, it means that quantization (quantization is a type of model compression technology) has no effect on the accuracy of the model itself, that is, the model still has room for compression, so the model has redundancy. When the similarity measure between the two output results is greater than or equal to the threshold #2, that is, even a small degree of compression at the parameter level such as quantization can have a greater impact on the result of the model itself, it means that the model has no redundancy and cannot be compressed from the model structure level.

[0097] It is understood that the similarity measure of two output results in this application is a numerical value used to measure the proximity of the two output results, and the similarity measure involved in this application is a parameter value. Exemplarily, the similarity measure between the two output results can be obtained by calculating the difference between the two output results, or it can also be obtained by other methods of calculating the similarity measure of the neural network, which is not limited in this application.

[0098] 3. Compressible network layer.

[0099] In this application, repeated computational structural units with the same network structure in a neural network are referred to as compressible network layers; or, network layers with consistent types and parameter settings are referred to as compressible network layers. Among them, the types of network layers include convolutional layers, pooling layers, activation layers, etc. For example, compressible network layers include convolutional layers with consistent parameter settings. It should be noted that compressible network layers do not include network layers that process input or output.

[0100] 4. Model compression technology involved in this application

[0101] (1) Quantification.

[0102] Quantization can convert floating-point numbers stored in the model layer into fixed-point numbers. Specifically, it converts floating-point values ​​into bit-type integer values, thereby increasing the speed of calculation and reducing the cost of calculation. For example, converting floating-point values ​​such as single-precision floating-point numbers (float) and double-precision floating-point numbers (double) into integer values ​​such as 16-bit integers (int16) and 8-bit integers (int8). A 32-bit floating-point number (float32) occupies 4 bytes, while an 8-bit integer (int8) occupies only 1 byte at most. Therefore, the advantage of quantization is not only that integer values ​​are several times faster than floating-point values ​​in memory reading, but also that fixed-point operations are easier to implement hardware acceleration than floating-point operations.

[0103] However, although quantization can improve model performance, it will cause a loss of model accuracy. For example, when converting a 32-bit floating point number (float32) to an 8-bit integer (int8), because the range of values ​​that a 32-bit floating point number (float32) can maintain is larger than that of an 8-bit integer (int8), a large number of values ​​cannot be represented by an 8-bit integer (int8) and can only be rounded to an 8-bit integer (int8) value.

[0104] Quantization includes post-training quantization (PTQ) and quantization-aware training (QAT). PTQ is a form of offline quantization that is separate from actual training. QAT combines the model's own training with the quantization process and is a form of online quantization.

[0105] (2) Unstructured pruning.

[0106] Pruning refers to pruning a neural network model, which can be roughly divided into structured pruning and unstructured pruning. Among them, structured pruning improves the reasoning speed of the neural network model by pruning the structure of the neural network model and reducing the number of model parameters. Therefore, structured pruning will change the structure of the neural network model. Assume that the neural network model (such as a convolutional neural network model) contains 5 convolutional layers, namely [L1, L2, L3, L4, L5]. Among them, L2 contains 8 convolution kernels of shape 3*3, then each of the 8 convolution kernels of L2 has 3*3=9 weight parameters (that is, it can be regarded as a 3*3 matrix), then the weight of L2 can be regarded as an 8*3*3 matrix. Structured pruning can directly prune a convolution layer of the neural network model, such as pruning L4, and the pruned neural network model will only have 4 convolutional layers, such as [L1, L2, L3, L5]. Structured pruning can also prune the convolution kernels of one or more convolution layers of the neural network model. For example, pruning the 8 convolution kernels in the L2 layer to 5 convolution layers. After pruning, there are 5 convolution kernels of shape 3*3 left in the L2 layer of the neural network model. Structured pruning can also prune the shape of the convolution kernels of one or more convolution layers in the neural network model. For example, pruning the shape of the 8 convolution kernels in the L2 layer from 3*3 to 2*2. After pruning, each of the 8 convolution kernels in the L2 layer of the neural network model has 2*2=4 weight parameters (that is, it can be regarded as a 2*2 matrix). In summary, the above three structured pruning methods will change the structure of the neural network model.

[0107] (3) Distillation training technology.

[0108] The distillation training involved in this application refers to pre-training a teacher network, and then in the training process of the super-resolution network, the teacher network is used as the teacher model and the super-resolution network is used as the student model, so that the teacher network guides the super-resolution network for training. Among them, the number of network layers included in the teacher network is significantly greater than the number of network layers included in the super-resolution network, and the number of parameters of the teacher network is significantly greater than the number of parameters of the super-resolution network; in other words, the network structure of the teacher network is significantly more complex than the network structure of the super-resolution network. Specifically, in the training process of the super-resolution network, the output result of the super-resolution network is no longer fitted to the target data, but the output result of the teacher network when the same input data is input. That is, the output result of the teacher network is used as a supervisory signal for the output result of the super-resolution network to supervise the training of the super-resolution network. The purpose of distillation training is to improve the fitting ability of the super-resolution network, that is, the learning ability. Since the number of parameters of the super-resolution network is relatively small, the learning ability is relatively weak. It is difficult to converge to the same degree as the teacher network by directly training the super-resolution network, that is, the error between the output result of the super-resolution network and the actual value is difficult to drop to the same degree as the error between the output result of the teacher network and the actual value. Therefore, during the training process, the output of the teacher network when inputting the same input data is used as the label of the input data (or supervisory signal), and the output of the super-resolution network is used to fit the output of the teacher network, and converge to a degree similar to that of the teacher network, which can improve the learning ability of the super-resolution network. The above process of using a larger network structure as a teacher model to guide the learning of the student model is called distillation training.

[0109] Figure 3 This is a schematic diagram of an example of a traditional distillation training architecture. Figure 3 As shown, the same input data is input to the teacher network and the super-resolution network, and the output results of the teacher network and the super-resolution network are output result #3 and output result #4 respectively. Output result #3 is used as a supervision signal to supervise the training of the super-resolution network. In addition, back propagation is performed based on the loss function of the distillation training architecture to update the parameters of the teacher network and the super-resolution network. In this architecture, the loss function of the distillation training architecture consists of two parts, one is the difference between output result #3 and output result #4, and the other is the loss function of training the super-resolution network to convergence.

[0110] During the distillation training process, for each set of training data, the input data is used not only to calculate the output result #4 of the super-resolution network, but also to synchronously calculate the output result #3 of the teacher network. And because the teacher network has a larger number of parameters, it takes longer to calculate the output result #3. Therefore, the time taken for distillation training is more than doubled that of directly training the super-resolution network.

[0111] As described in the background technology, when it is necessary to deploy a model for pixel-level image tasks such as optimizing image resolution on a mobile terminal such as a mobile phone, a GPU server will be used to compress the model and deploy the compressed model to the mobile terminal. For example, in a mobile game scenario, it is necessary to optimize the resolution of several frames of images. Generally, the mobile terminal first separates the several frames of images into text data and image data, and then inputs the image data into a neural network model, which outputs the image data with optimized resolution. The optimization of the resolution of an image by a neural network is a pixel-level image task, which is very demanding on the computing power and power consumption of the mobile terminal.

[0112] In view of this, the present application provides an image processing method and a model compression method.

[0113] According to the model compression method provided by the present application, when it is analyzed that the neural network to be compressed has redundancy through the method of analyzing model redundancy provided by the present application as described above, the first network layer in the current compressible network layer in the neural network to be compressed is deleted, and neural network #1 is trained. Among them, the current compressible network layer is a network layer with the same current type and parameter settings in the neural network to be compressed. Then, through the method of analyzing model redundancy provided by the present application as described above, it is analyzed whether neural network #1 has redundancy. If there is redundancy, the first network layer in the current compressible network layer in neural network #1 is deleted, and so on; if there is no redundancy, the model compression method is stopped, and neural network #1 is used as the compressed neural network finally obtained by the model compression method.

[0114] It can be understood that the output results of the current neural network before and after quantization are used to determine whether there is redundancy, and if there is redundancy, a network layer is deleted, and if there is no redundancy, the model compression is stopped, so as to achieve adaptive and accurate pruning, thereby achieving model redundancy compression. The above scheme does not need to compare the output results of the current neural network with another neural network to determine whether there is redundancy in the current neural network. Figure 1 The method of analyzing model redundancy in the model can improve the efficiency of analyzing model redundancy and deleting redundant network layers, thereby improving the efficiency of model compression. Compared with traditional structured pruning, the above solution can achieve no manual intervention and save labor costs. Therefore, by deleting redundant network structures and compressing the model, the computing efficiency and memory optimization of the GPU server can be improved.

[0115] Optionally, in the model compression method, model quantization, model distillation and model redundancy compression can be integrated together to achieve model compression at multiple levels, thereby greatly optimizing model performance.

[0116] According to the image processing method provided by the present application, the first neural network obtained by the above-mentioned model compression method is used to process the image, and the image processed by the first neural network is output.

[0117] It can be understood that the first neural network obtained by the above-mentioned model compression method reduces redundant network layers and has a simpler network structure compared to the model to be compressed, thereby saving computing power on the mobile terminal and reducing power consumption on the mobile terminal.

[0118] The image processing method provided in the embodiment of the present application is applicable to a variety of electronic devices, such as mobile phones, tablet computers, cameras, smart wearable devices, etc., and the embodiment of the present application is not limited to this.

[0119] It should be noted that the model compression method provided in this application is a general compression scheme for artificial intelligence models. The model compression method is highly adaptable and can achieve adaptive model redundancy compression for artificial intelligence models of different application types, greatly reducing the power consumption of the model generated by calculation on the mobile terminal. The image processing method provided in this application can be understood as an example of an application scenario applicable to the model compression method provided in this application, and does not limit the application scenarios applicable to the model compression method.

[0120] The following will be combined FIG. 4A to FIG. 8 , the model compression method provided in the embodiment of the present application is introduced in detail.

[0121] Figure 4A and Figure 4B A schematic flow chart of a model compression method provided in an embodiment of the present application. Figure 4B , Figure 4A The relationship between the various steps of the model compression method can be more clearly seen; compared to Figure 4A , Figure 4B It is more consistent with the following text and is more convenient to clearly introduce the model compression method. Figure 4A and Figure 4B As shown, the model compression method includes the following steps:

[0122] 1. The neural network before quantization passes through the PTQ module to obtain the quantized neural network.

[0123] 2. Input the input data into the neural network before quantization and the neural network after quantization, and obtain output result #a and output result #b respectively. Specifically, step 2.1 inputs the input data into the neural network before quantization to obtain output result #a. Step 2.2 inputs the input data into the neural network after quantization to obtain output result #b.

[0124] 3. The judgment module #1 judges whether there is redundancy in the neural network before quantization according to the output result #a and the output result #b. Specifically, the judgment module judges whether the similarity measure of the output result #a and the output result #b is less than the first threshold.

[0125] 3.1. If the judgment result in step 3 is "yes", the deletion module deletes the first network layer in the compressible network layer of the neural network before quantization, that is, the network structure is updated, and the training module trains the neural network after the network structure is updated to obtain a trained neural network.

[0126] 3.1.1. Judgment module #2 determines whether the accuracy of the trained neural network is significantly worse than that of the neural network before quantization.

[0127] 3.1.1.1. If the accuracy of the neural network after training in step 3.1.1 is not significantly worse than that of the neural network before quantization, the trained neural network is used as the neural network before quantization in the next round of the process, and steps 1 to 3 are repeated, and step 3.1 or step 3.2 is repeated according to the judgment result in step 3.

[0128] 3.1.1.2. If the accuracy of the neural network after training in step 3.1.1 is significantly lower than that of the neural network before quantization, return to the deletion module, and use the deletion module to restore the first network layer deleted in step 3.1, that is, output the neural network before quantization, and use it as the final model compression to obtain the neural network, and stop the process of the model compression method.

[0129] 3.2. If the judgment result in step 3 is "no", the neural network before quantization is output as the final model compression neural network, and the process of the model compression method is stopped.

[0130] It is understandable that the model compression method provided in this application can be Figure 4A and Figure 4B The process in is repeated for multiple rounds. Figure 5 , taking the i-th round process in the model compression method provided in this application as an example for introduction, i≥1 and i is an integer.

[0131] Figure 5 A schematic diagram of a model compression method 100 provided in an embodiment of the present application. Exemplarily, the method 100 may be executed by a GPU server.

[0132] S101, inputting input data into a second neural network to obtain a first output result, wherein the second neural network includes a first compressible network layer, and the first compressible network layer includes a plurality of first network layers with consistent types and parameters.

[0133] The second neural network and the first output result in S101 can be understood as the neural network before quantization and the output result #a in the i-th round of the process, respectively.

[0134] Fig. 6A Schematic diagram of an example of a second neural network provided in an embodiment of the present application. Fig. 6A As shown, the first compressible network layer includes Y first network layers of the same type and parameters. Fig. 6A The network layers filled with left slashes shown in are some of the i-1 network layers that have been deleted in the previous i-1 round of the process, and the other deleted network layers are replaced by ellipsis.

[0135] In a possible implementation, the Y first network layers included in the first compressible network layer all include second identification information, and the second identification information is used to indicate that the current convolutional layer is deletable. Exemplarily, the first identification information and the second identification information are implemented by means of a dropout operation. Specifically, a dropout layer is configured in each of the multiple first network layers to control the training parameters, or to mark whether the parameters of the layer are to be updated, or to mark whether the layer is to be trained. Wherein, when the parameter of the dropout layer is set to 1, the dropout layer is the second identification information, indicating that the current network layer configured with the dropout layer can be deleted. When the parameter of the dropout layer is set to 0, the dropout layer is the first identification information, indicating that the corresponding network layer parameters will not be updated, or the network will skip the layer for training, or the network layer will not affect the final training result of the network, or the network layer is deleted from the network. The dropout layer is configured for the Y first network layers and marked, and the parameter of the dropout layer defaults to 1. The deleted i-1 network layers all include first identification information, and the first identification information is used to indicate that the current network layer is deleted. Exemplarily, the deleted i-1 network layers are marked by adding a dropout layer, and the parameter of the dropout layer is 0.

[0136] As an example, compressible information corresponding to all processes of the model compression method is maintained, and the compressible information is used to record the compressible status of all compressible network layers in all processes of the model compression method. The number of sub-information in the compressible information is the same as the number of all compressible network layers in all processes of the model compression method, and the sub-information in the compressible information corresponds one-to-one to all compressible network layers in all processes of the model compression method. Among them, the compressible status includes any of the following: deleted, optionally including the order of deletion; or, deletable or not deleted. In the i-th round of the process, in the first compressible network layer of the second neural network, the compressible status of the first i-1 network layers is deleted, and the compressible status of the Y first network layers is deletable or not deleted.

[0137] For example, each sub-information in the compressible information may be a status bit, and the status bit is used to indicate the compressible status of the corresponding network layer. For example, if the status bit is 1, it means that the compressible status of the network layer is: deleted; if the status bit is 0, it means that the compressible status of the network layer is: deletable or not deleted.

[0138] For another example, the sub-information in the compressible information that indicates that the compressible state of the network layer has been deleted is called sub-information #1, and the arrangement order of sub-information #1 can represent the order in which the deleted network layers have been deleted.

[0139] Figure 6B A schematic diagram of an example of compressible information corresponding to a second neural network provided in an embodiment of the present application. The compressible information is taken as a vector, and each sub-information in the compressible information is taken as each column of the vector. Figure 6B As shown, each column of the vector corresponds to the first i-1 network layers that have been deleted and the Y first network layers that have not been deleted in the first compressible module. The values ​​of the parameters corresponding to the first i-1 network layers that have been deleted in the vector are 1, and the order of the values ​​1 corresponding to each network layer in the first i-1 network layers in the vector indicates the order in which each network layer in the first i-1 network layers is deleted. Figure 6B As shown, the value of the parameter corresponding to each first network layer in the Y first network layers in the vector is 0.

[0140] S102, performing quantization processing on the second neural network after training to obtain a third neural network, wherein the parameters of the second neural network are all floating-point values, and the parameters of the third neural network model are all integer values.

[0141] The third neural network in S102 can be understood as the quantized neural network in the i-th round of the process.

[0142] As mentioned above for post-training quantization (PTQ), converting floating-point values ​​into integer values ​​can improve computing speed and reduce computing costs. The advantage of quantization is not only that integer values ​​are several times faster than floating-point values ​​in memory reading, but also that fixed-point operations are easier to implement hardware acceleration than floating-point operations.

[0143] S103, inputting the input data into the third neural network to obtain a second output result.

[0144] The second output result in S103 can be understood as the output result #b in the i-th round of the process.

[0145] S104: When the similarity measure between the first output result and the second output result is less than a first threshold, skip the first first network layer among the plurality of first network layers to train the second neural network to obtain the first neural network.

[0146] The first neural network in S104 can be understood as the trained neural network in step 3.1 in the i-th round of the process. Exemplarily, the epochs for training the second neural network can be set according to the amount of data of the second neural network. For example, the second neural network can be trained for 100 epochs.

[0147] Among them, according to the similarity measure between the first output result and the second output result being less than the first threshold, the second identification information in the first first network layer is updated to the first identification information. The above scheme, by adding identification information (including the first identification information or the second identification information) to the Y first network layers respectively, to delete the first network layer including the first identification information, and not deleting the first network layer including the second identification information for training, can make the second neural network layer adaptively try to delete each network layer in the compressible network layer, that is, it can automatically delete the redundant layers during the training process, which is more efficient than the traditional method of manually modifying the network structure during the model redundancy compression process.

[0148] Fig. 7A FIG. 1 is a schematic diagram of another example of a second neural network provided in an embodiment of the present application. Fig. 7AAs shown, the second identification information in the first first network layer is updated to the first identification information, for example, the parameter of the dropout layer configured for the first first network layer is set to 1 by default, and the parameter of the dropout layer configured for the first first network layer is set to 0. Thus, when the second neural network is trained, the first first network layer is automatically skipped for training according to the first identification information included in the first first network layer. In other words, the first first network layer is deleted from the second neural network, while the other first network layers except the first first network layer in the plurality of first network layers are not deleted, or the second first network layer to the Yth first network layer in the Y first network layers are not deleted, or the second first network layer to the Yth first network layer in the Y first network layers are trained to update the parameters of the second neural network.

[0149] It can be understood that the above scheme uses the elimination property of the dropout operation to adaptively try to delete each network layer in the compressible network layer, that is, it can automatically delete redundant layers during the training process, which is more efficient than manually modifying the network structure during the traditional model redundancy compression process.

[0150] Exemplarily, when the second identification information in the first first network layer is updated to the first identification information, the pre-maintained compressible information needs to be updated. The updated compressible information is used to indicate that i network layers have been deleted, and the first first network layer is the i-th network layer deleted among the i network layers, 1≤i≤Y+i-1 and i is an integer. The compressible information includes Y+i-1 sub-information, and each sub-information in the Y+i-1 sub-information is used to indicate the compressible state of the Y+i-1 network layers; the first i-1 sub-information in the Y+i-1 sub-information is used to indicate that the corresponding i-1 network layers have been deleted, and the order in which the i-1 network layers are deleted, and the i+1th to Y+i-1th sub-information in the Y+i-1 sub-information is used to indicate that the corresponding Y-1 network layers have not been deleted.

[0151] Figure 7B A schematic diagram of another example of compressible information corresponding to the second neural network provided in an embodiment of the present application. Figure 7B As shown, while the second identification information in the first first network layer is updated to the first identification information, the value of the i-th column parameter in the pre-maintained vector is updated from 0 to 1. Therefore, the values ​​of the first i-th column parameters of the vector are all 1. The pre-maintained vector is used to indicate that i network layers have been deleted, and the first first network layer is the i-th network layer deleted among the i network layers.

[0152] It can be understood that the above scheme synchronously updates the compressible information according to the deletion of the network layer, so as to record the deletion of the network layer in all processes of the model compression method in real time and improve the accuracy of model redundancy compression.

[0153] In a possible implementation of S104, based on a first loss function value, the first first network layer is skipped to train the second neural network, wherein the first loss function value includes a similarity measure between the first output result and the second output result.

[0154] It can be understood that the process of outputting the first output result by the second neural network and the second output result by the third neural network is differentiable, so the similarity measure of the two output results can be used for back propagation. In other words, taking the similarity measure of the two output results as the difference between the two output results as an example, the difference between the first output result and the second data result can be used as the loss function value for back propagation.

[0155] The above scheme, by applying the similarity measure of the output results of the second neural network before and after quantization to the loss function value used to train the second neural network and performing back propagation, can reduce the difference between the output results of the second neural network and the third neural network before and after quantization, thereby accelerating the convergence in the process of training the second neural network.

[0156] In an implementation parallel to S104, when the similarity measure between the first output result and the second output result is greater than or equal to the first threshold, it indicates that there is no redundancy in the second neural network, that is, the second neural network has reached the simplest network structure, and the second neural network is output as the neural network finally obtained by the model compression method. This implementation can be understood as a possible example of step 3.2 in the above process.

[0157] Optionally, the method 100 may further include:

[0158] Step 1-1, save the first loss function value used to train the second neural network.

[0159] Wherein, step 1-1 can be performed during the process of training the second neural network.

[0160] Step 1-2, obtaining a second loss function value for training the first neural network.

[0161] The second loss function value can be saved during the process of training the first neural network.

[0162] Step 1-3, determine whether the difference between the first loss function value and the second loss function value is less than a second threshold.

[0163] Among them, steps 1-3 can be understood as a possible implementation method of step 3.1.1 in the above process.

[0164] It can be understood that by judging whether the difference between the first loss function value and the second loss function value is less than the second threshold, it can be determined whether deleting the first network layer makes the accuracy of the first neural network significantly worse than that of the second neural network. Thus, it is possible to reduce the situation where the accuracy is obviously lost due to blindly deleting redundant network layers, and achieve a balance between accuracy and performance. In addition, the above scheme adaptively attempts to delete each network layer in the compressible network layer, and analyzes whether there will be a significant loss of accuracy after deletion, so that the number of redundant layers can be automatically deleted during the training process and the deletion results can be judged, which is more efficient than the traditional method of manually modifying the network structure during the model redundancy compression process.

[0165] Regarding possible situation 1 of steps 1-3, as a possible implementation of step 3.1.1.1 in the above process:

[0166] If the difference between the first loss function value and the second loss function value is less than the second threshold, it means that the accuracy of the first neural network is not significantly worse than that of the second neural network. Therefore, it is determined whether there is still redundancy in the first neural network, and if there is redundancy, the redundant network layer is continuously deleted. For the first neural network, the steps as performed for the second neural network in S101 to S104 are executed, that is, the i+1th round of the process is executed.

[0167] Specifically, the input data is input into the first neural network to obtain the third output result, wherein the first neural network includes a second compressible network layer, the second compressible network layer includes Y-1 second network layers of the same type and parameters, and the Y-1 second network layers correspond one-to-one to the second to the Yth first network layers in the Y first network layers. The first neural network is quantized after training to obtain the fifth neural network, the parameters of the first neural network are all floating-point values, and the parameters of the fifth neural network model are all integer values; the input data is input into the fifth neural network to obtain the fifth output result. In one possible case, the similarity measure between the third output result and the fifth output result is less than the first threshold, and the first second network layer in the Y-1 second network layers is skipped to train the first neural network to obtain the sixth neural network. In parallel, in another possible case, the similarity measure between the third output result and the fifth output result is greater than or equal to the first threshold, the first neural network is used as the final neural network, and the model compression method is stopped. It can be understood that when the similarity measure between the third output result and the fifth output result is greater than or equal to the first threshold, it means that there is no redundancy in the first neural network, that is, the first neural network has almost reached the simplest network structure, or is extremely close to the simplest network structure, and the first neural network is output as the neural network finally obtained by the model compression method. This implementation method can be understood as another possible example of step 3.2 in the above process.

[0168] Correspondingly, it is also necessary to update the pre-maintained compressible information according to the processing of the first neural network. Specifically, the pre-maintained compressible information is updated, and the updated compressible information is used to indicate that i+1 network layers have been deleted, and the first second network layer is the i+1th network layer deleted in the i+1 network layers, 1≤i≤Y+i-1 and i is an integer; wherein the compressible information includes Y+i-1 sub-information, and each sub-information in the Y+i-1 sub-information is used to indicate the compressible state of the Y+i-1 network layer; wherein the first i sub-information in the Y+i-1 sub-information is used to indicate that the corresponding i network layers have been deleted, and the order in which the i network layers are deleted, and the i+2th to Y+i-1th sub-information in the Y+i-1 sub-information is used to indicate that the corresponding Y-2 network layers have not been deleted.

[0169] It should be noted that the description of the (i+1) round of the process can be understood by referring to the corresponding contents in S101 to S104, and will not be elaborated here.

[0170] It can be understood that when the accuracy of the first neural network is not significantly worse than that of the second neural network, the model redundancy compression of the first neural network will continue to be automatically performed. Compared with traditional structured pruning, this can be done without any human intervention and can also save labor costs.

[0171] Regarding possible situation 2 of steps 1-3, as a possible implementation of step 3.1.1.2 in the above process:

[0172] If the difference between the first loss function value and the second loss function value is greater than or equal to the second threshold, it means that the accuracy of the first neural network is significantly worse than that of the second neural network. Therefore, for the first neural network, the most recently deleted network layer is restored, or the last deleted network layer among all deleted network layers is restored; or in other words, the first neural network is abandoned, and the second neural network used to generate the first neural network is used as the final neural network obtained by the process of the entire model compression method, and the process is stopped.

[0173] Specifically, the first identification information in the first first network layer is updated to the second identification information, the second neural network is used as the final neural network, and the model compression method is stopped.

[0174] In a possible implementation, the last deleted network layer among all network layers that have been deleted in all processes of the model compression method can be determined based on the pre-maintained compressible information.

[0175] Specifically, as can be seen from the above, the compressible information is used to indicate that i network layers have been deleted, and the first first network layer is the i-th network layer deleted from the i network layers. Therefore, according to the pre-maintained compressible information, it is determined that the last deleted network layer in the deleted network layers in the model compression method is the first first network layer, and then the first identification information in the i-th first network layer is updated to the second identification information, i≥1 and i is an integer.

[0176] Accordingly, the pre-maintained compressible information also needs to be updated. Specifically, when i>1, the compressible information is updated, and the updated compressible information is used to indicate that the corresponding i-1 network layers have been deleted, and the order in which the i-1 network layers are deleted.

[0177] It can be understood that when the accuracy of the first neural network is significantly worse than that of the second neural network, restoring the network layer that was most recently deleted can reduce the obvious loss of accuracy due to blindly deleting redundant network layers, and achieve a balance between accuracy and performance.

[0178] In a possible implementation of method 100, the first neural network, the second neural network, or the third neural network involved in method 100 are all obtained through distillation training.

[0179] It can be understood that by integrating model quantization, model distillation and model redundancy compression into the above solution, the three methods of model compression technology are merged to achieve model compression at multiple levels, thereby greatly optimizing model performance.

[0180] Optionally, the distillation training is modular distillation training. The following is an example of skipping the first i first network layers among Y first network layers to perform modular distillation training on the second neural network to obtain the first neural network. The network layer is a convolutional layer as an example.

[0181] Specifically, each of the X first convolution modules included in the fourth neural network is used as a teacher model, and each of the X second convolution modules included in the second neural network is used as a student model, input data is input for each first convolution module, and input data is input for the convolution layer other than the first first network layer in each second convolution module, the X second convolution modules correspond to the X first convolution modules one by one, each first convolution module includes M convolution layers, each second convolution module includes N convolution layers, X>1, M>N≥1, and X, M and N are all integers; according to the first output result of each first convolution module and the second output result of each second convolution module, the parameters of the second neural network are updated to obtain the first neural network.

[0182] Figure 8 A schematic diagram of an example of a modular distillation training architecture provided in an embodiment of the present application. Figure 8 As shown, the fourth neural network includes 3 first convolution modules, and each first convolution module includes 4 convolution layers, and the second neural network includes 3 second convolution modules, and each second convolution module includes 2 convolution layers. The three first convolution modules correspond one to one with the three second convolution modules.

[0183] Among them, in addition to the convolutional layer, the second neural network and the fourth neural network also include other layers, such as activation layer, fully connected layer, pooling layer, etc.

[0184] In which, the i-th first convolution module among the X first convolution modules is taken as the teacher model, and the i-th second convolution module among the X second convolution modules is taken as the student model. The same input data is input to the i-th first convolution module and the i-th second convolution module respectively to obtain the i-th first output result and the i-th second output result. With the i-th first output result as the supervision signal, the i-th second output result of the i-th second convolution module is used to fit the i-th first output result of the i-th first convolution module, and converges to a degree close to the i-th first convolution module. In which, 1≤i≤X and i is an integer.

[0185] It can be understood that, on the one hand, by dividing the second neural network and the fourth neural network into multiple convolution modules respectively, the computational complexity of each second convolution module is much lower than the computational complexity of the entire second neural network, and the computational complexity of each first convolution module is also much lower than the computational complexity of the entire fourth neural network, thereby saving training time compared to the commonly used distillation training; on the other hand, the calculations of each second convolution module can be independent of each other so that each second convolution module can be calculated in parallel, and each first convolution module is similar thereto, which can also save time compared to the commonly used distillation training architecture in which the calculations of each convolution layer of the teacher network and the super-resolution network are interrelated. Therefore, the training efficiency can be significantly improved, thereby significantly improving the computing efficiency and memory optimization of the GPU server.

[0186] Optionally, in order to compensate for the accuracy loss that may be caused by quantization and distillation after training, back propagation of modules can be performed. First, according to the i-th first output result and the i-th second output result, the i-th first sub-loss function value is obtained. For example, the difference between the i-th first output result and the i-th second output result is determined as the i-th first sub-loss function value. Then, according to the i-th second output result and the target data, the i-th second sub-loss function value is obtained. For example, the difference between the i-th second output result and the target data is determined as the i-th second sub-loss function value. Thus, according to the i-th first sub-loss function value and the i-th second sub-loss function value, the i-th first loss function value is obtained. For example, the sum of the i-th first sub-loss function value and the i-th second sub-loss function value is determined as the i-th first loss function value. Back propagation is performed based on the i-th first loss function value to update the parameters of the i-th second convolution module. Optionally, back propagation is performed based on the i-th first loss function value to update the parameters of the i-th first convolution module.

[0187] Optionally, to further compensate for the possible accuracy loss caused by post-training quantization and distillation, global backpropagation can also be performed.

[0188] The following will be combined Fig. 9 , the image processing method provided in the embodiments of the present application is introduced in detail. Fig. 9 Schematic diagram of an image processing method 200 provided in an embodiment of the present application. Fig. 9 As shown, the method 200 includes S201 to S203. The method 200 is executed by an electronic device.

[0189] S201, acquiring an image.

[0190] It is understandable that the electronic device can obtain the image processed by the first neural network model in a variety of ways, such as the electronic device can capture the image through an image capturing device (such as a camera); for example, the electronic device can download the image, etc.; for example, the electronic device can exchange information such as images with other electronic devices through some applications. This application does not limit this.

[0191] S202: Process the image based on a first neural network model.

[0192] The first neural network is obtained by performing redundant compression processing on the first compressible network layer included in the second neural network, and the first compressible network layer includes multiple first network layers with the same type and parameters. Specifically, the first neural network is obtained by skipping the first first network layer among the multiple first network layers to train the second neural network when the similarity measure between the first output result and the second output result is less than or equal to the first threshold, wherein the first output result is obtained by inputting the input data into the second neural network, the second output result is obtained by inputting the input data into the third neural network, the third neural network is obtained by quantizing the second neural network after training, the parameters of the second neural network are all floating point values, and the parameters of the third neural network model are all integer values.

[0193] Optionally, the first neural network is obtained by training the second neural network by skipping the first first network layer based on a first loss function value when a similarity measure between the first output result and the second output result is less than or equal to a first threshold, wherein the first loss function value includes a similarity measure between the first output result and the second output result.

[0194] Specifically, the method for obtaining the first neural network through redundant compression processing can refer to the relevant description of the model compression method 100 above. In one possible implementation, the first neural network is obtained when the similarity measure between the first output result and the second output result is less than a first threshold, and the similarity measure between the third output result and the fifth output result is greater than or equal to the first threshold. In another possible implementation, the first neural network is obtained when the similarity measure between the first output result and the second output result is less than a first threshold, and the difference between the first loss function value and the second loss function value is greater than or equal to the second threshold. S203, output the image processed by the first neural network model.

[0195] As an example of method 200, after the electronic device acquires an image, it obtains image data (hereinafter referred to as image data #1 for convenience of explanation), and image data #1 includes data of multiple pixels in the image. For example, the image data #1 can be in raw (RAW) format or red, green and blue (RGB) format. Image data #1 is input into the first neural network, and the image data is calculated by the Z network layers in the first neural network. Specifically, the input layer first parses and processes the image data #1, and outputs a feature map that can be processed by the network layer (including convolution layer, pooling layer, activation layer, etc.), the first network layer among the Z network layers calculates the vector, and the calculation result output by the first network layer is input into the second network layer for calculation... The jth network layer calculates the vector, and the calculation result output by the jth network layer is input into the j+1th network layer and calculated by the j+1th network layer, 1≤j≤Z-1, Z>1 and j and Z are integers. The calculation result output by the Zth network layer is input to the output layer, and the output layer parses and processes the calculation result to output the image processed by the first neural network (hereinafter referred to as image data #2 for convenience of explanation). In one possible implementation, the image processed by the first neural network can be image data in RAW format or RGB format.

[0196] For example, in a mobile game scenario, the resolution of an image is optimized by the first neural network. Generally, the mobile terminal first separates the acquired frames of images into text data and image data #1, and then inputs the image data #1 into the neural network model. The first neural network calculates the image data #1 in the manner described in the above example and outputs the image data #2 with optimized resolution.

[0197] In the image processing method provided in the embodiment of the present application, the first neural network obtained by the above-mentioned model compression method reduces redundant network layers and has a simpler network structure compared to the model to be compressed, thereby saving computing power of the mobile terminal and reducing power consumption of the mobile terminal.

[0198] Please refer to Fig.10 , Fig.10 A hardware structure diagram of an electronic device 1000 provided in an embodiment of the present application. Fig.10The electronic device 1000 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0199] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 1000. In other embodiments of the present application, the electronic device 1000 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0200] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc.

[0201] The controller may be the nerve center and command center of the electronic device 1000. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0202] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0203] In some embodiments, the processor 110 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0204] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to realize the shooting function of the electronic device 1000. The processor 110 and the display screen 194 communicate via the DSI interface to realize the display function of the electronic device 1000.

[0205] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 1000. In other embodiments of the present application, the electronic device 1000 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0206] The charging management module 140 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193 and the wireless communication module 160.

[0207] The wireless communication function of the electronic device 1000 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0208] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 1000 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0209] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 1000. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1.

[0210] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 1000. The wireless communication module 160 can be one or more devices integrating at least one communication processing module.

[0211] The electronic device 1000 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0212] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 1000 may include 1 or N display screens 194, where N is an integer greater than 1.

[0213] The electronic device 1000 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0214] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0215] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 1000 may include 1 or N cameras 193, where N is an integer greater than 1.

[0216] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 1000 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0217] Video codecs are used to compress or decompress digital videos. The electronic device 1000 may support one or more video codecs. Thus, the electronic device 1000 may play or record videos in various coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0218] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 1000 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0219] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 1000. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0220] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 1000 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created by the electronic device 1000 during use (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0221] The electronic device 1000 can implement audio functions, such as music playback, recording, etc., through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone interface 170D and the application processor. The button 190 includes a power button, a volume button, etc. The button 190 can be a mechanical button or a touch button. The electronic device 1000 can receive key input and generate key signal input related to the user settings and function control of the electronic device 1000. The motor 191 can generate a vibration prompt. The indicator 192 can be an indicator light, which can be used to indicate the charging status, the change of power, and can also be used to indicate messages, missed calls, notifications, etc. The SIM card interface 195 is used to connect a SIM card.

[0222] The sensor module 180 may include one or more sensors, which may be of the same type or of different types. Fig.10 The sensor module 180 shown is only an exemplary division method. There may be other division methods, which are not limited in this application.

[0223] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. When a touch operation is applied to the display screen 194, the electronic device detects the touch operation intensity according to the pressure sensor 180A. The electronic device can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions.

[0224] The gyro sensor 180B can be used to determine the motion posture of the electronic device. In some embodiments, the angular velocity of the electronic device around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for anti-shake shooting.

[0225] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device in all directions (generally three axes). When the electronic device is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0226] The distance sensor 180F is used to measure the distance. The electronic device can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0227] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device, which is different from the position of the display screen 194.

[0228] The air pressure sensor 180C is used to measure air pressure. The magnetic sensor 180D includes a Hall sensor. The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The electronic device uses a photodiode to detect infrared reflected light from nearby objects. The ambient light sensor 180L is used to sense the brightness of ambient light. The fingerprint sensor 180H is used to obtain fingerprints. The temperature sensor 180J is used to detect temperature. The bone conduction sensor 180M can obtain vibration signals.

[0229] Next, the software system of the electronic device 1000 will be described.

[0230] Exemplarily, the electronic device 1000 may be a mobile phone. The software system of the electronic device 1000 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software system of the electronic device 1000.

[0231] Fig.11 This is a block diagram of a software system of an electronic device 1000 provided in an embodiment of the present application. Fig.11 The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: application layer, application framework layer, Android runtime and system layer, kernel layer and hardware abstraction layer (HAL).

[0232] The application layer can include a series of application packages. Fig.11 As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0233] The application framework layer provides application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Fig.11 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, etc. The window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc. The content provider is used to store and obtain data and make these data accessible to applications. These data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc. The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build the display interface of the application, and the display interface can be composed of one or more views, such as a view for displaying a text message notification icon, a view for displaying text, and a view for displaying pictures. The phone manager is used to provide communication functions of the electronic device 1000, such as management of call status (including connecting, hanging up, etc.). The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, etc. The notification manager enables the application to display notification information in the status bar, which can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of an icon or scroll bar text, such as a notification of an application running in the background. The notification manager can also be a notification that appears on the screen in the form of a dialog window, such as a text message in the status bar, a reminder sound, an electronic device vibrating, an indicator light flashing, etc.

[0234] Android runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system. The core library consists of two parts: one is the function that the Java language needs to call, and the other is the Android core library. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0235] The system library can include multiple functional modules, such as: surface manager, media library, 3D graphics processing library (such as: OpenGL ES), 2D graphics engine (such as: SGL), etc. The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing, etc. The 2D graphics engine is a drawing engine for 2D drawing.

[0236] The Hardware Abstraction Layer (HAL) is an interface layer between the operating system kernel and the upper software, and its purpose is to abstract the hardware. The hardware abstraction layer is an abstract interface driven by the device kernel, which is used to implement the application programming interface that provides access to the underlying device to the higher-level Java API framework. HAL contains multiple library modules, such as camera, display, Bluetooth, audio, etc. Each library module implements an interface for a specific type of hardware component. When the system framework layer API requires access to the hardware of the portable device, the Android operating system will load the library module for the hardware component.

[0237] The kernel layer is the layer between hardware and software. The kernel layer contains at least device drivers such as camera driver, processor driver, display driver, audio driver, etc. The device driver is the interface between the I / O system and related hardware, used to drive the corresponding hardware devices.

[0238] It should be noted that the application provides Fig.11 The software structure diagram of the electronic device shown is only an example and does not limit the specific module division in different layers of the Android operating system. For details, please refer to the introduction of the Android operating system software structure in conventional technology.

[0239] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0240] The above are optional embodiments provided for the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the technical scope disclosed in the present application shall be included in the protection scope of the present application.

Claims

1. An image processing method, applied to an electronic device, characterized in that: The method comprises: Get the image; Processing the image based on a first neural network; Outputting an image processed by the first neural network, wherein the first neural network is obtained by performing redundant compression processing on a first compressible network layer included in the second neural network, wherein the first compressible network layer includes a plurality of first network layers having the same type and parameters; The first neural network is obtained by performing redundant compression processing on the first compressible network layer included in the second neural network, including: The first neural network is obtained by skipping the first first network layer among the multiple first network layers and training the second neural network when the similarity measure between the first output result and the second output result is less than or equal to a first threshold, wherein the first output result is obtained by inputting input data into the second neural network, the second output result is obtained by inputting the input data into a third neural network, and the third neural network is obtained by quantizing the second neural network after training.

2. The method according to claim 1, characterized in that The processing of the image based on the first neural network includes: Obtain image data according to the image, wherein the image data includes data of a plurality of pixels in the image; Inputting the image data into the first neural network; The image data is calculated by Z network layers in the first neural network, wherein the calculation result of the jth network layer among the Z network layers is input into the j+1th network layer and calculated by the j+1th network layer, 1≤j≤Z-1, Z>1 and j and Z are integers.

3. The method according to claim 1 or 2, characterized in that The first neural network is obtained by training the second neural network by skipping the first first network layer among the plurality of first network layers when the similarity measure between the first output result and the second output result is less than or equal to a first threshold, including: The first neural network is obtained by training the second neural network by skipping the first first network layer based on a first loss function value when the similarity measure between the first output result and the second output result is less than or equal to a first threshold, wherein the first loss function value includes the similarity measure between the first output result and the second output result.

4. The method according to claim 1, characterized in that The multiple first network layers are Y first network layers, Y>1 and Y is an integer, the first first network layer includes first identification information, the first identification information is used to indicate to delete the current first network layer, and the second neural network is obtained by skipping the first first network layer among the multiple first network layers to train the second neural network, including: the second neural network is obtained by skipping the first first network layer among the multiple first network layers based on the first identification information; The 2nd first network layer to the Yth first network layer among the Y first network layers all include second identification information, and the second identification information is used to indicate that the current network layer is deletable.

5. The method according to claim 1 or 2, characterized in that: The first neural network is obtained by performing redundancy compression processing on a first compressible network layer included in the second neural network, including: The first neural network is obtained by performing redundant compression processing on a first compressible network layer included in the second neural network based on pre-maintained compressible information, wherein the compressible information is used to indicate that i network layers have been deleted, and the first first network layer is the i-th network layer deleted from the i network layers, 1≤i≤Y+i-1 and i is an integer; The compressible information includes Y+i-1 sub-information, and each sub-information in the Y+i-1 sub-information is used to indicate the compressible state of Y+i-1 network layers; Among them, the first i-1 sub-information among the Y+i-1 sub-information are respectively used to indicate that the corresponding i-1 network layers have been deleted, and the order in which the i-1 network layers are deleted, and the i+1th to Y+i-1th sub-information among the Y+i-1 sub-information are respectively used to indicate that the corresponding Y-1 network layers have not been deleted.

6. The method according to claim 1 or 2, characterized in that: A difference between a first loss function value and a second loss function value is smaller than a second threshold, the first loss function is used to train the second neural network, and the second loss function value is used to train the first neural network.

7. The method according to claim 1 or 2, characterized in that: The first neural network is obtained by training the second neural network by skipping the first first network layer among the plurality of first network layers when the similarity measure between the first output result and the second output result is less than or equal to a first threshold, including: The first neural network is obtained by performing distillation training on the second neural network by skipping the first first network layer among the multiple first network layers when the similarity measure between the first output result and the second output result is less than or equal to a first threshold.

8. The method according to claim 7, characterized in that The distillation training is modular distillation training, wherein each of the X first convolution modules included in the fourth neural network is a teacher model, and each of the X second convolution modules included in the second neural network is a student model. The first neural network is obtained by updating the parameters of the second neural network according to the first output result of each first convolution module and the second output result of each second convolution module, wherein the first output result is obtained by inputting the input data to each first convolution module, and the second output result is obtained by inputting the input data to the convolution layers other than the first first network layer in each second convolution module. The X second convolution modules correspond to the X first convolution modules one by one, each first convolution module includes M convolution layers, and each second convolution module includes N convolution layers, X>1, M>N≥1, and X, M and N are all integers.

9. A model compression method, characterized in that: include: Inputting the input data into a second neural network to obtain a first output result, wherein the second neural network includes a first compressible network layer, and the first compressible network layer includes a plurality of first network layers with the same type and parameters; Performing post-training quantization processing on the second neural network to obtain a third neural network, wherein the parameters of the second neural network are all floating-point values, and the parameters of the third neural network model are all integer values; Inputting the input data into the third neural network to obtain a second output result; When the similarity measure between the first output result and the second output result is less than a first threshold, the first first network layer among the multiple first network layers is skipped to train the second neural network to obtain a first neural network.

10. The method according to claim 9, characterized in that The skipping the first first network layer among the plurality of first network layers to train the second neural network comprises: Based on a first loss function value, skipping the first first network layer to train the second neural network, wherein the first loss function value includes a similarity measure between the first output result and the second output result.

11. The method according to claim 9, characterized in that The multiple first network layers are Y first network layers, Y>1 and Y is an integer, the first first network layer includes first identification information, the first identification information is used to indicate that the current first network layer is to be deleted, and the second first network layer to the Yth first network layer of the Y first network layers all include second identification information, and the second identification information is used to indicate that the current network layer is deletable; The skipping the first first network layer among the Y first network layers to train the second neural network includes: According to the first identification information included in the first first network layer, the second neural network is trained by skipping the first first network layer.

12. The method according to claim 11, characterized in that The method further comprises: According to the similarity measure between the first output result and the second output result being less than the first threshold, the second identification information in the first first network layer is updated to the first identification information.

13. The method according to claim 12, characterized in that The method further comprises: Updating the pre-maintained compressible information, wherein the updated compressible information is used to indicate that i network layers have been deleted, and the first first network layer is the i-th network layer deleted from the i network layers; The compressible information includes Y+i-1 sub-information, and each sub-information in the Y+i-1 sub-information is used to indicate the compressible state of Y+i-1 network layers; wherein, The first i-1 sub-information among the Y+i-1 sub-information are respectively used to indicate that the corresponding i-1 network layers have been deleted, and the order in which the i-1 network layers are deleted. The i+1th to Y+i-1th sub-information among the Y+i-1 sub-information are respectively used to indicate that the corresponding Y-1 network layers have not been deleted, 1≤i≤Y+i-1 and i is an integer.

14. The method according to any one of claims 9 to 13, characterized in that The method further comprises: Saving a first loss function value used for training the second neural network; Obtaining a second loss function value for training the first neural network; Determine whether the difference between the first loss function value and the second loss function value is less than a second threshold.

15. The method according to claim 14, characterized in that The plurality of first network layers are Y first network layers, and when a difference between the first loss function value and the second loss function value is less than the second threshold, the method further includes: Inputting the input data into the first neural network to obtain a third output result, wherein the first neural network includes a second compressible network layer, the second compressible network layer includes Y-1 second network layers of the same type and parameters, and the Y-1 second network layers correspond one-to-one to the second first network layer to the Yth first network layer in the Y first network layers; Performing post-training quantization processing on the first neural network to obtain a fifth neural network, wherein the parameters of the first neural network are all floating-point values, and the parameters of the fifth neural network model are all integer values; Inputting the input data into the fifth neural network to obtain a fifth output result; When the similarity measure between the third output result and the fifth output result is less than the first threshold, skipping the first second network layer among the Y-1 second network layers to train the first neural network, so as to obtain a sixth neural network; Alternatively, when the similarity measure between the third output result and the fifth output result is greater than or equal to the first threshold, the first neural network is used as the final neural network, and the model compression method is stopped.

16. The method according to claim 15, characterized in that The method further comprises: Updating the pre-maintained compressible information, the updated compressible information is used to indicate that i+1 network layers have been deleted, and the first second network layer is the i+1th network layer deleted from the i+1 network layers, 1≤i≤Y+i-1 and i is an integer; The compressible information includes Y+i-1 sub-information, and each sub-information in the Y+i-1 sub-information is used to indicate the compressible state of Y+i-1 network layers; Among them, the first i sub-information among the Y+i-1 sub-information are respectively used to indicate that the corresponding i network layers have been deleted and the order in which the i network layers are deleted, and the i+2th to Y+i-1th sub-information among the Y+i-1 sub-information are respectively used to indicate that the corresponding Y-2 network layers have not been deleted.

17. The method according to claim 14, characterized in that When the difference between the first loss function value and the second loss function value is greater than or equal to the second threshold, the method further includes: Update the first identification information in the first first network layer to the second identification information, wherein the first identification information is used to indicate that the current first network layer is deleted, and the second identification information is used to indicate that the current network layer is deletable; The second neural network is used as the final neural network, and the model compression method is stopped.

18. The method according to claim 17, characterized in that Updating the first identification information in the first first network layer to the second identification information includes: Determining, according to the pre-maintained compressible information, that the last deleted network layer among the network layers that have been deleted in the model compression method is the first first network layer, the compressible information is used to indicate that i network layers have been deleted, and the first first network layer is the i-th network layer deleted among the i network layers; Update the first identification information in the i-th first network layer to the second identification information, where i≥1 and i is an integer.

19. The method according to claim 18, characterized in that In the case where i>1, the method further comprises: The compressible information is updated, and the updated compressible information is used to indicate that the corresponding i-1 network layers have been deleted, and the order in which the i-1 network layers are deleted.

20. The method of claim 9, wherein: The second neural network is obtained through distillation training, and the first neural network is obtained by performing distillation training on the second neural network.

21. The method of claim 20, wherein: The skipping of the first first network layer among the plurality of first network layers to train the second neural network to obtain the first neural network includes: Each of the X first convolution modules included in the fourth neural network is used as a teacher model, and each of the X second convolution modules included in the second neural network is used as a student model, the input data is input for each of the first convolution modules, and the input data is input for the convolution layers other than the first first network layer in each of the second convolution modules, the X second convolution modules correspond to the X first convolution modules one by one, each of the first convolution modules includes M convolution layers, and each of the second convolution modules includes N convolution layers, X>1, M>N≥1, and X, M and N are all integers; According to the first output result of each of the first convolution modules and the second output result of each of the second convolution modules, the parameters of the second neural network are updated to obtain the first neural network.

22. An electronic device comprising a memory and one or more processors, characterized in that: The memory is used to store a computer program; the processor is used to call the computer program so that the electronic device executes the method described in any one of claims 1 to 8, or executes the method described in any one of claims 9 to 21.

23. A computer storage medium, characterized in that include: Computer instructions; when the computer instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 8, or executes the method according to any one of claims 9 to 21.

Images