Image processing method, model compression method and related device

Abstract

The application provides an image processing method, a model compression method and related equipment. The image processing method is based on a first neural network to process an image, the first neural network is obtained by performing redundancy compression processing on a neural network with more complex structure, and can save the computing power of a mobile terminal and reduce the power consumption of the mobile terminal. Optionally, the redundancy compression processing is combined with quantization and distillation training to greatly optimize the performance of the first neural network and further reduce the power consumption of the mobile terminal. The model compression method is used to determine whether there is redundancy according to the output results of the current neural network before and after quantization, and if there is redundancy, the redundancy is deleted, pruning is realized without manual intervention, the operation efficiency of a GPU server and the memory optimization degree can be improved, and the labor cost can be saved. Optionally, whether the accuracy of the neural network after deleting the redundancy decreases significantly is evaluated, and if the accuracy decreases significantly, the network layer that is deleted last time is restored to balance the accuracy and performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to an image processing method, a model compression method and related equipment. BACKGROUND

[0002] With the development of artificial intelligence (AI) and neural network (NN) technologies, neural network models are widely used in AI applications in the fields of image processing, speech recognition and natural language processing, etc., so that AI applications using neural network models are increasing year by year.

[0003] The parameters of a neural network model are usually in the order of millions, ten millions or hundreds of millions, so the storage and computing power requirements of a terminal device running an AI application using a neural network model are relatively high, which limits the use of neural network models on terminal devices. In order to reduce the operation cost and power consumption, an artificial intelligence model can be compressed, and the compressed convolutional neural network model can be deployed to a mobile terminal. For example, when a model for performing pixel-level image tasks such as optimizing image resolution needs to be deployed on a terminal device such as a mobile phone, a device with a graphics processing unit (GPU) server can be used to compress the model, and the compressed model can be deployed to a mobile terminal. SUMMARY

[0004] The present application provides an image processing method, a model compression method and related equipment. Among them, the image processing method can save the computing power of the mobile terminal and reduce the power consumption of the mobile terminal; the model compression method can significantly improve the operation efficiency and memory optimization degree of the GPU server, and save the labor cost.

[0005] It should be noted that the model compression method provided by the present application is a general compression scheme for artificial intelligence models. The model compression method is highly adaptable, and for artificial intelligence models of different application types, adaptive model redundancy compression can be realized, greatly reducing the power consumption generated by the calculation of the model on the mobile terminal. The image processing method provided in the present application can be understood as an example of an application scenario applicable to the model compression method provided by the present application, and does not limit the application scenarios applicable to the model compression method.

[0006] In a first aspect, the present application provides an image processing method applied to an electronic device, the method comprising:

[0007] obtaining an image; processing the image based on a first neural network; and outputting the image processed by the first neural network, the first neural network being obtained by performing a redundant compression on a first compressible network layer included in a second neural network, the first compressible network layer including a plurality of first network layers of the same type and parameters, wherein the first neural network is obtained by performing the redundant compression on the first compressible network layer included in the second neural network, including: the first neural network is obtained by training the second neural network by skipping a first one of the plurality of first network layers, in a case that a similarity measure of a first output result and a second output result is less than or equal to a first threshold, wherein the first output result is obtained by inputting input data into the second neural network, and the second output result is obtained by inputting the input data into a third neural network, the third neural network being obtained by performing a quantization on the second neural network after training the second neural network.

[0008] In the embodiments of the present application, the first neural network is obtained by redundant compression, so that the first neural network has fewer redundant network layers than the neural network before redundant compression, has a simpler network structure, and has improved performance, thereby saving the computing power of the mobile terminal and reducing the power consumption of the mobile terminal.

[0009] It can be understood that the training of the second neural network by skipping the first one of the plurality of first network layers means that the first one of the plurality of first network layers is deleted, and then the remaining network layers in the second neural network are trained. In other words, the first one of the plurality of first network layers is not used for training the second neural network.

[0010] For example, the parameters of the second neural network are all floating-point values, and the parameters of the third neural network model are all integer values. It can be understood that the training and quantization can improve the operation speed and reduce the operation cost.

[0011] In combination with the first aspect, in a possible embodiment, the processing of the image based on the first neural network includes:

[0012] obtaining image data from the image, the image data including data of a plurality of pixels in the image; inputting the image data into the first neural network; and calculating the image data by Z network layers in the first neural network, wherein a calculation result of a jth network layer in the Z network layers is input into a (j+1)th network layer and calculated by the (j+1)th network layer, 1≤j≤Z-1, Z>1, and j and Z are integers.

[0013] The electronic device obtains image data (referred to as image data #1 for convenience) from the image. The image data #1 includes data of a plurality of pixels in the image. For example, the image data #1 can be in a RAW format or a red-green-blue (RGB) format. The image data #1 is input into the first neural network, and the image data is calculated by Z network layers in the first neural network. Specifically, the image data #1 is parsed and processed by an input layer, and a feature map for processing by a network layer (including a convolution layer, a pooling layer, an activation layer, etc.) is output. The first network layer of the Z network layers calculates the feature map, and the calculation result output by the first network layer is input into the second network layer for calculation. The jth network layer calculates the feature map, and the calculation result output by the jth network layer is input into the (j+1)th network layer and calculated by the (j+1)th network layer, where 1≤j≤Z-1, Z>1, and j and Z are integers. The calculation result output by the Zth network layer is input into an output layer, and the calculation result is parsed and processed by the output layer. An image processed by the first neural network (referred to as image data #2 for convenience) is output. In one possible implementation, the image processed by the first neural network can be image data in a RAW format or an RGB format.

[0014] For example, in a mobile game scenario, the resolution of an image is optimized by the first neural network. A plurality of frames of pictures obtained by a mobile terminal are separated into text data and image data #1. The image data #1 is input into the neural network model, and the image data #1 is calculated by the first neural network in the manner described in the foregoing example. Image data #2 with optimized resolution is output.

[0015] In combination with the first aspect, in one possible implementation, the first neural network is obtained by training the second neural network by skipping the first network layer of the plurality of first network layers when the similarity measure of the first output result and the second output result is less than or equal to the first threshold value, and the first neural network includes the following.

[0016] The first neural network is obtained by training the second neural network by skipping the first network layer based on the first loss function value when the similarity measure of the first output result and the second output result is less than or equal to the first threshold value, and the first loss function value includes the similarity measure of the first output result and the second output result.

[0017] The scheme can narrow the difference between the output results of the second neural network before and after quantization and the third neural network, thereby accelerating the convergence in the process of training the second neural network, improving the accuracy of the first neural network, and improving the effect of image processing.

[0018] In combination with the first aspect, in a possible implementation, the plurality of first network layers are Y first network layers, Y>1 and Y is an integer, the first first network layer includes first identification information, the first identification information is used to indicate that the current first network layer is deleted, the second first network layer to the Yth first network layer in the Y first network layers all include second identification information, and the second identification information is used to indicate that the current network layer is deletable.

[0019] For example, the first identification information and the second identification information are implemented by means of a dropout operation. Specifically, a dropout layer is configured in each of the plurality of first network layers to control the training parameters, or to mark whether the parameters of the layer are to be updated, or to mark whether the layer is to be trained.

[0020] When the parameter of the dropout layer is set to 1, the dropout layer is the second identification information, indicating that the current network layer with the configured dropout layer is deletable. When the parameter of the dropout layer is set to 0, the dropout layer is the first identification information, indicating that the corresponding network layer parameter will not be updated, or that the network will skip the layer for training, or that the network layer will not affect the final training result of the network, or that the network layer is deleted from the network. Therefore, in the above scheme, the second first network layer to the Yth first network layer are all configured with the dropout layer with the parameter set to 1. The first first network layer is configured with the dropout layer with the parameter set to 0.

[0021] In combination with the first aspect, in a possible implementation, the first neural network is obtained by performing redundant compression processing on the first compressible network layer included in the second neural network, and the method comprises:

[0022] The first neural network is obtained by performing redundant compression processing on a first compressible network layer included in the second neural network based on pre-maintained compressible information, where the compressible information is used to indicate that i network layers have been deleted, and the first first network layer is the i-th network layer deleted among the i network layers, 1≤i≤Y+i-1 and i is an integer; where the compressible information includes Y+i-1 sub-information, each of the Y+i-1 sub-information is used to indicate the compressible state of Y+i-1 network layers; where the first i-1 sub-information in the Y+i-1 sub-information is used to indicate that the corresponding i-1 network layers have been deleted, and the order in which the i-1 network layers are deleted, and the i+1 sub-information to the Y+i-1 sub-information in the Y+i-1 sub-information is used to indicate that the corresponding Y-1 network layers are not deleted.

[0023] The above scheme synchronously updates the compressible information according to the deletion of the network layer, so as to record the deletion of the network layer in all processes of the model compression method in real time, and improve the accuracy of the model redundant compression. Thus, it is helpful to improve the accuracy of the first neural network, and further improve the effect of image processing.

[0024] In combination with the first aspect, in a possible implementation, a difference between the first loss function value and the second loss function value is less than a second threshold value, the first loss function is used to train the second neural network, and the second loss function value is used to train the first neural network.

[0025] The above scheme can determine whether deleting the first first network layer makes the accuracy of the first neural network significantly worse than the accuracy of the second neural network by judging whether the difference between the first loss function value and the second loss function value is less than the second threshold value. If the difference between the first loss function value and the second loss function value is less than the second threshold value, it indicates that the accuracy of the first neural network is not significantly worse than the accuracy of the second neural network. Thus, it can reduce the case of significantly losing accuracy due to blindly deleting redundant network layers, obtain the balance between accuracy and performance, and further ensure the effect of image processing.

[0026] In combination with the first aspect, in a possible implementation, the first neural network is obtained by training the second neural network by skipping the first first network layer in the plurality of first network layers under the condition that the similarity measurement of the first output result and the second output result is less than or equal to a first threshold value, and the first neural network includes the following steps:

[0027] The first neural network is obtained by distilling training the second neural network by skipping the first first network layer in the plurality of first network layers under the condition that the similarity measurement of the first output result and the second output result is less than or equal to a first threshold value.

[0028] The scheme integrates model quantization, model distillation and model redundancy compression together, thereby combining the three ways of model compression technology, realizing model compression in multiple aspects, and further optimizing the performance of the model, and further saving the computing power of the mobile terminal and reducing the power consumption of the mobile terminal.

[0029] In combination with the first aspect, in a possible implementation, the distillation training is modular distillation training, where each of the X first convolution modules included in the fourth neural network is a teacher model, each of the X second convolution modules included in the second neural network is a student model, and the first neural network is obtained by updating the parameters of the second neural network according to the first output result of each first convolution module and the second output result of each second convolution module, where the first output result is obtained by inputting the input data into each first convolution module, the second output result is obtained by inputting the input data into the convolution layers of each second convolution module except the first network layer, the X second convolution modules correspond to the X first convolution modules one by one, each first convolution module includes M convolution layers, each second convolution module includes N convolution layers, X>1, M>N≥1, and X, M and N are integers.

[0030] In the scheme, compared with the difference in the complexity of the network structure between the second neural network and the fourth neural network in the traditional distillation training architecture, the difference in the complexity of the network structure between the first convolution module and the corresponding second convolution module is much smaller. Therefore, compared with the traditional distillation training architecture in which the output result of the second neural network is used to fit the output result of the fourth neural network, the output result of the second convolution module can better fit the output result of the first convolution module, thereby improving the accuracy of the output result of the second convolution module and the accuracy of the final output result of the first neural network obtained by updating the parameters of the second neural network. Therefore, the image processing effect can be further improved.

[0031] In a second aspect, the present application provides a model compression method, comprising:

[0032] inputting the input data into the second neural network to obtain a first output result, where the second neural network includes a first compressible network layer, and the first compressible network layer includes a plurality of first network layers with consistent types and parameters; performing quantization processing on the second neural network after training to obtain a third neural network, where the parameters of the second neural network are floating-point values, and the parameters of the third neural network model are integer values; inputting the input data into the third neural network to obtain a second output result; and in a case where the similarity measure of the first output result and the second output result is less than a first threshold, skipping the first network layer in the plurality of first network layers to train the second neural network to obtain a first neural network.

[0033] The above scheme determines whether there is redundancy for the output results of the current neural network before and after quantization, and deletes one network layer in the case of redundancy, realizes adaptive and accurate pruning, and thus realizes model redundancy compression. The above scheme does not need to compare the output results of the current neural network and another neural network to determine whether the current neural network has redundancy, compared with the traditional way of analyzing model redundancy, which can improve the efficiency of analyzing model redundancy and deleting redundant network layers, and thus improve the efficiency of model compression. Compared with the traditional structured pruning, the above scheme can be completely free of human intervention and can save labor costs. Therefore, by deleting redundant network structures, the model compression can improve the operation efficiency of the GPU server and the degree of memory optimization. In addition, through the post-training quantization process, the operation speed and the operation cost can be reduced, and the operation efficiency of the GPU server and the degree of memory optimization can be improved.

[0034] It can be understood that the first first network layer in the plurality of first network layers is skipped to train the second neural network, that is, after the first first network layer is deleted, the remaining other network layers in the second neural network are trained. In other words, the first first network layer is not used for training the second neural network layer.

[0035] In combination with the second aspect, in a possible embodiment, skipping the first first network layer in the plurality of first network layers to train the second neural network comprises:

[0036] Based on the first loss function value, the first first network layer is skipped to train the second neural network, wherein the first loss function value comprises a similarity measure of the first output result and the second output result.

[0037] The above scheme can reduce the difference between the output results of the second neural network before and after quantization and the third neural network by applying the similarity measure of the output results of the second neural network before and after quantization to the loss function value used to train the second neural network and performing back propagation, thereby accelerating the convergence in the process of training the second neural network.

[0038] It can be understood that the first output result is output by the second neural network, and the second output result is output by the third neural network, and this process is differentiable, so the similarity measure of the two output results can be used for back propagation. In other words, taking the difference between the similarity measure of the two output results as an example, the difference between the first output result and the second data result can be used as a loss function value for back propagation.

[0039] With reference to the second aspect, in a possible implementation, the first network layers are Y first network layers, Y > 1 and Y is an integer, the first first network layer includes first identification information, the first identification information is used to indicate that the current first network layer is deleted, the second first network layer to the Yth first network layer in the Y first network layers all include second identification information, the second identification information is used to indicate that the current network layer is deletable; and the training of the second neural network by skipping the first first network layer includes: training the second neural network by skipping the first first network layer according to the first identification information included in the first first network layer.

[0040] The above scheme adds identification information (including first identification information or second identification information) to the Y first network layers respectively, so as to realize skipping the training of the first network layer including the first identification information and not skipping the training of the first network layer including the second identification information, which can enable the second neural network layer to adaptively attempt to delete each network layer in the compressible network layer, that is, the redundant number of layers can be automatically reduced during the training process, which is more efficient than the way of manually modifying the network structure in the traditional model redundancy compression process.

[0041] For example, the first identification information and the second identification information are realized by means of a dropout operation. Specifically, a dropout layer is configured in each of the first network layers to control the training parameters, or to mark whether the parameters of the layer are to be updated, or to mark whether the layer is to be trained.

[0042] When the parameter of the dropout layer is set to 1, the dropout layer is the second identification information, indicating that the current network layer with the configured dropout layer is deletable. The second first network layer to the Yth first network layer are all configured with the dropout layer with the parameter set to 1. When the parameter of the dropout layer is set to 0, the dropout layer is the first identification information, indicating that the corresponding network layer parameters will not be updated, or that the network will skip the layer for training, or that the network layer will not affect the final training result of the network, or that the network layer is deleted from the network. The first first network layer is configured with the dropout layer with the parameter set to 0.

[0043] With reference to the second aspect, in a possible implementation, the method further includes:

[0044] According to the similarity measure of the first output result and the second output result being less than the first threshold, updating the second identification information in the first first network layer to the first identification information.

[0045] For example, the transformation between the first identification information and the second identification information is implemented by means of a dropout operation. The parameter of the dropout layer configured in the first first network layer is changed from 1 to 0, that is, the transformation from the second identification information to the first identification information is implemented.

[0046] The above scheme uses the dropout operation to adaptively attempt to delete each network layer in the compressible network layer, that is, the redundant number of layers can be automatically reduced in the training process, which is more efficient than the way of manually modifying the network structure in the traditional model redundancy compression process.

[0047] With reference to the second aspect, in a possible implementation, the method further includes: updating the pre-maintained compressible information, and the updated compressible information is used to indicate that i network layers have been deleted, and the first first network layer is the i-th deleted network layer in the i network layers, 1≤i≤Y+i-1 and i is an integer; wherein the compressible information includes Y+i-1 pieces of sub-information, each piece of sub-information in the Y+i-1 pieces of sub-information is used to indicate the compressible state of Y+i-1 network layers; wherein the first i-1 pieces of sub-information in the Y+i-1 pieces of sub-information are used to indicate that the corresponding i-1 network layers have been deleted and the order in which the i-1 network layers are deleted, and the i+1-th to Y+i-1-th pieces of sub-information in the Y+i-1 pieces of sub-information are used to indicate that the corresponding Y-1 network layers are not deleted.

[0048] The above scheme updates the compressible information according to the deletion of the network layers, so as to record the deletion of the network layers in all processes of the model compression method in real time, and improve the accuracy of the model redundancy compression.

[0049] With reference to the second aspect, in a possible implementation, the method further includes: saving a first loss function value used for training the second neural network; obtaining a second loss function value used for training the first neural network; and determining whether a difference between the first loss function value and the second loss function value is less than a second threshold value.

[0050] The above scheme determines whether the deletion of the first first network layer causes the accuracy of the first neural network to be significantly worse than the accuracy of the second neural network by determining whether the difference between the first loss function value and the second loss function value is less than the second threshold value. Therefore, the case that the accuracy is significantly lost due to the deletion of the redundant network layer can be reduced, and the balance between the accuracy and the performance is obtained. In addition, the above scheme adaptively attempts to delete each network layer in the compressible network layer, and analyzes whether the accuracy is significantly lost after the deletion, so that the redundant number of layers can be automatically reduced in the training process, and the judgment on the reduction result can be automatically performed, which is more efficient than the way of manually modifying the network structure in the traditional model redundancy compression process.

[0051] With reference to the second aspect, in a possible implementation, the plurality of first network layers is Y first network layers, and in a case where the difference between the first loss function value and the second loss function value is less than a second threshold, the method further includes:

[0052] inputting the input data into the first neural network to obtain a third output result, wherein the first neural network includes a second compressible network layer, the second compressible network layer includes Y-1 second network layers of the same type and parameters, and the Y-1 second network layers correspond to the 2nd to Yth first network layers in the Y first network layers one by one; performing a training and quantization process on the first neural network to obtain a fifth neural network, parameters of the first neural network are all floating-point values, and parameters of the fifth neural network model are all integer values; inputting the input data into the fifth neural network to obtain a fifth output result; in a case where a similarity measure of the third output result and the fifth output result is less than a first threshold, training the first neural network by skipping a first second network layer in the Y-1 second network layers to obtain a sixth neural network; or in a case where the similarity measure of the third output result and the fifth output result is greater than or equal to the first threshold, taking the first neural network as a final obtained neural network, and stopping execution of the model compression method.

[0053] The above scheme automatically continues to perform model redundancy compression on the first neural network in a case where the accuracy of the first neural network does not significantly deteriorate compared to the accuracy of the second neural network, can be completely free of manual intervention compared to a traditional structured pruning, and can save labor costs.

[0054] With reference to the second aspect, in a possible implementation, the method further includes:

[0055] updating the pre-maintained compressible information, and the updated compressible information is used to indicate that i+1 network layers have been deleted, and the first second network layer is the i+1th network layer that has been deleted in the i+1 network layers, 1≤i≤Y+i-1 and i is an integer; wherein the compressible information includes Y+i-1 pieces of sub-information, each piece of sub-information in the Y+i-1 pieces of sub-information is used to indicate a compressible state of Y+i-1 network layers; wherein the first i pieces of sub-information in the Y+i-1 pieces of sub-information are used to indicate that the corresponding i network layers have been deleted and the order in which the i network layers are deleted, and the i+2th to Y+i-1th pieces of sub-information in the Y+i-1 pieces of sub-information are used to indicate that the corresponding Y-2 network layers have not been deleted.

[0056] The above scheme updates the compressible information according to the deletion of the network layers, so as to record the deletion of the network layers in all processes of the model compression method in real time, and improve the accuracy of model redundancy compression.

[0057] With reference to the second aspect, in a possible implementation, when the difference between the first loss function value and the second loss function value is greater than or equal to the second threshold, the method further includes:

[0058] updating the first identification information in the first network layer to second identification information, wherein the first identification information is used to indicate that the current first network layer is deleted, and the second identification information is used to indicate that the current network layer is deletable; taking the second neural network as a final obtained neural network, and stopping the execution of the model compression method.

[0059] The above scheme, in the case that the precision of the first neural network is significantly deteriorated compared with the precision of the second neural network, restores the last deleted network layer, can reduce the case that the precision is significantly deteriorated due to the deletion of redundant network layers, and balances the precision and performance.

[0060] With reference to the second aspect, in a possible implementation, updating the first identification information in the first network layer to second identification information includes: determining, according to pre-maintained compressible information, that the last deleted network layer in the deleted network layers in the model compression method is the first network layer, the compressible information is used to indicate that i network layers have been deleted, and the first network layer is the i-th deleted network layer in the i network layers; and updating the first identification information in the i-th network layer to the second identification information, i≥1 and i is an integer.

[0061] The above scheme, through the pre-maintained compressible information, can quickly find the last deleted network layer, and improves the efficiency of redundant compression.

[0062] With reference to the second aspect, in a possible implementation, when i>1, the method further includes:

[0063] updating the compressible information, and the updated compressible information is used to indicate that the corresponding i-1 network layers have been deleted, and the order in which the i-1 network layers are deleted.

[0064] The above scheme, according to the deletion of the network layers, synchronously updates the compressible information, so as to record the deletion of the network layers in all processes of the model compression method in real time, and improves the accuracy of model redundant compression.

[0065] With reference to the second aspect, in a possible implementation, the second neural network is obtained through distillation training, and the first neural network is obtained through distillation training on the second neural network.

[0066] The above scheme integrates model quantization, model distillation, and model redundant compression together, thereby combining the three ways of model compression technology, realizing model compression in multiple aspects, and further realizing great optimization of model performance.

[0067] With reference to the second aspect, in a possible implementation, the training of the first neural network by skipping the first first network layer in the plurality of first network layers comprises:

[0068] each first convolutional module included in the fourth neural network is taken as a teacher model, each second convolutional module included in the second neural network is taken as a student model, input data is input to each first convolutional module, and input data is input to convolutional layers other than the first network layer in each second convolutional module, the X second convolutional modules correspond to the X first convolutional modules one by one, each first convolutional module includes M convolutional layers, each second convolutional module includes N convolutional layers, X>1, M>N≥1, and X, M, and N are integers; and parameters of the second neural network are updated according to first output results of each first convolutional module and second output results of each second convolutional module to obtain the first neural network.

[0069] In the above solution, on the one hand, the second neural network and the fourth neural network are divided into a plurality of convolutional modules respectively, the calculation complexity of each second convolutional module is much lower than the calculation complexity of the entire second neural network, and the calculation complexity of each first convolutional module is much lower than the calculation complexity of the entire fourth neural network, thereby saving training time compared with the commonly used distillation training; on the other hand, the calculations of the second convolutional modules can be independent of each other, so that the second convolutional modules can be calculated in parallel, and the first convolutional modules are similar, and compared with the commonly used distillation training architecture, the calculations between the convolutional layers of the teacher network and the super-resolution network are associated with each other, thereby saving time. Therefore, the training efficiency can be significantly improved, thereby significantly improving the operation efficiency of the GPU server and the memory optimization degree.

[0070] In a third aspect, an electronic device is provided, which includes one or more processors and one or more memories; the one or more memories are coupled to the one or more processors, and the one or more memories are configured to store computer program codes, the computer program codes including computer instructions, which, when executed by the one or more processors, cause the electronic device to perform the method described in the first aspect or the second aspect or any possible implementation manner of the first aspect or the second aspect.

[0071] In a fourth aspect, an embodiment of the present application provides a chip system applied to an electronic device, the chip system including one or more processors, the processor being configured to invoke computer instructions to cause the electronic device to perform the method described in the first aspect or the second aspect or any possible implementation manner of the first aspect or the second aspect.

[0072] In a fifth aspect, a computer-readable storage medium is provided, and the computer-readable storage medium stores instructions which, when executed on a computer, cause the computer to perform the optimization method of the image processing model of the first aspect or the second aspect.

[0073] In a sixth aspect, a computer program product is provided, and the computer program product contains instructions which, when executed on a computer, cause the computer to perform the optimization method of the image processing model of the first aspect or the second aspect.

[0074] The technical effects obtained by the third aspect, the fourth aspect, the fifth aspect and the sixth aspect are similar to the technical effects obtained by the corresponding technical means in the first aspect or the second aspect, and will not be described here again. BRIEF DESCRIPTION OF DRAWINGS

[0075] Figure 1 A schematic diagram of an example of model redundancy provided by an embodiment of the present application;

[0076] Figure 2 A schematic diagram of an example of a way of analyzing model redundancy provided by an embodiment of the present application;

[0077] Figure 3 A schematic diagram of an example of a traditional distillation training architecture;

[0078] Figure 4A And Figure 4B A schematic flowchart of a model compression method provided by an embodiment of the present application;

[0079] Figure 5 A schematic diagram of an example of a model compression method provided by an embodiment of the present application;

[0080] Figure 6A A schematic diagram of an example of a second neural network provided by an embodiment of the present application;

[0081] Figure 6B A schematic diagram of an example of compressible information corresponding to the second neural network provided by an embodiment of the present application;

[0082] Figure 7A A schematic diagram of another example of a second neural network provided by an embodiment of the present application;

[0083] Figure 7B A schematic diagram of another example of compressible information corresponding to the second neural network provided by an embodiment of the present application;

[0084] Figure 8 A schematic diagram of an example of a distillation training architecture provided by an embodiment of the present application;

[0085] Figure 9A schematic diagram of an example of the image processing method provided by the embodiments of the present application;

[0086] Figure 10 A schematic diagram of a hardware structure of an electronic device 1000 provided by the embodiments of the present application;

[0087] Figure 11 A block diagram of a software system of an electronic device 1000 provided by the embodiments of the present application. DETAILED DESCRIPTION

[0088] To make the objectives, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0089] It should be understood that the "multiple" mentioned in the present application refers to two or more than two. In the description of the present application, unless otherwise specified, " / " represents the meaning of or, for example, A / B can represent A or B; "and / or" in the present application is only a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the three cases of A alone, A and B together, and B alone. In addition, in order to clearly describe the technical solutions of the present application, the same items or similar items with basically the same functions and roles are distinguished by using "first", "second", etc. The skilled in the art can understand that "first", "second", etc. do not limit the quantity and execution order, and "first", "second", etc. also do not limit the difference.

[0090] Model redundancy is a problem that often exists in the design of a neural network model, and model redundancy can reduce the computing efficiency of a terminal device and cause unnecessary power consumption overhead of the terminal device. The traditional model compression for model redundancy obtains a final compressed neural network model by manually modifying the network structure of the neural network model and trying to modify the network structure multiple times. In order to more clearly illustrate the embodiments of the present application, before introducing the embodiments of the present application, the related content of model redundancy, the concept of compressible module and several model compression technologies involved in the present application will be described.

[0091] 1. Model redundancy.

[0092] Figure 1 A schematic diagram of an example of model redundancy provided by the embodiments of the present application. Given two neural network models, model #1 f1(·) and model #2 f2(·), for the same input data I, if the difference #1 between the output results O1, O2 is less than a threshold #1, and o(f1(·))>o(f2(·)), it is said that there is redundancy in model #1 f1(·). Wherein, o(f1(·)) is the complexity of f1(·), and o(f2(·)) is the complexity of f2(·).

[0093] Because the terminal device has higher requirements on the performance and power consumption of the neural network model, the analysis and removal of model redundancy is of great significance to the terminal device. There are various ways to analyze model redundancy, including the way of calculating according to the flow shown in Figure 1 The analysis method of model redundancy provided by the embodiments of the present application.

[0094] 2. The analysis method of model redundancy provided by the present application.

[0095] By comparing the output results of the model before and after quantization, it is determined whether the model has redundancy.

[0096] Figure 2 The schematic diagram of an example of the analysis method of model redundancy provided by the embodiments of the present application. As shown in Figure 2 , first, an input data is given to obtain the output result #1 of the model #3 which has not been quantized. The model #3 is quantized by a quantization model to obtain the model #3'. The same input data is input into the model #3' to obtain a new output result, i.e., the output result #2. By comparing the two output results, if the similarity measure of the output result #1 and the output result #2 is less than a threshold value #2, it means that quantization (quantization belongs to one of the model compression techniques) does not affect the accuracy of the model itself, that is, the model still has compressible space, so the model has redundancy. When the similarity measure between the two output results is greater than or equal to the threshold value #2, that is, a small degree of compression in the parameter layer can greatly affect the result of the model itself, so it means that the model has no redundancy and cannot be compressed from the model structure level.

[0097] It can be understood that the similarity measure of the two output results in the present application is a numerical value for measuring the closeness of the two output results, and the similarity measure involved in the present application is a parameter value. Illustratively, the similarity measure between the two output results can be obtained by calculating the difference between the two output results, or it can also be obtained by other ways of calculating the similarity measure of the neural network, which is not limited in the present application.

[0098] 3. Compressible network layer.

[0099] In the present application, the repeated calculation structure unit with the same network structure in the neural network is referred to as a compressible network layer; or the network layer with consistent type and parameter setting is referred to as a compressible network layer. The type of network layer includes convolution layer, pooling layer, activation layer, etc., for example, the compressible network layer includes the convolution layer with consistent parameter setting. It should be noted that the compressible network layer does not include the network layer for processing input or output.

[0100] 4. The model compression technology involved in the present application

[0101] (1) Quantization.

[0102] Quantization can convert the floating-point numbers stored in the model layer into fixed-point numbers, specifically, convert the floating-point (float) numerical values into bit integer (int) numerical values, thereby improving the operation speed and reducing the operation cost. For example, the numerical values are converted from floating-point numerical values such as single-precision floating-point numbers (float), double-precision floating-point numbers (double) to integer numerical values such as 16-bit integers (int16), 8-bit integers (int8). A 32-bit floating-point number (float32) occupies 4 bytes, while an 8-bit integer (int8) occupies at most 1 byte, so the advantage of quantization is not only that the integer numerical values are several times faster than the floating-point numerical values in memory reading, but also that fixed-point operation is easier to realize hardware acceleration than floating-point operation.

[0103] However, although quantization can improve the performance of the model, it will cause the loss of the accuracy of the model. For example, converting a 32-bit floating-point number (float32) to an 8-bit integer (int8), because the numerical range that a 32-bit floating-point number (float32) can maintain is more than that of an 8-bit integer (int8), there will be a large number of numerical values that cannot be represented by an 8-bit integer (int8), and can only be rounded to an 8-bit integer (int8) value.

[0104] Quantization includes post-training quantization (PTQ) and quantization-aware training (QAT). PTQ is an offline quantization form separate from actual training. QAT combines the training of the model itself with the quantization process, which is an online quantization form.

[0105] (2) Unstructured pruning.

[0106] Pruning refers to pruning a neural network model, which can be roughly divided into structured pruning and unstructured pruning. Among them, structured pruning changes the structure of the neural network model by pruning the structure of the neural network model to reduce the parameter quantity of the model to improve the inference speed of the neural network model. Assuming that a neural network model (such as a convolutional neural network model) contains 5 convolutional layers, which are [L1, L2, L3, L4, L5] respectively. Among them, L2 contains 8 convolutional kernels with a shape of 3*3, so each of the 8 convolutional kernels in L2 has 3*3=9 weight parameters (i.e., can be regarded as a 3*3 matrix), and the weights of L2 can be regarded as an 8*3*3 matrix. Structured pruning can directly prune a convolutional layer of a neural network model, such as pruning L4, and the pruned neural network model only has 4 convolutional layers, such as [L1, L2, L3, L5]. Structured pruning can also prune the convolutional kernels of one or more convolutional layers of a neural network model, such as pruning 8 convolutional kernels in L2 to 5 convolutional layers, and the pruned neural network model L2 layer is left with 5 convolutional kernels with a shape of 3*3. Structured pruning can also prune the shape of the convolutional kernels of one or more convolutional layers in the neural network model, such as pruning the shape of the 8 convolutional kernels in L2 from 3*3 to 2*2, and the pruned neural network model L2 layer has 8 convolutional kernels, each of which has 2*2=4 weight parameters (i.e., can be regarded as a 2*2 matrix). In summary, the above three structured pruning methods will change the structure of the neural network model.

[0107] (3) Distillation training technology.

[0108] The distillation training involved in this application refers to pre-training a teacher network, and then using the teacher network as the teacher model and the super-resolution network as the student model during the training of the super-resolution network, allowing the teacher network to guide the training of the super-resolution network. Specifically, the teacher network has a significantly larger number of layers and parameters than the super-resolution network; in other words, the teacher network's network structure is significantly more complex than the super-resolution network's. Specifically, during the training of the super-resolution network, instead of fitting the super-resolution network's output to the target data, it fits the teacher network's output when given the same input data. That is, the teacher network's output is used as a supervisory signal for the super-resolution network's output, supervising its training. The purpose of distillation training is to improve the super-resolution network's fitting ability, i.e., its learning ability. Because the super-resolution network has relatively fewer parameters, its learning ability is relatively weak, and directly training the super-resolution network makes it difficult to converge to the same level as the teacher network; that is, the error between the super-resolution network's output and the actual value is difficult to reduce to the same level as the error between the teacher network's output and the actual value. Therefore, during training, the output of the teacher network when given the same input data is used as the label (or supervision signal) of the input data. The super-resolution network's output is then fitted to the teacher network's output until it converges to a similar level, thus improving the super-resolution network's learning ability. This process of using a larger network structure as the teacher model to guide the student model's learning is called distillation training.

[0109] Figure 3 This is a schematic diagram of an example of a traditional distillation training architecture. (For example...) Figure 3 As shown, the same input data is given to both the teacher network and the super-resolution network. The outputs of the teacher network and the super-resolution network are output result #3 and output result #4, respectively. Output result #3 is used as a supervision signal to supervise the training of the super-resolution network. Furthermore, backpropagation is performed based on the loss function of this distillation training architecture to update the parameters of both the teacher network and the super-resolution network. In this architecture, the loss function of the distillation training architecture consists of two parts: one is the difference between output result #3 and output result #4, and the other is the loss function used to train the super-resolution network to convergence.

[0110] During the distillation training process, the input data for each set of training data is used not only to calculate the output result #4 of the super-resolution network, but also to simultaneously calculate the output result #3 of the teacher network. Furthermore, because the teacher network has a larger number of parameters, it takes longer to calculate the output result #3. Therefore, the distillation training process takes more than twice as long as the direct training of the super-resolution network.

[0111] As described in the background, when a model for performing a pixel-level image task such as optimizing image resolution needs to be deployed on a mobile terminal such as a mobile phone, a GPU server is used to compress the model, and the compressed model is deployed to the mobile terminal. For example, in a mobile terminal game scenario, the resolution of a plurality of frames of pictures needs to be optimized, and generally the mobile terminal first separates the plurality of frames of pictures into text data and image data, and then inputs the image data into a neural network model to output image data with optimized resolution. The optimization of the resolution of the image by the neural network is a pixel-level image task, which tests the computing power and power consumption of the mobile terminal.

[0112] Therefore, the present application provides an image processing method and a model compression method.

[0113] According to the model compression method provided by the present application, when it is analyzed by the above-mentioned method for analyzing model redundancy provided by the present application that the to-be-compressed neural network has redundancy, the first network layer in the current compressible network layer in the to-be-compressed neural network is deleted, and a neural network #1 is obtained by training. The current compressible network layer is a network layer with the same type and parameter setting in the to-be-compressed neural network. Then, the above-mentioned method for analyzing model redundancy provided by the present application is used to analyze whether the neural network #1 has redundancy. If the neural network #1 has redundancy, the first network layer in the current compressible network layer in the neural network #1 is deleted, and the process is repeated. If the neural network #1 does not have redundancy, the model compression method is stopped, and the neural network #1 is taken as the compressed neural network finally obtained by the model compression method.

[0114] It can be understood that the output results before and after the current neural network is quantized are determined to determine whether there is redundancy, and in the case of redundancy, a network layer is deleted, and in the case of no redundancy, the model compression is stopped, realizing adaptive and accurate pruning, thereby realizing model redundancy compression. The above-mentioned scheme does not need to compare the output results of the current neural network and another neural network to determine whether the current neural network has redundancy, compared with Figure 1 The method for analyzing model redundancy in the above-mentioned scheme can improve the efficiency of analyzing model redundancy and deleting redundant network layers, thereby improving the efficiency of model compression. Compared with the traditional structured pruning, the above-mentioned scheme can be completely free of human intervention and can save labor costs. Therefore, by deleting redundant network structures, the compression of the model is realized, and the computing efficiency of the GPU server and the memory optimization degree are improved.

[0115] Optionally, in the model compression method, model quantization, model distillation, and model redundancy compression can be integrated together to realize model compression in multiple aspects, thereby greatly optimizing the performance of the model.

[0116] According to the image processing method provided in the present application, the first neural network obtained through the model compression method is used to process the image, and the image processed by the first neural network is output.

[0117] It can be understood that, compared with the model to be compressed, the first neural network obtained through the model compression method reduces redundant network layers and has a simpler network structure, thereby saving the computing power of the mobile terminal and reducing the power consumption of the mobile terminal.

[0118] The image processing method provided in the embodiments of the present application is applicable to various electronic devices, such as mobile phones, tablet computers, cameras, smart wearable devices, etc., and the embodiments of the present application do not limit this.

[0119] It should be noted that the model compression method provided in the present application is a general compression scheme for artificial intelligence models. The model compression method is highly adaptable, and can adaptively compress the model redundancy for artificial intelligence models of different application types, thereby greatly reducing the power consumption of the model caused by computing on the mobile terminal. The image processing method provided in the present application can be understood as an example of an application scenario of the model compression method provided in the present application, and does not limit the application scenarios of the model compression method.

[0120] The model compression method provided in the embodiments of the present application will be described in detail below. Figures 4A to 8

[0121] Figure 4A and Figure 4B are schematic flowcharts of the model compression method provided in the embodiments of the present application. Compared with Figure 4B , Figure 4A the relationship between the steps of the model compression method can be more clearly seen; compared with Figure 4A , Figure 4B the following is more corresponding and is more convenient for clearly introducing the model compression method. As shown in Figure 4A and Figure 4B , the model compression method comprises the following steps:

[0122] 1. The neural network before quantization passes through the PTQ module to obtain the neural network after quantization.

[0123] 2. The input data is input into the neural network before quantization and the neural network after quantization respectively to obtain output result#a and output result#b. Specifically, step 2.1 inputs the input data into the neural network before quantization to obtain output result#a. Step 2.2 inputs the input data into the neural network after quantization to obtain output result#b.

[0124] ​3. The determining module #1 determines whether there is redundancy in the neural network before quantization according to the output result #a and the output result #b. Specifically, the determining module determines whether the similarity measure of the output result #a and the output result #b is less than a first threshold value.

[0125] 3.1. If the determination result in step 3 is "yes", the first network layer in the compressible network layers of the neural network before quantization is deleted by the pruning module, that is, the network structure is updated, and the neural network after the network structure is updated is trained by the training module to obtain a trained neural network.

[0126] 3.1.1. The determining module #2 determines whether the accuracy of the trained neural network is obviously deteriorated compared with the neural network before quantization.

[0127] 3.1.1.1. If the accuracy of the trained neural network is not obviously deteriorated compared with the neural network before quantization in step 3.1.1, the trained neural network is taken as the neural network before quantization in the next round of process, and the above steps 1 to 3 are repeatedly continued, and step 3.1 or step 3.2 is repeatedly performed according to the determination result in step 3.

[0128] 3.1.1.2. If the accuracy of the trained neural network is obviously deteriorated compared with the neural network before quantization in step 3.1.1, the first network layer deleted in step 3.1 is recovered by the pruning module, that is, the neural network before quantization is output as the final model compression obtained neural network, and the process of the model compression method is stopped.

[0129] 3.2. If the determination result in step 3 is "no", the neural network before quantization is output as the final model compression obtained neural network, and the process of the model compression method is stopped.

[0130] It can be understood that the model compression method provided in the present application can be repeatedly performed for multiple rounds according to the process in Figure 4A and Figure 4B The model compression method provided in the present application is introduced by taking the i-th round process in the model compression method as an example, i≥1 and i is an integer. Figure 5

[0131] Figure 5 A schematic diagram of the model compression method 100 provided in the present application is shown. For example, the method 100 can be executed by a GPU server.

[0132] S101, inputting input data into a second neural network to obtain a first output result, wherein the second neural network comprises a first compressible network layer, and the first compressible network layer comprises a plurality of first network layers with consistent types and parameters.

[0133] ​The second neural network in S101 and the first output result can be understood as the neural network before quantization and the output result #a in the i-th round of the process, respectively.

[0134] Figure 6A A schematic diagram of an example of the second neural network provided by an embodiment of the present application is shown in FIG. 3. As shown in FIG. 3, the first compressible network layer includes Y first network layers with the same type and parameters. Figure 6A Figure 6A The network layers filled with left oblique lines shown in FIG. 3 are part of the i-1 network layers that have been deleted in the previous i-1 rounds of the process, and the other deleted network layers are replaced by ellipses.

[0135] In a possible implementation, the Y first network layers included in the first compressible network layer each include second identification information, and the second identification information is used to indicate that the current convolutional layer is deletable. For example, the first identification information and the second identification information are implemented by means of a dropout operation. Specifically, a dropout layer is configured in each of the plurality of first network layers to control the training parameters, or to mark whether the parameters of the layer are to be updated, or to mark whether the layer is to be trained. When the parameter of the dropout layer is set to 1, the dropout layer is the second identification information, indicating that the current network layer with the configured dropout layer is deletable. When the parameter of the dropout layer is set to 0, the dropout layer is the first identification information, indicating that the corresponding network layer parameters will not be updated, or that the network will skip the layer for training, or that the network layer will not affect the final training result of the network, or that the network layer is deleted from the network. The Y first network layers are marked by configuring dropout layers, and the parameters of the dropout layers are set to 1 by default. The i-1 network layers that have been deleted each include the first identification information, and the first identification information is used to indicate that the current network layer is deleted. For example, the i-1 network layers that have been deleted are marked by adding dropout layers, and the parameters of the dropout layers are set to 0.

[0136] ​As an example, maintain compressible information corresponding to all processes of the model compression method, the compressible information being used to record the compressible state of all compressible network layers in all processes of the model compression method. The number of sub-information in the compressible information is the same as the number of all compressible network layers in all processes of the model compression method, and the sub-information in the compressible information corresponds to all compressible network layers in all processes of the model compression method one by one. The compressible state includes any one of the following: has been deleted, optionally including the order of deletion; or, is deletable or not deleted. In the i th round of process, the compressible state of the first i-1 network layers in the first compressible network layer of the second neural network is that the network layers have been deleted, and the compressible state of the Y first network layers is that the network layers are deletable or not deleted.

[0137] For example, each sub-information in the compressible information can be a state bit used to indicate the compressible state of the corresponding network layer. For example, if the state bit takes the value 1, it means that the compressible state of the network layer is that it has been deleted; if the state bit takes the value 0, it means that the compressible state of the network layer is that it is deletable or not deleted.

[0138] For another example, the sub-information in the compressible information used to indicate that the compressible state of the network layer is that it has been deleted is referred to as sub-information #1, and the arrangement order of the sub-information #1 can represent the deletion order of the network layers that have been deleted.

[0139] Figure 6B A schematic diagram of an example of the compressible information corresponding to the second neural network provided by the embodiments of the present application is shown. Taking the compressible information as a vector and each column of the vector as an example, the compressible information is explained. As shown in Figure 6B Each column of the vector corresponds to the first i-1 network layers that have been deleted and the Y first network layers that have not been deleted in the first compressible module one by one. The value of the parameter corresponding to the first i-1 network layers that have been deleted in the vector is 1, and the order of the value 1 corresponding to each network layer in the first i-1 network layers in the vector, i.e., the order of the deletion of each network layer in the first i-1 network layers. As shown in Figure 6B The value of the parameter corresponding to each first network layer in the Y first network layers in the vector is 0.

[0140] S102, after training the second neural network, performing quantization processing to obtain a third neural network, the parameters of the second neural network being floating-point values, and the parameters of the third neural network model being integer values.

[0141] The third neural network in S102 can be understood as a quantized neural network in the i th round of process.

[0142] As introduced above for post-training quantization (i.e. PTQ), converting float numbers to int numbers can improve the operation speed and reduce the operation cost. The advantage of quantization is not only that int numbers are several times faster than float numbers in memory reading, but also that fixed-point operations are easier to achieve hardware acceleration than floating-point operations.

[0143] S103, inputting the input data into the third neural network to obtain a second output result.

[0144] The second output result in S103 can be understood as the output result #b in the i-th round of the process.

[0145] S104, in a case where the similarity measure of the first output result and the second output result is less than a first threshold, skipping the first one of the plurality of first network layers to train the second neural network to obtain a first neural network.

[0146] The first neural network in S104 can be understood as the trained neural network in step 3.1 in the i-th round of the process. For example, the number of epochs for training the second neural network can be set according to the data volume of the second neural network. For example, the second neural network can be trained for 100 epochs.

[0147] The second identification information in the first one of the plurality of first network layers is updated to the first identification information according to the similarity measure of the first output result and the second output result being less than the first threshold. The above scheme can achieve the training of the first network layer including the first identification information and the deletion of the first network layer including the second identification information by adding identification information (including the first identification information or the second identification information) to each of the Y first network layers, so that the second neural network layer can adaptively attempt to delete each network layer in the compressible network layer, i.e. the redundant number of layers can be automatically reduced during the training process, which is more efficient than the traditional manual modification of the network structure in the model redundancy compression process.

[0148] Figure 7A Another example of the second neural network provided by the embodiments of the present application is shown in the schematic diagram. Figure 7AAs shown, the second identification information in the first first network layer is updated to the first identification information, for example, the parameter of the dropout layer configured for the first first network layer is set to 1 by default, and the parameter of the dropout layer configured for the first first network layer is set to 0. Therefore, when the second neural network is trained, the first first network layer is automatically skipped for training according to the first identification information included in the first first network layer. In other words, the first first network layer is deleted from the second neural network, or the second first network layer to the Yth first network layer in the Y first network layers are not deleted, or the second first network layer to the Yth first network layer in the Y first network layers are trained to update the parameters of the second neural network.

[0149] It can be understood that the above scheme uses the dropout operation to adaptively attempt to delete each network layer in the compressible network layer, that is, the redundant number of layers can be automatically reduced during the training process, which is more efficient than the manual modification of the network structure in the traditional model redundancy compression process.

[0150] As shown, the second identification information in the first first network layer is updated to the first identification information, for example, the parameter of the dropout layer configured for the first first network layer is set to 1 by default, and the parameter of the dropout layer configured for the first first network layer is set to 0. Therefore, when the second neural network is trained, the first first network layer is automatically skipped for training according to the first identification information included in the first first network layer. In other words, the first first network layer is deleted from the second neural network, or the second first network layer to the Yth first network layer in the Y first network layers are not deleted, or the second first network layer to the Yth first network layer in the Y first network layers are trained to update the parameters of the second neural network.

[0151] Figure 7B Another example of the compressible information corresponding to the second neural network provided by the embodiment of the present application is shown in the schematic diagram. As shown in the schematic diagram, Figure 7B As shown, the second identification information in the first first network layer is updated to the first identification information, for example, the parameter of the dropout layer configured for the first first network layer is set to 1 by default, and the parameter of the dropout layer configured for the first first network layer is set to 0. Therefore, when the second neural network is trained, the first first network layer is automatically skipped for training according to the first identification information included in the first first network layer. In other words, the first first network layer is deleted from the second neural network, or the second first network layer to the Yth first network layer in the Y first network layers are not deleted, or the second first network layer to the Yth first network layer in the Y first network layers are trained to update the parameters of the second neural network.

[0152] It can be understood that the above scheme synchronously updates the compressible information according to the deletion of the network layer, so as to record the deletion of the network layer in all processes of the model compression method in real time, and improve the accuracy of the model redundancy compression.

[0153] In a possible implementation of S104, the second neural network is trained by skipping the first network layer based on the first loss function value, where the first loss function value includes a similarity measure of the first output result and the second output result.

[0154] It can be understood that the first output result is output by the second neural network, and the second output result is output by the third neural network, and the process is differentiable, so that the similarity measure of the two output results can be used for back propagation. In other words, taking the difference between the two output results as an example, the difference between the first output result and the second data result can be used as a loss function value for back propagation.

[0155] The above scheme can reduce the difference between the output results of the second neural network before and after quantization and the third neural network, thereby accelerating the convergence in the process of training the second neural network, by applying the similarity measure of the output results of the second neural network before and after quantization to the loss function value for training the second neural network and performing back propagation.

[0156] In an implementation parallel to S104, if the similarity measure of the first output result and the second output result is greater than or equal to the first threshold, it indicates that there is no redundancy in the second neural network, i.e., the second neural network has reached the simplest network structure, and the second neural network is output as the final neural network obtained by the model compression method. This implementation can be understood as a possible example of step 3.2 in the above process.

[0157] Optionally, the method 100 can further include:

[0158] Step 1-1, saving the first loss function value for training the second neural network.

[0159] Step 1-1 can be performed during the training of the second neural network.

[0160] Step 1-2, obtaining a second loss function value for training the first neural network.

[0161] The second loss function value can be saved during the training of the first neural network.

[0162] Step 1-3, determining whether the difference between the first loss function value and the second loss function value is less than a second threshold.

[0163] The steps 1-3 can be understood as a possible implementation of the step 3.1.1 in the above process.

[0164] It can be understood that, by judging whether the difference between the first loss function value and the second loss function value is less than the second threshold value, it can be determined whether deleting the first network layer makes the precision of the first neural network significantly worse than the precision of the second neural network. Thus, it can reduce the case of significantly losing precision due to blindly deleting redundant network layers, and obtain a balance between precision and performance. In addition, the above scheme adaptively attempts to delete each network layer in the compressible network layer, and analyzes whether the precision will be significantly lost after deletion, so that the redundant number of layers can be automatically reduced and the judgment of the reduction result can be made in the training process, which is more efficient than the way of manually modifying the network structure in the traditional model redundancy compression process.

[0165] For the possible case one of the step 1-3, as a possible implementation of the step 3.1.1.1 in the above process:

[0166] If the difference between the first loss function value and the second loss function value is less than the second threshold value, it means that the precision of the first neural network is not significantly worse than that of the second neural network. Thus, it is judged whether there is still redundancy in the first neural network, and in the case of redundancy, the redundant network layer is continued to be deleted. For the first neural network, the steps performed on the second neural network in S101-S104 are executed, that is, the i+1 round process is executed.

[0167] Specifically, the input data is input into the first neural network to obtain a third output result, wherein the first neural network comprises a second compressible network layer, the second compressible network layer comprises Y-1 second network layers of the same type and parameters, and the Y-1 second network layers correspond to the 2nd to Yth first network layers in the Y first network layers one by one. After training the first neural network and performing quantization processing, a fifth neural network is obtained, the parameters of the first neural network are all floating-point values, and the parameters of the fifth neural network model are all integer values; the input data is input into the fifth neural network to obtain a fifth output result. In one possible case, the similarity measure of the third output result and the fifth output result is less than a first threshold, the first neural network is trained by skipping the first second network layer in the Y-1 second network layers to obtain a sixth neural network. In parallel, in another possible case, the similarity measure of the third output result and the fifth output result is greater than or equal to the first threshold, the first neural network is taken as the finally obtained neural network, and the model compression method is stopped. It can be understood that, in the case where the similarity measure of the third output result and the fifth output result is greater than or equal to the first threshold, it is indicated that the first neural network has no redundancy, that is, the first neural network has almost reached the simplest network structure or has been extremely close to the simplest network structure, and the first neural network is output as the finally obtained neural network of the model compression method. This implementation mode can be understood as another possible example of step 3.2 in the above process.

[0168] Accordingly, the pre-maintained compressible information also needs to be updated according to the processing of the first neural network. Specifically, the pre-maintained compressible information is updated, and the updated compressible information is used to indicate that i+1 network layers have been deleted, and the first second network layer is the i+1th network layer deleted from the i+1 network layers, 1≤i≤Y+i-1 and i is an integer; wherein the compressible information comprises Y+i-1 pieces of sub-information, each piece of sub-information in the Y+i-1 pieces of sub-information is used to indicate the compressible state of Y+i-1 network layers; wherein the first i pieces of sub-information in the Y+i-1 pieces of sub-information are used to indicate that the corresponding i network layers have been deleted and the order in which the i network layers are deleted, and the i+2th to Y+i-1th pieces of sub-information in the Y+i-1 pieces of sub-information are used to indicate that the corresponding Y-2 network layers are not deleted.

[0169] It should be noted that the description of the i+1th round of process can be understood with reference to the corresponding contents in S101 to S104, which will not be described here.

[0170] It can be understood that, in the case where the accuracy of the first neural network does not significantly deteriorate compared to the accuracy of the second neural network, the model redundancy compression of the first neural network is automatically continued, which can be completely free of human intervention compared to the traditional structured pruning, and can save labor costs.

[0171] For the possible case two of steps 1-3, as a possible implementation of step 3.1.1.2 in the above process:

[0172] If the difference between the first loss function value and the second loss function value is greater than or equal to the second threshold, it indicates that the accuracy of the first neural network is significantly worse than that of the second neural network. Therefore, for the first neural network, the last deleted network layer among all the deleted network layers is recovered, or the last deleted network layer among all the deleted network layers is recovered; or, the first neural network is abandoned, the second neural network used to generate the first neural network is taken as the final neural network obtained by the process of the whole model compression method, and the process is stopped.

[0173] Specifically, the first identification information in the first network layer is updated to the second identification information. The second neural network is taken as the final neural network, and the model compression method is stopped.

[0174] In a possible implementation, the last deleted network layer among all the deleted network layers in the process of the model compression method can be determined according to the pre-maintained compressible information.

[0175] Specifically, as known from the above, the compressible information is used to indicate that i network layers have been deleted, and the first network layer is the i-th deleted network layer among the i network layers. Therefore, according to the pre-maintained compressible information, the last deleted network layer among the deleted network layers in the model compression method is determined to be the first network layer, and then the first identification information in the i-th network layer is updated to the second identification information, i≥1 and i is an integer.

[0176] Correspondingly, the pre-maintained compressible information also needs to be updated. Specifically, in the case of i>1, the compressible information is updated, and the updated compressible information is used to indicate that i-1 network layers have been deleted and the order in which the i-1 network layers are deleted.

[0177] It can be understood that in the case where the accuracy of the first neural network is significantly worse than that of the second neural network, recovering the last deleted network layer can reduce the case of significantly losing accuracy due to blindly deleting redundant network layers, and balance the accuracy and performance.

[0178] In a possible implementation of the method 100, the first neural network or the second neural network or the third neural network involved in the method 100 is obtained by distillation training.

[0179] It can be understood that the model quantization, the model distillation and the model redundancy compression are integrated in the above scheme, so that the three ways of model compression technology are combined, the model compression in multiple aspects is realized, and the great optimization of the model performance is realized.

[0180] Optionally, the distillation training is a modular distillation training. The following takes the modular distillation training of the second neural network by skipping the first i network layers in the Y first network layers as an example for description. The network layer is taken as a convolutional layer as an example for description.

[0181] Specifically, each first convolutional module in X first convolutional modules included in the fourth neural network is taken as a teacher model, each second convolutional module in X second convolutional modules included in the second neural network is taken as a student model, input data is input to each first convolutional module, and the input data is input to the convolutional layers in each second convolutional module except the first network layer, the X second convolutional modules correspond to the X first convolutional modules one by one, each first convolutional module includes M convolutional layers, each second convolutional module includes N convolutional layers, X>1, M>N≥1, and X, M and N are integers; the parameters of the second neural network are updated according to the first output result of each first convolutional module and the second output result of each second convolutional module, and the first neural network is obtained.

[0182] Figure 8 A schematic diagram of an example of the modular distillation training architecture provided by the embodiments of the present application is shown in FIG. 1. Figure 8 As shown in FIG. 1, the fourth neural network includes three first convolutional modules, and each first convolutional module includes four convolutional layers, and the second neural network includes three second convolutional modules, and each second convolutional module includes two convolutional layers.

[0183] In the second neural network and the fourth neural network, in addition to the convolutional layers, other layers such as activation layers, fully connected layers, and pooling layers are also included.

[0184] As an example, the i-th first convolutional module in the X first convolutional modules is taken as a teacher model, and the i-th second convolutional module in the X second convolutional modules is taken as a student model, the same input data is input to the i-th first convolutional module and the i-th second convolutional module, and the i-th first output result and the i-th second output result are obtained. The i-th second output result of the i-th second convolutional module is fitted to the i-th first output result of the i-th first convolutional module under the supervision of the i-th first output result, and converges to a similar degree as the i-th first convolutional module. 1≤i≤X and i is an integer.

[0185] It can be understood that, on the one hand, by dividing the second neural network and the fourth neural network into a plurality of convolution modules respectively, the calculation complexity of each second convolution module is much lower than the calculation complexity of the entire second neural network, and the calculation complexity of each first convolution module is also much lower than the calculation complexity of the entire fourth neural network, thereby saving training time compared with the commonly used distillation training; on the other hand, the calculations of each second convolution module can be independent of each other so that each second convolution module can be calculated in parallel, and each first convolution module is similar, and compared with the commonly used distillation training architecture, the calculations between the convolution layers of the teacher network and the super-resolution network are associated with each other, and time can also be saved. Therefore, the training efficiency can be significantly improved, thereby significantly improving the operation efficiency of the GPU server and the optimization degree of the memory.

[0186] Optionally, in order to make up for the precision loss that may be caused by post-training quantization and distillation, backward propagation can be performed in a sub-module manner. First, the i-th first sub-loss function value is obtained according to the i-th first output result and the i-th second output result. For example, the difference between the i-th first output result and the i-th second output result is determined as the i-th first sub-loss function value. Then, the i-th second sub-loss function value is obtained according to the i-th second output result and the target data. For example, the difference between the i-th second output result and the target data is determined as the i-th second sub-loss function value. Thus, the i-th first loss function value is obtained according to the i-th first sub-loss function value and the i-th second sub-loss function value. For example, the sum of the i-th first sub-loss function value and the i-th second sub-loss function value is determined as the i-th first loss function value. Backward propagation is performed based on the i-th first loss function value to update the parameters of the i-th second convolution module. Optionally, backward propagation is performed based on the i-th first loss function value to update the parameters of the i-th first convolution module.

[0187] Optionally, in order to further make up for the precision loss that may be caused by post-training quantization and distillation, global backward propagation can also be performed.

[0188] The image processing method provided in the embodiments of the present application will be described in detail below. Figure 9 The image processing method provided in the embodiments of the present application will be described in detail below. Figure 9 The schematic diagram of the image processing method 200 provided in the embodiments of the present application is shown in FIG. 2. As shown in FIG. 2, the method 200 includes S201 to S203. The method 200 is performed by an electronic device. Figure 9

[0189] S201, an image is acquired.

[0190] ​It is understood that electronic devices can acquire images processed by the first neural network model in various ways. For example, an electronic device can capture images using an image capturing device (such as a camera); an electronic device can download images; and an electronic device can interact with other electronic devices to exchange images and other information through applications. This application does not limit this.

[0191] S202, The image is processed based on the first neural network model.

[0192] The first neural network is obtained by redundantly compressing the first compressible network layer of the second neural network. The first compressible network layer includes multiple first network layers of the same type and parameters. Specifically, the first neural network is trained by skipping the first first network layer among multiple first network layers when the similarity metric between the first output result and the second output result is less than or equal to a first threshold. The first output result is obtained by inputting the input data into the second neural network, and the second output result is obtained by inputting the input data into the third neural network. The third neural network is obtained by quantizing the second neural network after training it. The parameters of the second neural network are all floating-point numbers, and the parameters of the third neural network model are all integer numbers.

[0193] Optionally, the first neural network is obtained by training the second neural network by skipping the first network layer based on the first loss function value, provided that the similarity measure between the first output result and the second output result is less than or equal to a first threshold. The first loss function value includes the similarity measure between the first output result and the second output result.

[0194] Specifically, the method for obtaining the first neural network through redundant compression processing can be found in the description of model compression method 100 above. In one possible implementation, the first neural network is obtained when the similarity metric between the first and second output results is less than a first threshold, and the similarity metric between the third and fifth output results is greater than or equal to the first threshold. In another possible implementation, the first neural network is obtained when the similarity metric between the first and second output results is less than the first threshold, and the difference between the first loss function value and the second loss function value is greater than or equal to a second threshold. S203, output the image processed by the first neural network model.

[0195] As an example of method 200, after the electronic device acquires an image, it obtains image data (hereinafter referred to as image data #1 for ease of explanation). Image data #1 includes data of multiple pixels in the image. For example, image data #1 can be in raw (RAW) format or red-green-blue (RGB) format. Image data #1 is input into a first neural network, where Z network layers in the first neural network perform calculations on the image data. Specifically, the input layer first parses and processes image data #1, outputting a feature map that can be processed by network layers (including convolutional layers, pooling layers, activation layers, etc.). The first network layer in the Z network layers performs calculations on this vector, and the calculation result output by the first network layer is input into the second network layer for calculation, and so on. The j-th network layer performs calculations on this vector, and the calculation result output by the j-th network layer is input into the (j+1)-th network layer for calculation, where 1 ≤ j ≤ Z-1, Z > 1, and j and Z are integers. The calculation result output from the Zth network layer is input into the output layer, which then parses and processes the result to output the image processed by the first neural network (hereinafter referred to as image data #2 for ease of explanation). In one possible implementation, the image processed by the first neural network can be image data in RAW or RGB format.

[0196] For example, in a mobile game scenario, the resolution of an image is optimized using a first neural network. Typically, the mobile device first separates several frames of images into text data and image data #1. Then, the image data #1 is input into the neural network model, and the first neural network calculates the image data #1 in the manner described in the example above, outputting the resolution-optimized image data #2.

[0197] In the image processing method provided in this application embodiment, the first neural network obtained by the above model compression method has fewer redundant network layers and a simpler network structure compared to the model to be compressed, thereby saving computing power on the mobile device and reducing the power consumption of the mobile device.

[0198] Please refer to Figure 10 , Figure 10 This is a schematic diagram of the hardware structure of an electronic device 1000 provided in an embodiment of this application. See also... Figure 10The electronic device 1000 can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset jack 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 can include a pressure sensor 180A, a gyro sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0199] It can be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device 1000. In other embodiments of the present application, the electronic device 1000 can include more or fewer components than illustrated, or combine certain components, or split certain components, or different arrangement of components. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0200] The processor 110 can include one or more processing units, such as: the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc.

[0201] The controller can be the nerve center and command center of the electronic device 1000. The controller can generate operation control signals according to instruction operation codes and timing signals, and complete the control of fetching instructions and executing instructions.

[0202] The processor 110 can also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can hold instructions or data that the processor 110 has just used or is using repeatedly. If the processor 110 needs to use the instructions or data again, it can be directly called from the memory. This avoids repeated access and reduces the waiting time of the processor 110, thereby improving the efficiency of the system.

[0203] In some embodiments, the processor 110 can include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0204] The MIPI interface can be used to connect the processor 110 and peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate through the CSI interface to implement the shooting function of the electronic device 1000. The processor 110 and the display screen 194 communicate through the DSI interface to implement the display function of the electronic device 1000.

[0205] It can be understood that the interface connection relationship between the modules shown in the embodiments of the present application is only illustrative and does not constitute a structural limitation of the electronic device 1000. In some other embodiments of the present application, the electronic device 1000 can also use different interface connection methods or combinations of multiple interface connection methods in the above embodiments.

[0206] The charging management module 140 is configured to receive charging input from a charger. The charger can be a wireless charger or a wired charger. The power management module 141 is configured to connect the battery 142 to the processor 110 and the charging management module 140. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, and the wireless communication module 160.

[0207] The wireless communication function of the electronic device 1000 can be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor.

[0208] The antenna 1 and the antenna 2 are configured to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 1000 can be configured to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example, the antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in combination with a tuning switch.

[0209] The mobile communication module 150 can provide a solution for wireless communication including 2G / 3G / 4G / 5G, etc. applied to the electronic device 1000. The mobile communication module 150 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and perform filtering, amplification, etc. on the received electromagnetic waves, and transmit the processed electromagnetic waves to the modem processor for demodulation. The mobile communication module 150 can also amplify signals modulated by the modem processor, and convert the amplified signals into electromagnetic waves to be radiated through the antenna 1.

[0210] The wireless communication module 160 can provide a solution for wireless communication including wireless local area networks (WLAN) (e.g., wireless fidelity (Wi-Fi) network), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc. applied to the electronic device 1000. The wireless communication module 160 can be one or more devices integrated with at least one communication processing module.

[0211] The electronic device 1000 implements a display function through a GPU, a display screen 194, and an application processor, etc. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs that execute program instructions to generate or change display information.

[0212] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flex light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light emitting diode (QLED), etc. In some embodiments, the electronic device 1000 can include 1 or N display screens 194, and N is an integer greater than 1.

[0213] The electronic device 1000 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor, etc.

[0214] The ISP is used to process data fed back by the camera 193. For example, when taking a photo, the shutter is opened, the light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electric signal, and the camera photosensitive element transmits the electric signal to the ISP for processing to convert it into an image visible to the naked eye. The ISP can also optimize the algorithm for noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature, and other parameters of the shooting scene. In some embodiments, the ISP can be arranged in the camera 193.

[0215] The camera 193 is configured to capture still images or videos. An object projects an optical image through a lens to a photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, which is then transmitted to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into a standard image signal in RGB, YUV, or the like. In some embodiments, the electronic device 1000 can include one or N cameras 193, where N is an integer greater than 1.

[0216] The digital signal processor is configured to process digital signals, including digital image signals. For example, when the electronic device 1000 is selecting a frequency point, the digital signal processor is configured to perform Fourier transform on the frequency point energy, and the like.

[0217] The video codec is configured to compress or decompress digital videos. The electronic device 1000 can support one or more video codecs. In this way, the electronic device 1000 can play or record videos in multiple encoding formats, such as moving picture experts group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, and the like.

[0218] The NPU is a neural-network (NN) computing processor that is configured to process input information quickly by imitating the structure of a biological neural network, such as the transmission mode between human brain neurons, and to continuously self-learn. Through the NPU, the electronic device 1000 can implement intelligent cognitive applications, such as image recognition, face recognition, voice recognition, text understanding, and the like.

[0219] The external memory interface 120 can be configured to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 1000. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, music, video, and the like can be saved in the external memory card.

[0220] The internal memory 121 can be used to store computer executable program codes including instructions. The processor 110 performs various functional applications and data processing of the electronic device 1000 by executing the instructions stored in the internal memory 121. The internal memory 121 can include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program (such as a sound play function, an image play function, etc.) required by a function, etc. The data storage area can store data (such as audio data, a phone book, etc.) created by the electronic device 1000 during use, etc. In addition, the internal memory 121 can include a high-speed random access memory, and can further include a non-volatile memory such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0221] The electronic device 1000 can implement an audio function such as music play, recording, etc. through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone interface 170D, an application processor, etc. The keys 190 include a power-on key, a volume key, etc. The keys 190 can be mechanical keys or touch keys. The electronic device 1000 can receive a key input and generate a key signal input related to user settings and function control of the electronic device 1000. The motor 191 can generate a vibration prompt. The indicator 192 can be an indicator light and can be used to indicate a charging state, a power change, or can be used to indicate a message, a missed call, a notification, etc. The SIM card interface 195 is used to connect a SIM card.

[0222] The sensor module 180 can include one or more sensors, which can be of the same type or different types. It can be understood that Figure 10 The illustrated sensor module 180 is only an exemplary division manner, and other division manners are also possible, which are not limited in the present application.

[0223] The pressure sensor 180A is used to sense a pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194. When a touch operation acts on the display screen 194, the electronic device detects the touch operation intensity according to the pressure sensor 180A. The electronic device can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions.

[0224] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device. In some embodiments, the angular velocity of the electronic device around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for shooting anti-shake.

[0225] The acceleration sensor 180E can detect the magnitude of acceleration of the electronic device in various directions (generally three axes). The magnitude and direction of gravity can be detected when the electronic device is stationary. It can also be used to identify the posture of the electronic device, applied to landscape / portrait switching, pedometer, etc.

[0226] The distance sensor 180F is used to measure distance. The electronic device can measure distance by infrared or laser. In some embodiments, the scene is shot, and the electronic device can use the distance sensor 180F to measure distance to achieve fast focusing.

[0227] The touch sensor 180K, also known as a "touch panel". The touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also known as a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the touch event type. The visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device, which is different from the position where the display screen 194 is located.

[0228] The barometric pressure sensor 180C is used to measure air pressure. The magnetic sensor 180D includes a Hall sensor. The proximity light sensor 180G can include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The electronic device uses the photodiode to detect infrared reflected light from nearby objects. The ambient light sensor 180L is used to sense the brightness of the ambient light. The fingerprint sensor 180H is used to acquire fingerprints. The temperature sensor 180J is used to detect temperature. The bone conduction sensor 180M can acquire vibration signals.

[0229] Next, the software system of the electronic device 1000 is described.

[0230] For example, the electronic device 1000 can be a mobile phone. The software system of the electronic device 1000 can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. The embodiments of the present application take the Android system with a layered architecture as an example to exemplarily describe the software system of the electronic device 1000.

[0231] Figure 11 A block diagram of a software system of an electronic device 1000 provided by an embodiment of the present application is shown. Referring toFigure 11 The layered architecture divides software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, the application layer, the application framework layer, the Android runtime and the system layer, the kernel layer and the hardware abstraction layer (HAL).

[0232] The application layer can include a series of application packages. As shown in Figure 11 , the application package can include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0233] The application framework layer provides application programming interface (API) and programming framework for the application of the application layer. The application framework layer includes some pre-defined functions. As shown in Figure 11 , the application framework layer can include window manager, content provider, view system, phone manager, resource manager, notification manager, etc. The window manager is used to manage the window program. The window manager can get the size of the display screen, judge whether there is a status bar, lock the screen, intercept the screen, etc. The content provider is used to store and obtain data, and make these data accessible to the application, which can include video, image, audio, dialing and answering phone, browsing history and bookmark, phone book, etc. The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build the display interface of the application, which can be composed of one or more views, such as a view for displaying a short message notification icon, a view for displaying text, and a view for displaying a picture. The phone manager is used to provide the communication function of the electronic device 1000, such as the management of the call state (including connection, hang up, etc.). The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, etc. The notification manager enables the application to display notification information in the status bar, which can be used to convey the type of message, which can automatically disappear after a short stay without user interaction. For example, the notification manager is used to inform the completion of download, message reminder, etc. The notification manager can also be a notification in the form of a chart or a scroll bar text in the top status bar of the system, such as the notification of the application running in the background. The notification manager can also be a notification in the form of a dialog window on the screen, such as a text message in the status bar, a prompt sound, an electronic device vibration, a flashing indicator light, etc.

[0234] The Android runtime includes core libraries and a virtual machine. The Android runtime is responsible for scheduling and managing the Android system. The core libraries contain two parts: one part is the function function that the java language needs to call, and the other part is the core library of Android. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the java file of the application layer and the application framework layer into a binary file. The virtual machine is used to manage the object life cycle, stack management, thread management, security and exception management, and garbage collection and the like.

[0235] The system library can include a plurality of functional modules, such as: a surface manager, media libraries, a three-dimensional graphics processing library (such as: OpenGL ES), a 2D graphics engine (such as: SGL), and the like. The surface manager is used for managing the display subsystem, and provides a plurality of applications with the fusion of 2D and 3D layers. The media library supports a plurality of commonly used audio, video format playback and recording, and static image files and the like. The media library can support a plurality of audio and video coding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG and the like. The three-dimensional graphics processing library is used for realizing three-dimensional graphics drawing, image rendering, synthesis, and layer processing and the like. The 2D graphics engine is a drawing engine for 2D drawing.

[0236] The hardware abstraction layer (HAL) is an interface layer between the operating system kernel and the upper layer software, and its purpose is to abstract the hardware. The hardware abstraction layer is an abstract interface of the device kernel driver, and is used to realize the application programming interface for providing access to the underlying device to the higher level Java API framework. The HAL contains a plurality of library modules, such as cameras, display screens, Bluetooth, audio and the like. Each library module realizes an interface for a specific type of hardware component. When the system framework layer API requires access to the hardware of the portable device, the Android operating system will load the library module for the hardware component.

[0237] The kernel layer is the layer between hardware and software. The kernel layer at least contains camera drivers, processor drivers, display drivers, audio drivers and the like device drivers. The device driver is an interface between the I / O system and the related hardware, and is used to drive the corresponding hardware device.

[0238] It should be noted that the software structure of the electronic device provided in the present application Figure 11 It should be noted that the software structure of the electronic device provided in the present application

[0239] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general purpose computer, a special purpose computer, a computer network or other programmable apparatus. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, such as the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium accessible by a computer, or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be magnetic media (such as floppy disk, hard disk, magnetic tape), optical media (such as digital versatile disc (DVD)) or semiconductor media (such as solid state disk (SSD)) and the like.

[0240] The above is an optional embodiment provided by the present application, and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the technical scope disclosed by the present application shall be included in the protection scope of the present application.

Claims

1. An image processing method applied to an electronic device, comprising: The method comprises: ​ acquiring an image; processing the image based on a first neural network; outputting the image processed by the first neural network, the first neural network being obtained by performing redundant compression processing on a first compressible network layer included in a second neural network, the first compressible network layer comprising a plurality of first network layers of the same type and parameters; wherein the first neural network is obtained by performing redundant compression processing on the first compressible network layer included in the second neural network, comprising: the first neural network is obtained by training the second neural network by skipping the first first network layer in the plurality of first network layers in the case that the similarity measure of the first output result and the second output result is less than or equal to the first threshold, wherein the first output result is obtained by inputting input data into the second neural network, and the second output result is obtained by inputting the input data into a third neural network, and the third neural network is obtained by quantization processing after training the second neural network.

2. The method of claim 1, wherein, The processing of the image based on the first neural network comprises: obtaining image data from the image, the image data comprising data of a plurality of pixels in the image; inputting the image data into the first neural network; calculating the image data by Z network layers in the first neural network, wherein the calculation result of the jth network layer in the Z network layers is input into the j+1th network layer and calculated by the j+1th network layer, 1≤j≤Z-1, Z>1 and j and Z are integers.

3. The method of claim 1 or 2, wherein, The first neural network is obtained by training the second neural network by skipping the first first network layer in the plurality of first network layers in the case that the similarity measure of the first output result and the second output result is less than or equal to the first threshold, comprising: the first neural network is obtained by training the second neural network by skipping the first first network layer based on the first loss function value in the case that the similarity measure of the first output result and the second output result is less than or equal to the first threshold, wherein the first loss function value comprises the similarity measure of the first output result and the second output result.

4. The method of claim 1, wherein, The plurality of first network layers are Y first network layers, Y>1 and Y is an integer, the first first network layer comprises first identification information, the first identification information is used to indicate deleting the current first network layer, and the training of the second neural network by skipping the first first network layer in the plurality of first network layers comprises: training the second neural network by skipping the first first network layer in the plurality of first network layers based on the first identification information; the second first network layer to the Yth first network layer in the Y first network layers all comprise second identification information, and the second identification information is used to indicate that the current network layer is deletable.

5. The method of claim 1 or 2, wherein, the first neural network is obtained by performing redundant compression processing on the first compressible network layer included in the second neural network, comprising: The first neural network is obtained by performing redundant compression processing on a first compressible network layer included in the second neural network based on pre-maintained compressible information, wherein the compressible information is used to indicate that i network layers have been deleted, and the first network layer is the i-th network layer that is deleted among the i network layers, 1≤i≤Y+i-1 and i is an integer; The compressible information includes Y+i-1 pieces of sub-information, and each piece of the Y+i-1 pieces of sub-information is used to indicate the compressible state of a network layer. The first i-1 pieces of sub-information in the Y+i-1 pieces of sub-information are used to indicate that the corresponding i-1 network layers have been deleted, and the order in which the i-1 network layers are deleted, and the i+1-th piece of sub-information to the Y+i-1-th piece of sub-information in the Y+i-1 pieces of sub-information are used to indicate that the corresponding Y-1 network layers are not deleted.

6. The method of claim 1 or 2, wherein The difference between the first loss function value and the second loss function value is less than a second threshold value, the first loss function is used to train the second neural network, and the second loss function value is used to train the first neural network.

7. The method of claim 1 or 2, wherein, The first neural network is obtained by training the second neural network by skipping the first network layer in the plurality of first network layers under the condition that the similarity measure of the first output result and the second output result is less than or equal to a first threshold value, comprising: The first neural network is obtained by distillation training the second neural network by skipping the first network layer in the plurality of first network layers under the condition that the similarity measure of the first output result and the second output result is less than or equal to a first threshold value.

8. The method of claim 7, wherein, The distillation training is a modular distillation training, wherein each first convolutional module in X first convolutional modules included in a fourth neural network is a teacher model, and each second convolutional module in X second convolutional modules included in the second neural network is a student model, the first neural network is obtained by updating parameters of the second neural network according to a first output result of each first convolutional module and a second output result of each second convolutional module, wherein the first output result is obtained by inputting the input data to each first convolutional module, the second output result is obtained by inputting the input data to convolutional layers in each second convolutional module except the first network layer, the X second convolutional modules correspond to the X first convolutional modules one by one, each first convolutional module includes M convolutional layers, each second convolutional module includes N convolutional layers, X>1, M>N≥1 and X, M and N are integers.

9. An electronic device, comprising a memory, and one or more processors, wherein the one or more processors are configured to: The memory is used to store a computer program; and the processor is used to call the computer program, so that the electronic device executes the method of any one of claims 1 to 8.

10. A computer storage medium, characterized in that The computer program includes: Computer instructions; when the computer instructions run on an electronic device, the electronic device executes the method of any one of claims 1 to 8.

Images