Network model processing method and device, computer equipment and readable storage medium

Abstract

The invention relates to a network model processing method and device, computer equipment and a readable storage medium. The method comprises the following steps: based on respective relative sensitivity parameters of weight elements included in each layer of structure in a target network model, determining respective weight sparsity rates corresponding to each layer of structure; for each layer structure, determining a sparse mask according to the weight sparse rate corresponding to the layer structure; performing single-layer pruning on the weight elements included in the layer structure based on the sparse mask to obtain sparse weight elements of the layer structure after single-layer pruning; performing single-layer quantization on the sparse weight elements to obtain target weight elements corresponding to the layer structure; and updating the target network model according to the target weight element corresponding to each layer structure to obtain an optimized target network model for image processing. By adopting the method, the expenditure cost can be reduced while the model precision is ensured.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a network model processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the rapid development of artificial intelligence technology, neural network models have achieved remarkable results in various fields such as image recognition, speech recognition, and natural language processing. Among them, neural network models usually have a large number of redundant weight parameters, which leads to high requirements for the storage space and computing resources of the model when it is actually deployed in edge devices, resulting in high hardware storage and transmission bandwidth delays and slow model reasoning. In order to reduce the complexity of neural network models and improve operating efficiency, pruning technology that reduces the amount of calculation by removing redundant parameters and structures in the model, and quantization technology that simplifies the computing and storage resources required for the model by replacing high-precision propagated values ​​in the model with low-precision data forms have become two important model compression technologies.

[0003] However, the traditional pruning and quantization techniques are decoupled during deployment, and each requires corresponding overhead costs, which complicates the entire workflow of the model. The integrated offline design of pruning and quantization is of great value for practical application deployment. Recently, some methods have also considered the joint optimization effects of quantization and pruning, but they require complex training and fine-tuning costs, consume more data and computing resources, and have obvious disadvantages in deployment. Summary of the invention

[0004] Based on this, it is necessary to provide a network model processing method, device, computer equipment, computer-readable storage medium and computer program product that can reduce overhead costs while ensuring model accuracy in response to the above technical problems.

[0005] In a first aspect, the present application provides a network model processing method, comprising:

[0006] Obtain the target network model, and determine the weight sparsity rate corresponding to each layer structure based on the relative sensitivity parameters of the weight elements included in each layer structure in the target network model; the relative sensitivity parameters are determined based on the numerical relationship between the weight elements and the weight elements of the same layer, and the weight elements of the same layer belong to the same layer structure;

[0007] For each layer structure in each layer structure, determine the sparse mask of the layer structure according to the weight sparse rate corresponding to the layer structure;

[0008] Performing single-layer pruning on the weight elements included in the targeted layer structure based on the sparse mask to obtain sparse weight elements of the targeted layer structure after single-layer pruning;

[0009] Perform single-layer quantization on the sparse weight elements to obtain the target weight elements corresponding to the targeted layer structure;

[0010] The target network model is updated according to the target weight elements corresponding to each layer structure to obtain an optimized target network model; the optimized target network model includes an image processing model, which is used to process the input image to output the image processing result.

[0011] In a second aspect, the present application also provides a network model processing device, including:

[0012] The layer sparsity rate determination module is used to obtain the target network model, and determine the weight sparsity rate corresponding to each layer structure based on the relative sensitivity parameters of the weight elements included in each layer structure in the target network model; the relative sensitivity parameters are determined based on the numerical relationship between the weight element and the weight element of the same layer, and the weight element of the same layer belongs to the same layer structure;

[0013] A sparse mask determination module, used for determining, for each layer structure in each layer structure, a sparse mask of the layer structure according to a weight sparse rate corresponding to the layer structure;

[0014] A single-layer pruning module is used to perform single-layer pruning on the weight elements included in the targeted layer structure based on the sparse mask to obtain the sparse weight elements of the targeted layer structure after the single-layer pruning;

[0015] A single-layer quantization module is used to perform single-layer quantization on sparse weight elements to obtain target weight elements corresponding to the targeted layer structure;

[0016] The model updating module is used to update the target network model according to the target weight elements corresponding to each layer structure to obtain an optimized target network model; the optimized target network model includes an image processing model, and the image processing model is used to process the input image to output the image processing result.

[0017] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above network model processing method are implemented.

[0018] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the above network model processing method are implemented.

[0019] In a fifth aspect, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the above network model processing method are implemented.

[0020] The above-mentioned network model processing method, device, computer equipment, computer-readable storage medium and computer program product determine the weight sparsity rate corresponding to each layer structure based on the relative sensitivity parameters of the weight elements included in each layer structure in the target network model. The relative sensitivity parameters are determined based on the numerical relationship between the weight elements and the same-layer weight elements belonging to the same layer structure. For each layer structure, the sparse mask of the layer structure is determined according to the weight sparsity rate corresponding to the layer structure. Based on the sparse mask, the weight elements included in the layer structure are single-layer pruned to obtain the sparse weight elements of the layer structure after single-layer pruning. The sparse weight elements are single-layer quantized to obtain the target weight elements corresponding to the layer structure. After traversing each layer structure to obtain the corresponding target weight elements, the target network model is updated according to the target weight elements corresponding to each layer structure to obtain the optimized target network model, so that the input image can be processed by the optimized target network model. The weight sparsity rate corresponding to each layer structure is determined based on the relative sensitivity parameters of the weight elements in the target network model. The weight sparsity rate can be adaptively determined based on the numerical relationship of the weight elements in the same layer structure to ensure the reasonable distribution of the sparsity rate among the various layer structures. Then, based on the weight sparsity rate of each layer structure, single-layer pruning and single-layer quantization are alternately optimized in turn according to the level of the layer structure, thus realizing the logical integration of pruning and quantization processing. There is no need to train and fine-tune the model, which reduces the overhead cost of network model processing while ensuring the model accuracy of the network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 An application environment diagram of a network model processing method in an embodiment;

[0023] Figure 2 A schematic diagram of a flow chart of a network model processing method in one embodiment;

[0024] Figure 3 A schematic diagram of a process for determining a weight sparsity rate in one embodiment;

[0025] Figure 4 A schematic diagram of a process of deploying a network model on an edge device in one embodiment;

[0026] Figure 5 A flowchart of a network model processing method in another embodiment;

[0027] Figure 6 A schematic diagram of a flow chart of determining a weight sparsity rate for a layer structure in one embodiment;

[0028] Figure 7 A schematic diagram of a process of iteratively performing single-layer pruning and single-layer quantization processing on a layer structure in one embodiment;

[0029] Figure 8 is a structural block diagram of a network model processing device in one embodiment;

[0030] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0032] The network model processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers.

[0033] The server 104 can obtain the target network model, specifically, the target network model can be obtained from the data storage system. The server 104 determines the weight sparse rate corresponding to each layer structure based on the relative sensitivity parameters of the weight elements included in each layer structure in the target network model. The relative sensitivity parameters are determined based on the numerical relationship between the weight elements and the weight elements of the same layer belonging to the same layer structure. For each layer structure, the server 104 determines the sparse mask of the layer structure according to the weight sparse rate corresponding to the layer structure, performs single-layer pruning on the weight elements included in the layer structure based on the sparse mask, obtains the sparse weight elements of the layer structure after single-layer pruning, and performs single-layer quantization on the sparse weight elements to obtain the target weight elements corresponding to the layer structure. After traversing each layer structure to obtain the corresponding target weight elements, the target network model is updated according to the target weight elements corresponding to each layer structure to obtain the optimized target network model. The server 104 can deploy the optimized target network model to the terminal 102 so that the terminal 102 can directly apply the optimized target network model locally. For example, if the target network model is a pre-trained image recognition model, the server 104 can deploy the optimized image recognition model to the terminal 102 after pruning and quantizing the image recognition model, so that the terminal 102 can recognize the image uploaded by the user based on the optimized image recognition model.

[0034] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0035] In an exemplary embodiment, Figure 2 As shown, a network model processing method is provided, which is applied to Figure 1 The server in the example is used to illustrate, including the following steps 202 to 210. Among them:

[0036] Step 202, obtain the target network model, and determine the weight sparsity rate corresponding to each layer structure based on the relative sensitivity parameters of the weight elements included in each layer structure in the target network model; the relative sensitivity parameter is determined based on the numerical relationship between the weight element and the weight element of the same layer, and the weight element of the same layer belongs to the same layer structure.

[0037] The target network model may be an artificial neural network model that needs to be optimized for compression, and may specifically include models constructed based on various algorithms, such as at least one of a feedforward neural network (FNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory network (LSTM), or a generative adversarial network (GAN). The target network model may include at least two layer structures, each of which may include different weight elements. The weight elements are parameters that connect different layer structures in the target network model, and they determine the signal transmission strength between different layer structures. Each layer structure may be used as a neuron, and each neuron receives weighted input from the previous layer of neurons and generates output through an activation function. The weight element is the core parameter of the target network model, which determines how the target network model processes input data and generates corresponding output.

[0038] The relative sensitivity parameter is used to measure the importance of the weight element in the layer structure to which it belongs. The relative sensitivity parameter is determined based on the numerical relationship between the weight element and the weight element of the same layer. The weight element of the same layer refers to the weight element that belongs to the same layer structure as the weight element. The relative sensitivity parameter is determined based on the numerical values ​​of the weight elements included in the same layer structure, specifically based on the numerical relationship between each weight element. The weight sparsity rate refers to the sparsity rate of the weight elements corresponding to the layer structure, that is, the proportion of the number of weight elements with a value of 0 in the layer structure corresponding to the weight elements. The higher the weight sparsity rate, the greater the proportion of 0, which can effectively reduce the number of model processing, thereby improving the model processing efficiency.

[0039] Optionally, the server can obtain the target network model that needs to be compressed and optimized, and determine the weight elements included in each layer structure in the target network model. For each layer structure, the server can determine the relative sensitivity parameters of each weight element in the layer structure in turn. The specific server can determine the weight elements belonging to the same layer structure, and determine the relative sensitivity parameters of each weight element based on the numerical relationship of each weight element. The server can determine the weight sparsity rate of each layer structure based on the relative sensitivity parameters of each weight element. The specific server can use the relative sensitivity parameters of all weight elements in the target network model to screen all weight elements, and determine the weight sparsity rate of the layer structure based on the weight elements retained after screening. In some embodiments, when screening all weight elements, it can be ensured that each layer structure includes at least one weight element to avoid the entire layer structure being pruned.

[0040] Step 204 : for each layer structure in each layer structure, determine a sparse mask for the layer structure according to a weight sparse rate corresponding to the layer structure.

[0041] Among them, the sparse mask is calculated based on the weight sparsity rate and is used to prune the weight elements of the layer structure. Specifically, the relatively unimportant values ​​in the weight elements can be set to zero through the relative sensitivity parameters, so as to perform pruning based on the sparse mask. Exemplarily, the server can traverse each layer structure in the target network model, that is, perform compression processing on each layer structure respectively. Specifically, the server can determine the weight sparsity rate of the layer structure, and perform mask calculation based on the weight sparsity rate to determine the sparse mask of the layer structure.

[0042] Step 206: Perform single-layer pruning on the weight elements included in the targeted layer structure based on the sparse mask to obtain sparse weight elements of the targeted layer structure after single-layer pruning.

[0043] Among them, single-layer pruning is a hierarchical pruning process for the layer structure, that is, pruning is performed on the layer structure as a whole. Single-layer pruning can be implemented step by step in the layer, so that each layer structure can be traversed to achieve independent pruning of each layer structure. Sparse weight elements are the pruning results obtained after single-layer pruning of the weight elements in the layer structure. Optionally, for the targeted layer structure, the server can perform single-layer pruning on the included weight elements based on the sparse mask of the layer structure. Specifically, single-layer pruning can be performed on the layer structure based on the alternating direction multiplier method to remove some weight elements in the layer structure to obtain sparse weight elements after single-layer pruning.

[0044] Step 208, single-layer quantization is performed on the sparse weight elements to obtain target weight elements corresponding to the targeted layer structure.

[0045] Among them, single-layer quantization is a hierarchical quantization process for the layer structure, that is, the layer structure is quantized as a whole, and single-layer quantization can be implemented step by step in the layer, so that each layer structure can be traversed to achieve independent quantization processing of each layer structure. The target weight element is the optimization result obtained after the original weight element in the layer structure is compressed and optimized by single-layer pruning and single-layer quantization in turn. Optionally, the server can perform single-layer quantization on the sparse weight elements of the layer structure, specifically, it can perform single-layer quantization on the sparse weight elements based on the alternating direction multiplier method to obtain the target weight element corresponding to the layer structure.

[0046] Step 210, updating the target network model according to the target weight elements corresponding to each layer structure to obtain an optimized target network model; the optimized target network model includes an image processing model, and the image processing model is used to process the input image to output the image processing result.

[0047] Exemplarily, the server traverses each layer structure in the target network model, and each layer structure can independently perform hierarchical pruning and hierarchical quantization processing in sequence to obtain corresponding target weight elements. The server can update the target network model through the target weight elements corresponding to each layer structure. Specifically, the weight elements in the target network model can be updated to the target weight elements corresponding to each layer structure, thereby compressing and optimizing the target network model to obtain an optimized target network model. The optimized target network model includes an image processing model, which is used to process an input image to output an image processing result. For example, the image processing model may include an image recognition model to identify the object category included in the image from the input image; for example, the image processing model may include an image classification model to perform image classification on the input image.

[0048] In the above network model processing method, based on the relative sensitivity parameters of the weight elements included in each layer structure in the target network model, the weight sparsity rate corresponding to each layer structure is determined, and the relative sensitivity parameter is determined based on the numerical relationship between the weight element and the weight element of the same layer belonging to the same layer structure. For each layer structure, the sparse mask of the targeted layer structure is determined according to the weight sparsity rate corresponding to the targeted layer structure, and the weight elements included in the targeted layer structure are single-layer pruned based on the sparse mask to obtain the sparse weight elements of the targeted layer structure after single-layer pruning, and the sparse weight elements are single-layer quantized to obtain the target weight elements corresponding to the targeted layer structure. After traversing each layer structure to obtain the corresponding target weight elements, the target network model is updated according to the target weight elements corresponding to each layer structure to obtain the optimized target network model. The weight sparsity rate corresponding to each layer structure is determined based on the relative sensitivity parameters of the weight elements in the target network model. The weight sparsity rate can be adaptively determined based on the numerical relationship of the weight elements in the same layer structure to ensure the reasonable distribution of the sparsity rate among the various layer structures. Then, based on the weight sparsity rate of each layer structure, single-layer pruning and single-layer quantization are alternately optimized in turn according to the level of the layer structure, thus realizing the logical integration of pruning and quantization processing. There is no need to train and fine-tune the model, which reduces the overhead cost of network model processing while ensuring the model accuracy of the network model.

[0049] In an exemplary embodiment, Figure 3 As shown, the process of determining the weight sparsity rate, that is, determining the weight sparsity rate corresponding to each layer structure based on the relative sensitivity parameters of the weight elements included in each layer structure in the target network model, includes steps 302 to 308. Among them:

[0050] Step 302, determining a target sparsity rate for the entire target network model.

[0051] Among them, the target sparsity rate is the overall sparsity rate for compression optimization of the target network model, that is, the degree of compression optimization required for the target network model as a whole. Optionally, the server can determine the target sparsity rate of the target network model as a whole, and the target sparsity rate can be flexibly configured by the user according to actual needs. The target sparsity rate is the compression optimization requirement of the target network model as a whole. The target sparsity rate can be allocated to each layer structure of the target network model, that is, each layer structure can be configured with a different weighted sparsity rate so that the target network model as a whole can meet the target sparsity rate.

[0052] Step 304, for each weight element included in each layer structure in the target network model, calculate the relative sensitivity parameter of the targeted weight element based on the numerical relationship between the targeted weight element and the weight elements in the same layer.

[0053] Exemplarily, the server can calculate the corresponding relative sensitivity parameter for each weight element. Specifically, for each weight element, the server can determine the weight element of the same layer that belongs to the same layer structure as the weight element, and based on the numerical relationship between the weight element and the weight element of the same layer, specifically based on the numerical size relationship between the weight element and the weight element of the same layer, calculate the relative sensitivity parameter of the weight element. After traversing each weight element, the server can obtain the relative sensitivity parameters of each weight element in the target network model.

[0054] Step 306 , based on the target sparsity rate and the relative sensitivity parameters of each weight element in the target network model, an adaptive weight element is obtained by screening out from each weight element.

[0055] Among them, the adaptive weight element is the weight element that needs to be retained that is screened out from the various weight elements of the target network model. Optionally, the server can perform an overall global screening of the weight elements in the target network model according to the target sparsity rate, and specifically can screen each weight element based on the relative sensitivity parameters of each weight element to obtain the adaptive weight elements that are screened and retained. In some embodiments, the server can sort the relative sensitivity parameters of each weight element according to the numerical value, and remove the weight elements corresponding to the values ​​less than the global threshold, so as to retain the adaptive weight elements.

[0056] Step 308: Determine the weight sparsity rate corresponding to each layer structure based on the adaptive weight element.

[0057] Optionally, the adaptive weight elements retained by screening may come from different layer structures, and the number of weight elements retained in different layer structures may be different. The server may determine the weight sparsity rate of each layer structure based on the adaptive weight elements. The specific server may determine the layer structure to which each adaptive weight element belongs, and count the number of adaptive weight elements retained in each layer structure, thereby calculating the corresponding weight sparsity rate based on the number of adaptive weight elements.

[0058] In this embodiment, the server selects adaptive weight elements from each weight element based on the overall target sparsity rate of the target network model and the relative sensitivity parameters of each weight element, and determines the weight sparsity rate of each layer structure based on the adaptive weight elements. The server can adaptively calculate the non-uniform sparsity rate for each layer structure of the target network model, alleviate the problem of accuracy loss caused by unreasonable sparsity rate distribution, ensure the rationality of the sparsity rate distribution, and thus ensure the model accuracy of the network model.

[0059] In an exemplary embodiment, a sparse mask of a targeted layer structure is determined according to a weight sparsity rate corresponding to the targeted layer structure, including: determining the number of pruning iterations for the targeted layer structure; when the number of pruning iterations does not meet a pruning end condition, based on the weight sparsity rate corresponding to the targeted layer structure, updating the pre-pruning weight elements of the current pruning iteration of the targeted layer structure to obtain a sparse mask of the current pruning iteration of the targeted layer structure.

[0060] The number of pruning iterations is the number of times the pruning iteration process has been performed on the weight elements of the layer structure, and the pruning end condition is used to determine whether it is necessary to continue the pruning iteration process on the weight elements of the layer structure, and specifically may include a pruning iteration number threshold. When the pruning iteration number exceeds the pruning iteration number threshold, it can be considered that the pruning end condition is satisfied. This pruning iteration refers to the current pruning iteration process, and the pre-pruning weight element is the initial weight element of the layer structure before this pruning iteration process.

[0061] Exemplarily, when determining the sparse mask of each layer structure for single-layer pruning, multiple pruning iterations can be performed on the weight elements of the layer structure, and the sparse mask needs to be determined in each pruning iteration. The server can determine the number of pruning iterations for the pruning iteration process. When the number of pruning iterations does not meet the pruning end condition, it indicates that the pruning iteration process has not ended. The server can determine the pre-pruning weight element of this pruning iteration and update the pre-pruning weight element of this pruning iteration based on the weight sparsity of the layer structure. Specifically, the sparse mask of this pruning iteration can be obtained by mapping based on the pre-pruning weight element. If this pruning iteration is the kth pruning iteration process, the pre-pruning weight element can be , the sparse mask M of this pruning iteration can be obtained.

[0062] Further, based on the sparse mask, single-layer pruning is performed on the weight elements included in the targeted layer structure to obtain the sparse weight elements of the targeted layer structure after single-layer pruning, including: based on the sparse mask of the current pruning iteration of the targeted layer structure, single-layer pruning is performed on the weight elements included in the targeted layer structure of this pruning iteration to obtain the pruned weight elements of this pruning iteration; the number of pruning iterations is updated, and when the updated number of pruning iterations does not meet the pruning end condition, the pruned weight elements of this pruning iteration are used as the pre-pruning weight elements of the next pruning iteration, and the weight sparsity rate corresponding to the targeted layer structure is returned, and the pre-pruning weight elements of the current pruning iteration of the targeted layer structure are updated to obtain the sparse mask of the current pruning iteration of the targeted layer structure, so as to perform the next pruning iteration; when the updated number of pruning iterations meets the pruning end condition, the sparse weight elements of the targeted layer structure after single-layer pruning are obtained according to the pruned weight elements of the last pruning iteration.

[0063] Exemplarily, during this pruning iteration, the server can perform single-layer pruning on the weight elements included in the layer structure based on the sparse mask of this pruning iteration. Specifically, the pruning problem of the layer weights can be regarded as minimizing the loss between the layer outputs before and after pruning, and single-layer pruning is performed on the weight elements included in the layer structure based on the alternating direction multiplier method to obtain the pruned weight elements of this pruning iteration. For example, if this pruning iteration is the kth pruning iteration, the weight elements before pruning can be , then the pruned weight element of this pruning iteration can be .

[0064] Furthermore, the server may update the number of pruning iterations, such as adding 1 to the number of pruning iterations to obtain an updated number of pruning iterations. The server may determine whether the updated number of pruning iterations satisfies the pruning end condition. When the updated number of pruning iterations still does not satisfy the condition, the next pruning iteration may be performed on the layer structure. The specific server may use the pruned weight element of the pruning iteration as the pre-pruning weight element of the next pruning iteration, and return the weight sparsity rate corresponding to the targeted layer structure, update the pre-pruning weight element of the current pruning iteration of the targeted layer structure, and obtain the sparse mask processing of the current pruning iteration of the targeted layer structure, so that the pruned weight element of this pruning iteration can be used for the next pruning iteration, thereby realizing iterative pruning processing of the layer structure.

[0065] Furthermore, when the updated number of pruning iterations meets the pruning end condition, it indicates that the pruning iteration processing can be ended, and the server can obtain the pruned weight element of the last pruning iteration, that is, it can obtain the pruned weight element of the last pruning iteration, and based on the pruned weight element of the last pruning iteration, obtain the sparse weight element of the layer structure after single-layer pruning, thereby realizing single-layer pruning processing for the layer structure.

[0066] In this embodiment, the server performs multiple pruning iterations on the layer structure, and the sparse mask can be updated in each pruning iteration, so as to obtain sparse weight elements that meet the pruning requirements and ensure the model accuracy of the network model.

[0067] In an exemplary embodiment, based on the sparse mask of the targeted layer structure in this pruning iteration, single-layer pruning of the current pruning iteration is performed on the weight elements included in the targeted layer structure to obtain the pruned weight elements of the current pruning iteration, including: determining the pruning optimization parameter and pruning multiplier parameter of the current pruning iteration by alternating direction multiplier method; performing single-layer pruning calculation based on the sparse mask, pruning optimization parameter and pruning multiplier parameter of the targeted layer structure in this pruning iteration to obtain the pruned weight elements of the current pruning iteration; updating the pruning optimization parameter based on the pruned weight elements, pruning multiplier parameter and sparse mask to obtain the pruning optimization parameter of the next pruning iteration; obtaining the pruning multiplier parameter of the next pruning iteration based on the pruned weight elements, the pruning optimization parameter and pruning multiplier parameter of the next pruning iteration.

[0068] Among them, the alternating direction multiplier method is usually used in mathematics to solve the optimization problem of convex functions. Assuming that the objective function has two variables, but it is difficult to optimize them at the same time, it may be simpler to fix one of the variables to solve the minimization problem about the other variable, so it is usually considered to alternately minimize the two variables. In this embodiment, by splitting the network model into independent layer structures, the pruning or quantization of each layer structure can also be processed by the alternating direction multiplier method. The pruning optimization amount parameter and the pruning multiplier parameter are intermediate parameters in the pruning iterative process determined based on the alternating direction multiplier method, which are used to optimize the weight elements.

[0069] Exemplarily, the server may determine the pruning optimization amount parameter for this pruning iteration based on the alternating direction multiplier method: and the pruning multiplier parameter , the server can optimize the amount of pruning parameters based on , pruning multiplier parameters The server can calculate the pruned weight elements of this pruning iteration based on the sparse mask of this pruning iteration. and the sparse mask pair pruning optimization parameter of this pruning iteration The server can further update the pruning multiplier parameter based on the pruned weight element, the pruning optimization parameter of the next pruning iteration, and the sparse mask of this pruning iteration. The pruning optimization amount parameters of the next pruning iteration are updated to obtain the pruning optimization amount parameters of the next pruning iteration, and the pruning optimization amount parameters of the next pruning iteration are used for the single-layer pruning calculation in the next pruning iteration processing.

[0070] Furthermore, when the updated number of pruning iterations meets the pruning end condition, the sparse weight elements of the targeted layer structure after single-layer pruning are obtained according to the pruned weight elements of the last pruning iteration, including: when the updated number of pruning iterations meets the pruning end condition, the sparse weight elements of the targeted layer structure after single-layer pruning are calculated according to the pruned weight elements of the last pruning iteration, the pruning multiplier parameters of the last pruning iteration, and the sparse mask of the last pruning iteration.

[0071] Optionally, when the updated number of pruning iterations meets the pruning end condition, indicating that the pruning iteration processing needs to be ended, the server can obtain the last pruning iteration parameters in the last pruning iteration processing, specifically including the pruned weight elements, pruning multiplier parameters and sparse masks. The server can calculate the sparse weight elements of the layer structure after single-layer pruning based on the last pruning iteration parameters.

[0072] In this embodiment, the server determines the pruning optimization amount parameter and the pruning multiplier parameter based on the alternating direction multiplier method to perform multiple pruning iterations on the layer structure, thereby obtaining sparse weight elements that meet the pruning requirements and ensuring the model accuracy of the network model.

[0073] In an exemplary embodiment, a single-layer quantization is performed on a sparse weight element to obtain a target weight element corresponding to the targeted layer structure, including: determining the number of outer-layer quantization iterations for the targeted layer structure; when the number of outer-layer quantization iterations does not meet the outer-layer iteration end condition, determining the quantization optimization parameter and the quantization multiplier parameter of this outer-layer quantization iteration by an alternating direction multiplier method; performing inner-layer quantization iteration based on the quantization multiplier parameter of this outer-layer quantization iteration and the initial sparse weight element to obtain an inner-layer quantization iteration variable; performing an inner-layer quantization iteration based on the inner-layer quantization iteration variable, the quantization multiplier parameter, the quantization optimization parameter and the initial sparse weight element Quantization iteration calculation is performed to obtain the quantization weight of this outer quantization iteration; the number of outer quantization iterations is updated. When the updated number of outer quantization iterations does not meet the outer iteration end condition, the quantization weight of this outer quantization iteration is used as the initial sparse weight element of the next outer quantization iteration, and the step of determining the quantization optimization parameter and quantization multiplier parameter of this outer quantization iteration by the alternating direction multiplier method is returned to perform the next outer quantization iteration; when the updated number of outer quantization iterations meets the outer iteration end condition, the target weight element corresponding to the targeted layer structure is obtained according to the quantization weight of the last outer quantization iteration.

[0074] The outer quantization iteration number is the number of times the outer quantization iteration processing has been performed on the weight elements of the layer structure. The outer iteration end condition is used to determine whether it is necessary to continue the outer quantization iteration processing on the weight elements of the layer structure. Specifically, it may include an outer quantization iteration number threshold. When the outer quantization iteration number exceeds the outer quantization iteration number threshold, it is considered that the outer iteration end condition is met. The quantization optimization parameter and the quantization multiplier parameter are intermediate parameters in the outer quantization iteration processing determined based on the alternating direction multiplier method, and are used to optimize the weight elements.

[0075] Optionally, after completing the single-layer pruning for the weight elements of the layer structure, a single-layer quantization process may be further performed, specifically, multiple outer-layer quantization iteration processes may be performed. The server may determine the number of outer-layer quantization iterations for the layer structure, and when the number of outer-layer quantization iterations does not meet the outer-layer iteration termination condition, it indicates that the outer-layer quantization iteration process needs to be continued. The server may determine the quantization optimization parameter of this outer-layer quantization iteration by the alternating direction multiplier method. and quantized multiplier parameters The server can be based on the quantization multiplier parameters of this outer quantization iteration And the initial sparse weight elements of this outer quantization iteration are processed by inner quantization iteration, so as to obtain inner quantization iteration variables. The inner quantization iteration processing can also include multiple iteration processing based on the alternating direction multiplication method. The initial sparse weight elements are the weight elements of the previous layer structure before the outer quantization iteration. When the outer quantization iteration is the first outer quantization iteration processing, the initial sparse weight elements can be the sparse elements obtained by single-layer pruning; when the outer quantization iteration is the nth (n>1) outer quantization iteration processing, the initial sparse weight elements can be the quantization weights obtained by the previous outer quantization iteration processing.

[0076] Furthermore, the server can update the number of outer layer quantization iterations. When the updated number of outer layer quantization iterations does not meet the outer layer iteration end condition, it indicates that the outer layer quantization iteration processing needs to continue. The server can use the quantization weight of this outer layer quantization iteration as the initial sparse weight element of the next outer layer quantization iteration, and return the processing of determining the quantization optimization parameter and quantization multiplier parameter of this outer layer quantization iteration through the alternating direction multiplier method, so as to perform the next outer layer quantization iteration processing. When the updated number of outer layer quantization iterations meets the outer layer iteration end condition, it indicates that the outer layer quantization iteration processing can be ended, and the server can determine the quantization weight of the last outer layer quantization iteration, that is, it can determine the quantization weight obtained by the last outer layer quantization iteration processing, and obtain the target weight element corresponding to the layer structure based on the quantization weight obtained by the last outer layer quantization iteration processing, so as to realize single-layer quantization processing for the layer structure.

[0077] In this embodiment, the server determines the quantization optimization parameter and the quantization multiplier parameter through the alternating direction multiplier method, and performs multiple outer quantization iterations based on the quantization optimization parameter and the quantization multiplier parameter. The quantization weights that meet the quantization requirements can be obtained through the outer quantization iterations, thereby ensuring the model accuracy of the network model.

[0078] In an exemplary embodiment, an inner quantization iteration is performed based on the quantization multiplier parameters and initial sparse weight elements of this outer quantization iteration to obtain an inner quantization iteration variable, including: determining the number of inner quantization iterations corresponding to this outer quantization iteration; when the number of inner quantization iterations does not meet the inner iteration end condition, determining the pre-iteration variable of this inner quantization iteration; based on the initial inner quantization iteration variable, the quantization multiplier parameters of this outer quantization iteration and the initial sparse weight elements, calculating the post-iteration variable of this inner quantization iteration; updating the number of inner quantization iterations, when the updated number of inner quantization iterations does not meet the inner iteration end condition, using the post-iteration variable of this inner quantization iteration as the pre-iteration variable of the next inner quantization iteration, returning to the step of determining the pre-iteration variable of this inner quantization iteration to perform the next inner quantization iteration; when the updated number of inner quantization iterations meets the inner iteration end condition, obtaining the inner quantization iteration variable according to the post-iteration variable of the last inner quantization iteration.

[0079] Among them, the number of inner layer quantization iterations is the number of times that inner layer quantization iteration processing has been performed on the weight elements of the layer structure. The inner layer iteration end condition is used to determine whether it is necessary to continue the inner layer quantization iteration processing on the weight elements of the layer structure. Specifically, it may include an inner layer quantization iteration number threshold. When the number of inner layer quantization iterations exceeds the inner layer quantization iteration number threshold, it can be considered that the inner layer iteration end condition is met.

[0080] Exemplarily, in each outer quantization iteration process, multiple inner quantization iteration processes are also included. When performing quantization iteration calculation based on the inner quantization iteration variable, quantization multiplier parameter, quantization optimization parameter and initial sparse weight element, the server can determine the number of inner quantization iterations corresponding to this outer quantization iteration. When the number of inner quantization iterations does not meet the inner iteration end condition, it indicates that inner quantization iteration processing is required. The server can determine the pre-iteration variable of this inner quantization iteration, and calculate based on the initial inner quantization iteration variable, the quantization multiplier parameter of this outer quantization iteration and the initial sparse weight element to obtain the post-iteration variable of this inner quantization iteration. The server can update the number of inner quantization iterations. When the updated number of inner quantization iterations does not meet the inner iteration end condition, it indicates that the next inner quantization iteration process needs to be continued. The server can use the post-iteration variable of this inner quantization iteration as the pre-iteration variable of the next inner quantization iteration, and return the process of determining the pre-iteration variable of this inner quantization iteration to perform the next inner quantization iteration.

[0081] Furthermore, when the updated number of inner layer quantization iterations meets the inner layer iteration end condition, indicating that the inner layer quantization iteration processing can be terminated, the server can determine the post-iteration variable of the last inner layer quantization iteration, specifically, it can determine the post-iteration variable of the last inner layer quantization iteration processing, and obtain the inner layer quantization iteration variable based on the post-iteration variable of the last inner layer quantization iteration processing, so as to perform the quantization iteration calculation in the outer layer quantization iteration processing based on the inner layer quantization iteration variable.

[0082] In this embodiment, in each outer layer quantization iteration processing, the server performs multiple inner layer quantization iteration processing, which can ensure the accuracy of the variables after iteration, so as to perform quantization iterative calculations in the outer layer quantization iterative processing, and obtain quantization weights that meet the quantization requirements, thereby ensuring the model accuracy of the network model.

[0083] The present application also provides an application scenario, and the application scenario applies the above-mentioned network model processing method. Specifically, the application of the network model processing method in the application scenario is as follows:

[0084] In the field of computer vision, existing neural network pruning and quantization technologies have the following problems: ① For weight pruning, the existing technology does not reasonably allocate the layer sparsity rate, resulting in serious loss of model accuracy, which requires training and fine-tuning after pruning. ② In terms of model compression and deployment, traditional practices usually consider designing the quantization module and the pruning module independently, which makes the model compression workflow more complicated and increases the logic design cost. ③ At present, most pruning and quantization algorithms are designed online, which requires complete training data and back-propagation processes, has higher requirements for computing resources, and is not friendly to model deployment.

[0085] The network model processing method provided in this embodiment relates to an integrated offline framework for weight pruning and quantization based on an alternating optimization strategy: a reasonable weight sensitivity score is proposed to adaptively calculate the sparsity rate of the layer, and an optimization strategy based on the alternating direction multiplier method is used to serially perform pruning and quantization on the weights of each layer of the model, so that good model accuracy can be guaranteed without training and fine-tuning. The network model processing method provided in this embodiment belongs to an offline framework, and only a small amount of unlabeled calibration sets are used to obtain the input of the model offline, which is sufficient for the pruning and quantization process. Moreover, the pruning and quantization of weights are logically designed as one, which reduces the overhead of independent design of training resources and modules during deployment.

[0086] In the field of computer vision, on the one hand, neural network models usually have a large number of redundant weight parameters, which leads to high requirements for the storage space and computing resources of the model when it is actually deployed in edge devices, resulting in high hardware storage and transmission bandwidth delays and slow model reasoning. As one of the model deployment compression technologies, model pruning is of great significance to reducing model bandwidth requirements and accelerating reasoning. Existing pruning schemes are generally divided into unstructured pruning (i.e. weight pruning) and structured pruning. The weight pruning scheme evaluates the importance of each weight and prunes the less important elements. This type of method has problems such as unreasonable sparsity rate distribution leading to precision loss and expensive training costs to compensate for the precision effect. The structured pruning scheme prunes redundant structures according to the importance of neurons or channels, which has a more serious impact on the model's expressiveness and still relies on training fine-tuning to alleviate precision loss. Offline pruning schemes that get rid of training have great advantages in saving resources and bandwidth in model deployment. Such as Figure 4 As shown in the figure, in the general process of model deployment and reasoning on edge devices, after the model structure design is completed, the model needs to be streamlined, including pruning of the pruning module and quantization of the quantization module, so as to perform compression acceleration based on the algorithm layer, and then convert the model file format, convert and optimize it through tools such as compilers, and then deploy it to the edge device for reasoning and operation.

[0087] On the other hand, quantization is a necessary step for model deployment and accelerated reasoning. Many quantization schemes still rely on training and computing resources. How to keep quantization efficient, fast, and accuracy-friendly is also critical. In traditional practices, the pruning module and the quantization module are decoupled during deployment, each requiring corresponding overhead costs, which complicates the entire workflow of the model. The integrated offline design of pruning and quantization is of great value for practical application deployment. Recently, some methods have also considered the joint optimization effects of quantization and pruning, but they require complex training and fine-tuning costs, consume more data and computing resources, and have obvious disadvantages in deployment.

[0088] In traditional technology, on the one hand, due to the redundant neural network parameters, the bandwidth during deployment becomes a bottleneck, and various weight pruning algorithms have been proposed one after another. However, the existing pruning algorithms usually perform unstructured pruning based on the absolute value of the weights, and most methods use uniform sparsity in each layer or simply use a global threshold to achieve non-uniform sparsity, without fully considering whether the sparsity rate of each layer is appropriately allocated. This single pruning operation coupled with unreasonable sparsity rate distribution can easily lead to serious loss of model accuracy, and usually requires a large amount of data and long-term training fine-tuning to restore some accuracy. On the other hand, due to the differences in the principles of pruning and quantization, the existing technology usually decouples pruning and quantization in terms of deployment, resulting in a more complicated reasoning process for the model and an increase in the design overhead for logically independent modules. Moreover, due to its reliance on online training, the existing technology has higher requirements for training data and computing resources and is not very scalable.

[0089] Based on this, the network model processing method provided in this embodiment proposes an effective weight sensitivity evaluation score for adaptively calculating the sparsity rate of each layer, alleviating the challenge of precision loss caused by unreasonable sparsity rate distribution; moreover, the network model processing method provided in this embodiment uses an optimization strategy based on the alternating direction multiplier method to unify the ideas of pruning and quantization, and integrates pruning and quantization into an integrated design to achieve fast layer-level weight sparsity plus quantization, without the need for costly training and fine-tuning to ensure good model accuracy, and more importantly, it is conducive to saving design overhead during deployment. In addition, the offline framework of the network model processing method provided in this embodiment does not rely on the back propagation of the neural network model, which reduces resource costs and enhances practicality and scalability.

[0090] Specifically, the network model processing method provided in this embodiment can be applied to the field of computer vision, and involves an integrated offline framework for weight pruning and quantization based on an alternating optimization strategy, which can be directly applied to the sparsity and quantization of onnx (Open Neural Network Exchange) models. First, the sparsity rate of each layer is adaptively calculated based on the proposed relative sensitivity score (RSS) of the weights, and then the optimization strategy based on the alternating direction multiplier method is used to efficiently implement layer-level weight pruning and quantization, without the need to spend training cost fine-tuning, and the accuracy effect that meets the application expectations can be achieved. Figure 5As shown in the figure, the weights of each layer (i.e., layer structure) are decoupled for the pre-trained model onnx, and the non-uniform sparse rate is adaptively calculated based on RSS, thereby decoupling the sparse rate of each layer. Furthermore, each layer structure is traversed, and the layer structures are pruned and quantized independently of each other. Specifically, the integrated offline framework based on the alternating optimization strategy performs weight pruning and quantization in series, and the model is updated after the quantized weight nodes of each layer are obtained, thereby obtaining the weight quantization model onnx after compression optimization.

[0091] The process of adaptively calculating the non-uniform sparsity rate may include:

[0092] (1) For each pre-trained neural network, the number of weights to be pruned is obtained given the target sparsity ratio of the entire network.

[0093] (2) The proposed weight sensitivity evaluation score is used to process the weight elements of each layer, and the relative sensitivity score (RSS) of each weight value in its layer is obtained.

[0094] The weighted relative sensitivity assessment score is expressed as:

[0095]

[0096] in, and They are all weight matrices of a certain layer The element value in , subscript and represents the index sorted by the square of the value, express The absolute value of the index weight is greater than The weight of the index. The numerator of one of the multipliers in this formula is the weight element The absolute value of the denominator is sorted by absolute value. The sum of all other weight elements before, according to the order of magnitude in pruning, the elements with smaller values ​​are more likely to be pruned, so this ratio measures the importance score of a weight element compared to other surviving weight elements in the layer. Another multiplier Input activation for this layer The L2 norm of the input activation is also of reference value in determining the output of the neuron, but in order to prevent the activation value from being too large, a balancing factor is used Therefore, RSS takes into account the impact of weights and activations on the layer output, and can be regarded as the relative sensitivity of a weight element in the current layer. The smaller the score, the less important the corresponding weight value.

[0097] (3) Then, the global threshold of the relative sensitivity score RSS is obtained according to the number of weights to be sparse in the entire network. The RSS of the entire network is sorted, and the corresponding weights with RSS smaller than the global threshold will be pruned, thereby realizing the adaptive calculation of non-uniform sparsity rates of each layer.

[0098] In extreme cases, the relative sensitivity score proposed in the present invention allows at least one weight value to survive in each layer of the neural network (when the ratio of the left half of the RSS formula is equal to 1), thereby avoiding the risk of the entire layer being pruned and making the sparse rate distribution of each layer more reasonable.

[0099] Specific as Figure 6 As shown, after flattening and sorting a certain layer weight W and a certain layer weight U, the relative sensitivity scores of the weight elements are calculated respectively, and the relative sensitivity scores are globally sorted, and the sparsity rate of each layer is adaptively calculated to obtain the weight sparsity rate of each layer.

[0100] Furthermore, for the integrated pruning and quantization framework based on the alternating optimization strategy, under the assumption that the weight matrices of each layer of the neural network are independent of each other, each layer of weights is operated independently, and the pruning and quantization process is performed serially on each layer of weights. This integrated framework aims to jointly realize layer-level weight sparsity and quantization. The present invention divides the different states of each layer of weights in the pruning and quantization process into three types: "initial state, update state, and completion state". The three states and their descriptions are as follows:

[0101] (1) Initial state: The weight state of this layer is the weight of the pre-trained model, and the relevant variables are initialized using the model input obtained from a small calibration set.

[0102] (2) Update state: The weights then enter the "update state", and the weight state of this layer is further divided into the pruned state and the quantized state, with pruning first and quantization later in a serial manner. This scheme adopts the iterative update idea of ​​the alternating direction multiplier method. For the pruning state: each iteration first calculates a more accurate sparse mask based on the proposed mask evolution strategy, and then uses the alternating direction multiplier method to quickly and efficiently update the remaining weights of the layer to ensure that the gap between the weights of this layer after the pruning operation and the original weights is minimized, thereby minimizing the accuracy loss of the entire neural network. For the quantized state: the weights after pruning are further quantized based on the update idea of ​​the alternating direction multiplier method for the inner and outer layers. Although both layers of loops involve optimization variables, the speed is faster.

[0103] (3) Completed state: The weight layer that has completed the "update state" becomes the "completed state", and pruning and quantization are completed.

[0104] Specifically, for the optimization strategy based on the alternating direction multiplier method, the alternating direction multiplier method is usually used in mathematics to solve the optimization problem of convex functions. Assuming that the objective function has two variables, but it is difficult to optimize them at the same time, it may be simpler to fix one of the variables and solve the minimization problem about the other variable, so it is usually considered to alternately minimize the two variables. Splitting the network into independent layers, pruning or quantizing each layer can actually be regarded as this type of optimization problem, which is referred to as the alternating optimization strategy in this embodiment.

[0105] In the layer pruning process, the pruning problem of layer weights is first regarded as a problem of minimizing the loss between the layer outputs before and after pruning, which is formalized as:

[0106]

[0107] in, Here, it represents the weight of this layer after final pruning. represents the input of this layer, represents the sparse mask of weights, is the same or operation, is the L2 norm.

[0108] The objective function is expressed as , and express the constraints as indicator functions:

[0109]

[0110] The pruning problem is further formalized as follows, which is converted into a constrained optimization problem that can be solved by the alternating direction multiplier method:

[0111]

[0112] According to the analysis of this type of optimization problem using the alternating direction multiplier method, the Lagrange multiplier is introduced The following steps will be used to iteratively resolve the variables involved in the above optimization problem: :

[0113]

[0114] Where k is the current iteration number, It is a penalty parameter, the default value is 1; represents the Euclidean projection on the constraint set C, which is determined based on the range of weights. In the pruning weight update problem, this scheme uses a mask to calculate the projection and The update of is similar to the update of the principal variable when solving the Lasso problem using the alternating direction multiplier method, so and The optimization can be further simplified. The update of each variable is explicitly expressed as follows:

[0115]

[0116] in, is the identity matrix, and These are all known quantities that can be directly calculated.

[0117] Furthermore, in the hierarchical quantization process, that is, the quantization of the hierarchical weights, it can also be regarded as a problem of minimizing the loss between the hierarchical outputs before and after quantization, which can be formalized as follows:

[0118]

[0119] Similarly, introducing auxiliary variables , In satisfying the constraints It takes 0 when , otherwise it takes infinity. is the scaling factor ( ), is the number of bits. The problem is further formalized as follows:

[0120]

[0121] Introducing Lagrange multipliers , using the alternating direction multiplier method to solve:

[0122]

[0123] The update of the above three variables is located in the outer loop of the iteration process. For the update of the quantized weights, the key is to analyze , an inner loop needs to be established to solve it.

[0124] make equal , then in the set The Euclidean projection on can be expressed as:

[0125]

[0126] The variable Separate from the constraints and let ,have to:

[0127]

[0128] The minimum value problem can be further solved locally using the alternating direction multiplication method (the following formula represents the number of iterations of the inner loop), and obtains the value of the variable The analytical solution of :

[0129]

[0130] in, and Iteration variable for inner quantization.

[0131] For the integrated solution of hierarchical weight pruning and quantization, the pruning process based on mask evolution and alternating direction multiplier method is shown in the principle. The entire iterative process of alternating direction multiplier method involves three optimization quantities: Each iteration requires mask calculation, and the sparse mask of the weights directly affects the update of the above variables. This scheme proposes to obtain a more accurate sparse mask by setting the weight relative sensitivity score RSS to zero relatively unimportant values, and as the mask obtained by iteration gradually becomes better, this scheme regards this process as a mask evolution strategy.

[0132] More specific steps are: ① Determine the initial value of the variable: Lagrange multiplier is an all-zero variable of the same size as the weight matrix, and the total number of iterations , calculated based on the input activation related variables ② Enter the iteration loop, execute mask evolution, and update variables: In the mask selection, calculate The weighted relative sensitivity score RSS is sorted and the positions with smaller scores are set to zero so that the sparsity reaches the sparse rate of the layer based on RSS, thus obtaining the sparse mask of the weight ; Then update the optimization amount according to the iterative update formula. ③ After the number of iterations is completed, use the mask to update the optimization amount again , as the weight after level sparseness, and end the update.

[0133] For the quantization state process based on alternating updates of the inner and outer layers, it can be seen from the principle that this solution uses two layers of loops to quantize and optimize variables. For the outer loop, the optimization quantity , is the sparse weight after pruning. The number of outer iterations is consistent with the pruning state, and both are taken ,in and Can be directly updated based on known quantities and iterative processes, variables The inner loop variables need to be further updated. For the inner loop, the optimization quantity , given the number of local iterations , first use the statistical maximum and minimum values ​​of the weight of this layer to initialize , and then solve the quantization weight in this case according to the update formula , and then obtain The internal and external parameters are updated alternately in a loop to efficiently quantize the sparse weights of this layer.

[0134] like Figure 7 As shown in the figure, in the process of single-layer pruning and single-layer quantization processing for the layer structure in sequence, the original weights of the layers of the layer structure go through the initial state, update state and completion state respectively, so as to obtain the weights after layer pruning and quantization. Among them, in the update state processing process, the initial value of each variable of the layer structure can be determined, and the pruning state processing flow can be realized based on the mask evolution and the alternating direction multiplier method. Specifically, the iterative processing is determined by the number of iterations K. When the number of iterations K is not reached, the sparse mask M can be determined based on the mask evolution strategy, and the number of iterations is returned after further updating each variable. After the iteration, the single-layer pruning is completed, and the sparse mask is used to update the optimization quantity. , as the weight after hierarchical sparseness. In the quantized state process based on alternating updates of inner and outer layers, the weight obtained based on single-layer pruning is used as the initial weight, and the iterative processing is determined by the outer layer iteration number K and the inner layer iteration number J. In the inner layer iteration processing, each inner layer quantized iteration variable is updated and , update each variable in the outer layer iterative processing, and obtain the hierarchical pruning quantization weight after the single layer quantization is completed, that is, obtain the target weight element corresponding to the layer structure.

[0135] For the strategy of non-uniform sparsity rate allocation, traditional technology generally allocates it globally based on the absolute value. That is, after using the target sparsity rate of the whole network to determine the global threshold, the value with the smaller absolute value of the weight is directly set to zero to achieve non-uniform sparsity in each layer. However, there are problems with such a simple and crude method: ① The sparse rate allocation is unreasonable, which may easily lead to some layers being set to zero, and the unstructured pruning is directly transformed into structured pruning. The rationality of the sparse rate allocation is not as good as the technical solution of the present invention; ② After the pruning operation, the model accuracy is seriously lost, and it usually takes longer to train and fine-tune.

[0136] In some embodiments, the hierarchical weight pruning and updating technology can also be implemented based on the Optimal Brain Compression (OBC) and SparseGPT methods. OBC and SparseGPT both use the weight importance formula and weight compensation formula based on the Hessian matrix to independently prune and update the weights of each layer of the neural network. Specifically,

[0137] The weight importance formula is used to determine the position of the cut weight as follows:

[0138]

[0139] in, is the pth weight and H represents the Hessian matrix of the loss of a given (dense) model. is the p-th diagonal entry of the inverse Hessian matrix.

[0140] The formula for updating the remaining weight according to the weight compensation formula is as follows:

[0141]

[0142] in, is the compensation value for the remaining weight, is the p-th column of the inverse Hessian matrix.

[0143] For hierarchical quantization technology, it can also be implemented based on the Optimal Brain Quantization (OBQ) and GPTQ methods. Similar to pruning, OBQ and GPTQ are based on the weight importance formula and weight compensation formula of the Hessian matrix to independently quantize and update the weights of each layer. The formula is as follows:

[0144]

[0145] Among them, quant(·) is the quantitative processing, is the pth weight, H F The Hessian matrix representing the loss for a given (dense) model, is the inverse matrix H F The pth diagonal item of ; is the compensation value for the remaining weight, is the inverse matrix H F The qth column of .

[0146] The network model processing method provided in this embodiment proposes a weight relative sensitivity evaluation score to more accurately measure the importance of weight elements, so as to adaptively calculate the appropriate sparsity rate for each layer and alleviate the problem of pruning accuracy loss caused by unreasonable sparsity rate distribution. In addition, this embodiment proposes an optimization strategy based on the alternating direction multiplier method to unify the ideas of pruning and quantization, design an efficient pruning and quantization integrated framework, and perform layer-level pruning and quantization processes offline in series, without the need for training and fine-tuning, to achieve the accuracy effect that meets the application expectations, which not only reduces the demand for training resources, but also reduces the overhead of deployment design.

[0147] The network model processing method provided in this embodiment alleviates the challenge of unreasonable sparse rate distribution of each layer of model pruning, realizes efficient weight pruning and quantization of visual neural networks, and ensures good precision effects. Moreover, the network model processing method provided in this embodiment is an offline technology, which does not involve the training and back propagation of neural networks, reduces the demand for training resources, and enhances practicality and scalability. In addition, the network model processing method provided in this embodiment logically integrates model pruning and quantization, which is beneficial to reducing deployment design overhead. Moreover, the sparse quantization model of the network model processing method provided in this embodiment effectively reduces the model storage and transmission bandwidth, which is beneficial to reasoning acceleration.

[0148] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0149] Based on the same inventive concept, the embodiment of the present application also provides a network model processing device for implementing the network model processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more network model processing device embodiments provided below can refer to the limitations on the network model processing method above, and will not be repeated here.

[0150] In an exemplary embodiment, Figure 8 As shown, a network model processing device 800 is provided, comprising: a layer sparse rate determination module 802, a sparse mask determination module 804, a single layer pruning module 806, a single layer quantization module 808 and a model updating module 810, wherein:

[0151] The layer sparsity rate determination module 802 is used to obtain the target network model, and determine the weight sparsity rate corresponding to each layer structure based on the relative sensitivity parameters of the weight elements included in each layer structure in the target network model; the relative sensitivity parameters are determined based on the numerical relationship between the weight element and the weight element of the same layer, and the weight element of the same layer belongs to the same layer structure;

[0152] A sparse mask determination module 804 is used to determine, for each layer structure in each layer structure, a sparse mask for the layer structure according to a weight sparse rate corresponding to the layer structure;

[0153] The single-layer pruning module 806 is used to perform single-layer pruning on the weight elements included in the targeted layer structure based on the sparse mask to obtain the sparse weight elements of the targeted layer structure after the single-layer pruning;

[0154] A single-layer quantization module 808 is used to perform single-layer quantization on the sparse weight elements to obtain target weight elements corresponding to the targeted layer structure;

[0155] The model updating module 810 is used to update the target network model according to the target weight elements corresponding to each layer structure to obtain an optimized target network model; the optimized target network model includes an image processing model, and the image processing model is used to process the input image to output the image processing result.

[0156] In some embodiments, the layer sparsity rate determination module 802 is also used to determine the target sparsity rate for the target network model as a whole; for each weight element in the weight elements included in each layer structure in the target network model, based on the numerical relationship between the targeted weight element and the weight elements of the same layer, calculate the relative sensitivity parameter of the targeted weight element; according to the target sparsity rate and the relative sensitivity parameters of each weight element in the target network model, filter out adaptive weight elements from each weight element; and determine the weight sparsity rate corresponding to each layer structure based on the adaptive weight elements.

[0157] In some embodiments, the sparse mask determination module 804 is further used to determine the number of pruning iterations for the targeted layer structure; when the number of pruning iterations does not meet the pruning end condition, based on the weight sparsity rate corresponding to the targeted layer structure, the pre-pruning weight elements of the targeted layer structure of the current pruning iteration are updated to obtain the sparse mask of the targeted layer structure of the current pruning iteration; the single-layer pruning module 806 is further used to perform single-layer pruning of the weight elements included in the targeted layer structure of the current pruning iteration based on the sparse mask of the targeted layer structure of the current pruning iteration to obtain the pruned weight elements of the current pruning iteration; The number of pruning iterations is updated. When the updated number of pruning iterations does not meet the pruning end condition, the pruned weight element of this pruning iteration is used as the pre-pruning weight element of the next pruning iteration, and the weight sparsity rate corresponding to the targeted layer structure is returned. The pre-pruning weight element of the pruning iteration of the targeted layer structure is updated to obtain the sparse mask of the pruning iteration of the targeted layer structure for the next pruning iteration; when the updated number of pruning iterations meets the pruning end condition, the sparse weight element of the targeted layer structure after single-layer pruning is obtained according to the pruned weight element of the last pruning iteration.

[0158] In some embodiments, the single-layer pruning module 806 is further used to determine the pruning optimization amount parameter and pruning multiplier parameter of this pruning iteration by means of an alternating direction multiplier method; perform single-layer pruning calculation based on the sparse mask, pruning optimization amount parameter and pruning multiplier parameter of this pruning iteration of the targeted layer structure to obtain the pruned weight element of this pruning iteration; update the pruning optimization amount parameter based on the pruned weight element, pruning multiplier parameter and sparse mask to obtain the pruning optimization amount parameter of the next pruning iteration; obtain the pruning multiplier parameter of the next pruning iteration based on the pruned weight element, the pruning optimization amount parameter and pruning multiplier parameter of the next pruning iteration; when the updated number of pruning iterations meets the pruning end condition, calculate the sparse weight element of the targeted layer structure after single-layer pruning based on the pruned weight element of the last pruning iteration, the pruning multiplier parameter of the last pruning iteration and the sparse mask of the last pruning iteration.

[0159] In some embodiments, the single-layer quantization module 808 is also used to determine the number of outer-layer quantization iterations for the targeted layer structure; when the number of outer-layer quantization iterations does not meet the outer-layer iteration end condition, determine the quantization optimization amount parameter and quantization multiplier parameter of this outer-layer quantization iteration by the alternating direction multiplier method; perform inner-layer quantization iteration based on the quantization multiplier parameter and the initial sparse weight element of this outer-layer quantization iteration to obtain the inner-layer quantization iteration variable; perform quantization iteration calculation based on the inner-layer quantization iteration variable, the quantization multiplier parameter, the quantization optimization amount parameter and the initial sparse weight element to obtain the quantization weight of this outer-layer quantization iteration; update the number of outer-layer quantization iterations, and when the updated number of outer-layer quantization iterations does not meet the outer-layer iteration end condition, use the quantization weight of this outer-layer quantization iteration as the initial sparse weight element of the next outer-layer quantization iteration, return to the step of determining the quantization optimization amount parameter and quantization multiplier parameter of this outer-layer quantization iteration by the alternating direction multiplier method to perform the next outer-layer quantization iteration; when the updated number of outer-layer quantization iterations meets the outer-layer iteration end condition, obtain the target weight element corresponding to the targeted layer structure according to the quantization weight of the last outer-layer quantization iteration.

[0160] In some embodiments, the single-layer quantization module 808 is also used to determine the number of inner-layer quantization iterations corresponding to the current outer-layer quantization iteration; when the number of inner-layer quantization iterations does not meet the inner-layer iteration end condition, determine the pre-iteration variable of the current inner-layer quantization iteration; based on the initial inner-layer quantization iteration variable, the quantization multiplier parameter of the current outer-layer quantization iteration and the initial sparse weight element, calculate the post-iteration variable of the current inner-layer quantization iteration; update the number of inner-layer quantization iterations, when the updated number of inner-layer quantization iterations does not meet the inner-layer iteration end condition, use the post-iteration variable of the current inner-layer quantization iteration as the pre-iteration variable of the next inner-layer quantization iteration, and return to the step of determining the pre-iteration variable of the current inner-layer quantization iteration to perform the next inner-layer quantization iteration; when the updated number of inner-layer quantization iterations meets the inner-layer iteration end condition, obtain the inner-layer quantization iteration variable according to the post-iteration variable of the last inner-layer quantization iteration.

[0161] Each module in the above network model processing device can be implemented in whole or in part by software, hardware and a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0162] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig. 9 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store various data involved in the network model processing method. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a network model processing method is implemented.

[0163] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0164] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0165] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0166] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0167] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0168] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0169] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0170] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A network model processing method, characterized in that: The method comprises: Obtain a target network model, and determine the weight sparsity rate corresponding to each layer structure based on the relative sensitivity parameters of the weight elements included in each layer structure in the target network model; the relative sensitivity parameters are determined based on the numerical relationship between the weight element and the weight element of the same layer, and the weight element of the same layer belongs to the same layer structure as the weight element; For each layer structure in each of the layer structures, determining a sparse mask of the targeted layer structure according to a weight sparse rate corresponding to the targeted layer structure; Performing single-layer pruning on the weight elements included in the targeted layer structure based on the sparse mask to obtain sparse weight elements of the targeted layer structure after single-layer pruning; Performing single-layer quantization on the sparse weight elements to obtain target weight elements corresponding to the targeted layer structure; The target network model is updated according to the target weight elements corresponding to each of the layer structures to obtain an optimized target network model; the optimized target network model includes an image processing model, which is used to process the input image to output an image processing result.

2. The method according to claim 1, characterized in that: The determining, based on the relative sensitivity parameters of the weight elements included in each layer structure in the target network model, the weight sparsity rates corresponding to each layer structure respectively include: Determining a target sparsity rate for the target network model as a whole; For each weight element included in each layer structure in the target network model, based on the numerical relationship between the targeted weight element and the weight element of the same layer, calculate the relative sensitivity parameter of the targeted weight element; According to the target sparsity rate and the relative sensitivity parameters of each of the weight elements in the target network model, selecting an adaptive weight element from each of the weight elements; The weight sparsity rates corresponding to the respective layer structures are determined based on the adaptive weight elements.

3. The method according to claim 1, characterized in that The step of determining the sparse mask of the targeted layer structure according to the weight sparse rate corresponding to the targeted layer structure includes: Determine the number of pruning iterations for the targeted layer structure; When the number of pruning iterations does not meet the pruning end condition, based on the weight sparsity rate corresponding to the targeted layer structure, the pre-pruning weight element of the targeted layer structure is updated to obtain the sparse mask of the targeted layer structure; The step of performing single-layer pruning on the weight elements included in the targeted layer structure based on the sparse mask to obtain the sparse weight elements of the targeted layer structure after single-layer pruning includes: Based on the sparse mask of the targeted layer structure in this pruning iteration, perform single-layer pruning of the targeted layer structure in this pruning iteration to obtain the pruned weight elements of this pruning iteration; The number of pruning iterations is updated. When the updated number of pruning iterations does not meet the pruning end condition, the post-pruning weight element of the current pruning iteration is used as the pre-pruning weight element of the next pruning iteration, the weight sparsity rate corresponding to the targeted layer structure is returned, the pre-pruning weight element of the current pruning iteration of the targeted layer structure is updated, and the sparse mask of the current pruning iteration of the targeted layer structure is obtained, so as to perform the next pruning iteration; When the updated number of pruning iterations satisfies the pruning end condition, the sparse weight elements of the targeted layer structure after single-layer pruning are obtained according to the pruned weight elements of the last pruning iteration.

4. The method according to claim 3, characterized in that The step of performing single-layer pruning of the weight elements included in the targeted layer structure based on the sparse mask of the targeted layer structure in the current pruning iteration to obtain the pruned weight elements of the current pruning iteration includes: The pruning optimization amount parameter and pruning multiplier parameter of this pruning iteration are determined by the alternating direction multiplier method; Performing single-layer pruning calculation based on the sparse mask of the current pruning iteration of the targeted layer structure, the pruning optimization amount parameter and the pruning multiplier parameter to obtain the pruned weight element of the current pruning iteration; Based on the pruned weight element, the pruning multiplier parameter and the sparse mask, the pruning optimization amount parameter is updated to obtain the pruning optimization amount parameter for the next pruning iteration; Obtaining a pruning multiplier parameter for the next pruning iteration based on the pruned weight element, the pruning optimization amount parameter for the next pruning iteration, and the pruning multiplier parameter; When the updated number of pruning iterations satisfies the pruning end condition, obtaining the sparse weight element of the targeted layer structure after single-layer pruning according to the pruned weight element of the last pruning iteration includes: When the updated number of pruning iterations meets the pruning end condition, the sparse weight elements of the targeted layer structure after single-layer pruning are calculated based on the pruned weight elements of the last pruning iteration, the pruning multiplier parameters of the last pruning iteration, and the sparse mask of the last pruning iteration.

5. The method according to any one of claims 1 to 4, characterized in that: The single-layer quantization is performed on the sparse weight element to obtain the target weight element corresponding to the targeted layer structure, including: Determine the number of outer layer quantization iterations for the targeted layer structure; When the number of outer layer quantization iterations does not meet the outer layer iteration termination condition, determining the quantization optimization parameter and the quantization multiplier parameter of this outer layer quantization iteration by an alternating direction multiplier method; Perform inner quantization iteration based on the quantization multiplier parameter of this outer quantization iteration and the initial sparse weight element to obtain an inner quantization iteration variable; Perform quantization iterative calculation based on the inner quantization iteration variable, the quantization multiplier parameter, the quantization optimization parameter and the initial sparse weight element to obtain the quantization weight of this outer quantization iteration; The outer quantization iteration number is updated. When the updated outer quantization iteration number does not meet the outer iteration end condition, the quantization weight of this outer quantization iteration is used as the initial sparse weight element of the next outer quantization iteration, and the step of determining the quantization optimization parameter and the quantization multiplier parameter of this outer quantization iteration by the alternating direction multiplier method is returned to perform the next outer quantization iteration; When the updated number of outer layer quantization iterations meets the outer layer iteration end condition, the target weight element corresponding to the targeted layer structure is obtained according to the quantization weight of the last outer layer quantization iteration.

6. The method according to claim 5, characterized in that The inner quantization iteration is performed based on the quantization multiplier parameter of the outer quantization iteration and the initial sparse weight element to obtain the inner quantization iteration variable, including: Determine the number of inner quantization iterations corresponding to this outer quantization iteration; When the number of inner layer quantization iterations does not satisfy the inner layer iteration end condition, determining the pre-iteration variable of this inner layer quantization iteration; Based on the initial inner quantization iteration variable, the quantization multiplier parameter of the current outer quantization iteration and the initial sparse weight element, the post-iteration variable of the current inner quantization iteration is calculated; The inner quantization iteration number is updated. When the updated inner quantization iteration number does not meet the inner iteration end condition, the post-iteration variable of the current inner quantization iteration is used as the pre-iteration variable of the next inner quantization iteration, and the step of determining the pre-iteration variable of the current inner quantization iteration is returned to perform the next inner quantization iteration; When the updated number of inner layer quantization iterations meets the inner layer iteration end condition, the inner layer quantization iteration variable is obtained according to the post-iteration variable of the last inner layer quantization iteration.

7. A network model processing device, characterized in that: The device comprises: A layer sparsity rate determination module is used to obtain a target network model, and determine the weight sparsity rate corresponding to each layer structure based on the relative sensitivity parameters of the weight elements included in each layer structure in the target network model; the relative sensitivity parameters are determined based on the numerical relationship between the weight elements and the weight elements of the same layer, and the weight elements of the same layer belong to the same layer structure as the weight elements; A sparse mask determination module, configured to determine, for each layer structure in each of the layer structures, a sparse mask of the layer structure according to a weight sparse rate corresponding to the layer structure; A single-layer pruning module, configured to perform single-layer pruning on the weight elements included in the targeted layer structure based on the sparse mask to obtain sparse weight elements of the targeted layer structure after single-layer pruning; A single-layer quantization module, used for performing single-layer quantization on the sparse weight elements to obtain target weight elements corresponding to the targeted layer structure; A model updating module is used to update the target network model according to the target weight elements corresponding to each of the layer structures to obtain an optimized target network model; the optimized target network model includes an image processing model, and the image processing model is used to process the input image to output the image processing result.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images