Network model compression method and device, electronic equipment and storage medium

Abstract

The application provides a network model compression method and device, electronic equipment and storage medium, wherein the network model compression method comprises the following steps: obtaining initial weight parameters and initial bias parameters of a network model to be compressed; determining a target inference model based on the initial weight parameters and the initial bias parameters, the target inference model being used to infer the state of different gates of the network model to be compressed at different time points; performing pruning on the target inference model, and determining target weight parameters and target bias parameters of a pruned model; and determining a target network model based on the target weight parameters and the target bias parameters. The method provided by the application not only avoids the problems of increased model parameter quantity and calculation quantity, and increased storage difficulty caused by the introduction of a sparse matrix, but also achieves the purposes of reducing calculation quantity and improving inference speed.

Description

Technical Field

[0001] This invention relates to the field of computer hardware acceleration technology, and in particular to a network model compression method, apparatus, electronic device, and storage medium. Background Technology

[0002] Currently, the Long Short-Term Memory (LSTM) model is used to solve the gradient explosion and gradient vanishing problems that occur during RNN training because it is a recurrent neural network (RNN) with added gating mechanism. However, the LSTM model introduces a huge number of parameters and computational cost during use, which increases the deployment requirements of embedded devices. Therefore, it is particularly important to find ways to reduce the number of parameters and computational cost of the LSTM model.

[0003] In related technologies, pruning is commonly used to prune the weight parameters of the LSTM model, setting unimportant weight parameters to zero, thereby reducing the number of parameters and computational cost of the LSTM model.

[0004] However, since the pruning method easily introduces sparse matrices during implementation, it not only increases the number of parameters and computation of the LSTM model, but also makes it more difficult to store the weight parameters. Summary of the Invention

[0005] This invention provides a network model compression method, apparatus, electronic device, and storage medium to address the shortcomings of existing technologies that introduce sparse matrices, which increase the number of model parameters, computational load, and storage difficulty, thereby achieving the goal of significantly reducing model parameters, reducing computational load, and improving computational speed.

[0006] This invention provides a network model compression method, comprising:

[0007] Obtain the initial weight parameters and initial bias parameters of the network model to be compressed;

[0008] Based on the initial weight parameters and the initial bias parameters, a target inference model is determined, which is used to infer the state of different gates in the network model to be compressed at different times.

[0009] The target inference model is pruned using a preset gate, and the target weight parameters and target bias parameters of the pruned model are determined.

[0010] The target network model is determined based on the target weight parameters and the target bias parameters.

[0011] According to a network model compression method provided by the present invention, obtaining the initial weight parameters and initial bias parameters of the network model to be compressed includes:

[0012] The initial LSTM model is trained using the sample dataset to obtain the network model to be compressed;

[0013] Obtain the input layer weight parameters, input layer bias parameters, hidden layer weight parameters, and hidden layer bias parameters of the network model to be compressed;

[0014] The input layer weight parameters and the hidden layer weight parameters are determined as the initial weight parameters of the network model to be compressed;

[0015] The input layer bias parameters and the hidden layer bias parameters are determined as the initial bias parameters of the network model to be compressed.

[0016] According to a network model compression method provided by the present invention, the step of pruning the target inference model by a preset gate and determining the target weight parameters and target bias parameters of the pruned model includes:

[0017] Based on the target inference model and the sample dataset, a fitting dataset for a preset time series is determined. The fitting dataset includes input data at different current times, hidden states at different previous times, cell states at different previous times, cell states at different current times, and hidden states at different current times in the preset time series.

[0018] The input and output gates of the target inference model are pruned;

[0019] The initial fully connected layer is trained based on the fitted dataset to determine the target fully connected layer;

[0020] Based on the parameters of the target fully connected layer, the target weight parameters and target bias parameters of the pruned model are determined.

[0021] According to a network model compression method provided by the present invention, the step of training an initial fully connected layer based on the fitted dataset to determine a target fully connected layer includes:

[0022] The input data at different current times, the hidden states at different previous times, and the cell states at different previous times are determined as the sample input set, and the cell states at different current times and the hidden states at different current times are determined as the sample output set.

[0023] The initial fully connected layer is trained using the sample input set to determine the new cell states and new hidden states at different times.

[0024] The new cell state and the new hidden state are validated using the sample output set to determine the target fully connected layer.

[0025] According to a network model compression method provided by the present invention, the step of determining the target inference model based on the initial weight parameters and the initial bias parameters includes inferring the input gate state, forget gate state, memory cell state, output gate state, cell state, and hidden state at time t using the following formula:

[0026] i t =sigmoid(W ii x t +b ii +W hi h t-1 +b hi );

[0027] f t =sigmoid(W if x t +b if +W hf h t-1 +b hf );

[0028] g t =tanh(W ig x t +b ig +W hg h t-1 +b hg );

[0029] o t =sigmoid(W io x t +b io +W ho h t-1 +b ho );

[0030] c t =f t ⊙c t-1 +i t ⊙g t ;

[0031] h t =o t ⊙tanh(c t );

[0032] Among them, i t Let f be the input gate state at time t. t Let g be the forget gate state at time t. t Let o be the state of the memory cell at time t.t Let c be the output gate state at time t. t Let h be the cell state at time t. t Let W be the hidden state at time t. ii b represents the initial weight parameters of the input layer at the input gate. ii W represents the initial bias parameters of the input layer at the input gate. hi b represents the initial weight parameters of the hidden layer at the input gate. hi x represents the initial bias parameters of the hidden layer at the input gate. t Let h be the input data at time t. t-1 Let W be the hidden state at time t-1. if W represents the initial weight parameters of the input layer at the forget gate. hf b represents the initial weight parameters of the hidden layer at the forget gate. if b represents the initial bias parameters of the input layer at the forget gate. hf W represents the initial bias parameters of the hidden layer at the forget gate. ig W represents the initial weight parameters of the input layer in the memory cells. hg b represents the initial weight parameters of the hidden layer in the memory cells. ig b represents the initial bias parameters of the input layer in the memory cells. hg W represents the initial bias parameters of the hidden layer in the memory cells. io W represents the initial weight parameters of the input layer at the output gate. ho b represents the initial weight parameters of the hidden layer at the output gate. io b represents the initial bias parameters of the input layer at the output gate. ho c represents the initial bias parameters of the hidden layer at the output gate. t-1 This represents the cell state at time t-1.

[0033] According to a network model compression method provided by the present invention, determining the target network model based on the target weight parameters and the target bias parameters includes determining the new forget gate state, the new memory cell state, the new cell state, and the new hidden state at time t using the following formula:

[0034] f t = sigmoid(W) if ′x t +b if ′+W hf ′h t-1 ′+b hf ′);

[0035] g t '=tanh(W ig 'x t +b ig '+W hg'h t-1 '+b hg ');

[0036] c t ′=f t ′⊙c t-1 ′+g t ′;

[0037] h t =tanh(c t ′);

[0038] Among them, f t 'G is the new forget gate state at time t, g t 'C represents the new memory cell state at time t.' t 'H represents the new cell state at time t.' t Let x be the new hidden state at time t. t Let h be the input data at time t. t-1 'W' represents the new hidden state at time t-1. if 'W represents the target weight parameters of the input layer at the forget gate.' ig 'b' represents the target weight parameters of the input layer in the memory cell. if ' is the target bias parameter of the input layer at the forget gate, b ig 'W represents the target bias parameter of the input layer in the memory cell.' hf 'W represents the target weight parameter of the hidden layer at the forget gate.' hg 'b' represents the target weight parameter of the hidden layer in the memory cell. hf 'b' represents the target bias parameter of the hidden layer at the forget gate. hg 'where c is the target bias parameter of the hidden layer in the memory cell.' t-1 'This represents the new cell state at time t-1.

[0039] The present invention also provides a network model compression device, comprising:

[0040] The acquisition module is used to obtain the initial weight parameters and initial bias parameters of the network model to be compressed;

[0041] The first determining module is used to determine the target inference model based on the initial weight parameters and the initial bias parameters. The target inference model is used to infer the state of different gates in the network model to be compressed at different times.

[0042] The pruning module is used to prune the target inference model using a preset gate, and to determine the target weight parameters and target bias parameters of the pruned model.

[0043] The second determining module is used to determine the target network model based on the target weight parameters and the target bias parameters.

[0044] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the network model compression method described above.

[0045] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the network model compression method as described above.

[0046] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the network model compression method as described above.

[0047] The present invention provides a network model compression method, apparatus, electronic device, and storage medium. The network model compression method first obtains the initial weight parameters and initial bias parameters of the network model to be compressed. Then, based on the initial weight parameters and initial bias parameters, a target inference model is determined. Since the target inference model is used to infer the states of different gates in the network model to be compressed at different times, it can combine inference techniques and model construction analysis to select pre-defined gates that can be pruned, and prune the target inference model using these pre-defined gates. Finally, based on the target weight parameters and target bias parameters determined by the pruned model, the compressed target network model is determined. This method of determining the compressed target network model by generating a lightweight inference model through pruning the inference model not only avoids the drawbacks of increasing the number of model parameters and computational load, as well as the increased storage difficulty caused by introducing sparse matrices, but also achieves the goal of reducing computational load and improving inference speed. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0049] Figure 1 This is a flowchart illustrating the network model compression method provided by the present invention;

[0050] Figure 2 This is a schematic diagram illustrating the process of building a target reasoning model provided by the present invention;

[0051] Figure 3 This is a schematic diagram illustrating the process of pruning the target inference model provided by the present invention;

[0052] Figure 4 This is a schematic diagram of the network model compression device provided by the present invention;

[0053] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0055] Recurrent Neural Networks (RNNs) are a type of neural network used to process sequential data. While they have achieved good results in fields such as speech recognition, speech models, and robotics due to their ability to extract temporal information from sequential data, RNNs suffer from gradient explosion and gradient vanishing problems as the length of the sequential data increases. Considering that the Long Short-Term Memory (LSTM) model is an RNN with added gating mechanisms, it can be used to solve the gradient explosion and gradient vanishing problems that occur during RNN training. However, the LSTM model introduces a huge number of parameters and computational costs during use, which increases the deployment requirements of embedded devices.

[0056] For hardware acceleration and model compression of LSTM models, pruning is commonly used to remove unimportant weights by setting them to zero. However, pruning can introduce sparse matrices, increasing the difficulty of storing these weights. Alternatively, gated recurrent neural networks (GRUs) with fewer weights can address the vanishing and exploding gradient problems. However, when the training dataset is large, GRUs are not more effective than LSTMs in resolving these issues. Therefore, reducing the number of parameters and computational cost of LSTM models, as well as simplifying parameter storage, becomes crucial.

[0057] Based on this, the present invention provides a network model compression method, apparatus, electronic device, and storage medium. The execution entity of the network model compression method can be a terminal device or a server. The terminal device can be a personal computer (PC), portable device, laptop, smartphone, tablet computer, portable wearable device, or other electronic device. The server can be a standalone server or a server cluster composed of multiple servers. For example, the server can be a physical server containing independent hosts, a virtual server hosted by a host cluster, a cloud server, etc. The present invention does not limit the specific form of the terminal device, nor does it specifically limit the specific form of the server.

[0058] The following is combined Figures 1-5 The network model compression method, apparatus, electronic device, and storage medium of the present invention are described below. The following method embodiments are illustrated with a terminal device as the execution subject, and the execution subject of the following method embodiments may be part or all of the terminal device.

[0059] Figure 1 This is a flowchart illustrating the network model compression method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the network model compression method includes the following steps:

[0060] Step 110: Obtain the initial weight parameters and initial bias parameters of the network model to be compressed.

[0061] Specifically, obtaining the initial weight parameters and initial bias parameters of the network model to be compressed can be achieved by training the neural network model using a sample dataset and stopping training when the number of training epochs reaches a preset threshold. The neural network model corresponding to the point where training stops is then identified as the network model to be compressed. At this point, the model parameters of the network model to be compressed include the input layer weight parameters, input layer bias parameters, hidden layer weight parameters, and hidden layer bias parameters. The input layer weight parameters and hidden layer weight parameters are used as the initial weight parameters of the network model to be compressed, and the input layer bias parameters and hidden layer bias parameters are used as the initial bias parameters of the network model to be compressed.

[0062] Step 120: Based on the initial weight parameters and the initial bias parameters, determine the target inference model. The target inference model is used to infer the state of different gates in the network model to be compressed at different times.

[0063] Specifically, the terminal device can construct a target inference model for the network model to be compressed based on the obtained input layer weight parameters, input layer bias parameters, hidden layer weight parameters, and hidden layer bias parameters. The target inference model is used to perform forward inference based on the input layer weight parameters, input layer bias parameters, hidden layer weight parameters, hidden layer bias parameters, and sample dataset, thereby inferring the state of different gates in the network model to be compressed at different times.

[0064] It should be noted that the sample dataset can be an image dataset, text dataset, speech dataset, or video dataset, depending on the application scenario of the network model to be compressed. This invention does not specifically limit the type of sample dataset. Furthermore, the sample dataset can be an existing dataset or obtained through data collection based on the application scenario. This invention also does not specifically limit the method of obtaining the sample dataset.

[0065] Step 130: Prune the target inference model using preset gates, and determine the target weight parameters and target bias parameters of the pruned model.

[0066] Specifically, since neural network models typically contain different gates, the different values ​​of the weight and bias parameters in the model parameters, as well as the different input data at different times, will all affect the state of the corresponding gates at different times. Therefore, the state of different gates at different times can be inferred by reasoning the constructed target inference model, so as to determine the way of lightweighting the model. For example, the preset gates in the target inference model can be pruned to achieve the purpose of lightweighting the model and reducing the model parameters, and further determine the target weight parameters and target bias parameters of the pruned model.

[0067] Step 140: Determine the target network model based on the target weight parameters and target bias parameters.

[0068] Specifically, since the target weight parameters and target bias parameters are determined based on the pruned model, they can be used as parameters of the model obtained after compressing the network model to be compressed. In other words, the target network model can be constructed based on the target weight parameters and target bias parameters. The target network model can not only reflect the significant reduction in model parameters, but also the new states of the remaining gates in the target inference model at different times, excluding the pruned preset gates, thereby achieving the purpose of model compression.

[0069] The network model compression method provided in this invention first obtains the initial weight parameters and initial bias parameters of the network model to be compressed. Then, based on the initial weight parameters and initial bias parameters, a target inference model is determined. Since the target inference model is used to infer the states of different gates in the network model to be compressed at different times, it is possible to combine inference techniques and model construction analysis to select preset gates that can be pruned, and to prune the target inference model using preset gates. Finally, based on the target weight parameters and target bias parameters determined by the pruned model, the compressed target network model is determined. By generating a lightweight inference model through pruning the inference model, the compressed target network model is determined. This not only avoids the drawbacks of increasing the number of model parameters and computational load, as well as the increased storage difficulty caused by introducing sparse matrices, but also achieves the goal of reducing computational load and improving inference speed.

[0070] Optionally, the specific implementation process of step 110 may include:

[0071] First, the initial LSTM model is trained using a sample dataset to obtain the network model to be compressed. Then, the input layer weight parameters, input layer bias parameters, hidden layer weight parameters, and hidden layer bias parameters of the network model to be compressed are obtained. Then, the input layer weight parameters and hidden layer weight parameters are determined as the initial weight parameters of the network model to be compressed, and the input layer bias parameters and hidden layer bias parameters are determined as the initial bias parameters of the network model to be compressed.

[0072] Specifically, the initial LSTM model is trained using a sample dataset. First, the dataset is divided into two subsets: one for training samples and the other for test samples. The LSTM model is then trained using the training samples. Once all training samples have been used in one training pass, the initial LSTM model has completed one round of training. The intermediate network model obtained after this first training pass is then tested using the test samples. The accuracy of the test results is then compared to the actual accuracy of the test samples. If the accuracy matches, the intermediate network model at that point is considered the trained model to be compressed. Conversely, if the accuracy does not match, an updated intermediate network model is obtained, and the updated model is then trained and tested again using the sample dataset. This process continues until the accuracy of the test results matches the actual accuracy of the test samples.

[0073] Furthermore, since the network model to be compressed is obtained by training an LSTM model, its network structure is the same as that of the LSTM model, i.e., it contains input gates, forget gates, memory cells, and output gates. Therefore, the input layer weight parameters and input layer bias parameters, as well as the hidden layer weight parameters and hidden layer bias parameters of the network model to be compressed, will also exist in the input gate, forget gate, memory cell, and output gate, respectively. Thus, the initial weight parameters of the network model to be compressed include the initial weight parameters of the input layer at the input gate, the initial weight parameters of the hidden layer at the input gate, the initial weight parameters of the input layer at the forget gate, and the initial weight parameters of the hidden layer at the forget gate. The initial weight parameters of the forget gate, the initial weight parameters of the input layer in the memory cells, the initial weight parameters of the hidden layer in the memory cells, the initial weight parameters of the input layer in the output gate, and the initial bias parameters of the hidden layer in the output gate are included. The initial bias parameters of the network model to be compressed include the initial bias parameters of the input layer in the input gate, the initial bias parameters of the hidden layer in the input gate, the initial bias parameters of the input layer in the forget gate, the initial bias parameters of the hidden layer in the forget gate, the initial bias parameters of the input layer in the memory cells, the initial bias parameters of the hidden layer in the memory cells, the initial bias parameters of the input layer in the output gate, and the initial bias parameters of the hidden layer in the output gate.

[0074] The network model compression method provided in this invention obtains the initial weight parameters and initial bias parameters of the network model to be compressed by training an LSTM model. By combining this with the network structure and model training of the LSTM model, the accuracy and reliability of obtaining the weight parameters and bias parameters can be improved, and the foundation can be laid for improving the forward inference speed in the future.

[0075] Optionally, the specific implementation of step 120 may include: inferring the input gate state, forget gate state, memory cell state, output gate state, cell state, and hidden state at time t using the following formula:

[0076] i t =sigmoid(W ii x t +b ii +W hi h t-1 +b hi );

[0077] f t =sigmoid(W if x t +b if +W hf h t-1 +b hf );

[0078] g t =tanh(Wig x t +b ig +W hg h t-1 +b hg );

[0079] o t =sigmoid(W io x t +b io +W ho h t-1 +b ho );

[0080] c t =f t ⊙c t-1 +i t ⊙g t ;

[0081] h t =o t ⊙tanh(c t );

[0082] Among them, i t Let f be the input gate state at time t. t Let g be the forget gate state at time t. t Let o be the state of the memory cell at time t. t Let c be the output gate state at time t. t Let h be the cell state at time t. t Let W be the hidden state at time t. ii b represents the initial weight parameters of the input layer at the input gate. ii W represents the initial bias parameters of the input layer at the input gate. hi b represents the initial weight parameters of the hidden layer at the input gate. hi x represents the initial bias parameters of the hidden layer at the input gate. t Let h be the input data at time t. t-1 Let W be the hidden state at time t-1. if W represents the initial weight parameters of the input layer at the forget gate. hf b represents the initial weight parameters of the hidden layer at the forget gate. if b represents the initial bias parameters of the input layer at the forget gate. hf W represents the initial bias parameters of the hidden layer at the forget gate. ig W represents the initial weight parameters of the input layer in the memory cells. hg b represents the initial weight parameters of the hidden layer in the memory cells. ig b represents the initial bias parameters of the input layer in the memory cells. hg W represents the initial bias parameters of the hidden layer in the memory cells.io W represents the initial weight parameters of the input layer at the output gate. ho b represents the initial weight parameters of the hidden layer at the output gate. io b represents the initial bias parameters of the input layer at the output gate. ho c represents the initial bias parameters of the hidden layer at the output gate. t-1 represents the cell state at time t-1; sigmoid is the activation function operator, and tanh is the tangent function operator.

[0083] Specifically, the process of building a target inference model for the network model to be compressed, based on the initial weight parameters and initial bias parameters of the network model to be compressed, can be as follows: Figure 2 As shown, in Figure 2 In the sample dataset, the input data x at time t is used. t The input gate (i), forget gate (f), memory cell (g), and output gate (o) of the network model to be compressed, as well as the corresponding operations (e.g., ...) through different gates. Figure 2 The tangent, activation, addition, and multiplication operations shown can be based on the cell state c at time t-1. t-1 and the hidden state h at time t-1 t-1 Infer the input gate state i at time t t The forget gate state f at time t t The state of memory cells at time t t The output gate state at time t t Based on this, it can be determined that the target inference model is specifically a forward inference model for the network model to be compressed.

[0084] The network model compression method provided in this invention constructs a target inference model by inferring the input gate state, forget gate state, memory cell state, output gate state, cell state, and hidden state at time t using the input data at time t and the corresponding operations of the input gate, forget gate, memory cell, and output gate. This improves the inference speed and provides a theoretical basis for subsequent compressed models.

[0085] Optionally, the specific implementation process of step 130 may include:

[0086] First, based on the target inference model and the sample dataset, a fitting dataset for the preset time series is determined. The fitting dataset includes the input data at different current times, the hidden states at different previous times, the cell states at different previous times, the cell states at different current times, and the hidden states at different current times in the preset time series. Then, the input and output gates of the target inference model are further pruned. Next, the initial fully connected layer is trained based on the fitting dataset to determine the target fully connected layer. Finally, based on the parameters of the target fully connected layer, the target weight parameters and target bias parameters of the pruned model are determined.

[0087] Specifically, based on Figure 2 The target inference model shown can determine the fitting dataset for a preset time series. The preset time series can include multiple different times, and the number of times can be set manually according to actual conditions; no specific limit is made here. Based on this, the fitting dataset can include the input data x at time t. t The hidden state h at time t-1 t-1 Cell state at time t-1 t-1 Cell state c at time t t and the hidden state h at time t t t = 1, 2, ..., T, where T is the maximum time in the preset time series.

[0088] Furthermore, by pruning the input and output gates of the target inference model, it is possible to... Figure 2 The target inference model shown in the diagram is processed by pruning the input gate, its corresponding sigmoid activation function, and its corresponding multiplication operation, and by separately acquiring the output gate, its corresponding sigmoid activation function, and its corresponding multiplication operation, thereby obtaining the following... Figure 3 The model shown is obtained by cutting out the material. Figure 3 In the cropped model shown, the dashed lines represent the cropped portion. The initial fully connected layer is trained using the fitted dataset. The model at the end of training is the target fully connected layer. The parameters of the target fully connected layer are used as new weight parameters and new bias parameters for the forget gate and memory cells, thereby determining the target weight parameters and target bias parameters of the cropped model.

[0089] The network model compression method provided in this invention determines the target fully connected layer by training the initial fully connected layer with a determined fitting dataset, and determines the target weight parameters and target bias parameters of the target inference model obtained by pruning the model through the input and output gates based on the parameters of the target fully connected layer. By combining weight parameter fitting and fully connected layer fitting, the method aims to reduce the model parameters by half, while also significantly reducing the amount of computation and improving the operation speed.

[0090] Optionally, the initial fully connected layer is trained based on the fitted dataset to determine the target fully connected layer, including:

[0091] First, the input data at different current times, the hidden states at different previous times, and the cell states at different previous times are determined as the sample input set, and the cell states at different current times and the hidden states at different current times are determined as the sample output set. Then, the initial fully connected layer is trained using the sample input set to determine the new cell states and new hidden states at different times. Finally, the new cell states and new hidden states are validated using the sample output set to determine the target fully connected layer.

[0092] Specifically, given the input data x at time t... t The hidden state h at time t-1 t-1 and cell state c at time t-1 t-1 Given the sample input set, determine the cell state c at time t. t and the hidden state h at time t t Given a sample output set, t = 1, 2, ..., T, where T is the maximum time in the preset time series, the input data x at time t is used. t The hidden state h at time t-1 t-1 and cell state c at time t-1 t-1 After training the initial fully connected layer for the tth time, the new cell state c at time t can be output at the end of the tth training. t 'and the new hidden state h at time t t ', and then based on the cell state c at time t t and the new cell state c at time t t ', and the hidden state h at time t. t and the new hidden state h at time t t 'Calculate the mean squared error of the hidden state at time t and the mean squared error of the cell state at time t. The method for calculating the mean squared error can refer to existing methods for calculating mean squared error, and will not be elaborated here. Further, determine whether the closeness between the mean squared error of the hidden state at time t and the mean squared error of the cell state at time t meets a preset requirement. If it does, then determine the fully connected layer corresponding to the preset requirement as the trained target fully connected layer; otherwise, if it does not meet the requirement, then adjust the parameters of the fully connected layer after the t-th training, increment the value of t by 1, and repeat the process using the input data x at time t. t The hidden state h at time t-1 t-1 and cell state c at time t-1 t-1 Perform the t-th training iteration and use the new cell state c at time t, output at the end of the t-th training iteration. t 'and the new hidden state h at time tt The process of calculating the mean square error of the hidden state and the mean square error of the cell state at time t; until the mean square error of the hidden state and the mean square error of the cell state are close enough to meet the preset requirements.

[0093] The network model compression method provided in this invention determines the trained target fully connected layer by training the initial fully connected layer using a sample training set of a fitted dataset and validating the new cell states and new hidden states at different time points using a sample output set. This method, combined with a simulated dataset and training the fully connected layer using mean squared error as the loss function, improves the training accuracy and reliability of the fully connected layer and further enhances the stability and accuracy of the compressed model.

[0094] Optionally, based on the target weight parameters and target bias parameters, the target network model is determined. This process may include determining the new forget gate state, new memory cell state, new cell state, and new hidden state at time t using the following formula:

[0095] f t = sigmoid(W if 'x t +b if '+W hf 'h t-1 '+b hf ');

[0096] g t '=tanh(W ig 'x t +b ig '+W hg 'h t-1 '+b hg ');

[0097] c t '=f t '⊙c t-1 '+g t ';

[0098] h t =tanh(c t ');

[0099] Among them, f t 'G is the new forget gate state at time t, g t 'C represents the new memory cell state at time t.' t 'H represents the new cell state at time t.' t Let x be the new hidden state at time t. t Let h be the input data at time t. t-1′ represents the new hidden state at time t-1, W if ′ represents the target weight parameters of the input layer at the forget gate, W ig ′ represents the target weight parameter of the input layer in the memory cell, b if ′ represents the target bias parameter of the input layer at the forget gate, b ig ′ represents the target bias parameter of the input layer in the memory cell, W hf ′ represents the target weight parameter of the hidden layer at the forget gate, W hg ′ represents the target weight parameter of the hidden layer in the memory cell, b hf ′ represents the target bias parameter of the hidden layer at the forget gate, b hg ′ represents the target bias parameter of the hidden layer in the memory cell, and c t-1 ′ represents the new cell state at time t-1.

[0100] Specifically, based on Figure 3 The cropped model shown can be rebuilt into the forward inference model to obtain the target network model. The target weight parameters and target bias parameters of the target network model are the parameters of the cropped model, including the target weight parameter W of the input layer at the forget gate. if The input layer has target weight parameters W for memory cells. ig The input layer's target bias parameter b at the forget gate if The input layer has a target bias parameter b in the memory cell. ig The target weight parameter W of the hidden layer in the forget gate hf The hidden layer has target weight parameters W for memory cells. hg The target bias parameter b of the hidden layer at the forget gate hf The target bias parameter b of the hidden layer and the memory cell. hg The new forget gate state f at time t is inferred by using the target weight parameter and the target bias parameter. t The new memory cell state g at time t' t The new cell state c at time t' t The new hidden state h at time t and t t The '' algorithm can be used to evaluate the performance of the compressed network model, that is, to evaluate the performance of the target network model compressed from the network model to be compressed.

[0101] The network model compression method provided in this invention constructs a forward inference model for the network model to be compressed by using target weight parameters and target bias parameters. This determines the new forget gate state, new memory cell state, new cell state, and new hidden state at time t. By combining weight fitting and forward inference methods, the computation speed of the compressed model is improved, achieving the goal of reducing the number of parameters by half and reducing the amount of computation.

[0102] The network model compression device provided by the present invention is described below. The network model compression device described below can be referred to in correspondence with the network model compression method described above.

[0103] Figure 4 An example is a schematic diagram of a network model compression device, such as... Figure 4 As shown, the network model compression device 400 includes:

[0104] The acquisition module 410 is used to acquire the initial weight parameters and initial bias parameters of the network model to be compressed;

[0105] The first determining module 420 is used to determine the target inference model based on the initial weight parameters and the initial bias parameters. The target inference model is used to infer the state of different gates in the network model to be compressed at different times.

[0106] The pruning module 430 is used to prune the target inference model using preset gates and determine the target weight parameters and target bias parameters of the pruned model.

[0107] The second determination module 440 is used to determine the target network model based on the target weight parameters and the target bias parameters.

[0108] Optionally, module 410 can be used to train an initial LSTM model using a sample dataset to obtain the network model to be compressed; obtain the input layer weight parameters, input layer bias parameters, hidden layer weight parameters, and hidden layer bias parameters of the network model to be compressed; determine the input layer weight parameters and hidden layer weight parameters as the initial weight parameters of the network model to be compressed; and determine the input layer bias parameters and hidden layer bias parameters as the initial bias parameters of the network model to be compressed.

[0109] Optionally, the first determining module 420 can be used to infer the input gate state, forget gate state, memory cell state, output gate state, cell state, and hidden state at time t using the following formula:

[0110] i t =sigmoid(W ii x t +b ii +W hi h t-1 +b hi (1);

[0111] f t =sigmoid(W if x t +b if +W hf h t-1 +bhf (2);

[0112] g t =tanh(W ig x t +b ig +W hg h t-1 +b hg (3);

[0113] o t =sigmoid(W io x t +b io +W ho h t-1 +b ho (4);

[0114] c t =f t ⊙c t-1 +i t ⊙g t (5);

[0115] h t =o t ⊙tanh(c t (6);

[0116] In equations (1) to (6), i t Let f be the input gate state at time t. t Let g be the forget gate state at time t. t Let o be the state of the memory cell at time t. t Let c be the output gate state at time t. t Let h be the cell state at time t. t Let W be the hidden state at time t. ii b represents the initial weight parameters of the input layer at the input gate. ii W represents the initial bias parameters of the input layer at the input gate. hi b represents the initial weight parameters of the hidden layer at the input gate. hi x represents the initial bias parameters of the hidden layer at the input gate. t Let h be the input data at time t. t-1 Let W be the hidden state at time t-1. if W represents the initial weight parameters of the input layer at the forget gate. hf b represents the initial weight parameters of the hidden layer at the forget gate. if b represents the initial bias parameters of the input layer at the forget gate. hf W represents the initial bias parameters of the hidden layer at the forget gate. ig W represents the initial weight parameters of the input layer in the memory cells.hg b represents the initial weight parameters of the hidden layer in the memory cells. ig b represents the initial bias parameters of the input layer in the memory cells. hg W represents the initial bias parameters of the hidden layer in the memory cells. io W represents the initial weight parameters of the input layer at the output gate. ho b represents the initial weight parameters of the hidden layer at the output gate. io b represents the initial bias parameters of the input layer at the output gate. ho c represents the initial bias parameters of the hidden layer at the output gate. t-1 This represents the cell state at time t-1.

[0117] Optionally, the pruning module 430 can be used to determine a fitting dataset for a preset time series based on the target inference model and the sample dataset. The fitting dataset includes input data at different current times, hidden states at different previous times, cell states at different previous times, cell states at different current times, and hidden states at different current times in the preset time series; prune the input and output gates of the target inference model; train the initial fully connected layer based on the fitting dataset to determine the target fully connected layer; and determine the target weight parameters and target bias parameters of the pruned model based on the parameters of the target fully connected layer.

[0118] Optionally, the pruning module 430 can also be used to determine the input data at different current times, the hidden states at different previous times, and the cell states at different previous times as sample input sets, and to determine the cell states at different current times and the hidden states at different current times as sample output sets; to train the initial fully connected layer using the sample input sets, to determine the new cell states at different times and the new hidden states at different times; and to validate the new cell states and the new hidden states using the sample output sets, to determine the target fully connected layer.

[0119] Optionally, the second determining module 440 can be used to determine the new forget gate state, the new memory cell state, the new cell state, and the new hidden state at time t using the following formula:

[0120] f t = sigmoid(W) if ′x t +b if ′+W hf ′h t-1 ′+b hf ′) (7);

[0121] g t ′=tanh(W ig ′x t +b ig ′+Whg ′h t-1 ′+b hg ′) (8);

[0122] c t ′=f t ′⊙c t-1 ′+g t ′ (9);

[0123] h t ′=tanh(c t ′) (10);

[0124] In equations (7) to (10), f t 'G is the new forget gate state at time t, g t ′ represents the new memory cell state at time t, c t ′ represents the new cell state at time t, h t Let ' be the new hidden state at time t, and x t Let h be the input data at time t. t-1 ′ represents the new hidden state at time t-1, W if ′ represents the target weight parameters of the input layer at the forget gate, W ig ′ represents the target weight parameter of the input layer in the memory cell, b if ′ represents the target bias parameter of the input layer at the forget gate, b ig ′ represents the target bias parameter of the input layer in the memory cell, W hf ′ represents the target weight parameter of the hidden layer at the forget gate, W hg ′ represents the target weight parameter of the hidden layer in the memory cell, b hf ′ represents the target bias parameter of the hidden layer at the forget gate, b hg ′ represents the target bias parameter of the hidden layer in the memory cell, and c t-1 ′ represents the new cell state at time t-1.

[0125] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device 500 may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a network model compression method, which includes:

[0126] Obtain the initial weight parameters and initial bias parameters of the network model to be compressed;

[0127] Based on the initial weight parameters and initial bias parameters, the target inference model is determined. The target inference model is used to infer the state of different gates in the network model to be compressed at different times.

[0128] The target inference model is pruned using preset gates, and the target weight parameters and target bias parameters of the pruned model are determined.

[0129] The target network model is determined based on the target weight parameters and the target bias parameters.

[0130] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0131] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the network model compression method provided by the above methods, the method comprising:

[0132] Obtain the initial weight parameters and initial bias parameters of the network model to be compressed;

[0133] Based on the initial weight parameters and initial bias parameters, the target inference model is determined. The target inference model is used to infer the state of different gates in the network model to be compressed at different times.

[0134] The target inference model is pruned using preset gates, and the target weight parameters and target bias parameters of the pruned model are determined.

[0135] The target network model is determined based on the target weight parameters and the target bias parameters.

[0136] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the network model compression method provided by the methods described above, the method comprising:

[0137] Obtain the initial weight parameters and initial bias parameters of the network model to be compressed;

[0138] Based on the initial weight parameters and initial bias parameters, the target inference model is determined. The target inference model is used to infer the state of different gates in the network model to be compressed at different times.

[0139] The target inference model is pruned using preset gates, and the target weight parameters and target bias parameters of the pruned model are determined.

[0140] The target network model is determined based on the target weight parameters and the target bias parameters.

[0141] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0142] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0143] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A network model compression method, characterized in that, Applied to embedded devices, the method includes: Obtain the initial weight parameters and initial bias parameters of the network model to be compressed. The network model to be compressed is used to process image data, text data, speech data, or video data. The network model to be compressed is obtained by training an initial LSTM model using a sample dataset. Based on the initial weight parameters and the initial bias parameters, a target inference model is determined, which is used to infer the state of different gates in the network model to be compressed at different times. The target inference model is pruned using preset gates, and the target weight parameters and target bias parameters of the pruned model are determined, including: determining a fitting dataset for a preset time series based on the target inference model and the sample dataset, wherein the fitting dataset includes input data at different current times, hidden states at different previous times, cell states at different previous times, cell states at different current times, and hidden states at different current times in the preset time series; pruning the input and output gates of the target inference model; training an initial fully connected layer based on the fitting dataset to determine a target fully connected layer; and determining the target weight parameters and target bias parameters of the pruned model based on the parameters of the target fully connected layer. Based on the target weight parameters and the target bias parameters, a target network model is determined to reduce the parameter storage and computational load of the embedded device.

2. The network model compression method according to claim 1, characterized in that, The process of obtaining the initial weight parameters and initial bias parameters of the network model to be compressed includes: Obtain the input layer weight parameters, input layer bias parameters, hidden layer weight parameters, and hidden layer bias parameters of the network model to be compressed; The input layer weight parameters and the hidden layer weight parameters are determined as the initial weight parameters of the network model to be compressed; The input layer bias parameters and the hidden layer bias parameters are determined as the initial bias parameters of the network model to be compressed.

3. The network model compression method according to claim 1, characterized in that, The step of training the initial fully connected layer based on the fitted dataset to determine the target fully connected layer includes: The input data at different current times, the hidden states at different previous times, and the cell states at different previous times are determined as the sample input set, and the cell states at different current times and the hidden states at different current times are determined as the sample output set. The initial fully connected layer is trained using the sample input set to determine the new cell states and new hidden states at different times. The new cell state and the new hidden state are validated using the sample output set to determine the target fully connected layer.

4. The network model compression method according to any one of claims 1-3, characterized in that, The determination of the target inference model based on the initial weight parameters and the initial bias parameters includes inferring the input gate state, forget gate state, memory cell state, output gate state, cell state, and hidden state at time t using the following formula: ； ； ； ； ； ； in, Let be the input gate state at time t. Let t be the forget gate state. Let t represent the state of the memory cells at time t. The output gate state at time t. Let t represent the cell state at time t. Let be the hidden state at time t. These are the initial weight parameters of the input layer at the input gate. These are the initial bias parameters of the input layer at the input gate. These are the initial weight parameters of the hidden layer at the input gate. These are the initial bias parameters of the hidden layer at the input gate. The input data is at time t. The hidden state at time t-1 These are the initial weight parameters for the input layer at the forget gate. These are the initial weight parameters for the hidden layer at the forget gate. These are the initial bias parameters for the input layer at the forget gate. These are the initial bias parameters for the hidden layer at the forget gate. These are the initial weight parameters for the input layer in the memory cells. These are the initial weight parameters for the hidden layer in the memory cells. These are the initial bias parameters for the input layer in the memory cells. These are the initial bias parameters of the hidden layer in the memory cells. These are the initial weight parameters of the input layer at the output gate. These are the initial weight parameters of the hidden layer at the output gate. These are the initial bias parameters of the input layer at the output gate. These are the initial bias parameters of the hidden layer at the output gate. This represents the cell state at time t-1.

5. The network model compression method according to any one of claims 1-3, characterized in that, The determination of the target network model based on the target weight parameters and the target bias parameters includes determining the new forget gate state, new memory cell state, new cell state, and new hidden state at time t using the following formula: ； ； ； ； in, Let t be the new forget gate state. This represents the new state of the memory cell at time t. This represents the new cell state at time t. Let be the new hidden state at time t. The input data is at time t. This represents the new hidden state at time t-1. The target weight parameters of the input layer at the forget gate, The target weight parameters for the input layer in the memory cells, The target bias parameters of the input layer at the forget gate. The input layer is the target bias parameter in the memory cell. The target weight parameters of the hidden layer at the forget gate. The target weight parameters of the hidden layer in the memory cells, The target bias parameter of the hidden layer at the forget gate. The target bias parameters of the hidden layer in the memory cell. This represents the new cell state at time t-1.

6. A network model compression device, characterized in that, Applied to embedded devices, the device includes: The acquisition module is used to acquire the initial weight parameters and initial bias parameters of the network model to be compressed. The network model to be compressed is used to process image data, text data, speech data or video data. The network model to be compressed is obtained by training an initial LSTM model using a sample dataset. The first determining module is used to determine the target inference model based on the initial weight parameters and the initial bias parameters. The target inference model is used to infer the state of different gates in the network model to be compressed at different times. The pruning module is used to prune the target inference model by setting preset gates and determine the target weight parameters and target bias parameters of the pruned model. This includes: determining a fitting dataset for a preset time series based on the target inference model and the sample dataset; the fitting dataset including input data at different current times, hidden states at different previous times, cell states at different previous times, cell states at different current times, and hidden states at different current times in the preset time series; pruning the input and output gates of the target inference model; training an initial fully connected layer based on the fitting dataset to determine a target fully connected layer; and determining the target weight parameters and target bias parameters of the pruned model based on the parameters of the target fully connected layer. The second determining module is used to determine the target network model based on the target weight parameters and the target bias parameters, so as to reduce the parameter storage and computational load of the embedded device.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the network model compression method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the network model compression method as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the network model compression method as described in any one of claims 1 to 5.

Images