Online training method and system based on fine-grained structured sparsity

Abstract

The present invention proposes an online training method and system based on fine-grained structured sparsity, comprising: training a dense model based on a preset learning rate and weight gradient; performing fine-grained structured sparsity on the weights of the dense model to obtain a sparse mask, and training a sparse model based on the sparse mask, and updating the sparse mask and the model weights of the sparse model during the training of the sparse model; fixing the network structure of the sparse model and retraining the sparse model, and only updating the model weights of the sparse model during the retraining of the sparse model, and saving the model weights and sparse mask of the retrained sparse model as the online training result. The sparse model is smaller in scale than the dense model and has fewer channels than the dense model. When a sparse model is used when deploying a model, the amount of calculation can be effectively reduced due to the small scale of the sparse model, thereby achieving the technical effect of accelerating the running speed of the model.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence accelerated computing and image classification technology, and in particular to an online training method and system based on fine-grained structured sparsity. Background Art

[0002] Deep neural networks have achieved tremendous success in fields such as computer vision and natural language processing. However, as deep neural networks grow in size, the number of parameters and computational complexity increases, restricting their use on resource-constrained hardware platforms, such as edge and mobile devices. Network sparsification technology can effectively reduce the number of parameters in a deep neural network by pruning neural network weights based on specific rules, while only causing a slight loss in accuracy. Therefore, network sparsification technology is of great significance to the application of deep neural networks in scenarios such as edge and mobile devices.

[0003] Current network sparsification techniques focus on two aspects: sparsity granularity and training methods.

[0004] In terms of sparsity granularity, existing network sparsification technologies are mainly divided into two categories: fine-grained sparsification and coarse-grained sparsification. Fine-grained sparsification is to sparsify the weights at different positions in the neural network. Generally, the accuracy loss of the model after sparsification is very small, or even no accuracy loss occurs. However, the sparsification method of fine-grained sparsification is unstructured, and the sparsified model parameters are not hardware-friendly. There are some problems during deployment, which will restrict the actual performance improvement on the hardware. Coarse-grained sparsification is to sparsify the entire convolution kernel of the neural network weights. It is a structured sparsification method, so the performance improvement after actual hardware deployment is more obvious. However, the sparsification method of coarse-grained sparsification will lose important weights in the convolution kernel when sparsifying the entire convolution kernel. Therefore, the accuracy of the model after coarse-grained sparsification is generally slightly lower than that of fine-grained sparsification.

[0005] In terms of training methods, existing sparse technologies are mainly divided into two types: offline training and online training. Offline training is also called a two-stage training method. In the first stage, a dense model is trained, and then the sparse structure of the model is obtained using specific rules based on the values ​​of the model weights obtained through training. In the second stage, the sparse structure model is fine-tuned to finally obtain the sparse model weights, that is, a sparse mask (sparse structure) is obtained after the first stage, and model weights are obtained after the second stage. Online training is also called a one-stage training method. It does not require training a dense model, but directly trains a sparse model. The model weights and sparse masks are continuously updated during the training process. After the training, the sparse mask and model weights are obtained at the same time. In comparison, the offline training method takes a longer time to train, but may obtain a sparse model with higher accuracy than online training.

[0006] Current sparsification techniques are primarily used to improve model performance during inference. However, how to leverage sparsification techniques to improve performance during training has yet to be fully addressed. Summary of the Invention

[0007] To address the problems of high precision but poor performance in the aforementioned fine-grained sparse approach, and good performance but low precision in the coarse-grained sparse approach, this paper focuses on fine-grained structured sparse technology. This technology groups and sparses convolution weights during sparsification, such as grouping four consecutive weights and sparsifying two of them. This approach can retain more important weights like fine-grained sparse, while also making the sparsified model structured and more hardware-friendly. Therefore, fine-grained structured sparse technology can strike a balance between precision and performance.

[0008] In view of the shortcomings of the existing technology, the present invention proposes an online training method based on fine-grained structured sparsity, which includes:

[0009] Step 1: Train the dense model based on the preset learning rate and weight gradient;

[0010] Step 2: Perform fine-grained structured sparseness on the weights of the dense model to obtain a sparse mask, and train a sparse model based on the sparse mask. During the training of the sparse model, the sparse mask and the model weights of the sparse model are updated.

[0011] Step 3: Fix the network structure of the sparse model and train the sparse model again. During the retraining process of the sparse model, only the model weights of the sparse model are updated, and the model weights and sparse masks of the retrained sparse model are saved as the online training results.

[0012] In the online training method based on fine-grained structured sparsity, step 1 includes training and updating the weights of the dense model based on the following formula:

[0013]

[0014] The weight of the model is W dense , the learning rate is l, and the gradient of the weight is

[0015] The online training method based on fine-grained structured sparsity, wherein the sparse mask of the dense model in step 1 is an all-1 matrix.

[0016] The online training method based on fine-grained structured sparsity, wherein step 2 includes:

[0017] Step 21: Get the weight W of the dense model dense and learning rate l;

[0018] Step 22: For W dense , use multiple preset templates to calculate the cumulative sum of the absolute values ​​of each group of weights under each template, take the template with the largest cumulative sum among all preset templates as the sparse mask m of the group of weights, and get the sparse weight W unpruned =W dense ⊙m, where ⊙ represents element-by-element multiplication, and the sparsely populated portion of the weight is recorded as W pruned =0;

[0019] Step 23: Using sparse weights W unpruned Perform the forward propagation of the network. The forward propagation formula is:

[0020] x i+1 =x i *W unpruned

[0021] Where x i is the input of the i-th layer in the network;

[0022] Step 24: The network performs back propagation, using the topdiff propagated from the i+1th layer in the network, recorded as Calculate the bottomdiff of layer i, denoted as and the gradient of the weights The formula is:

[0023]

[0024]

[0025] Step 25: According to the gradient of weight And the current sparse mask updates the sparse weight W unpruned and the sparse part of the weight W pruned , the formula is:

[0026]

[0027]

[0028] Determine whether this is the end of an epoch. If so, execute step 26; otherwise, execute step 23 again.

[0029] Step 26: Using sparse weights W unpruned and the sparse part of the weight W pruned To update the sparse mask, first W unpruned and W pruned Add up to get W dense , and then use all preset templates to calculate W respectively denseThe cumulative sum of the absolute values ​​of the weights of each group of weights under each preset template is taken as the sparse mask of the group of weights, thereby obtaining the updated sparse mask m;

[0030] Step 27: Update W using sparse mask m unpruned and W pruned , the formula is

[0031] W unpruned =W dense ⊙m

[0032] W pruned =0

[0033] Determine whether the set number of iterations has been reached. If so, execute step 3; otherwise, execute step 23 again.

[0034] This step 3 involves fixing the sparse mask and updating the sparse model weights by:

[0035]

[0036] The online training method based on fine-grained structured sparsity, wherein the dense model in step 1 is a convolutional neural network for image classification; step 3 includes constructing a sparse image classification network based on the online training results to classify the images to be classified.

[0037] The present invention also proposes an online training system based on fine-grained structured sparsity, comprising:

[0038] The inception module is used to train dense models with preset learning rates and weight gradients;

[0039] A first training module is configured to perform fine-grained structured sparsification on the weights of the dense model to obtain a sparse mask, train a sparse model based on the sparse mask, and update the sparse mask and the model weights of the sparse model during the training of the sparse model;

[0040] The second training module is used to fix the network structure of the sparse model and retrain the sparse model, and only update the model weights of the sparse model during the retraining of the sparse model, and save the model weights and sparse masks of the retrained sparse model as the online training results.

[0041] The online training system based on fine-grained structured sparsity, wherein the initial module includes training and updating the weights of the dense model based on the following formula:

[0042]

[0043] The weight of the model is W dense, the learning rate is l, and the gradient of the weight is

[0044] The online training system based on fine-grained structured sparsity, wherein the sparse mask of the dense model in the initial module is an all-1 matrix.

[0045] In the online training system based on fine-grained structured sparsity, the first training module is used to:

[0046] Get the weight W of the dense model dense and learning rate l;

[0047] For W dense , use multiple preset templates to calculate the cumulative sum of the absolute values ​​of each group of weights under each template, take the template with the largest cumulative sum among all preset templates as the sparse mask m of the group of weights, and get the sparse weight W unpruned =W dense ⊙m, where ⊙ represents element-by-element multiplication, and the sparsely populated portion of the weight is recorded as W pruned =0;

[0048] Using sparse weights W unpruned Perform the forward propagation of the network. The forward propagation formula is:

[0049] x i+1 =x i *W unpruned

[0050] Where x i is the input of the i-th layer in the network;

[0051] The network performs back propagation, using the topdiff propagated from the i+1th layer in the network, denoted as Calculate the bottomdiff of the i-th layer, denoted as and the gradient of the weights The formula is:

[0052]

[0053]

[0054] According to the gradient of weight And the current sparse mask updates the sparse weight W unpruned and the sparse part of the weight W pruned , the formula is:

[0055]

[0056]

[0057] Using sparse weights W unpruned and the sparse part of the weight W pruned To update the sparse mask, first W unpruned and W pruned Add up to get W dense , and then use all preset templates to calculate W respectively dense The cumulative sum of the absolute values ​​of the weights of each group of weights under each preset template is taken as the sparse mask of the group of weights, thereby obtaining the updated sparse mask m;

[0058] Update W using sparse mask m unpruned and W pruned , the formula is

[0059] W unpruned =W dense ⊙m

[0060] W pruned =0

[0061] This module 3 involves fixing the sparse mask and updating the sparse model weights by the following formula:

[0062]

[0063] The online training system based on fine-grained structured sparsity, wherein the dense model in the initial module is a convolutional neural network for image classification; the second training module is used to construct a sparse image classification network based on the online training results to classify the images to be classified.

[0064] The present invention also proposes a storage medium for storing a program for executing any one of the online training methods based on fine-grained structured sparsity.

[0065] The present invention also proposes a client for use in any one of the online training systems based on fine-grained structured sparsity.

[0066] From the above scheme, it can be seen that the advantages of the present invention are:

[0067] A sparse model is obtained by applying sparse processing to a dense model. Sparse models are smaller in size and have fewer channels than dense models. Using a sparse model when deploying a model can effectively reduce computational effort due to its smaller size, thus accelerating model execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0069] In order to make the above features and effects of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.

[0070] In order to utilize sparsification technology to improve the performance of the training phase, the present invention proposes a fine-grained structured sparse online training algorithm, thereby improving the performance of model training with less accuracy loss.

[0071] Specifically, it includes the following three aspects of technological innovation:

[0072] 1. A sparse method in which both the input and output channels of the convolution kernel are fine-grained and structured can accelerate both forward and backward propagation during neural network training.

[0073] 2. Propose a three-stage training method that is suitable for sparse online training.

[0074] 3. Propose a fine-grained structured sparse weight update strategy and mask update strategy suitable for online training.

[0075] 1. Sparse method

[0076] To improve performance during the training phase, we proposed a sparse method that sparsifies both the input and output channels of the convolution kernel to meet fine-grained structuring requirements. For example, using the 4-choose-2 fine-grained structuring method, this sparsification method requires that each 4x4 block in the convolution kernel retain 2 values ​​out of 4 in both the input and output channel dimensions, while sparsifying 2 values. There are 90 4x4 templates that meet this requirement. When calculating the sparse mask, we use these 90 preset templates and calculate the sum of the absolute values ​​of the weights for each 4x4 block under each template. The template with the largest sum among the 90 templates is used as the sparse mask for that 4x4 block.

[0077] Calculating this template translates to assigning values ​​of 0 or 1 to a 4x4 grid, with 1 representing a reserved position and 0 representing an unreserved position. If each row and column has two 0s and two 1s, the requirement is met. Traversing all possible cases yields 90 that meet the requirement.

[0078] 2. Three-stage training method

[0079] The fine-grained structured sparse online training scheme proposed in this paper adopts a three-stage training method to achieve a sparse strategy while training network weights during online training:

[0080] Phase 1: Training a non-sparse dense model. This dense model can be, for example, a dense model of a convolutional neural network model for image classification.

[0081] The second stage: obtain the weights of the dense model to obtain the sparse mask and train the sparse model. In this stage, the sparse structure of the network is changing dynamically, that is, the sparse mask and network weights are changing.

[0082] The third stage: fixed sparse mask training sparse model, that is, fixed sparse mask, the sparse structure of the network no longer changes, only the network weights are updated.

[0083] 3. Weight update strategy and mask update strategy

[0084] The weight update strategy and mask update strategy in each of the above stages are different and are introduced separately in this section.

[0085] In the first stage, the dense model is trained, and the weight of the model is set to W dense , the learning rate is l, and the gradient of the weight is The weight update formula is:

[0086]

[0087] At this time, since it is a dense model, the sparse mask can be regarded as an all-1 matrix, and L represents the loss value between the model prediction value and the manually labeled true value.

[0088] In the second stage, the sparse mask and network weights are updated simultaneously, which is performed as follows:

[0089] Step 1: Get the dense model calculated after the first stage, with weight W dense , the learning rate is l.

[0090] Step 2: For W dense , using the preset 90 templates, calculate the cumulative sum of the absolute value of each weight group (for example, 4x4 weight) under each template, and take the template with the largest cumulative sum among the 90 templates as the sparse mask of the 4x4, thereby obtaining the sparse mask m and sparse weight W unpruned =W dense ⊙m, where ⊙ represents the element-by-element multiplication operation. The sparsely populated part of the weight is recorded as W pruned =0.

[0091] Step 3: Using sparse weights W unpruned Execute the forward propagation process of the sparse network, and set the input of this layer to be x i , the output of this layer is x i+1 , then the forward propagation process formula of this layer is

[0092] x i+1 =x i *W unpruned

[0093] Step 4: Calculate the back propagation process of the network, using the topdiff (partial derivative of L with respect to x^(i+1)) propagated from the previous layer, recorded as Calculate the bottomdiff of this layer (partial derivative of L with respect to x^i), recorded as ) and the gradient of the weights The formula is

[0094]

[0095]

[0096] Step 5: According to the gradient of weight And the current sparse mask updates the sparse weight W unpruned and the sparse part of the weight W pruned , the formula is

[0097]

[0098]

[0099] Determine whether this is the end of a training cycle epoch. If so, execute step 6. If not, return to step 3.

[0100] Step 6: Using sparse weights W unpruned and the sparse part of the weight W pruned To update the sparse mask, first W unpruned and W pruned Add up to get W dense , and then use the preset 90 templates to calculate W dense The weight of each 4x4 in each template is the cumulative sum of the absolute values ​​of the weights, and the template with the largest cumulative sum among the 90 templates is taken as the sparse mask of the 4x4, thereby obtaining the updated sparse mask m.

[0101] Step 7: Update W using sparse mask m unpruned and W pruned , the formula is

[0102] W unpruned =W dense ⊙m

[0103] W pruned =0

[0104] Determine whether the number of training iterations set in the second stage has been reached. If so, enter the third stage; otherwise, return to step three.

[0105] In the third stage, the sparse mask is fixed and the sparse model weight is updated. The formula is:

[0106]

[0107] Specifically, if Figure 1 As shown, the present invention proposes an online training method based on fine-grained structured sparsity, comprising:

[0108] Step 1: Train the dense model based on the preset learning rate and weight gradient;

[0109] Step 2: Perform fine-grained structured sparseness on the weights of the dense model to obtain a sparse mask, and train a sparse model based on the sparse mask. During the training of the sparse model, the sparse mask and the model weights of the sparse model are updated.

[0110] Step 3: Fix the network structure of the sparse model and train the sparse model again. During the retraining process of the sparse model, only the model weights of the sparse model are updated, and the model weights and sparse masks of the retrained sparse model are saved as the online training results.

[0111] Finally, after obtaining the sparse mask and model weights, a sparse model can be obtained. The sparse mask is a 0-1 value. Only the model weights corresponding to the positions with a value of 1 in the sparse mask are extracted to obtain a sparse model. The sparse model is smaller in scale than the dense model. The number of channels of the dense model in the 4-choose-2 method of the present invention is 4. The number of channels of the sparse model obtained after sparseness is 2, which is half the number of channels of the dense model. When deploying the model, a sparse model is deployed. Due to its small scale, it can effectively reduce the amount of calculation and achieve the purpose of acceleration.

[0112] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiment.

[0113] The present invention also proposes an online training system based on fine-grained structured sparsity, comprising:

[0114] The inception module is used to train dense models with preset learning rates and weight gradients;

[0115] A first training module is configured to perform fine-grained structured sparsification on the weights of the dense model to obtain a sparse mask, train a sparse model based on the sparse mask, and update the sparse mask and the model weights of the sparse model during the training of the sparse model;

[0116] The second training module is used to fix the network structure of the sparse model and retrain the sparse model, and only update the model weights of the sparse model during the retraining of the sparse model, and save the model weights and sparse masks of the retrained sparse model as the online training results.

[0117] The online training system based on fine-grained structured sparsity, wherein the initial module includes training and updating the weights of the dense model based on the following formula:

[0118]

[0119] The weight of the model is W dense , the learning rate is l, and the gradient of the weight is

[0120] The online training system based on fine-grained structured sparsity, wherein the sparse mask of the dense model in the initial module is an all-1 matrix.

[0121] In the online training system based on fine-grained structured sparsity, the first training module is used to:

[0122] Get the weight W of the dense model dense and learning rate l;

[0123] For W dense , use multiple preset templates to calculate the cumulative sum of the absolute values ​​of each group of weights under each template, take the template with the largest cumulative sum among all preset templates as the sparse mask m of the group of weights, and get the sparse weight W unpruned =W dense ⊙m, where ⊙ represents element-by-element multiplication, and the sparsely populated portion of the weight is recorded as W pruned =0;

[0124] Using sparse weights W unpruned Perform the forward propagation of the network. The forward propagation formula is:

[0125] x i+1 =x i *W unpruned

[0126] Where x i is the input of the i-th layer in the network;

[0127] The network performs back propagation, using the topdiff propagated from the i+1th layer in the network, denoted as Calculate the bottomdiff of the i-th layer, denoted as and the gradient of the weights The formula is:

[0128]

[0129]

[0130] According to the gradient of weight And the current sparse mask updates the sparse weight W unpruned and the sparse part of the weight W pruned , the formula is:

[0131]

[0132]

[0133] Using sparse weights W unpruned and the sparse part of the weight W pruned To update the sparse mask, first W unpruned and W pruned Add up to get W dense , and then use all preset templates to calculate W respectively dense The cumulative sum of the absolute values ​​of the weights of each group of weights under each preset template is taken as the sparse mask of the group of weights, thereby obtaining the updated sparse mask m;

[0134] Update W using sparse mask m unpruned and W pruned , the formula is

[0135] W unpruned =W dense ⊙m

[0136] W pruned =0

[0137] This module 3 involves fixing the sparse mask and updating the sparse model weights by the following formula:

[0138]

[0139] The online training system based on fine-grained structured sparsity, wherein the dense model in the initial module is a convolutional neural network for image classification; the second training module is used to construct a sparse image classification network based on the online training results to classify the images to be classified.

[0140] The present invention also proposes a storage medium for storing a program for executing any one of the online training methods based on fine-grained structured sparsity.

[0141] The present invention also proposes a client for use in any one of the online training systems based on fine-grained structured sparsity.

Claims

1. An online training method based on fine-grained structured sparsity, characterized in that: include: Step 1: Train the dense model based on the preset learning rate and weight gradient; Step 2: Perform fine-grained structured sparseness on the weights of the dense model to obtain a sparse mask, and train a sparse model based on the sparse mask. During the training of the sparse model, the sparse mask and the model weights of the sparse model are updated. Step 3: Fix the network structure of the sparse model and train the sparse model again. During the retraining process, only the model weights of the sparse model are updated, and the model weights and sparse masks of the retrained sparse model are saved as the online training results. This step 2 includes: Step 21: Get the weight W of the dense model dense and learning rate l; Step 22: For W dense , use multiple preset templates to calculate the cumulative sum of the absolute values ​​of each group of 4x4 weights under each template, where each row and column of the preset template satisfies two 0s and two 1s, 1 represents the retained position, and 0 represents the unreserved position; take the template with the largest cumulative sum among all the preset templates as the sparse mask m of the group of weights, and get the sparse weight W unpruned =W dense ⊙m, where ⊙ represents element-by-element multiplication, and the sparsely populated portion of the weight is recorded as W pruned =0; Step 23: Using sparse weights W unpruned Perform the forward propagation of the network. The forward propagation formula is: x i+1 =x i *W unpruned Where x i is the input of the i-th layer in the network; Step 24: The network performs back propagation, using the topdiff propagated from the i+1th layer in the network, recorded as Calculate the bottomdiff of the i-th layer, denoted as and the gradient of the weights The formula is: Step 25: According to the gradient of weight And the current sparse mask updates the sparse weight W unpruned and the sparse part of the weight W pruned , the formula is: Determine whether this is the end of an epoch. If so, execute step 26; otherwise, execute step 23 again. Step 26: Using sparse weights W unpruned and the sparse part of the weight W pruned To update the sparse mask, first W unpruned and W pruned Add up to get W dense , and then use all preset templates to calculate W respectively dense The cumulative sum of the absolute values ​​of the weights of each group of weights under each preset template is taken as the sparse mask of the group of weights, thereby obtaining the updated sparse mask m; Step 27: Update W using sparse mask m unpruned and W pruned , the formula is IN unpruned =In dense ⊙m W pruned ＝0 Determine whether the set number of iterations has been reached. If so, execute step 3; otherwise, execute step 23 again. This step 3 involves fixing the sparse mask and updating the sparse model weights by: The dense model in step 1 is a convolutional neural network for image classification; and step 3 includes constructing a sparse image classification network based on the online training results to classify the images to be classified.

2. The online training method based on fine-grained structured sparsity according to claim 1, characterized in that Step 1 involves training and updating the weights of the dense model based on the following formula: The weight of the model is W dense , the learning rate is l, and the gradient of the weight is 3. The online training method based on fine-grained structured sparsity according to claim 1, characterized in that The sparse mask of the dense model in step 1 is an all-one matrix.

4. An online training system based on fine-grained structured sparsity, characterized by: include: The inception module is used to train dense models with preset learning rates and weight gradients; A first training module is configured to perform fine-grained structured sparsification on the weights of the dense model to obtain a sparse mask, train a sparse model based on the sparse mask, and update the sparse mask and the model weights of the sparse model during the training of the sparse model; The second training module is used to fix the network structure of the sparse model and retrain the sparse model, and only update the model weights of the sparse model during the retraining of the sparse model, and save the model weights and sparse masks of the retrained sparse model as the online training result; The first training module is used to: Get the weight W of the dense model dense and learning rate l; For W dense , use multiple preset templates to calculate the cumulative sum of the absolute values ​​of each group of 4x4 weights under each template, where each row and column of the preset template satisfies two 0s and two 1s, 1 represents the retained position, and 0 represents the unreserved position; take the template with the largest cumulative sum among all the preset templates as the sparse mask m of the group of weights, and get the sparse weight W unpruned =W dense ⊙m, where ⊙ represents element-by-element multiplication, and the sparsely populated portion of the weight is recorded as W pruned =0; Using sparse weights W unpruned Perform the forward propagation of the network. The forward propagation formula is: x i+1 =x i *W unpruned Where x i is the input of the i-th layer in the network; The network performs back propagation, using the topdiff propagated from the i+1th layer in the network, denoted as Calculate the bottomdiff of layer i, denoted as and the gradient of the weights The formula is: According to the gradient of weight And the current sparse mask updates the sparse weight W unpruned and the sparse part of the weight W pruned , the formula is: Using sparse weights W unpruned and the sparse part of the weight W pruned To update the sparse mask, first W unpruned and W pruned Add up to get W dense , and then use all preset templates to calculate W respectively dense The cumulative sum of the absolute values ​​of the weights of each group of weights under each preset template is taken as the sparse mask of the group of weights, thereby obtaining the updated sparse mask m; Update W using sparse mask m unpruned and W pruned , the formula is IN unpruned =In dense ⊙m W pruned ＝0 Determine whether the set number of iterations has been reached at this time. If so, call the second training module; The second training module is used to fix the sparse mask and update the sparse model weights by the following formula: The dense model in the initial module is a convolutional neural network for image classification; the second training module is used to construct a sparse image classification network based on the online training results to classify the images to be classified.

5. The online training system based on fine-grained structured sparsity as claimed in claim 4, characterized in that The initial module involves training and updating the weights of the dense model based on the following formula: The weight of the model is W dense , the learning rate is l, and the gradient of the weight is 6. The online training system based on fine-grained structured sparsity as claimed in claim 4, characterized in that The sparse mask of this dense model in the initial module is an all-one matrix.

7. A storage medium for storing a program for executing any one of the online training methods based on fine-grained structured sparsity as claimed in claims 1 to 3.

8. A client, used for any one of the online training systems based on fine-grained structured sparseness in claims 4 to 6.

Images