Coarse-grained neural network sparsification method and system based on two-stage weight group attenuation
Through a two-stage weight group attenuation method, the attenuation amplitude of the structural unit is dynamically controlled, which solves the problem of structural unit importance evaluation and recovery, realizes efficient neural network sparsification and acceleration, and is suitable for model compression in resource-constrained environments.
Patent Information
- Application Number
- CN202510708079.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-29
- Publication Date
- 2025-10-10
AI Technical Summary
In existing neural network sparsification technologies, the importance of structural units is difficult to accurately evaluate, removed structural units are difficult to restore, and the model requires a lot of fine-tuning after pruning to restore performance, resulting in low sparsification efficiency and difficulty in achieving efficient compression and performance assurance in resource-constrained environments.
A two-stage weight group decay method is adopted, including model pre-training, fine-grained weight decay in the first stage and coarse-grained decay of structural units in the second stage. The decay amplitude of each structural unit is dynamically controlled, and the exponential decay function and importance evaluation are combined to gradually guide the sparsification process.
It improves the stability and accuracy of the sparsification process, reduces fine-tuning overhead, achieves efficient model compression and computational acceleration, and is suitable for efficient deployment in resource-constrained environments.
Smart Images
Figure CN120764598A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of neural network model compression and acceleration, and specifically relates to a coarse-grained neural network sparsification method based on two-stage weight set decay, which is used to achieve model compression and deployment acceleration. Background Art
[0002] Existing deep neural network models generally have the problems of large number of parameters and high computational complexity. Especially when deployed in edge computing devices (such as mobile terminals and embedded devices), they face challenges such as large model size, slow computing speed, and high power consumption. In order to improve the inference efficiency of the model and reduce the complexity of the model, neural network compression technology has emerged, among which structured pruning is the main method currently studied and applied. Compared with fine-grained pruning (weight pruning), structured pruning helps maintain a regular network structure and facilitates efficient hardware acceleration. This method uses preset evaluation criteria to evaluate the importance of specific structural units in the neural network (such as neurons, channels, filters, etc.), and removes structural units that have little contribution to model performance at one time or iteratively. Ultimately, the number of model parameters and computational overhead are significantly reduced without reducing the model accuracy.
[0003] Although existing structured pruning technologies have made some progress, the following technical challenges still exist in practical applications:
[0004] 1) The importance of structural units is difficult to accurately assess: In existing technologies, the contribution of structural units to model performance changes dynamically during training, making it difficult to accurately judge their ultimate impact on model performance in the early stages of training. This may result in some potentially important connections being mistakenly removed in the early stages, thereby affecting model performance.
[0005] 2) Removed structural units are difficult to recover: In order to reduce the accuracy loss caused by incorrect pruning, some studies have attempted to allow the recovery of removed structural units during training, but the recovery effect of such methods is usually limited.
[0006] 3) The model requires extensive fine-tuning to restore performance after pruning: The pruning operation destroys the parameter space of the original model. Especially in pruning tasks with high sparsity rates, the model performance degrades significantly, requiring a large number of fine-tuning rounds to restore accuracy, thereby increasing training costs and time overhead.
[0007] The above limitations greatly affect the deployment efficiency of neural network sparsification technology in practical applications, especially in resource-constrained environments where it is difficult to achieve efficient compression and performance assurance. Summary of the Invention
[0008] In order to solve the problems of low efficiency of neural network sparsification and difficulty in achieving efficient compression and performance assurance due to the difficulty in accurately evaluating the importance of structural units, the difficulty in restoring removed structural units, and the need for a large amount of fine-tuning of the model after pruning to restore performance, the present invention provides a coarse-grained neural network sparsification method based on two-stage weight group attenuation. By introducing a two-stage adjustable mixed attenuation mechanism during the training process, the attenuation amplitude of each structural unit is dynamically controlled, and the sparsification process is gradually guided while maintaining the tightly connected structure of the network, thereby achieving model compression and acceleration without significantly affecting the model performance.
[0009] According to one aspect of the present invention, a coarse-grained neural network sparsification method based on two-stage weight set decay is provided, comprising:
[0010] Model pre-training: Pre-train the initial neural network for several cycles to achieve initial convergence on the target task;
[0011] First-stage decay: Based on the pre-trained model, a first-stage fine-grained weight decay mechanism is introduced, which assigns different decay factors according to the relative magnitude of each weight.
[0012] Second-stage attenuation: Based on the preset importance evaluation criteria, the second-stage coarse-grained attenuation operation is performed with the structural unit as the granularity. The ownership weight within each structural unit is considered as a whole, and a uniform attenuation factor is applied according to its importance level. This further weakens the contribution of low-importance structural units to model performance, gradually achieving a structured sparse process.
[0013] Parameter fine-tuning: Fine-tune the sparse model to recover the accuracy loss caused by model sparsification.
[0014] As a further technical solution, the first stage of attenuation further includes:
[0015] Arrange all weights in each layer of the network in ascending order of magnitude and calculate the relative position factor of each weight;
[0016] The relative position factor is used as the input of the exponential decay function to calculate the decay factor of each weight;
[0017] Update the weight magnitude.
[0018] As a further technical solution, the first stage of attenuation further includes:
[0019] For weights with smaller weight magnitudes, a larger attenuation factor is assigned to them; for weights with larger weight magnitudes, a smaller attenuation factor is assigned to them.
[0020] As a further technical solution, the second stage of attenuation further includes:
[0021] In the weight update phase, the importance of each structural unit in each layer is calculated according to the predefined pruning criteria;
[0022] The importance of each structural unit is sorted in ascending order, and the relative position factor of each structural unit is calculated in the same way. Then, the attenuation factor is obtained by mapping it through the exponential attenuation function.
[0023] As a further technical solution, the second stage of attenuation further includes:
[0024] For relatively unimportant structural units, a larger attenuation factor is assigned to them; for relatively important structural units, a smaller attenuation factor is applied to them.
[0025] As a further technical solution, the method further includes:
[0026] The attenuation function is shifted and scaled to dynamically adjust the attenuation range.
[0027] According to one aspect of the present invention, a coarse-grained neural network sparsification system based on two-stage weight group decay is provided, and the system is used to implement the coarse-grained neural network sparsification method based on two-stage weight group decay.
[0028] As a further technical solution, the system includes:
[0029] The model pre-training module is used to pre-train the initial neural network for several cycles to make the model initially converge on the target task;
[0030] The first-stage attenuation module is used to introduce the first-stage fine-grained weight attenuation mechanism based on the pre-trained model, assigning different attenuation factors according to the relative magnitude of each weight;
[0031] The second-stage attenuation module is used to perform the second-stage coarse-grained attenuation operation based on the preset importance evaluation criteria and the structural unit as the granularity. The ownership weight within each structural unit is regarded as a whole, and a uniform attenuation factor is applied according to its importance level, further weakening the contribution of low-importance structural units to the model performance, gradually realizing the structured sparse process;
[0032] The parameter fine-tuning module is used to fine-tune the sparse model and recover the accuracy loss caused by model sparsification.
[0033] According to one aspect of the present invention, a coarse-grained neural network sparsification system based on two-stage weight group decay is provided, comprising a memory and a processor, wherein the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the steps of the coarse-grained neural network sparsification method based on two-stage weight group decay.
[0034] According to one aspect of the present invention, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to execute the steps of the coarse-grained neural network sparsification method based on two-stage weight group decay.
[0035] Compared with the prior art, the present invention has the following beneficial effects:
[0036] 1. Improving the retention rate of important connections: By dynamically adjusting the attenuation strength of each structural unit during training, the problem of incorrectly removing potentially important structural units in the early stages of training is effectively avoided, improving the stability and accuracy of the sparsification process.
[0037] 2. Reduce fine-tuning overhead: The sparse model generated by the present invention has a reasonable structure and stable parameter distribution. Only a few fine-tuning training cycles are needed to restore the model accuracy, which significantly reduces the optimization cost after model compression.
[0038] 3. Achieve efficient compression: While ensuring that model performance is basically unaffected, it can achieve a high parameter compression rate and computational acceleration ratio, which is suitable for efficient model deployment in resource-constrained environments. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief introduction will be given below to the drawings used in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0040] Figure 1 This is a flowchart of the existing sparsification technology.
[0041] Figure 2 This is a flowchart of the sparsification technology provided by an embodiment of the present invention.
[0042] Figure 3 This is a schematic diagram of attenuation provided by an embodiment of the present invention.
[0043] Figure 4 Schematic diagram of the structural unit attenuation provided in an embodiment of the present invention applicable to various sparse granularities and evaluation criteria.
[0044] Figure 5 A schematic diagram of dynamically adjusting the attenuation interval provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0045] The application provides a coarse-grained neural network sparsification method based on two-stage weight group attenuation.
[0046] The method is suitable for various neural network structures and sparse application scenarios, and has good universality and deployment performance. Specifically, the method comprises the following steps:
[0047] 1) Model pre-training: the initial neural network is pre-trained for several cycles, so that the model is preliminarily converged on the target task, and a stable parameter basis is provided for the subsequent two-stage attenuation process;
[0048] 2) First-stage attenuation: a first-stage fine-grained weight attenuation mechanism is introduced on the basis of the pre-trained model, different attenuation factors are allocated according to the relative amplitude of each weight, and greater attenuation is applied to the weights with smaller amplitudes, thereby providing a buffer for the subsequent structural unit attenuation and reducing the influence of small-amplitude weights in the subsequent structural unit evaluation.
[0049] 3) Second-stage attenuation: based on a preset importance evaluation standard, a second-stage coarse-grained attenuation operation is performed on the structural units (such as filters), all weights in each structural unit are regarded as a whole, and a uniform attenuation factor is applied according to the importance level, thereby further weakening the contribution of low-importance structural units to the model performance and gradually realizing the structured sparsification process;
[0050] 4) Parameter fine-tuning: the sparse model is fine-tuned to recover the accuracy loss caused by model sparsification.
[0051] To make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application, taking filter sparsification as an example. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application. In addition, the technical features in each embodiment or single embodiment provided by the present application can be combined with each other to form new technical solutions. This combination is not restricted by the order of steps and / or structure composition mode, but must be based on the realization by those skilled in the art. When the combination of technical solutions contradicts each other or cannot be realized, it should be considered that the combination of technical solutions does not exist and is not within the scope of protection required by the present application.
[0052] Figure 1 The following figure shows a typical iterative pruning framework in the prior art, which generally includes the following three steps: 1) Model pre-training: training a baseline network model as the initial model for subsequent pruning; 2) Network pruning: removing unimportant structural units according to pruning criteria to obtain a sparse subnetwork; 3) Parameter fine-tuning: fine-tuning the pruned model to recover performance degradation caused by pruning. These pruning and fine-tuning steps typically require multiple rounds of iteration to gradually increase model sparsity, but the overall training cost is high and, as mentioned in the background, there are problems such as mis-pruning of important connections and limited recovery effects.
[0053] To address the above problems, the present invention proposes a coarse-grained neural network sparsification and inference acceleration method based on two-stage attenuation. This method maintains the integrity of the network structure during most training cycles, and only adjusts the attenuation factors of different structural units to guide the weight amplitudes in some structural units to gradually decrease. In the later stages of training, the sparsification operation is gradually performed, thereby achieving effective compression while ensuring model performance. The sparse framework of this method is as follows: Figure 2 As shown, it is mainly divided into four steps. Taking filter sparseness as an example, the specific steps and descriptions are as follows:
[0054] 1. Pre-training model: Starting from a randomly initialized neural network model, pre-training is performed for about 10 to 20 training cycles to make the model initially converge on the target task and provide a stable parameter foundation for the subsequent two-stage attenuation process.
[0055] 2. Two-stage attenuation: The specific attenuation process is as follows Figure 3 As shown, this is done in two stages.
[0056] 1) First-stage fine-grained attenuation: The first-stage attenuation uses the weight of each layer as the attenuation unit.
[0057] First, all weights in each layer of the network are arranged in ascending order of magnitude, according to the formula Calculate the relative position factor of each weight in Represents the weights in the sorted weight vector The corresponding index position, len(w l ) is the length of the weight vector. Then, the relative position factor s is used as an exponential decay function The input is used to calculate the attenuation factor of each weight, which is in the range of (0, 1). Where ε represents the weight regularization coefficient, and its value range is usually 10 -5 ~10 -3 , S target Indicates the target sparsity. For example, if the target sparsity is 95%, the weight regularization coefficient is 10 -4 , then the decay function Weights with smaller magnitudes will be given a larger attenuation factor to accelerate their decay to zero, while weights with larger magnitudes will be assigned a smaller attenuation factor to prevent them from decaying too quickly. Finally, according to the formula And the formula (w t+1 ←w t+1 ⊙(1-f)) updates the weight amplitude, where η represents the learning rate, Represents the gradient of the loss function with respect to the weight, w t+1 Represents the weight value at time t+1.
[0058] 2) Second-stage coarse-grained attenuation: The second-stage attenuation uses preset structural units (here, the filters in each layer) as attenuation units.
[0059] In the weight update stage, the importance of each filter in each layer is first calculated based on the predefined pruning criteria (such as L1 norm). Then, the importance of each filter is sorted in ascending order, and the relative position factor of each filter is calculated in the same way, and then mapped through the exponential decay function to obtain an attenuation factor in the range of (0, 1). For relatively unimportant filters, a larger attenuation factor is assigned to them to reduce the weight amplitude of the entire filter so that it decays rapidly to zero. On the contrary, for relatively important filters, a smaller attenuation factor is applied to them to protect the weight amplitude in the filter from being affected almost. In order to reduce the accuracy loss caused by one-time sparsity and ensure that other filters can be effectively attenuated, this method dynamically adjusts the attenuation interval by performing appropriate displacement and scale transformation on the attenuation function, such as Figure 5 The adjusted curve formula is where f i is the attenuation function of the ith dynamic change, s i is the horizontal coordinate of the dotted line position during the ith dynamic change, according to the formula Determine. Among them, t i Indicates the current (i-th moment) dynamic adjustment round; g is a custom parameter used to control the speed of the dotted line movement, and T represents the total number of dynamic adjustment rounds. The above attenuation process has good flexibility and can be combined with common evaluation criteria to apply to sparse tasks of different granularities (such as channel level, core level, etc.). Figure 4 shown.
[0060] Fine-tuning: After completing the structural sparsity processing, fine-tune the sparse model to recover the accuracy loss that may be caused by the pruning process and improve the performance of the final model on the target task.
[0061] The implementation basis of each embodiment of the present invention is achieved through programmed processing by a device with processor functionality. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the present invention are encapsulated into various modules. Based on this reality, on the basis of the above-mentioned embodiments, an embodiment of the present invention provides a coarse-grained neural network sparsification system based on two-stage weight group decay, which is used to execute a coarse-grained neural network sparsification method based on two-stage weight group decay in the above-mentioned method embodiment.
[0062] The system includes: a model pre-training module, which is used to pre-train the initial neural network for several cycles so that the model can initially converge on the target task; a first-stage attenuation module, which is used to introduce the first-stage fine-grained weight attenuation mechanism based on the pre-trained model, and assign different attenuation factors according to the relative amplitude of each weight; a second-stage attenuation module, which is used to perform a second-stage coarse-grained attenuation operation based on the preset importance evaluation standard and the structural unit as the granularity, and regards all the weights in each structural unit as a whole, and applies a uniform attenuation factor according to its importance level, further weakening the contribution of low-importance structural units to the model performance, and gradually realizing the structured sparse process; a parameter fine-tuning module, which is used to fine-tune the sparse model and restore the accuracy loss caused by model sparsification.
[0063] An embodiment of the present invention provides a coarse-grained neural network sparsification system based on two-stage weight group decay. It addresses the problem that the sparsification efficiency of neural networks is low, and it is difficult to achieve efficient compression and performance assurance due to the difficulty in accurately evaluating the importance of structural units, the difficulty in restoring removed structural units, and the need for a lot of fine-tuning of the model after pruning to restore performance. By adopting the above-mentioned several modules, a two-stage adjustable mixed decay mechanism is introduced during the training process to dynamically control the decay amplitude of each structural unit. While maintaining the tightly connected structure of the network, the sparsification process is gradually guided, thereby achieving model compression and acceleration without significantly affecting the model performance.
[0064] It should be noted that the system embodiments provided by the present invention are not only used to implement the methods in the above-mentioned method embodiments, but also used to implement the methods in other method embodiments provided by the present invention. The only difference is the setting of corresponding functional modules. The principles thereof are basically the same as those of the above-mentioned system embodiments provided by the present invention. As long as those skilled in the art refer to the specific technical solutions in other method embodiments on the basis of the above-mentioned system embodiments, obtain corresponding technical means and technical solutions composed of these technical means by combining technical features, and on the premise of ensuring the practicality of the technical solutions, improve the modules in the above-mentioned system embodiments to obtain corresponding system-type embodiments, which are used to implement the methods in other method-type embodiments. For example:
[0065] Based on the content of the above system embodiment, as a preferred embodiment, the two-stage weight group attenuation-based coarse-grained neural network sparsification system provided in the embodiment of the application further comprises:
[0066] arranging all the weights in each layer of the network in ascending order of amplitude, and calculating the relative position factor of each weight;
[0067] taking the relative position factor as the input of the exponential attenuation function, and calculating the attenuation factor of each weight;
[0068] updating the weight amplitude.
[0069] Based on the content of the above system embodiment, as a preferred embodiment, the two-stage weight group attenuation-based coarse-grained neural network sparsification system provided in the embodiment of the application further comprises:
[0070] a larger attenuation factor is assigned to a weight with a smaller weight amplitude, and a smaller attenuation factor is assigned to a weight with a larger weight amplitude.
[0071] Based on the content of the above system embodiment, as a preferred embodiment, the two-stage weight group attenuation-based coarse-grained neural network sparsification system provided in the embodiment of the application further comprises:
[0072] in the weight updating stage, the importance of each structural unit in each layer is calculated according to a predefined pruning criterion;
[0073] arranging the importance of each structural unit in ascending order, calculating the relative position factor of each structural unit, and then mapping the attenuation factor through the exponential attenuation function.
[0074] Based on the content of the above system embodiment, as a preferred embodiment, the two-stage weight group attenuation-based coarse-grained neural network sparsification system provided in the embodiment of the application further comprises:
[0075] a larger attenuation factor is assigned to a relatively unimportant structural unit, and a smaller attenuation factor is applied to a relatively important structural unit.
[0076] Based on the content of the above system embodiment, as a preferred embodiment, the two-stage weight group attenuation-based coarse-grained neural network sparsification system provided in the embodiment of the application further comprises:
[0077] performing displacement and scale transformation on the attenuation function to dynamically adjust the attenuation interval.
[0078] Based on the same inventive concept as the above-mentioned embodiment, an embodiment of the present invention also provides a coarse-grained neural network sparsification system based on two-stage weight group decay, including a memory and a processor, wherein the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the steps of the coarse-grained neural network sparsification method based on two-stage weight group decay.
[0079] Based on the same inventive concept as the above embodiment, an embodiment of the present invention further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions cause the computer to execute the following steps of the sheep body three-dimensional point cloud registration method based on local overlapping areas:
[0080] Model pre-training: Pre-train the initial neural network for several cycles to achieve initial convergence on the target task;
[0081] First-stage decay: Based on the pre-trained model, a first-stage fine-grained weight decay mechanism is introduced, which assigns different decay factors according to the relative magnitude of each weight.
[0082] Second-stage attenuation: Based on the preset importance evaluation criteria, the second-stage coarse-grained attenuation operation is performed with the structural unit as the granularity. The ownership weight within each structural unit is considered as a whole, and a uniform attenuation factor is applied according to its importance level. This further weakens the contribution of low-importance structural units to model performance, gradually achieving a structured sparse process.
[0083] Parameter fine-tuning: Fine-tune the sparse model to recover the accuracy loss caused by model sparsification.
[0084] In summary, the present invention introduces a two-stage adjustable hybrid attenuation mechanism during training, dynamically controlling the attenuation amplitude of each structural unit. This gradually guides the sparsification process while maintaining the network's densely connected structure, thereby achieving model compression and acceleration without significantly affecting model performance. The method described in the present invention is applicable to a variety of neural network structures and sparsification application scenarios, and has good versatility and deployment performance.
[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.
Claims
1. A coarse-grained neural network sparsification method based on two-stage weight group decay, characterized by: include: Model pre-training: Pre-train the initial neural network for several cycles to achieve initial convergence on the target task; First-stage decay: Based on the pre-trained model, a first-stage fine-grained weight decay mechanism is introduced, which assigns different decay factors according to the relative magnitude of each weight. Second-stage attenuation: Based on the preset importance evaluation criteria, the second-stage coarse-grained attenuation operation is performed with the structural unit as the granularity. The ownership weight within each structural unit is considered as a whole, and a uniform attenuation factor is applied according to its importance level. This further weakens the contribution of low-importance structural units to model performance, gradually achieving a structured sparse process. Parameter fine-tuning: Fine-tune the sparse model to recover the accuracy loss caused by model sparsification.
2. The coarse-grained neural network sparsification method based on two-stage weight set decay according to claim 1 is characterized in that: The first stage attenuation further includes: Arrange all weights in each layer of the network in ascending order of magnitude and calculate the relative position factor of each weight; The relative position factor is used as the input of the exponential decay function to calculate the decay factor of each weight; Update the weight magnitude.
3. The coarse-grained neural network sparsification method based on two-stage weight set decay according to claim 2, characterized in that: The first stage attenuation further includes: For weights with smaller weight magnitudes, a larger attenuation factor is assigned to them; for weights with larger weight magnitudes, a smaller attenuation factor is assigned to them.
4. The coarse-grained neural network sparsification method based on two-stage weight set decay according to claim 1, characterized in that: The second stage attenuation further includes: In the weight update phase, the importance of each structural unit in each layer is calculated according to the predefined pruning criteria; The importance of each structural unit is sorted in ascending order, and the relative position factor of each structural unit is calculated in the same way. Then, the attenuation factor is obtained by mapping it through the exponential attenuation function.
5. The coarse-grained neural network sparsification method based on two-stage weight set decay according to claim 4 is characterized in that: The second stage attenuation further includes: For relatively unimportant structural units, a larger attenuation factor is assigned to them; for relatively important structural units, a smaller attenuation factor is applied to them.
6. The coarse-grained neural network sparsification method based on two-stage weight set decay according to claim 1, characterized in that: The method further comprises: The attenuation function is shifted and scaled to dynamically adjust the attenuation range.
7. A coarse-grained neural network sparsification system based on two-stage weight set decay, characterized by: The system is used to implement the coarse-grained neural network sparsification method based on two-stage weight group decay as described in any one of claims 1 to 6.
8. The coarse-grained neural network sparsification system based on two-stage weight set decay according to claim 7, characterized in that: The system comprises: The model pre-training module is used to pre-train the initial neural network for several cycles to make the model initially converge on the target task; The first-stage attenuation module is used to introduce the first-stage fine-grained weight attenuation mechanism based on the pre-trained model, assigning different attenuation factors according to the relative magnitude of each weight; The second-stage attenuation module is used to perform the second-stage coarse-grained attenuation operation based on the preset importance evaluation criteria and the structural unit as the granularity. The ownership weight within each structural unit is considered as a whole, and a uniform attenuation factor is applied according to its importance level, further weakening the contribution of low-importance structural units to the model performance, gradually achieving a structured sparse process; The parameter fine-tuning module is used to fine-tune the sparse model and recover the accuracy loss caused by model sparsification.
9. A coarse-grained neural network sparsification system based on two-stage weight set decay, characterized in that: The invention comprises a memory and a processor, wherein the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the steps of the coarse-grained neural network sparsification method based on two-stage weight group decay according to any one of claims 1 to 6.
10. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores computer instructions, which enable the computer to execute the steps of the coarse-grained neural network sparsification method based on two-stage weight group decay according to any one of claims 1 to 6.