Activation sparse method and device based on weight perception, equipment and storage medium
By employing a weight-aware activation sparsity method, which combines activation values and weight information to dynamically evaluate the importance of activation values and perform sparsity processing, the large error and resource constraints caused by existing methods are resolved, achieving efficient acceleration and improved accuracy of deep neural networks.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-12
- Publication Date
- 2026-04-10
AI Technical Summary
Existing activation sparsity methods rely solely on the magnitude of activation values to determine importance, leading to large output errors in deep neural networks and making them difficult to deploy in resource-constrained environments.
The weight-aware activation sparsity method determines the importance score by obtaining the absolute value of the activation value and the L1 norm of the weight column, and performs sparsity processing based on the importance threshold. Combined with the increasing sparsity of different depth blocks from shallow to deep, a sparsity strategy from fine to coarse is formed.
It reduces the amount of computation in the inference process, improves the output accuracy of deep neural networks, and protects the shallow feature extraction capability while accelerating inference.
Smart Images
Figure CN121835774A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of deep neural network technology, and in particular to a weight-aware activation sparsity method, apparatus, device and storage medium. Background Technology
[0002] As the parameter size of deep neural networks continues to increase, their demand for computing resources during inference also increases, making it difficult to deploy large-scale deep neural networks in resource-constrained environments. To address this challenge, activation sparsity processing is usually applied to deep neural networks. However, existing activation sparsity methods typically determine the importance of activation values based solely on their magnitude information, which leads to large output errors in deep neural networks. Summary of the Invention
[0003] This application provides a weight-aware activation sparse method, apparatus, device, and storage medium to solve the problems mentioned in the background art.
[0004] In a first aspect, this application provides a weight-aware activation sparsity method for a deep neural network, the deep neural network comprising multiple blocks from shallow to deep, each block comprising multiple linear layers, the sparsity of each block increasing from shallow to deep, the method comprising: During the forward propagation of the deep neural network, for each linear layer, the activation value vector of the input of each linear layer is obtained; For each activation value in the same activation value vector, the L1 norm of the absolute value of the activation value and the weight column corresponding to the activation value is determined as the importance score corresponding to the activation value. For each of the linear layers, activation sparsity is applied to the linear layer based on the importance threshold corresponding to the linear layer and the importance score corresponding to each activation value in the activation value vector input to the linear layer.
[0005] Secondly, this application provides a weight-aware activation sparsity device for a deep neural network, the deep neural network comprising multiple blocks from shallow to deep, each block comprising multiple linear layers, the sparsity of each block increasing from shallow to deep, the device comprising: The acquisition module is used to acquire the activation value vector of each linear layer during the forward propagation of the deep neural network. The determination module is used to determine, for each activation value in the same activation value vector, the absolute value of the activation value and the L1 norm of the weight column corresponding to the activation value as the importance score corresponding to the activation value; The activation sparsity processing module is used to perform activation sparsity on each of the linear layers based on the importance threshold corresponding to the linear layer and the importance score corresponding to each activation value in the activation value vector input to the linear layer.
[0006] Thirdly, this application provides a terminal device, the terminal device including a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the weight-aware activation sparsity method as described in any of the preceding claims.
[0007] Fourthly, this application provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the weight-aware activation sparsity method as described in any of the preceding claims.
[0008] This application provides a weight-aware activation sparsity method, apparatus, device, and storage medium. The method is used in a deep neural network, which includes multiple blocks from shallow to deep, each block including multiple linear layers, and the sparsity of each block increasing from shallow to deep. The method includes: during the forward propagation of the deep neural network, obtaining the activation value vector of the input of each linear layer; and for each activation value in the same activation value vector, determining the importance score corresponding to the activation value as the L1 norm of the absolute value of the activation value and the weight column corresponding to the activation value. For each linear layer, activation sparsity is applied to the linear layer based on the importance threshold corresponding to the linear layer and the importance score corresponding to each activation value in the activation value vector input to the linear layer. This method, on the one hand, introduces the L1 norm of the weight column to jointly determine the importance score of the activation value, simultaneously considering the dynamic activation intensity and the influence of the static weight structure in the sparsity decision, making the importance assessment more comprehensive and theoretically forming an upper bound for minimizing the output error; on the other hand, by performing sparsity judgment based on the importance threshold, activation values with low importance scores are set to zero, avoiding multiplication and addition operations related to these zero-value activations and reducing the actual computational load during inference; furthermore, by setting a sparsity rate that increases from shallow to deep for blocks of different depths, an overall sparsity strategy from fine to coarse is formed, adapting to the differences in the importance of feature extraction functions at different levels in the deep neural network. While achieving the overall acceleration goal, it protects the key feature extraction capabilities of shallow layers, helping to improve the output accuracy of the deep neural network after sparsification. Attached Figure Description
[0009] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0010] Figure 1 A flowchart illustrating the weight-aware activation sparsity method provided in this application embodiment; Figure 2 A schematic block diagram of the structure of a weight-aware activation sparse device provided in the embodiments of this application; Figure 3 A schematic block diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation
[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0012] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the described order. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change based on the actual situation.
[0013] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.
[0014] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0015] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features described herein can be combined with each other.
[0016] Please see Figure 1 , Figure 1This is a flowchart illustrating a weight-aware activation sparsity method provided in an embodiment of this application. The method is used in a deep neural network, which includes multiple blocks of varying depths. Each block comprises multiple linear layers, and the sparsity of each block increases from shallow to deep. Figure 1 As shown, the weight-aware activation sparsity method provided in this application includes steps 1 to 3.
[0017] Step 1: During the forward propagation of the deep neural network, obtain the activation value vector of the input of each linear layer.
[0018] The deep neural network is a pre-trained deep neural network, which can be any one of the following: a large language model based on a Transformer architecture, an image recognition model based on a convolutional neural network structure, etc.
[0019] Specifically, when the deep neural network processes input data and performs inference tasks, the data flows layer by layer along the network structure. When the data flows through any linear layer, the complete input data of that linear layer is intercepted and captured, that is, the activation value vector corresponding to that linear layer.
[0020] Step 2: For each activation value in the same activation value vector, determine the importance score corresponding to the activation value as the L1 norm of the absolute value of the activation value and the weight column corresponding to the activation value.
[0021] Wherein, the activation absolute value refers to the absolute value of the activation value.
[0022] Specifically, for each activation value in the same activation value vector, the L1 norm of the weight column corresponding to the activation value is the sum of the absolute values of all weight elements in the column of the weight matrix of the linear layer corresponding to the input channel where the activation value is located.
[0023] Step 3: For each of the linear layers, activate the linear layer sparsly based on the importance threshold corresponding to the linear layer and the importance score corresponding to each activation value in the activation value vector input to the linear layer.
[0024] Specifically, the activation sparsity of the linear layer based on the importance threshold corresponding to the linear layer and the importance scores corresponding to each activation value in the activation value vector input to the linear layer includes: Step 31: For each activation value in the activation value vector input to the linear layer, determine whether the importance score corresponding to the activation value is less than the importance threshold corresponding to the linear layer.
[0025] Step 32: If yes, set the activation value to zero; Step 33: If not, retain the activation value.
[0026] The method provided in this embodiment, on the one hand, introduces the L1 norm of the weight column to jointly determine the importance score of the activation value, and simultaneously considers the dynamic activation strength and the influence of the static weight structure in the sparsity decision, making the importance assessment more comprehensive and theoretically forming an upper bound for minimizing the output error; on the other hand, by performing sparsity judgment based on the importance threshold, activation values with low importance scores are set to zero, avoiding multiplication and addition operations related to these zero-value activations and reducing the actual computational load in the inference process; furthermore, by setting a sparsity rate that increases from shallow to deep for blocks of different depths, an overall sparsity strategy from fine to coarse is formed, adapting to the differences in the importance of feature extraction functions at different levels in the deep neural network, achieving the overall acceleration goal while protecting the key feature extraction capabilities of shallow layers, which helps to improve the output accuracy of the deep neural network after sparsification.
[0027] In some embodiments, the method for determining the importance threshold corresponding to each linear layer in the deep neural network includes: Step A: For each linear layer in the deep neural network, perform probability density modeling on the importance scores corresponding to each activation bit of the linear layer to obtain the cumulative distribution function corresponding to the linear layer.
[0028] Specifically, the step of performing probability density modeling on the importance scores corresponding to each activation position of the linear layer to obtain the cumulative distribution function corresponding to the linear layer includes: Step A1: Generate an importance score sample set corresponding to each linear layer of the deep neural network based on a preset validation dataset.
[0029] Specifically, step A1 includes: Step A11: For each input value in the verification dataset, input the input value into the deep neural network to make the deep neural network perform forward propagation, and obtain the verification activation value vector of each linear layer input during the forward propagation process.
[0030] The verification dataset includes multiple input values. For each input value in the verification dataset, after the input value is input into the deep neural network, each linear layer of the deep neural network corresponds to a verification activation value vector. It can be understood that for each linear layer in the deep neural network, the number of verification activation value vectors corresponding to the linear layer is the same as the number of input values in the verification dataset.
[0031] Step A12: For each of the linear layers, multiply the absolute value of the verification activation of each verification activation value in the verification activation value vector corresponding to the linear layer by the L1 norm of the weight column corresponding to the verification activation value to obtain the importance score corresponding to each verification activation value in the verification activation value vector corresponding to the linear layer; the importance scores corresponding to each verification activation value in the verification activation value vector corresponding to the linear layer constitute the importance score sample set corresponding to the linear layer.
[0032] Wherein, the absolute value of the verification activation refers to the absolute value corresponding to the verification activation value.
[0033] Specifically, multiplying the absolute value of the verification activation of each verification activation value in each verification activation value vector corresponding to the linear layer with the L1 norm of the weight column corresponding to the verification activation value means multiplying the absolute value of the verification activation corresponding to each verification activation value in each verification activation value vector corresponding to the linear layer with the L1 norm of the weight column corresponding to the verification activation value.
[0034] Understandably, steps A11 to A12, on the one hand, by using an independent validation dataset for forward propagation to collect validation activation values, provide a large number of real importance score observation samples for each linear layer, which helps ensure that the data basis for subsequent probability density modeling is statistically significant; on the other hand, by independently calculating and aggregating its own set of importance score samples for each linear layer, the differences that may exist in the function and data distribution of different linear layers are respected, laying the foundation for building an independent and accurate probability model for each linear layer.
[0035] Step A2: For each of the linear layers, a parameter estimation method is used to fit the set of importance score samples corresponding to the linear layer into a probability density function, and the probability density function is integrated to obtain the cumulative distribution function corresponding to the linear layer.
[0036] The parameter estimation method is either the maximum likelihood estimation method or the moment estimation method, and the probability density function is either the Gaussian distribution function or the Laplace distribution function.
[0037] Understandably, steps A1 to A2, on the one hand, establish a complete statistical model for the importance score of each linear layer by independently fitting the probability density function and integrating it to obtain the cumulative distribution function, which can accurately describe the probability distribution of any importance score within that layer; on the other hand, the obtained cumulative distribution function links the macroscopic objective of sparsity rate with the microscopic threshold required to achieve that objective, providing a reliable mathematical tool for subsequently solving the corresponding threshold accurately based on the sparsity rate of any objective.
[0038] Step B: Based on the target sparsity corresponding to the deep neural network and the constraint that the block sparsity of each block increases from shallow to deep, a tree-structured Bayesian optimizer is used to assign a block sparsity to each block in the deep neural network; the arithmetic mean of the block sparsity is the target sparsity.
[0039] Specifically, step B includes: Step B1: Obtain the total number of blocks in the deep neural network, the verification dataset, and the maximum number of optimization iterations.
[0040] Step B2: In each optimization iteration, a block sparsity sequence satisfying the constraints is generated by the tree-structured Bayesian optimizer, and the block sparsity sequence is configured in the deep neural network. The perplexity corresponding to the block sparsity sequence is evaluated based on the validation dataset, and the probability model of the tree-structured Bayesian optimizer is updated based on the evaluation results. The constraints include that the arithmetic mean of the block sparsity is the target sparsity, and that the block sparsity of each block increases from shallow to deep.
[0041] Specifically, generating the block sparsity rate sequence that satisfies the constraints through the tree-structured Bayesian optimizer includes: Step B21, based on the target sparsity and preset offset Determine the candidate sparsity of the first block of the deep neural network. The tree-structured Bayesian optimizer in the interval The candidate sparsity rate corresponding to the first block of the deep neural network is collected internally.
[0042] The preset offset δ is 0.05.
[0043] Step B22: Determine the first... Candidate sparsity of each block The tree-structured Bayesian optimizer in the interval The deep neural network's first internal acquisition The candidate sparsity rate corresponding to each block; where It is a positive integer greater than 1 and less than or equal to the total number of blocks in the deep neural network.
[0044] Step B23: Calculate the arithmetic mean of the candidate sparsity rates corresponding to each block, and calculate the absolute value of the difference between the arithmetic mean of the candidate sparsity rates corresponding to each block and the target sparsity rate.
[0045] Step B24: Determine whether the absolute value is less than a preset absolute value.
[0046] Step B25: If yes, arrange the candidate sparsity rates corresponding to each block in sequence based on the depth corresponding to each block in the deep neural network to obtain the block sparsity rate sequence; it can be understood that when the absolute value is less than the preset absolute value, the arithmetic mean of the candidate sparsity rates corresponding to each block is close to the target sparsity rate. At this time, the arithmetic mean of the candidate sparsity rates corresponding to each block can be defaulted as the target sparsity rate.
[0047] Step B26: If not, resample the candidate sparsity rates corresponding to each block; it can be understood that if the absolute value is not less than a preset absolute value, continue to execute steps B21 to B25 to achieve resampling of the candidate sparsity rates corresponding to each block.
[0048] Understandably, steps B21 to B26, on the one hand, by setting a sampling interval for the first block that is no higher than the target sparsity, protect the importance of shallow networks at the beginning of the search and reduce the risk of the search process damaging the basic feature extraction capability of deep neural networks; on the other hand, by dynamically setting the sampling lower bound of subsequent blocks to the candidate sparsity of the previous block, a monotonically non-decreasing constraint is directly embedded in the sampling process, ensuring that any generated candidate sparsity sequence satisfies the characteristic of increasing from shallow to deep; furthermore, by calculating the arithmetic mean of the candidate sequences and comparing it with the target sparsity with tolerance, invalid schemes that do not meet the global constraints are filtered out before evaluation, improving the search efficiency and resource utilization of the optimization process.
[0049] The specific method for updating the probabilistic model of the tree-structured Bayesian optimizer based on the evaluation results is a dynamic learning and self-improvement process. Its core lies in using new data obtained from each trial to refine the optimizer's understanding of the parameter space. Specifically, the block sparsity sequence obtained in each optimization iteration and its corresponding model perplexity evaluation result are added as new data points to the historical observation record library. Based on all current historical records, all these block sparsity sequences are sorted according to the perplexity index, and the historical data are divided into two subsets according to a preset proportional quantile: one is a "high-performing" set composed of block sparsity sequences with low perplexity, and the other is a "relatively poor-performing" set composed of block sparsity sequences with high perplexity. The optimizer then uses kernel density estimation, a nonparametric statistical method, to reconstruct two independent probability density models for these two sets respectively: one model aims to describe the distribution of historically "high-performing" block sparsity sequences in the parameter space, i.e., in which regions they tend to appear; this model is called the likelihood function; the other model aims to describe the distribution of "relatively poor-performing" block sparsity sequences. In this way, the optimizer's internal biprobabilistic model is refreshed, enabling it to more accurately identify potential parameter space regions and regions to be avoided. This updated probabilistic model will directly guide the sampling in the next iteration, making the sampler more inclined to generate new candidate points from "high-performing" distributions, thereby gradually focusing the search scope on block sparsity configurations that are more likely to produce low perplexity, i.e., high performance, achieving intelligent convergence of the search process.
[0050] Step B3: When the optimization iteration process reaches the maximum number of optimization iterations, determine the block sparsity corresponding to each block based on the block sparsity sequence corresponding to the lowest perplexity.
[0051] Specifically, based on the depth of each block in the deep neural network, the block sparsity rates in the block sparsity rate sequence corresponding to the lowest perplexity are sequentially assigned to each block; wherein, the depth of the block in the deep neural network is positively correlated with the sequence number of the block sparsity rate assigned to the block in the block sparsity rate sequence.
[0052] Understandably, steps B1 to B3, on the one hand, use a tree-structured Bayesian optimizer to search in a constrained high-dimensional space, which can automatically find the block sparsity allocation scheme that satisfies the overall sparsity objective and monotonicity conditions, thus avoiding the subjectivity and inefficiency of manual trial and error; on the other hand, using perplexity as a performance-oriented evaluation index ensures that the optimization process always takes maintaining the output quality of the deep neural network as the core objective, thus ensuring that the obtained sparsity allocation scheme can maintain the original performance of the deep neural network to the maximum extent while accelerating inference.
[0053] Step C: For each block, based on the block sparsity rate corresponding to the block, a greedy algorithm is used to assign a layer sparsity rate to each linear layer within the block; the weighted average of the layer sparsity rates is the block sparsity rate corresponding to the block.
[0054] Specifically, the step of allocating layer sparsity to each linear layer within the block using a greedy algorithm based on the block sparsity corresponding to the block includes: Step C1: Calculate the total number of parameters for each linear layer within the block.
[0055] Step C2: For each linear layer, determine the ratio between the number of parameters of the linear layer and the total number of parameters as the sparsity weight of the linear layer.
[0056] Step C3: For each linear layer, calculate the ratio between the total number of parameters and the number of parameters of the linear layer, and determine the product of the ratio and the preset sparsity increment as the target sparsity increment corresponding to the linear layer.
[0057] Step C4: Input a specified input value into the deep neural network to generate a standard output vector corresponding to the block.
[0058] Step C5: Set the initial sparsity of all linear layers in the block to zero, and enter the iterative process of layer sparsity allocation based on the sparsity weight, the target sparsity increment of each linear layer and the standard output vector, until the weighted average of the layer sparsity of each linear layer in the block is the block sparsity.
[0059] Specifically, the iterative process of layer sparsity allocation based on the sparsity weights, the target sparsity increments corresponding to each of the linear layers, and the standard output vector, until the weighted average of the layer sparsities of each linear layer within the block is the block sparsity, includes: Step C51: During each iteration of layer sparsity allocation, when the weighted average of the layer sparsities of each linear layer within the block is less than the block sparsity, for each linear layer within the block, after adding its corresponding target sparsity increment to the current layer sparsity of the linear layer, the specified input value is input to the block to generate an output vector corresponding to the block. The Euclidean distance between the output vector and the standard output vector is determined as the error of the linear layer in this iteration. After permanently adding its corresponding target sparsity increment to the current layer sparsity of the linear layer with the smallest error, the weighted average of the current layer sparsities of each linear layer within the block is updated.
[0060] Wherein, the current layer sparsity corresponding to the linear layer refers to the layer sparsity determined for the linear layer during the previous layer sparsity allocation iteration.
[0061] In this context, permanently increasing the current layer sparsity of the linear layer corresponding to the minimum error by its corresponding target sparsity increment means that in each iteration of layer sparsity allocation, only the current layer sparsity of the linear layer corresponding to the minimum error is increased by its corresponding target sparsity increment, while the current layer sparsity of other linear layers remains unchanged.
[0062] Step C52: Determine whether the updated weighted average value is less than the block sparsity corresponding to the block.
[0063] Step C53: If yes, continue the iterative process of layer sparsity allocation.
[0064] Step C54: If not, stop the iterative process of layer sparsity allocation and determine whether the updated weighted average value is greater than the block sparsity corresponding to the block.
[0065] Step C55: If not, output the layer sparsity rate of each linear layer of the block determined in the last iteration.
[0066] Step C56: If yes, optimize the layer sparsity rate determined in the last iteration for each linear layer of the block so that the weighted average of the layer sparsity rates of each linear layer in the block is the block sparsity rate corresponding to the block, and output the optimized layer sparsity rate corresponding to each linear layer of the block.
[0067] Understandably, steps C51 to C56, on the one hand, ensure the accurate achievement of the overall sparsity target within the block by iterative allocation under the constraint of parameter size as weight, thus guaranteeing the effective implementation of the upper-level inter-block allocation strategy; on the other hand, by selecting the linear layer that causes the least change to the current block output at each step to increase its sparsity, a locally optimal greedy strategy is implemented, so that the performance loss caused by each step in the process of approaching the total sparsity target is minimized under the current circumstances; furthermore, by introducing the target sparsity increment with parameter size normalization, the sparsity increase rate of linear layers with different parameter sizes is comparable, avoiding the search process from biasing towards larger matrices with greater influence.
[0068] Specifically, optimizing the layer sparsity rate of each linear layer in the block during the last iteration, so that the weighted average of the layer sparsity rates of each linear layer in the block is the block sparsity rate, includes: Step C561: Count the number of times each linear layer is permanently increased by its corresponding target sparsity increment during the entire layer sparsity allocation iteration process.
[0069] Step C562: Reduce the layer sparsity of the linear layer whose corresponding target sparsity increment has been increased the fewest times during the last iteration, so that the weighted average of the layer sparsities of the linear layers in the block is the block sparsity corresponding to the block.
[0070] Understandably, steps C561 to C562, by reducing the layer sparsity determined in the last iteration of the linear layer whose corresponding target sparsity increment has been increased the fewest times, so that the weighted average of the layer sparsity of each linear layer in the block is the block sparsity, in a fine-tuning manner with minimal impact on overall performance, make the weighted average sparsity of the block accurately fall back to the block sparsity, which helps to improve the output accuracy of the deep neural network after sparsification.
[0071] Step D: For each linear layer in the deep neural network, input the layer sparsity rate corresponding to the linear layer into the cumulative distribution function corresponding to the linear layer to obtain the importance threshold corresponding to the linear layer.
[0072] Understandably, steps A through D, on the one hand, construct a multi-level, fully automated sparse strategy generation system by moving from micro-level activation value statistical modeling to macro-level inter-block optimization allocation, and then to micro-level intra-block greedy allocation, achieving a precise transformation from the overall acceleration goal of deep neural networks to the specific execution threshold of each linear layer; on the other hand, by using the cumulative distribution function as a mathematical tool to transform the optimized layer sparsity rate into an executable importance threshold, enabling all the previous optimization results to be applied to the dynamic sparse operations during inference, completing the closed loop from offline optimization configuration to online efficient inference; furthermore, by comprehensively considering activation value and weight information and adopting a hierarchical optimization strategy, under the same overall sparsity rate, it can allocate sparse resources more intelligently compared to traditional methods, achieving the acceleration goal while better maintaining the original output accuracy of the model.
[0073] Please see Figure 2 , Figure 2 This is a schematic block diagram of a weight-aware activation sparsity device 100 provided in an embodiment of this application. The weight-aware activation sparsity device 100 is used in a deep neural network, which includes multiple blocks from shallow to deep. Each block includes multiple linear layers, and the sparsity of each block increases from shallow to deep. Figure 2 As shown, the weight-aware activation sparsity device 100 includes: The acquisition module 110 is used to acquire the activation value vector of each linear layer input during the forward propagation process of the deep neural network.
[0074] The determination module 120 is used to determine, for each activation value in the same activation value vector, the absolute value of the activation value and the L1 norm of the weight column corresponding to the activation value as the importance score corresponding to the activation value.
[0075] The activation sparsity processing module 130 is used to perform activation sparsity on each of the linear layers based on the importance threshold corresponding to the linear layer and the importance score corresponding to each activation value in the activation value vector input to the linear layer.
[0076] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device and each module described above can be referred to the corresponding process in the aforementioned embodiment of the weight-aware activation sparsity method, and will not be repeated here.
[0077] The terminal device 200 provided in the above embodiments can be implemented as a computer program, which can be used in, for example... Figure 3 The terminal device 200 shown is running on it.
[0078] Please see Figure 3 , Figure 3 The present invention provides a schematic block diagram of the structure of a terminal device 200. The terminal device 200 includes a processor 201 and a memory 202, which are connected via a device bus 203. The memory 202 may include a non-volatile storage medium and internal memory.
[0079] The non-volatile storage medium can store a computer program. The computer program includes program instructions that, when executed by the processor 201, cause the processor 201 to perform any of the aforementioned weight-aware activation sparsity methods.
[0080] The processor 201 provides computing and control capabilities to support the operation of the entire terminal device 200.
[0081] The internal memory provides an environment for the execution of computer programs in non-volatile storage media. When the computer program is executed by the processor 201, the processor 201 can execute any of the weight-aware activation sparsity methods mentioned above.
[0082] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the terminal device 200 involved in the present application. The specific terminal device 200 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0083] It should be understood that processor 201 can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, the general-purpose processor can be a microprocessor or any conventional processor.
[0084] In some embodiments, the processor 201 is configured to run a computer program stored in memory to perform the following steps: During the forward propagation of the deep neural network, for each linear layer, the activation value vector of the input of each linear layer is obtained; For each activation value in the same activation value vector, the L1 norm of the absolute value of the activation value and the weight column corresponding to the activation value is determined as the importance score corresponding to the activation value. For each of the linear layers, activation sparsity is applied to the linear layer based on the importance threshold corresponding to the linear layer and the importance score corresponding to each activation value in the activation value vector input to the linear layer.
[0085] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the terminal device 200 described above can be referred to the corresponding process of the aforementioned weight-aware activation sparsity method, and will not be repeated here.
[0086] This application also provides a computer-readable storage medium storing a computer program that, when executed by one or more processors, causes the one or more processors to implement the weight-aware activation sparsity method provided in this application.
[0087] The computer-readable storage medium can be an internal storage unit of the terminal device 200 in the aforementioned embodiments, such as a hard disk or memory of the terminal device 200. The computer-readable storage medium can also be an external storage device of the terminal device 200, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided with the terminal device 200.
[0088] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A weight-aware activation sparsity method, characterized in that, The method is used in a deep neural network, which includes multiple blocks from shallow to deep, each block including multiple linear layers, and the sparsity of each block increasing from shallow to deep. The method includes: During the forward propagation of the deep neural network, for each linear layer, the activation value vector of the input of each linear layer is obtained; For each activation value in the same activation value vector, the L1 norm of the absolute value of the activation value and the weight column corresponding to the activation value is determined as the importance score corresponding to the activation value. For each of the linear layers, activation sparsity is applied to the linear layer based on the importance threshold corresponding to the linear layer and the importance score corresponding to each activation value in the activation value vector input to the linear layer.
2. The weight-aware activation sparsity method according to claim 1, characterized in that, The activation sparsity of the linear layer based on the importance threshold corresponding to the linear layer and the importance scores corresponding to each activation value in the activation value vector input to the linear layer includes: For each activation value in the activation value vector input to the linear layer, determine whether the importance score corresponding to the activation value is less than the importance threshold corresponding to the linear layer; If so, set the activation value to zero; If not, retain the activation value.
3. The weight-aware activation sparsity method according to claim 1, characterized in that, The method for determining the importance threshold corresponding to each linear layer in the deep neural network includes: For each linear layer in the deep neural network, probability density modeling is performed on the importance scores corresponding to each activation bit of the linear layer to obtain the cumulative distribution function corresponding to the linear layer. Based on the target sparsity corresponding to the deep neural network and the constraint that the block sparsity of each block increases from shallow to deep, a tree-structured Bayesian optimizer is used to assign a block sparsity to each block in the deep neural network; the arithmetic mean of the block sparsity is the target sparsity. For each block, based on the block sparsity corresponding to the block, a greedy algorithm is used to assign a layer sparsity to each linear layer within the block; the weighted average of the layer sparsities is the block sparsity corresponding to the block. For each linear layer in the deep neural network, the layer sparsity rate corresponding to the linear layer is input into the cumulative distribution function corresponding to the linear layer to obtain the importance threshold corresponding to the linear layer.
4. The weight-aware activation sparsity method according to claim 3, characterized in that, The step of performing probability density modeling on the importance scores corresponding to each activation site of the linear layer to obtain the cumulative distribution function corresponding to the linear layer includes: Based on a preset validation dataset, generate a set of importance score samples corresponding to each linear layer of the deep neural network; For each of the linear layers, a parameter estimation method is used to fit the set of importance score samples corresponding to the linear layer into a probability density function, and the probability density function is integrated to obtain the cumulative distribution function corresponding to the linear layer.
5. The weight-aware activation sparsity method according to claim 4, characterized in that, The method of generating an importance score sample set corresponding to each linear layer of the deep neural network based on a preset verification dataset includes: For each input value in the verification dataset, the input value is input into the deep neural network to enable the deep neural network to perform forward propagation, and during the forward propagation process, the verification activation value vector of each input of the linear layer is obtained; For each of the linear layers, the absolute value of the verification activation of each verification activation value in the verification activation value vector corresponding to the linear layer is multiplied by the L1 norm of the weight column corresponding to the verification activation value to obtain the importance score corresponding to each verification activation value in the verification activation value vector corresponding to the linear layer; the importance scores corresponding to each verification activation value in the verification activation value vector corresponding to the linear layer constitute the importance score sample set corresponding to the linear layer.
6. The weight-aware activation sparsity method according to claim 4, characterized in that, The probability density function is a Gaussian distribution function or a Laplace distribution function.
7. The weight-aware activation sparsity method according to claim 3, characterized in that, The constraint that the block sparsity of each block increases from shallow to deep, based on the target sparsity corresponding to the deep neural network and the block sparsity of each block, is used to assign a block sparsity to each block in the deep neural network through a tree-structured Bayesian optimizer, including: Obtain the total number of blocks, the validation dataset, and the maximum number of optimization iterations for the deep neural network; In each optimization iteration, a block sparsity sequence satisfying the constraints is generated by the tree-structured Bayesian optimizer, and the block sparsity sequence is configured in the deep neural network. The perplexity corresponding to the block sparsity sequence is evaluated based on the validation dataset, and the probabilistic model of the tree-structured Bayesian optimizer is updated based on the evaluation results. The constraints include that the arithmetic mean of the block sparsity is the target sparsity, and that the block sparsity of each block increases from shallow to deep. When the optimization iteration process reaches the maximum number of optimization iterations, the block sparsity corresponding to each block is determined based on the block sparsity sequence corresponding to the lowest perplexity.
8. The weight-aware activation sparsity method according to claim 7, characterized in that, The step of generating a block sparsity rate sequence that satisfies the constraints through the tree-structured Bayesian optimizer includes: Based on the target sparsity and preset offset Determine the candidate sparsity of the first block of the deep neural network. The tree-structured Bayesian optimizer in the interval The candidate sparsity rate corresponding to the first block of the deep neural network is collected internally. Determine the first of the deep neural network Candidate sparsity of each block The tree-structured Bayesian optimizer in the interval The deep neural network's first internal acquisition The candidate sparsity rate corresponding to each block; where A positive integer greater than 1 and less than or equal to the total number of blocks in the deep neural network; Calculate the arithmetic mean of the candidate sparsity rates corresponding to each block, and calculate the absolute value of the difference between the arithmetic mean of the candidate sparsity rates corresponding to each block and the target sparsity rate; Determine whether the absolute value is less than a preset absolute value; If so, the candidate sparsity rates corresponding to each block are arranged sequentially based on the depth corresponding to each block in the deep neural network to obtain the block sparsity rate sequence; If not, resample the candidate sparsity rates corresponding to each block.
9. The weight-aware activation sparsity method according to claim 8, characterized in that, The preset offset δ is 0.
05.
10. The weight-aware activation sparsity method according to claim 3, characterized in that, The process of allocating layer sparsity rates to each linear layer within the block based on the block sparsity rate corresponding to the block, using a greedy algorithm, includes: Calculate the total number of parameters for each linear layer within the block; For each of the linear layers, the ratio between the number of parameters in the linear layer and the total number of parameters is determined as the sparsity weight of the linear layer; For each of the linear layers, the ratio between the total number of parameters and the number of parameters of the linear layer is calculated, and the product of the ratio and the preset sparsity increment is determined as the target sparsity increment for the linear layer. A specified input value is input into the deep neural network to generate a standard output vector corresponding to the block; The initial sparsity of all linear layers within the block is set to zero, and an iterative process of layer sparsity allocation is entered based on the sparsity weights, the target sparsity increments corresponding to each linear layer, and the standard output vector, until the weighted average of the layer sparsity of each linear layer within the block is the block sparsity corresponding to the block.
11. The weight-aware activation sparsity method according to claim 10, characterized in that, The iterative process of layer sparsity allocation based on the sparsity weights, the target sparsity increments corresponding to each linear layer, and the standard output vector, until the weighted average of the layer sparsities of each linear layer within the block is the block sparsity, includes: In each iteration of layer sparsity allocation, when the weighted average of the layer sparsities of each linear layer within the block is less than the block sparsity corresponding to the block, for each linear layer within the block, after adding its corresponding target sparsity increment to the current layer sparsity corresponding to the linear layer, the specified input value is input to the block to generate an output vector corresponding to the block, and the Euclidean distance between the output vector and the standard output vector is determined as the error corresponding to the linear layer in this iteration, and after permanently adding its corresponding target sparsity increment to the current layer sparsity of the linear layer corresponding to the minimum error, the weighted average of the current layer sparsities of each linear layer within the block is updated; Determine whether the updated weighted average is less than the block sparsity corresponding to the block; If so, continue the iterative process of layer sparsity allocation; If not, stop the iterative process of layer sparsity allocation and determine whether the updated weighted average value is greater than the block sparsity corresponding to the block; If not, output the layer sparsity of each linear layer of the block as determined in the last iteration; If so, optimize the layer sparsity rate determined in the last iteration of each linear layer of the block so that the weighted average of the layer sparsity rates of each linear layer in the block is the block sparsity rate corresponding to the block, and output the optimized layer sparsity rate corresponding to each linear layer of the block.
12. The weight-aware activation sparsity method according to claim 11, characterized in that, The optimization of the layer sparsity rate determined in the last iteration of each linear layer of the block, so that the weighted average of the layer sparsity rates of each linear layer within the block is the block sparsity rate, includes: The number of times each of the linear layers is permanently increased by its corresponding target sparsity increment during the entire layer sparsity allocation iteration process is counted. The layer sparsity determined in the last iteration of the linear layer whose corresponding target sparsity increment is increased the fewest times is reduced, so that the weighted average of the layer sparsities of the linear layers in the block is the block sparsity corresponding to the block.
13. A weight-aware activation sparsity device, characterized in that, The device is used in a deep neural network, the deep neural network comprising multiple blocks from shallow to deep, each block comprising multiple linear layers, the sparsity of each block increasing from shallow to deep, the device comprising: The acquisition module is used to acquire the activation value vector of each linear layer during the forward propagation of the deep neural network. The determination module is used to determine, for each activation value in the same activation value vector, the absolute value of the activation value and the L1 norm of the weight column corresponding to the activation value as the importance score corresponding to the activation value; The activation sparsity processing module is used to perform activation sparsity on each of the linear layers based on the importance threshold corresponding to the linear layer and the importance score corresponding to each activation value in the activation value vector input to the linear layer.
14. A terminal device, characterized in that, The terminal device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the weight-aware activation sparsity method as described in any one of claims 1 to 12.
15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the weight-aware activation sparsity method as described in any one of claims 1 to 12.