Neural network sparsification method, medium and computer program product

Abstract

The invention discloses a neural network sparsification method, a medium and a computer program product. The neural network rarefaction method comprises the following steps: setting a corresponding bias item for each neuron in a network unit to be rarefied in a neural network system, wherein the value of the bias item is dynamically updated based on the activation frequency of the corresponding neuron; calculating a band bias selection score corresponding to each neuron based on an input activation vector and a bias item of the to-be-sparsified network unit; based on the band bias selection score, performing a balanced TopK sparsification operation to select K neurons with the highest score, and generating a sparse mask for the to-be-sparsified network unit; and generating a sparse activation output of the network element to be sparse according to the sparse mask and the input activation vector.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence (AI), and more specifically to methods for sparse neural networks, computer-readable storage media, and computer program products. Background Technology

[0002] Neural network sparsification is a technique that reduces computation and storage by minimizing redundant or insignificant node connections in a model, while maintaining performance as much as possible. Many weights or activation values ​​in a neural network are often redundant or contribute very little. By identifying and pruning these unimportant parts, a dense network structure can be transformed into a sparse structure, significantly reducing computation and memory usage while preserving the model's expressive power. This is an important approach to achieving lightweight model deployment and efficient inference. Summary of the Invention

[0003] In one aspect, this application discloses a neural network sparsification method, comprising: setting a corresponding bias term for each neuron in the network unit to be sparsified in the neural network system, wherein the value of the bias term is dynamically updated based on the activation frequency of the corresponding neuron; calculating a biased selection score corresponding to each neuron based on the input activation vector and the bias term of the network unit to be sparsified; performing a balanced TopK sparsification operation based on the biased selection score to select the K neurons with the highest scores and generate a sparse mask for the network unit to be sparsified; and generating a sparse activation output of the network unit to be sparsified based on the sparse mask and the input activation vector.

[0004] In other respects, this application discloses a computer-readable storage medium storing instructions and a computer program product including instructions. These instructions, when executed by one or more processors, cause the processors to perform the methods described in this application. Attached Figure Description

[0005] When read in conjunction with the accompanying drawings, various aspects of this disclosure are best understood through the following detailed description. It should be noted that, in accordance with standard practice in the art, the features are not drawn to scale. In fact, for clarity of discussion, the dimensions of the features may be arbitrarily increased or decreased.

[0006] Figure 1 This schematically illustrates the neuron states that may occur in a neural network system where traditional TopK operations are applied for network sparsification.

[0007] Figure 2 A schematic diagram illustrating the workflow of implementing the balanced TopK sparsity operation in a neural network system according to an embodiment of this application is shown.

[0008] Figure 3A schematic diagram illustrating the workflow of implementing balanced TopK sparsity operation based on a neuron grouping strategy according to an embodiment of this application is shown.

[0009] Figure 4 A schematic diagram of a multilayer perceptron (MLP) layer architecture applying the SwiGLU activation function and equalizing TopK sparsity operations according to an embodiment of this application is shown.

[0010] Figure 5 This is a flowchart illustrating a neural network sparsification method according to an embodiment of this application.

[0011] Figure 6 This is a schematic diagram of a computing device that can be implemented according to an embodiment of the present invention. Detailed Implementation

[0012] The following disclosure provides numerous different embodiments or examples for implementing various features of the provided subject matter. Specific examples of components and arrangements are described below to simplify this disclosure. Of course, these are merely examples and not limiting.

[0013] Furthermore, the various operations will be described as multiple discrete operations in a manner most conducive to understanding the illustrative embodiments; however, the order of description should not be construed as implying that these operations must depend on the order. In particular, these operations do not need to be performed in the order presented.

[0014] Many weights or activation values ​​in neural networks are often redundant or contribute little. By identifying and pruning these unimportant parts, dense network structures can be transformed into sparse structures, thereby significantly reducing computation and memory usage while maintaining the model's expressive power. This is an important way to achieve lightweight model deployment and efficient inference.

[0015] TopK sparsity, as an effective model compression and acceleration technique, achieves sparse computation by retaining only the K largest elements in the activations or weights of a neural network and setting the rest to zero, thereby significantly reducing the computational load and memory consumption of the model. This technique is particularly valuable in accelerating the sparse training and inference of large-scale language models, facilitating the deployment of high-performance models in resource-constrained environments. However, despite its significant potential, existing TopK sparsity methods still face several challenges, limiting their further optimization and widespread application.

[0016] First, the non-differentiability of the TopK operation hinders gradient propagation. Since the TopK selection process is inherently discrete and discontinuous, gradient information cannot be directly passed to the preceding network layers during backpropagation. Existing solutions typically use a straight-through estimator (STE) for gradient approximation, but this method introduces significant gradient bias, affecting the accuracy and convergence stability of model optimization, and thus limiting the performance of the final model.

[0017] Secondly, the degradation problem caused by uneven distribution of neuron activation cannot be ignored. In traditional TopK training, due to the distribution characteristics of initial weights or input data, some neurons have persistently high activation values ​​and are repeatedly selected to participate in updates, while other neurons remain dormant for a long time. This "Matthew effect" intensifies as the training process progresses, eventually leading to two typical problems: one is "neuronal degradation," where neurons that have not been activated for a long time gradually lose their updating ability, and their corresponding weights stagnate, resulting in a permanent loss of the model's effective capacity; the other is "activation collapse," where the model's expression is dominated by a few active neurons, impairing overall diversity and generalization ability.

[0018] Figure 1 This schematically illustrates the possible neuron states in a neural network system where traditional TopK operations are applied for network sparsity reduction. For example... Figure 1 As shown, neurons in a neural network system may exhibit significant activation imbalances: some neurons are frequently selected and activated very frequently, while others remain dormant for extended periods and are activated very infrequently. After multiple training rounds, these dormant neurons gradually degenerate into "dead neurons." If a large number of neurons in a neural network fail to update effectively due to prolonged inactivity, it will ultimately impair the overall capacity and expressive power of the model.

[0019] Furthermore, current balancing strategies for mitigating uneven neuronal activation distribution have limited effectiveness. Existing studies have attempted to promote balanced neuronal utilization by introducing activation regularization, periodic re-initialization, or manually adjusting selection thresholds. However, these methods often require cumbersome hyperparameter tuning and cannot fundamentally change the competitive mechanism of neuronal activation. The balancing effect is unstable and sometimes even interferes with the main optimization trajectory of the model.

[0020] This application proposes an adaptive bias-based balanced TopK sparsity method. This method maintains a bias term for each neuron and dynamically updates the value of this bias term based on the historical activation frequency of each neuron, thereby adaptively adjusting the probability of each neuron being selected by the TopK operation. This mechanism can proactively balance the activation opportunities of neurons during training, avoiding the long-term suppression or overactivity of some neurons, thus effectively alleviating the problems of neuron degeneration and activation collapse, and improving the training stability and final accuracy of the sparse model.

[0021] Figure 2 A schematic diagram illustrating the workflow of implementing the balanced TopK sparsity operation in a neural network system according to an embodiment of this application is shown.

[0022] like Figure 2 As shown, the balanced TopK sparsity process according to an embodiment of this application first sets a bias term b_i for each neuron in the neural network system, and the initial value of the bias term can be set to 0. Next, in the forward propagation phase, biased TopK selection is performed, that is, for each activation value in the input activation vector a = [a_1, a_2, ..., a_n], a biased selection score s_i = a_i + b_i is calculated, and then all neurons are sorted according to the score, and only the K neurons with the highest scores are retained, thereby generating a sparse mask m. For neurons selected based on the biased selection score, m_i = 1, and for unselected neurons, m_i = 0. Finally, a sparse activation output vector output is generated based on the original input activation vector. Note that output_i = a_i × m_i, that is, the activation output value is the product of the original input activation value a_i and the mask m_i, and the bias term is only used to affect the selection process and does not participate in the final output.

[0023] Furthermore, according to embodiments of this application, the value of the bias term of each neuron can be dynamically updated based on the historical activation frequency of each neuron, thereby adaptively adjusting the probability of each neuron being selected by the TopK operation. For example... Figure 2 As shown, after the forward computation of one or more training batches is completed, the activation frequency count_i of each neuron in the current training batch (i.e., m_i=1) can be counted. Then, the value of the bias term can be updated according to the activation frequency as follows: for neurons with a high activation frequency (e.g., whose activation frequency count_i is higher than the target activation frequency), the value of the bias term of that neuron can be decreased, while for neurons with a low activation frequency (e.g., whose activation frequency count_i is lower than the target activation frequency), the value of the bias term of that neuron can be increased.

[0024] In one embodiment according to this application, the value of the neuron's bias term can be updated, for example, using the following formula: b_i = b_i – α×(count_i) The target activation frequency is defined as follows: α is the learning rate or step size of the bias update, and target is the predetermined target activation frequency. In one embodiment, the target activation frequency can be determined based on the number of samples in each training batch (batch_size), the number of neurons (K) to be retained in the balanced TopK sparsification operation, and the total number of neurons (n) in the network unit to be sparsified. For example, the target activation frequency, i.e., the desired uniform activation count, can be determined using the formula target = batch_size × (K / n).

[0025] Based on the dynamic updating of the bias terms of each neuron, the bias term value of a neuron with a high activation frequency is reduced, thereby decreasing the probability of that neuron being selected in the next round of sparsification training; conversely, the bias term value of a neuron with a low activation frequency is increased, thereby increasing the probability of that neuron being selected in the next round of sparsification training. Through this dynamic feedback mechanism, the system can adaptively guide all neurons towards uniform activation, thus effectively alleviating the neuron degeneration problem caused by the traditional TopK sparsification process.

[0026] According to one embodiment of this application, the network unit to be sparsified for the TopK sparsification operation may include the network layer to be sparsified in a neural network system. According to other embodiments of this application, the network layer to be sparsified may be divided into multiple neuron groups according to a preset grouping size, and each neuron group may be used as a network unit to be sparsified and the TopK sparsification operation may be performed independently.

[0027] Figure 3 A schematic diagram illustrating the workflow of a balanced TopK sparsity operation based on a neuron grouping strategy according to an embodiment of this application is shown. The diagram illustrates the grouped balanced TopK sparsity process designed to adapt to hardware parallel computing architectures. Figure 3 As shown, the network layer to be sparsified is divided into multiple neuron banks according to a preset grouping size B. For example, for an activation vector with a dimension of 1024, if the grouping size is 64, it is divided into 16 neuron banks. Figure 3 Only four neuron groups are shown, but it is easy to understand that the number of neuron groups can be set according to actual needs, and this invention does not limit this. Within each neuron group, independent execution is performed as follows: Figure 2The diagram illustrates the balanced TopK sparsity operation. For example, if each neuron group retains K_bank neurons, then the sparsity rate s = 1 – K_bank / B. Furthermore, the dynamic updates of the bias terms for neurons are performed independently within each neuron group to ensure a balanced activation distribution within each group.

[0028] This group-balanced TopK sparsification process distributes the global sparsification objective across fixed-size groups of neurons, offering several hardware advantages: Each neuron group's output exhibits a regular and predictable sparsity pattern (e.g., a fixed number of activation elements, K_bank). This regular sparsity pattern aligns perfectly with the data arrangement of the hardware's parallel computing units, simplifying data indexing and retrieval logic and facilitating efficient hardware implementation. Furthermore, compared to the global balanced TopK sparsification process, the group-balanced TopK sparsification process allows different groups to independently generate diverse sparse combinations, preserving richer feature representation paths while maintaining the overall sparsity rate. This enhances the model's expressive power and ultimately improves the accuracy of the final task.

[0029] According to some embodiments of this application, the following can also be used: Figure 2 and Figure 3 The equalization TopK sparsification process shown is combined with the SwiGLU (Switch-Gated Linear Unit) activation function commonly used in large language models to achieve efficient sparsification of multilayer perceptron (MLP) layers.

[0030] Figure 4 A schematic diagram of an MLP layer architecture applying the SwiGLU activation function and a balanced TopK sparsity operation according to an embodiment of this application is shown. The SwiGLU activation function can be decomposed into a Hadamard product (element-wise multiplication) of the Sigmoid activation function and the gate output. According to an embodiment of this application, a balanced TopK sparsity operation can be inserted after the Sigmoid activation function and before the Hadamard product.

[0031] like Figure 4As shown, the input activation vector is first normalized by an RMSnorm (Root Mean Square Normalization) layer. Then, the data flows in parallel into two branches: Up (projection upwards) and Gate (gating). The output of the Gate branch is processed by a Sigmoid activation function to generate gating weights. The output of the Sigmoid activation function is then subjected to a TopK equalization sparsity operation to generate a sparsity mask. This sparsity mask is simultaneously applied to the outputs of both the Up and Gate branches, achieving synchronous and position-corresponding sparsity for the output tensors of both branches. Next, the sparsified Up output tensor and the sparsified Gate output tensor are multiplied element-wise to obtain a sparsified gated fusion result tensor. Finally, this sparsity result tensor is passed to the Down (projection downwards) layer. The Down layer can skip the computation of rows corresponding to the masked positions, thus achieving end-to-end sparsity acceleration.

[0032] The following will further combine Figure 5 The overall flow of the neural network sparsification method according to embodiments of this application is described below.

[0033] Figure 5 This is a flowchart illustrating a neural network sparsity method 500 according to an embodiment of this application. Method 500 can be executed by a system or device for implementing network sparsity. Figure 5 In the illustrated embodiment, method 500 may include operations 510 to 540. In other embodiments, methods for implementing neural network sparsity may include more or fewer or different operations. This disclosure is not limiting in this respect.

[0034] At position 510, a corresponding bias term is set for each neuron in the network unit to be sparsified in the neural network system. The value of this bias term is dynamically updated based on the activation frequency of the corresponding neuron. According to some embodiments of this application, the network unit to be sparsified may include the network layer to be sparsified in the neural network system. According to other embodiments of this application, the network layer to be sparsified in the neural network system can be divided into multiple neuron groups according to a preset grouping size, wherein each neuron group is configured as a network unit to be sparsified to undergo a sparsification process independently, and then the sparse activation output of the network layer to be sparsified is generated based on the sparse activation output of each neuron group.

[0035] At position 520, the biased selection score for each neuron is calculated based on the input activation vector of the network unit to be sparsified and the bias term of each neuron.

[0036] At position 530, a balanced TopK sparsification operation is performed based on the biased selection score of each neuron to select the K neurons with the highest scores and generate a sparse mask for the network unit to be sparsified.

[0037] At position 540, the sparse activation output of the network unit to be sparsified is generated based on the sparse mask and the input activation vector.

[0038] According to an embodiment of this application, the neural network sparsification method may further include: after the forward computation of one or more training batches of the neural network system is completed, counting the activation frequency of each neuron in the current training batch; when the activation frequency of a neuron in the current training batch is higher than the target activation frequency, decreasing the value of the bias term corresponding to the neuron; and when the activation frequency of a neuron in the current training batch is lower than the target activation frequency, increasing the value of the bias term corresponding to the neuron.

[0039] According to an embodiment of this application, the initial value of the bias term for each neuron can be set to 0.

[0040] According to embodiments of this application, the target activation frequency can be determined based on the number of samples in each training batch, the number of neurons K to be retained in the balanced TopK sparsification operation, and the total number of neurons in the network unit to be sparsified.

[0041] According to embodiments of this application, the value of the bias term corresponding to each neuron can be decreased or increased based on a preset bias update step size and target activation frequency. For example, the value of the bias update step size can be set between 0.001 and 0.01.

[0042] Furthermore, according to embodiments of this application, a balanced TopK sparsity operation can be applied to an MLP layer employing the SwiGLU activation function. For example, a sparse mask can be generated based on the output of the Sigmoid activation function of the MLP layer by applying the balanced TopK sparsity operation, and this sparse mask can be applied to both the Up branch and the Gate branch of the SwiGLU activation function simultaneously.

[0043] Embodiments of this application may include a computer-readable storage medium. The medium stores instructions that, when executed by one or more processors, cause the processors to perform the neural network sparsification methods described herein.

[0044] Embodiments of this application may also include a computer program product. This computer program product includes instructions that, when executed by one or more processors, cause the processors to perform the neural network sparsification methods described herein.

[0045] The following describes an example process for implementing the neural network sparsification method proposed in this application in a neural network system. For example, the neural network sparsification process can be implemented in four stages: parameter configuration, model structure modification, model training, and inference acceleration.

[0046] First, the system allows for parameter configuration, setting parameters such as Bank size, sparsity rate, bias update step size, and statistical period, establishing basic rules for subsequent sparsity processing. Then, the model structure is modified by inserting a TopK balancing operation module at an appropriate location in the target network layer (e.g., an MLP layer). This includes, for example, initializing the bias terms of each neuron, connecting the original activation outputs to the input of the TopK balancing operation module, and then connecting the output of the TopK balancing operation module to subsequent network layers.

[0047] The model training phase can be further divided into the forward propagation phase, the back propagation phase, and the bias update phase. In the forward propagation phase, biased selection scores are calculated, TopK sparse selection is performed, sparse activation outputs are generated, and the activation frequency of each neuron in the current training batch is counted. In the back propagation phase, gradient information is passed through a pass-through estimator, which only propagates through activated locations. In the bias update phase, the target activation frequency is calculated after one or more training batches, and the bias term values ​​of each neuron are updated based on the calculated target activation frequency and a preset bias update step size, achieving balanced adjustment of neuron activation.

[0048] Finally, in the inference acceleration stage, the values ​​of the bias terms of each neuron are fixed and no longer updated. TopK sparse selection is performed based on the biased selection scores to generate a sparse mask, so that subsequent calculations can skip unnecessary operations, achieving efficient inference acceleration on actual hardware.

[0049] The balanced TopK sparsification scheme according to the embodiments of this application has many advantages over the traditional TopK sparsification scheme.

[0050] First, the adaptive bias update mechanism effectively addresses the issues of neuron degeneration and uneven activation distribution, ensuring that all neurons receive a balanced activation opportunity. This fully preserves model capacity and significantly improves training stability. Experiments show that performance remains stable even with high sparsity and large training data scales, without a sharp drop in accuracy or training crashes. Furthermore, by adjusting the bias update step size, the degree of balance in activation distribution can be flexibly controlled to adapt to different sparsity rates and model sizes.

[0051] Secondly, the TopK sparsity equalization scheme according to the embodiments of this application is simple to implement and has minimal overhead. It only requires maintaining a bias vector with the same activation dimension and can achieve dynamic equalization through lightweight arithmetic operations. The storage and computational overhead are very low, and it can be seamlessly integrated into existing deep learning training frameworks.

[0052] Furthermore, by incorporating a grouping strategy, the balanced TopK sparsity scheme according to embodiments of this application can provide more sparse combination possibilities compared to traditional sparsity patterns, while maintaining high model accuracy even in high sparsity scenarios (e.g., 75%, 87.5%). Moreover, the regular intra-group sparsity pattern facilitates hardware acceleration, fully adapting to dedicated sparse computing chips and delivering significant inference acceleration in practical deployments.

[0053] Figure 6 This is a schematic diagram of a computing device that can be implemented according to embodiments of the present invention, and can be used to implement the system or device for neural network sparsity described above. For example... Figure 6 As shown, computing device 600 may include bus 602 or other communication mechanism for transmitting information, and one or more hardware processors 604 coupled to bus 602 for processing information. The one or more hardware processors 604 may include, for example, one or more general-purpose microprocessors.

[0054] like Figure 6 As shown, in some embodiments, computing device 600 may further include main memory 606 coupled to bus 602. Main memory 606 is used to store information and instructions executed by one or more processors 604, such as random access memory (RAM), cache, and / or other dynamic storage devices. Main memory 606 may also be used to store temporary variables or other intermediate information during the execution of instructions executed by one or more processors 604. These instructions, when stored in storage media accessible to one or more processors 604, can cause computing device 600 to become a dedicated machine customized to perform the operations specified in the instructions. Storage device 608 may include non-volatile and / or volatile storage media. Non-volatile storage media may include, for example, optical discs or magnetic disks. Volatile storage media may include dynamic memory. Common forms of storage media may include, for example, floppy disks, hard disks, solid-state drives, magnetic tape, or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, DRAM, PROM, and EPROM, FLASH-EPROM, NVRAM, any other memory chip or cartridge, or their networking versions.

[0055] like Figure 6 As shown, in some embodiments, computing device 600 may further include one or more communication interfaces 610 coupled to bus 602. Communication interface 610 may provide bidirectional data communication coupling to one or more network links connected to one or more networks. As another example, communication interface 610 may be a local area network (LAN) card to provide data communication connectivity to a LAN-compatible (or WAN component communicating with a WAN) network. Wireless links may also be implemented.

[0056] The execution of certain operations can be distributed across processors rather than residing within a single machine, but rather deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0057] Each of the processes, methods, and algorithms described herein may be embodied in code modules executed by one or more computer systems or computer processors including computer hardware, and may be fully or partially automated by these code modules. The processes and algorithms may be implemented, partially or fully, in dedicated circuit systems.

[0058] When the functions disclosed herein are implemented as software functional units and sold or used as stand-alone products, they may be stored in a processor-executable, non-volatile, computer-readable storage medium. Specific technical solutions (all or part) disclosed herein, or aspects contributing to the prior art, may be embodied in the form of a software product. The software product may be stored in a storage medium and includes several instructions that cause a computing device (which may be a personal computer, server, network device, etc.) to perform all or some steps of the methods of the embodiments of this application. The storage medium may include a flash drive, portable hard disk drive, ROM, RAM, magnetic disk, optical disk, other media operable to store program code, or any combination thereof.

[0059] Specific embodiments further provide a system including a processor and a computer-readable storage medium storing instructions executable by the processor to cause the system to perform operations corresponding to steps in any method of the embodiments disclosed above. Specific embodiments further provide a computer-readable storage medium storing instructions executable by one or more processors to cause the one or more processors to perform operations corresponding to steps in any method of the embodiments disclosed above.

[0060] The embodiments disclosed herein can be implemented via a cloud platform, server, or server cluster (collectively referred to below as the "Service System") that interacts with a client. The client can be a terminal device or a client registered by a user at the platform, wherein the terminal device can be a mobile terminal, a personal computer (PC), or any device that can have the platform application installed.

[0061] The various features and processes described above can be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Additionally, certain method or process blocks may be omitted in some embodiments. The methods and processes described herein are not limited to any particular order, and their associated blocks or states may be executed in other suitable orders. For example, described blocks or states may be executed in an order other than that specifically disclosed, or multiple blocks or states may be combined into a single block or state. Example blocks or states may be executed sequentially, in parallel, or in some other manner. Blocks or states may be added to or removed from the disclosed example embodiments. The exemplary systems and components described herein may be configured differently than described. For example, components may be added to, removed from, or rearranged compared to the disclosed example embodiments.

[0062] The various operations of the exemplary methods described herein can be performed at least in part by an algorithm. The algorithm may be included in program code or instructions stored in memory (e.g., the aforementioned computer-readable storage medium). This algorithm may include a machine learning algorithm. In some embodiments, the machine learning algorithm may not explicitly turn the computer into an execution function but may learn from training data to produce a predictive model of the execution function.

[0063] The various operations of the exemplary methods described herein can be performed, at least in part, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, these processors can constitute an engine of processor implementation that operates to perform one or more of the operations or functions described herein.

[0064] Similarly, the methods described herein may be implemented at least in part by a processor, wherein one or more specific processors are instances of hardware. For example, at least some operations of the methods may be performed by one or more processors or an engine implemented by a processor. Furthermore, one or more processors may also be operable to support the execution of relevant operations in a “cloud computing” environment or as the execution of relevant operations in a “Software as a Service” (SaaS) context. For example, at least some operations may be performed by a group of computers (as an example of a machine containing processors), wherein these operations are accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application programming interfaces (APIs)).

[0065] The execution of certain operations can be distributed across processors rather than residing within a single machine, and can be deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0066] Throughout this specification, multiple instances may be implemented as components, operations, or structures described as a single instance. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of these individual operations may be performed simultaneously, and not necessarily in the order illustrated. Structures and functions presented as separate components in the example configuration may be implemented as composite structures or components. Similarly, structures and functions presented as single components may be implemented as single components. These and other variations, modifications, additions, and improvements fall within the scope of this document.

[0067] As used herein, "or" is inclusive rather than exclusive unless explicitly indicated by the context. Therefore, in this document, "A, B, or C" means "A, B, A and B, A and C, B and C, or A, B, and C" unless explicitly indicated by the context. Furthermore, "and" is combined and separate unless explicitly indicated by the context. Therefore, in this document, "A and B" means "A and B, combined or separate" unless explicitly indicated by the context. Additionally, multiple instances of resources, operations, or structures described herein may be provided as a single instance. Furthermore, the boundaries between various resources, operations, engines, and data storage devices are somewhat arbitrary and specific operations are illustrated within the context of a particular illustrative configuration. Other functional assignments are foreseeable and fall within the scope of various embodiments of this disclosure. Generally, structures and functions presented as individual resources in example configurations may be implemented as combined structures or resources. Similarly, structures and functions presented as single resources may be implemented as single resources. These and other changes, modifications, additions, and improvements fall within the scope of the embodiments of this disclosure as expressed in the appended claims. Therefore, this specification and drawings should be considered illustrative rather than restrictive.

[0068] The terms “comprising” or “including” are used to indicate the presence of a subsequently claimed feature, but do not preclude the addition of other features. Unless otherwise specifically stated or otherwise understood in the context in which they are used, conditional language such as “may,” “can,” “may,” and “can” is generally intended to convey that certain embodiments include certain features, components, and / or steps that are not included in other embodiments. Therefore, this conditional language is generally not intended to imply that one or more embodiments require features, components, and / or steps in any way, or that one or more embodiments must include logic for determining whether such features, components, and / or steps are included in or performed in any particular embodiment, with or without user input or prompts.

[0069] Although the general outline of the subject matter has been described with reference to specific exemplary embodiments, various modifications and changes may be made to these embodiments without departing from the broad scope of embodiments of this disclosure. Where more than one embodiment is disclosed, these embodiments of the subject matter may be referred to individually or collectively herein as the term "invention," this is for convenience only and is not intended to automatically limit the scope of this application to any single disclosure or concept.

[0070] The embodiments illustrated herein are described in detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. Therefore, “implementation” is not intended to be limiting, and the scope of the various embodiments is defined only by the appended claims and their full scope.

Claims

1. A method for sparsifying neural networks, characterized in that, The neural network sparsification method includes: For each neuron in the network unit to be sparsified in the neural network system, a corresponding bias term is set, wherein the value of the bias term is dynamically updated based on the activation frequency of the corresponding neuron; Based on the input activation vector of the network unit to be sparsified and the bias term, calculate the biased selection score corresponding to each neuron; Based on the biased selection scores, a balanced TopK sparsification operation is performed to select the K neurons with the highest scores, generating sparse masks for the network units to be sparsified; and The sparse activation output of the network unit to be sparsified is generated based on the sparse mask and the input activation vector.

2. The neural network sparsity method according to claim 1, characterized in that, The neural network sparsification method also includes: After the forward computation of one or more training batches of the neural network system is completed, the activation frequency of each neuron in the current training batch is counted. When the activation frequency of a neuron in the current training batch is higher than the target activation frequency, the value of the bias term corresponding to that neuron is decreased; and When the activation frequency of a neuron in the current training batch is lower than the target activation frequency, the value of the bias term corresponding to that neuron is increased.

3. The neural network sparsity method according to claim 1 or 2, characterized in that, The initial value of the bias term is set to 0.

4. The neural network sparsity method according to claim 2, characterized in that, The neural network sparsification method further includes determining the target activation frequency based on the number of samples in each training batch, the number of neurons K to be retained in the balanced TopK sparsification operation, and the total number of neurons in the network unit to be sparsified.

5. The neural network sparsity method according to claim 2, characterized in that, The neural network sparsification method further includes: reducing or increasing the value of the bias term corresponding to the neuron based on a preset bias update step size and the target activation frequency.

6. The neural network sparsity method according to claim 5, characterized in that, The value of the bias update step size is set to be between 0.001 and 0.

01.

7. The neural network sparsity reduction method according to claim 1, characterized in that, The network unit to be sparsified includes the network layer to be sparsified in the neural network system.

8. The neural network sparsity method according to claim 1, characterized in that, The neural network sparsification method also includes: The network layer to be sparsified in the neural network system is divided into multiple neuron groups according to a preset grouping size, wherein each neuron group is configured as a network unit to be sparsified; and The sparse activation output of the network layer to be sparsified is generated based on the sparse activation output of each neuron group.

9. The neural network sparsity method according to claim 7 or 8, characterized in that, The network layer to be sparsified includes a multilayer perceptron layer using the SwiGLU activation function.

10. The neural network sparsity method according to claim 9, characterized in that, The neural network sparsification method also includes: Based on the output of the Sigmoid activation function of the multilayer perceptron layer, a sparse mask is generated by applying the equalization TopK sparsification operation, so that the sparse mask can be applied to both the upward projection branch and the gated branch of the SwiGLU activation function.

11. A computer-readable storage medium storing instructions, characterized in that, When executed by one or more processors, the instructions cause the one or more processors to perform the neural network sparsification method according to any one of claims 1-10.

12. A computer program product, the computer program product comprising instructions, characterized in that, When executed by one or more processors, the instructions cause the one or more processors to perform the neural network sparsification method according to any one of claims 1-10.

Images