Large language model training method and device based on sparse neural network

Abstract

The invention provides a large language model training method and device based on a sparse neural network, and the method comprises the steps: determining the target sparseness according to the current training step number for each training step number under the condition that the current moment is a structure adjustment moment, determining a pruning edge set and a growth edge set according to the target sparseness and a structure evolution criterion, adjusting a network structure of a to-be-trained model according to the pruning edge set and the growth edge set; and under the condition that the current moment is not the structure adjustment moment, determining an attention step length by utilizing a mixed sparse attention mechanism according to the current training step number, and performing parameter updating on the to-be-trained model according to the attention step length. According to the method, the to-be-trained model is trained in combination with the dynamically evolved sparse network structure and the mixed sparse attention mechanism, so that the model performance is kept and the training calculation overhead is reduced to the maximum extent.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a large language model training method and device based on a sparse neural network. Background Art

[0002] In recent years, large language models (LLMs) have received extensive attention from academia and industry. Thanks to their outstanding performance in various language generation tasks, large language models have promoted the development of various artificial intelligence applications. However, the parameter scale of large language models is very large, resulting in very high training and inference costs, which poses huge challenges to deployment resources, user experience, and economic costs.

[0003] Modern neural networks are usually full of redundant structures, so people can often remove most of the redundant connections in the model without losing model accuracy. Compared with dense networks, this sparse model structure has higher parameter utilization and computational efficiency, and can also significantly reduce inference time in some cases. The current methods for obtaining sparse network models include dynamic sparse training algorithms and pruning methods.

[0004] The dynamic sparse training algorithm trains a sparse neural network model from scratch and dynamically adjusts and updates the sparse network structure during the training process to reduce the parameters and computational complexity of the model, thereby improving training efficiency and reducing the demand for hardware resources. The main idea of ​​this algorithm is to dynamically adjust the sparsity of the model according to the performance of the model and the demand for computing resources during the training process, so that the model can adapt to different computing environments while maintaining performance. The purpose of the dynamic sparse training algorithm is to obtain an efficient sparse model, which can also be used to reduce the computational load and memory usage during the entire training process. Pruning methods can also be applied to the compression of neural network models by eliminating redundant connections in the model. In pruning methods, it is usually necessary to first train a dense model and then sparse the model by pruning some of its connections. Existing pruning methods can be divided into three categories: structured pruning, semi-structured pruning, and unstructured pruning.

[0005] However, the dynamic sparse training algorithm cannot significantly save the training cost of large language models. Some studies have shown that the dynamic sparse training algorithm can significantly reduce the computational cost of the fully connected layer, and it can train 80% sparse networks such as ResNet-50 and MobileNet from scratch without reducing performance. However, directly applying the dynamic sparse training algorithm to the pre-training stage of large language models has achieved very limited gains. This is because large language models require a large number of parameters in complex language modeling tasks, making it very difficult to sparse large language models. Some studies have shown that the dynamic sparse training algorithm can usually only achieve a sparsity of no more than 20% without losing the performance of large language models. In addition, pruning methods can only reduce the computational cost of the inference stage of large language models, and there is no gain in saving the computational cost of the training stage. This is because pruning methods are mainly applicable to the model application stage after training, and are not practical for solving the high computational cost problem during the pre-training of large language models.

[0006] In summary, the existing large language model training technology has the problem of excessive computational overhead in the training phase. Summary of the invention

[0007] The present invention provides a large language model training method and device based on a sparse neural network, which are used to solve the defect of excessive computational overhead in the large language model training stage in the prior art, and achieve low computational overhead during the training process without obvious degradation of model performance after training.

[0008] The present invention provides a large language model training method based on a sparse neural network, comprising the following steps: For each training step, when the current moment is a structural adjustment moment, determine the target sparsity according to the current training step, determine the pruning edge set and the growing edge set according to the target sparsity and the structural evolution criterion, and adjust the network structure of the model to be trained according to the pruning edge set and the growing edge set; When the current moment is not a structural adjustment moment, the attention step length is determined according to the current training step number using a hybrid sparse attention mechanism, and the parameters of the model to be trained are updated according to the attention step length.

[0009] According to a large language model training method based on a sparse neural network provided by the present invention, the target sparsity is determined according to the current training step number, specifically including: When the current number of training steps is less than a first preset number of steps, calculating the target sparsity according to a first preset formula; When the current training step number is greater than or equal to the first preset step number and less than the second preset step number, the target sparsity is the same as the target sparsity at the previous moment; When the current number of training steps is greater than or equal to the second preset number of steps and less than the third preset number of steps, calculating the target sparsity according to a second preset formula; Wherein, the first preset formula includes:

[0010] In the formula, represents the number of training steps, , Indicates the first preset number of steps, represents the target sparsity, represents the maximum sparsity, represents the frequency of sparsity change, Indicates the pruning interval; The second preset formula includes:

[0011] In the formula, represents the number of training steps, , represents the growth frequency, Indicates the first preset number of steps, Indicates the second preset number of steps, Indicates the third preset number of steps.

[0012] According to a large language model training method based on a sparse neural network provided by the present invention, a pruning edge set and a growing edge set are determined according to the target sparsity and the structural evolution criterion, and the network structure of the model to be trained is adjusted according to the pruning edge set and the growing edge set, which specifically includes: According to the current sparsity and the target sparsity, the current sparsity and the target sparsity of each layer of the model to be trained are obtained; For each layer of the model to be trained, determining a pruning edge set and a growing edge set using a structural evolution criterion according to a structural update ratio, a current sparsity of the layer, and a target sparsity of the layer; The sparse mask of each layer is updated according to the growing edge set and the pruned edge set of the layer, and the structure of the model to be trained is updated according to the sparse mask until each layer of the model to be trained is adjusted.

[0013] According to a large language model training method based on a sparse neural network provided by the present invention, the structure update ratio is:

[0014] in, Represents the number of training steps, and there is a Make Indicates The size of the update ratio, and , …, .

[0015] According to a large language model training method based on a sparse neural network provided by the present invention, the attention step length is determined by using a hybrid sparse attention mechanism according to the current training step number, specifically including: When the current training step number is less than the second preset step number, the attention step length is the preset step length; When the current training step number is greater than or equal to the second preset step number, the attention step length is 1.

[0016] According to a large language model training method based on a sparse neural network provided by the present invention, according to the structure update ratio, the current sparsity of the layer and the target sparsity of the layer, the structure evolution criterion is used to determine the pruning edge set and the growing edge set, specifically including: Determine the number of pruning according to the structural update ratio, the current sparsity of the layer and the target sparsity of the layer; determine the pruned edge set according to the number of pruning; Determine the growth amount according to the structure update ratio, the current sparsity of the layer and the target sparsity of the layer; Determine a random growth quantity according to the growth quantity, and determine a gradient growth quantity according to the random growth quantity and the growth quantity; Determine a gradient growing edge set according to the pruning edge set and the gradient growing quantity; Determine a random growth edge set according to the pruned edge set, the gradient growth edge set and the random growth quantity; A growing edge set is determined according to the random growing edge set and the gradient growing edge set.

[0017] The present invention also provides a large language model training device based on a sparse neural network, comprising the following modules: A structure adjustment unit, for each training step, when the current moment is a structure adjustment moment, determines a target sparsity according to the current training step, determines a pruned edge set and a grown edge set according to the target sparsity and a structure evolution criterion, and adjusts the network structure of the to-be-trained model according to the pruned edge set and the grown edge set; A parameter updating unit is used to determine the attention step length using a hybrid sparse attention mechanism according to the current training step number when the current moment is not a structural adjustment moment, and to update the parameters of the model to be trained according to the attention step length.

[0018] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a large language model training method based on a sparse neural network as described above is implemented.

[0019] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described large language model training methods based on a sparse neural network.

[0020] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned large language model training methods based on a sparse neural network.

[0021] The large language model training method and device based on sparse neural network provided by the present invention, for each training step, when the current moment is a structural adjustment moment, the target sparsity is determined according to the current training step, the pruned edge set and the grown edge set are determined according to the target sparsity and the structural evolution criterion, and the network structure of the model to be trained is adjusted according to the pruned edge set and the grown edge set; when the current moment is not a structural adjustment moment, the attention step length is determined according to the current training step length using a hybrid sparse attention mechanism, and the parameters of the model to be trained are updated according to the attention step length. The present invention combines the dynamically evolving sparse network structure and the hybrid sparse attention mechanism to train the model to be trained, so as to simultaneously maintain the model performance and minimize the training calculation overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 This is one of the flow charts of the large language model training method based on sparse neural network provided by the present invention.

[0024] Figure 2 This is the second flow chart of the large language model training method based on sparse neural network provided by the present invention.

[0025] Figure 3 This is the third flow chart of the large language model training method based on sparse neural network provided by the present invention.

[0026] Figure 4 It is a schematic diagram of target sparsity change of the large language model training method based on sparse neural network provided by the present invention.

[0027] Figure 5 It is a schematic diagram of the structural update ratio change of the large language model training method based on sparse neural network provided by the present invention.

[0028] Figure 6 It is a schematic diagram of the attention step change of the large language model training method based on sparse neural network provided by the present invention.

[0029] Figure 7 It is a schematic diagram of a sparse attention mask of a large language model training method based on a sparse neural network provided by the present invention.

[0030] Figure 8 It is a structural schematic diagram of a large language model training device based on a sparse neural network provided by the present invention.

[0031] Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] Combine the following Figure 1-Figure 7 The present invention describes a large language model training method based on a sparse neural network.

[0034] Figure 1 This is one of the flow charts of the large language model training method based on sparse neural network provided by the present invention, such as Figure 1 As shown, the method comprises the following steps: Step 110: For each training step, when the current moment is a structural adjustment moment, determine the target sparsity according to the current training step, determine the pruned edge set and the grown edge set according to the target sparsity and the structural evolution criterion, and adjust the network structure of the model to be trained according to the pruned edge set and the grown edge set.

[0035] It should be noted that the present invention is particularly aimed at the training stage of the model to be trained, and operates on each training step. That is to say, before executing step 110 and step 120 mentioned in the present invention, it is first necessary to determine whether the current step is a training stage, such as Figure 2 and Figure 3 As shown, T is the total number of training steps.

[0036] After confirming that the current time is in the training phase, it is determined whether the current time is a structural adjustment time. The structural adjustment time means that the sparse network model will be structurally adjusted at the current time, rather than parameter update. Specifically, in a specific embodiment, the current time is recorded as ,if , then the current moment is considered to be the moment of structural adjustment, where represents the structural evolution interval.

[0037] When it is determined that the current moment is a structural adjustment moment, the model to be trained is structurally adjusted. In the process of structural adjustment, the target sparsity of each layer of the model to be trained needs to be determined first, and then the network structure is adjusted according to the target sparsity.

[0038] It should be noted that the adjustment of the network structure based on the target sparsity with the training model is completed through dynamic sparse training. Specifically, the training of the model to be trained mentioned in the present invention involves three different stages: pruning, sparse and growing stages. The pruning stage converts the dense model into a sparse model, aiming to establish a suitable initial sparse network structure. The sparse stage focuses on reducing computational overhead. The growth stage re-establishes connections and is responsible for compensating for the model performance degradation caused by sparsification. The interaction of these three stages ensures the reasonable conversion of the model parameter scale, enabling it to maintain performance while utilizing parameter redundancy.

[0039] In more detail, the distinction between the pruning, thinning and growing stages can be determined based on the number of training steps and the first preset number of steps, the second preset number of steps and the third preset number of steps. Then, the target sparsity is determined.

[0040] It should be noted that the target sparsity is calculated differently for different stages. The following is an explanation of how to determine the target sparsity. In some embodiments, the target sparsity is determined based on the current number of training steps, specifically including: When the current number of training steps is less than a first preset number of steps, calculating the target sparsity according to a first preset formula; When the current training step number is greater than or equal to the first preset step number and less than the second preset step number, the target sparsity is the same as the target sparsity at the previous moment; When the current number of training steps is greater than or equal to the second preset number of steps and less than the third preset number of steps, calculating the target sparsity according to a second preset formula; Wherein, the first preset formula includes:

[0041] In the formula, represents the number of training steps, , Indicates the first preset number of steps, represents the target sparsity, represents the maximum sparsity, represents the frequency of sparsity change, Indicates the pruning interval; The second preset formula includes:

[0042] In the formula, represents the number of training steps, , represents the growth frequency, Indicates the first preset number of steps, Indicates the second preset number of steps, Indicates the third preset number of steps.

[0043] Specifically, if Figure 3 As shown, the first preset number of steps is recorded as ,if , that is, when the number of training steps is greater than 0 and less than the first preset number of steps, it is determined that the current stage is pruning, and the target sparsity is calculated using the first preset formula. The target sparsity at this time can also be called pruning sparsity. At this stage, the dense model is converted into a sparse model to ensure that a suitable sparse initialization is obtained. The embodiment of the present invention adopts a step-by-step pruning strategy, which is divided into The number of training steps is to gradually introduce sparsity into the dense network. The sparsity It is determined by the cubic attenuation function, and the specific formula is shown in the first preset formula.

[0044] The first preset formula is:

[0045] in, , and Represent the maximum sparsity, sparsity change frequency and pruning interval respectively. The length of the pruning phase is Number of training steps.

[0046] The second preset number of steps is recorded as ,if , that is, when the number of training steps is greater than or equal to the first preset number of steps and less than the second preset number of steps, it is determined that the current stage is in the sparse stage, and the target sparsity is the same as the target sparsity at the previous moment.

[0047] It should be noted that after the pruning stage, the model enters the sparse stage, in which dynamic sparse training is performed on the sparse structure established in the pruning stage. The length of this stage is Number of training steps.

[0048] The third preset number of steps is recorded as ,if , that is, when the number of training steps is greater than or equal to the second preset number of steps and less than the third preset number of steps, it is determined that the current stage is in the growth stage, and the target sparsity is calculated using the second preset formula. The target sparsity at this time can also be called the growth sparsity. At this stage, in order to enhance the expressive power of the model and compensate for the performance loss that may occur in the pruning and sparse stages, the embodiment of the present invention will strategically restore the model connection. The growth stage adopts a progressive method symmetrical to the pruning stage. The sparsity of the sparse network is gradually reduced in the growth iterations. The specific formula is shown in the second preset formula.

[0049] The second preset formula is:

[0050] in, represents the growth frequency. The length of the growth phase is Number of training steps, is the frequency of sparsity change.

[0051] Furthermore, in some embodiments, the maximum number of training steps is recorded as ,if , that is, when the number of training steps is greater than or equal to the third preset number of steps but is still in the training stage, the target sparsity is set to 0.

[0052] In summary, the sparseness change scheme of the embodiment of the present invention is as follows: Figure 4 shown.

[0053] Based on the above embodiment, after the target sparsity is determined, the pruned edge set and the grown edge set are determined according to the target sparsity and the structural evolution criterion. It should be emphasized that the dynamic sparse training mentioned in the embodiment of the present invention requires structural adjustment of each layer according to the determined target sparsity.

[0054] It should be emphasized that during the training process, the embodiment of the present invention allows different numbers of connections to be pruned and grown, and for the connection growth strategy, the embodiment of the present invention adopts a hybrid strategy that combines gradient amplitude growth and random growth of the connection, and fuses the two in a specific proportion. It should be understood that the introduction of random growth effectively broadens the exploration range of the model structure and reduces the possibility of the model converging to a suboptimal structure.

[0055] In addition, for the pruning strategy, the embodiment of the present invention adopts a simple pruning criterion based on the weight amplitude of the connection. It should be understood that the pruning criterion does not introduce additional hyperparameters and still performs well.

[0056] The following further describes the determination of the pruning edge set and the growing edge set according to the target sparsity and the structural evolution criterion. In some embodiments, the pruning edge set and the growing edge set are determined according to the target sparsity and the structural evolution criterion, and the network structure of the model to be trained is adjusted according to the pruning edge set and the growing edge set, specifically including: According to the current sparsity and the target sparsity, the current sparsity and the target sparsity of each layer of the model to be trained are obtained; For each layer of the model to be trained, determining a pruning edge set and a growing edge set using a structural evolution criterion according to a structural update ratio, a current sparsity of the layer, and a target sparsity of the layer; The sparse mask of each layer is updated according to the growing edge set and the pruned edge set of the layer, and the structure of the model to be trained is updated according to the sparse mask until each layer of the model to be trained is adjusted.

[0057] Specifically, first, according to the current sparsity With target sparsity Get the first The current sparsity of the layer With layer target sparsity In some embodiments, the ER strategy (Erdős-Rény) is used in combination with the current sparsity With target sparsity The current sparsity of each layer With layer target sparsity to confirm.

[0058] After that, the predetermined structural evolution criteria are used to determine the pruning edge set and growing edge set of each layer according to the structural update ratio, the current sparsity of the layer, and the target sparsity of the layer. The pruning edge set and the growing edge set are used to update the sparse mask of the layer, and the sparse mask is used to update the structure of the model to be trained. The steps are repeated until each layer of the model to be trained is adjusted.

[0059] Different from the existing structural evolution schemes, the structural evolution criterion of the present invention introduces flexibility by allowing different numbers of pruning and growing connections, which facilitates adjusting the network sparsity according to the preset sparsity change pattern. In addition, after determining the total sparsity of the model, the structural evolution criterion of the present invention adopts the ER strategy to determine the sparsity distribution of each layer. When a given layer The layer target sparsity of The current sparsity of the layer At the same time, structural evolution will dynamically adjust the number of pruned or grown connections to match the target sparsity, which can achieve a higher model sparsity and significantly reduce the computational overhead of the training phase. Existing pruning techniques can only reduce the computational overhead of the inference phase of large language models, and there is no gain in saving the computational overhead of the training phase. In addition, the methods of the prior art are mainly applicable to the later stages of training, and are not practical for solving the problem of high computational costs during the pre-training of large language models. The present invention specifically designs structural evolution criteria for the inherent algorithmic efficiency problem of large language models, which can be applied to the pre-training stage of large language models.

[0060] Furthermore, in order to further illustrate the determination of the pruned edge set and the grown edge set, a specific embodiment is given. In this embodiment, the pruned edge set and the grown edge set are determined using a structural evolution criterion according to the structural update ratio, the current sparsity of the layer, and the target sparsity of the layer, specifically including: Determine the number of pruning according to the structural update ratio, the current sparsity of the layer and the target sparsity of the layer; determine the pruned edge set according to the number of pruning; Determine the growth amount according to the structure update ratio, the current sparsity of the layer and the target sparsity of the layer; Determine a random growth quantity according to the growth quantity, and determine a gradient growth quantity according to the random growth quantity and the growth quantity; Determine a gradient growing edge set according to the pruning edge set and the gradient growing quantity; Determine a random growth edge set according to the pruned edge set, the gradient growth edge set and the random growth quantity; A growing edge set is determined according to the random growing edge set and the gradient growing edge set.

[0061] Specifically, it should be explained that in each structural evolution step, the structural evolution criterion mentioned in the present invention removes a portion of existing connections with the smallest absolute weight amplitude, and also preferentially restores a certain number of idle connections with the largest gradient amplitude. The remaining connections are randomly generated in each layer. Structural update ratio Determined by a piecewise decaying cosine annealing scheme.

[0062] Record The number of parameters of the layer is , the parameters are , and the sparse mask (indicating whether the connection between neurons exists) is . Use Indicates the structural update ratio, Represents the proportion of connections through random growth.

[0063] For each layer in the neural network , the number of pruning branches is .

[0064] The growth amount is .

[0065] The number of random growths is .

[0066] The number of gradient growth is .

[0067] The pruned edge set is ,in is a function that takes the largest K elements.

[0068] The gradient growing edge set is ,in is the loss function.

[0069] The randomly grown edge set is ,in is a function that randomly selects K elements.

[0070] The growing edge set is .

[0071] Further, based on the above embodiment, according to the growth edge set and pruned edge sets Update sparse mask . .

[0072] In some embodiments, the structure update ratio is:

[0073] in, Represents the number of training steps, and there is a Make Indicates The size of the update ratio, and , , , …, , .

[0074] In a specific embodiment, the structure update ratio is as follows: Figure 5 As shown, the network is guaranteed to have robust evolutionary capabilities at different sparsity levels.

[0075] The present invention uses dynamic sparse training in some stages of training, and can use dynamic sparse training to significantly reduce the computational overhead of large language models, solving the problem that using dynamic sparse training throughout the process cannot bring about significant computational overhead savings. Existing methods directly apply dynamic sparse methods to the pre-training stage of large language models. Since large language models require a large number of parameters in complex language modeling tasks, the training overhead savings that can be achieved by dynamic sparse training algorithms are very limited without losing model performance. The present invention improves traditional full-process dynamic sparse training, proposes a sparsity change scheme, introduces a pruning stage at the beginning of training, introduces a growth stage at the end of training, and uses dynamic sparse training in the middle of training. This ensures that the performance of the model after training is not degraded, but fully utilizes the advantages of the dynamic sparse network to reduce the computational overhead during training.

[0076] Step 120: When the current moment is not a structural adjustment moment, the attention step length is determined using a hybrid sparse attention mechanism according to the current number of training steps, and the parameters of the model to be trained are updated according to the attention step length.

[0077] If the current moment is not a structural adjustment moment, the parameters of the model to be trained are updated.

[0078] In a specific embodiment, the current time is recorded as ,if , then the current moment is not considered to be a structural adjustment moment, where represents the structural evolution interval.

[0079] On this basis, more specifically, the present invention adopts a hybrid sparse attention mechanism to update the parameters of the model to be trained.

[0080] It is important to explain that in the early stages of training, the hybrid sparse attention mechanism uses a sparse self-attention mask to save the computational overhead of the self-attention operation. As the model becomes denser, the hybrid sparse attention mechanism transitions to a dense self-attention mask to ensure full recovery of the model performance. The embodiments of the present invention can efficiently save the computational overhead of the attention operation while maintaining optimal model performance.

[0081] It should be noted that the self-attention operation mentioned in the embodiment of the present invention includes: a self-attention layer embeds the input matrix Transformed into an output matrix, affected by the connection mode Control, where Represents the input embedding sequence Length, Representative The index set that the output vector focuses on. , and They are responsible for the given Transformed into a weight matrix for query, key, or value. The self-attention output at each position is calculated as the sum of values ​​weighted by the scaled dot product similarity of the key and the query:

[0082] in, ,

[0083] in, Represents an internal dimension.

[0084] It is understandable that in one embodiment, the model to be trained is an autoregressive model such as GPT-2, and the dense self-attention setting , allowing each element to focus on all previous positions, including itself.

[0085] In addition, the sparse self-attention mentioned in the embodiment of the present invention includes: different attention heads share the same sparse mask. In the sparse self-attention mentioned in the embodiment of the present invention, each Output vector Focus on the front Locations (such as Figure 7 ), and every one of the positions (such as Figure 7 ), where represents the self-attention step. Formally, for ,have .

[0086] Based on the above embodiments, Figure 6 Shown the variation of the self-attention mask step size during training for Hybrid Sparse Attention.

[0087] In some embodiments, the attention step length is determined using a hybrid sparse attention mechanism according to the current training step number, specifically including: When the current training step number is less than the second preset step number, the attention step length is the preset step length; When the current training step number is greater than or equal to the second preset step number, the attention step length is 1.

[0088] Specifically, the second preset number of steps is recorded as ,if , that is, when the number of training steps is less than the second preset number of steps, the attention step length is set to the preset step length The preset step length is predetermined, and the present invention does not limit its specific amount, and can be set according to actual conditions. It should be noted that when the step length is , the model uses a sparse self-attention mask.

[0089] if , that is, when the number of training steps is greater than or equal to the second preset number of steps and less than the total number of training steps, the attention step length is set to . It should be noted that when the step size is 1, the model uses a dense self-attention mask.

[0090] After obtaining the attention step, the parameters can be updated according to the attention step.

[0091] The present invention utilizes a hybrid sparse attention scheme to further reduce the computational overhead brought by the self-attention layer in the training stage of a large language model. It can be understood that, in addition to the fully connected layer, the self-attention layer contributes significantly to the computational overhead. The computational overhead of the fully connected layer comes from the weight update, while the computationally intensive self-attention operation brings the computational overhead of the self-attention layer. The prior art cannot significantly reduce the computational overhead brought by the self-attention layer in the training stage of a large language model while ensuring that the model performance does not decrease. The hybrid sparse attention scheme proposed in the present invention not only utilizes the advantage of low computational overhead brought by the sparse self-attention mechanism, but also compensates for the performance loss brought by the sparse self-attention mechanism through the dense self-attention mechanism in the final stage.

[0092] Further, jump to the next step and repeat steps 110-120 until the current step is greater than the total training steps and the training phase is completed.

[0093] In order to further demonstrate the superiority of the training method provided by the present invention, in a specific embodiment, a large language model training based on a sparse neural network provided by the present invention and a conventional training method were tested and compared based on the GPT-2 model. Table 1 shows the computational cost savings and model performance on different data sets brought about by using the training scheme of the present invention in the pre-training stage of the GPT-2 model.

[0094] Table 1 Comparison of computational overhead and model performance on GPT-2 model pre-training tasks

[0095] As can be seen from Table 1, the model performance obtained by the large language model training method based on sparse neural network provided by the present invention has no obvious loss compared with the original model, but greatly reduces the floating-point calculation amount in the pre-training stage.

[0096] The large language model training method based on sparse neural network provided by the present invention determines the target sparsity according to the current training step number when the current moment is a structural adjustment moment, determines the pruning edge set and the growing edge set according to the target sparsity and the structural evolution criterion, and adjusts the network structure of the model to be trained according to the pruning edge set and the growing edge set; when the current moment is not a structural adjustment moment, determines the attention step length according to the current training step number using the hybrid sparse attention mechanism, and updates the parameters of the model to be trained according to the attention step length. The present invention combines the dynamically evolving sparse network structure and the hybrid sparse attention mechanism to train the model to be trained, so as to simultaneously maintain the model performance and minimize the training calculation overhead.

[0097] The following is a description of the large language model training device based on a sparse neural network provided by the present invention. The large language model training device based on a sparse neural network described below and the large language model training method based on a sparse neural network described above can be referred to each other. Figure 8 As shown, the device includes the following modules: A structure adjustment unit 810 is used for, for each training step, when the current moment is a structure adjustment moment, determining a target sparsity according to the current training step, determining a pruning edge set and a growing edge set according to the target sparsity and a structure evolution criterion, and adjusting the network structure of the to-be-trained model according to the pruning edge set and the growing edge set; The parameter updating unit 820 is used to determine the attention step length using the hybrid sparse attention mechanism according to the current training step number when the current moment is not the structural adjustment moment, and update the parameters of the model to be trained according to the attention step length.

[0098] According to a large language model training device based on a sparse neural network provided by the present invention, the target sparsity is determined according to the current training step number, specifically including: When the current number of training steps is less than a first preset number of steps, calculating the target sparsity according to a first preset formula; When the current training step number is greater than or equal to the first preset step number and less than the second preset step number, the target sparsity is the same as the target sparsity at the previous moment; When the current number of training steps is greater than or equal to the second preset number of steps and less than the third preset number of steps, calculating the target sparsity according to a second preset formula; Wherein, the first preset formula includes:

[0099] In the formula, represents the number of training steps, , Indicates the first preset number of steps, represents the target sparsity, represents the maximum sparsity, represents the frequency of sparsity change, Indicates the pruning interval; The second preset formula includes:

[0100] In the formula, represents the number of training steps, , represents the growth frequency, Indicates the first preset number of steps, Indicates the second preset number of steps, Indicates the third preset number of steps.

[0101] According to a large language model training device based on a sparse neural network provided by the present invention, a pruned edge set and a grown edge set are determined according to the target sparsity and a structural evolution criterion, and a network structure of a model to be trained is adjusted according to the pruned edge set and the grown edge set, specifically comprising: According to the current sparsity and the target sparsity, the current sparsity and the target sparsity of each layer of the model to be trained are obtained; For each layer of the model to be trained, determining a pruning edge set and a growing edge set using a structural evolution criterion according to a structural update ratio, a current sparsity of the layer, and a target sparsity of the layer; The sparse mask of each layer is updated according to the growing edge set and the pruned edge set of the layer, and the structure of the model to be trained is updated according to the sparse mask until each layer of the model to be trained is adjusted.

[0102] According to a large language model training device based on a sparse neural network provided by the present invention, the structure update ratio is:

[0103] in, Represents the number of training steps, and there is a Make Indicates The size of the update ratio, and , , , …, , .

[0104] According to a large language model training device based on a sparse neural network provided by the present invention, the attention step length is determined by using a hybrid sparse attention mechanism according to the current training step number, specifically including: When the current training step number is less than the second preset step number, the attention step length is the preset step length; When the current training step number is greater than or equal to the second preset step number, the attention step length is 1.

[0105] According to a large language model training device based on a sparse neural network provided by the present invention, the pruning edge set and the growing edge set are determined by using a structural evolution criterion according to the structural update ratio, the current sparsity of the layer and the target sparsity of the layer, specifically including: Determine the number of pruning according to the structural update ratio, the current sparsity of the layer and the target sparsity of the layer; determine the pruned edge set according to the number of pruning; Determine the growth amount according to the structure update ratio, the current sparsity of the layer and the target sparsity of the layer; Determine a random growth quantity according to the growth quantity, and determine a gradient growth quantity according to the random growth quantity and the growth quantity; Determine a gradient growing edge set according to the pruning edge set and the gradient growing quantity; Determine a random growth edge set according to the pruned edge set, the gradient growth edge set and the random growth quantity; A growing edge set is determined according to the random growing edge set and the gradient growing edge set.

[0106] The large language model training device based on sparse neural network provided by the present invention determines the target sparsity according to the current training step for each training step, determines the pruning edge set and the growing edge set according to the target sparsity and the structural evolution criterion, and adjusts the network structure of the model to be trained according to the pruning edge set and the growing edge set; when the current moment is not a structural adjustment moment, determines the attention step length according to the current training step using the hybrid sparse attention mechanism, and updates the parameters of the model to be trained according to the attention step length. The present invention combines the dynamically evolving sparse network structure and the hybrid sparse attention mechanism to train the model to be trained, so as to simultaneously maintain the model performance and minimize the training calculation overhead.

[0107] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9As shown, the electronic device may include: a processor (processor) 910, a communication interface (Communications Interface) 920, a memory (memory) 930 and a communication bus 940, wherein the processor 910, the communication interface 920, and the memory 930 complete mutual communication through the communication bus 940. The processor 910 can call the logic instructions in the memory 930 to execute a large language model training method based on a sparse neural network, the method comprising: for each training step, when the current moment is a structural adjustment moment, determining the target sparsity according to the current training step, determining the pruning edge set and the growing edge set according to the target sparsity and the structural evolution criterion, and adjusting the network structure of the model to be trained according to the pruning edge set and the growing edge set; when the current moment is not a structural adjustment moment, determining the attention step length according to the current training step number using a hybrid sparse attention mechanism, and updating the parameters of the model to be trained according to the attention step length.

[0108] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0109] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the large language model training method based on a sparse neural network provided by the above methods, and the method includes: for each training step, when the current moment is a structural adjustment moment, determining the target sparsity according to the current training step, determining the pruned edge set and the growing edge set according to the target sparsity and the structural evolution criterion, and adjusting the network structure of the model to be trained according to the pruned edge set and the growing edge set; when the current moment is not a structural adjustment moment, determining the attention step according to the current training step using a hybrid sparse attention mechanism, and updating the parameters of the model to be trained according to the attention step.

[0110] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the large language model training method based on a sparse neural network provided by the above-mentioned methods, the method comprising: for each training step, when the current moment is a structural adjustment moment, determining the target sparsity according to the current training step, determining the pruned edge set and the growing edge set according to the target sparsity and the structural evolution criterion, and adjusting the network structure of the model to be trained according to the pruned edge set and the growing edge set; when the current moment is not a structural adjustment moment, determining the attention step length according to the current training step number using a hybrid sparse attention mechanism, and updating the parameters of the model to be trained according to the attention step length.

[0111] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0112] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A large language model training method based on sparse neural network, characterized in that: include: For each training step, when the current moment is a structural adjustment moment, determine the target sparsity according to the current training step, determine the pruning edge set and the growing edge set according to the target sparsity and the structural evolution criterion, and adjust the network structure of the model to be trained according to the pruning edge set and the growing edge set; When the current moment is not a structural adjustment moment, the attention step length is determined according to the current training step number using a hybrid sparse attention mechanism, and the parameters of the model to be trained are updated according to the attention step length.

2. The large language model training method based on sparse neural network according to claim 1 is characterized in that: Determine the target sparsity based on the current number of training steps, including: When the current number of training steps is less than a first preset number of steps, calculating the target sparsity according to a first preset formula; When the current training step number is greater than or equal to the first preset step number and less than the second preset step number, the target sparsity is the same as the target sparsity at the previous moment; When the current number of training steps is greater than or equal to the second preset number of steps and less than the third preset number of steps, calculating the target sparsity according to a second preset formula; Wherein, the first preset formula includes: ， In the formula, represents the number of training steps, , Indicates the first preset number of steps, represents the target sparsity, represents the maximum sparsity, represents the frequency of sparsity change, Indicates the pruning interval; The second preset formula includes: ， In the formula, represents the number of training steps, , represents the growth frequency, Indicates the first preset number of steps, Indicates the second preset number of steps, Indicates the third preset number of steps.

3. The large language model training method based on sparse neural network according to claim 1 is characterized in that: Determining a pruning edge set and a growing edge set according to the target sparsity and the structural evolution criterion, and adjusting the network structure of the to-be-trained model according to the pruning edge set and the growing edge set, specifically includes: According to the current sparsity and the target sparsity, the current sparsity and the target sparsity of each layer of the model to be trained are obtained; For each layer of the model to be trained, determining a pruning edge set and a growing edge set using a structural evolution criterion according to a structural update ratio, a current sparsity of the layer, and a target sparsity of the layer; The sparse mask of each layer is updated according to the growing edge set and the pruned edge set of the layer, and the structure of the model to be trained is updated according to the sparse mask until each layer of the model to be trained is adjusted.

4. The large language model training method based on sparse neural network according to claim 3 is characterized in that: The structural update ratio is: ， in, Represents the number of training steps, and there is a Make , Indicates The size of the update ratio, and .

5. The large language model training method based on sparse neural network according to claim 2 is characterized in that: The attention step length is determined by using the hybrid sparse attention mechanism according to the current training step number, including: When the current training step number is less than the second preset step number, the attention step length is the preset step length; When the current training step number is greater than or equal to the second preset step number, the attention step length is 1.

6. The large language model training method based on sparse neural network according to claim 3 is characterized in that: According to the structural update ratio, the current sparsity of the layer and the target sparsity of the layer, the pruning edge set and the growing edge set are determined using the structural evolution criterion, specifically including: Determine the number of pruning according to the structural update ratio, the current sparsity of the layer and the target sparsity of the layer; determine the pruned edge set according to the number of pruning; Determine the growth amount according to the structure update ratio, the current sparsity of the layer and the target sparsity of the layer; Determine a random growth quantity according to the growth quantity, and determine a gradient growth quantity according to the random growth quantity and the growth quantity; Determine a gradient growing edge set according to the pruning edge set and the gradient growing quantity; Determine a random growth edge set according to the pruned edge set, the gradient growth edge set and the random growth quantity; A growing edge set is determined according to the random growing edge set and the gradient growing edge set.

7. A large language model training device based on sparse neural network, characterized in that: include: A structure adjustment unit, for each training step, when the current moment is a structure adjustment moment, determines a target sparsity according to the current training step, determines a pruned edge set and a grown edge set according to the target sparsity and a structure evolution criterion, and adjusts the network structure of the to-be-trained model according to the pruned edge set and the grown edge set; A parameter updating unit is used to determine the attention step length using a hybrid sparse attention mechanism according to the current training step number when the current moment is not a structural adjustment moment, and to update the parameters of the model to be trained according to the attention step length.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the large language model training method based on a sparse neural network is implemented as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the large language model training method based on a sparse neural network is implemented as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the large language model training method based on a sparse neural network is implemented as described in any one of claims 1 to 6.

Images