Large model compression method and system based on adversarial learning and sparse regularization

Abstract

The invention provides a large model compression method and system based on adversarial learning and sparse regularization, and the method comprises the steps: cutting an attention head with less contribution through a multi-head attention mechanism with a mask, so as to reduce the amount of parameters; secondly, sparse constraint residual structure pruning is carried out on the FFN structure, and information flow blockage is avoided while the residual structure is optimized; a self-adaptive threshold value is also introduced to reduce invalid calculation; and finally, in combination with the regular constraint of the adversarial learning normal form and the source model characteristics, sharing the influence of the hyper-parameters on the pruning model into a discriminator, and optimizing the discriminator by using an FISTA optimization method, thereby further improving the stability of pruning training. According to the method, the adversarial learning normal form is applied to the pruning task of the large model, so that single-stage pruning of the large model is realized, the pruned model can keep a similar reasoning effect of a source model without fine tuning, and the design greatly reduces the calculation burden and the calculation cost.

Description

Technical Field

[0001] The present invention belongs to the technical field of large model compression, and particularly relates to a large model compression method and system based on adversarial learning and sparse regularization. Background Art

[0002] With the rapid development of large model technology, the scale and complexity of models have increased exponentially. For large language models, which have a huge number of parameters, training and deploying these models require consuming a large amount of GPU video memory resources and data storage space. This not only increases the hardware cost but also makes it extremely difficult to deploy and apply the models on edge devices and low-cost servers.

[0003] Large model compression significantly reduces the hardware cost and energy consumption by reducing the computational resource requirements, accelerating the inference speed, improving the deployment flexibility and transmission efficiency, enabling the wide application of large models on resource-constrained devices. Currently, the mainstream large model compression technologies include model pruning for model structure, model quantization for computational precision, and distillation technology based on knowledge transfer; among which the large model pruning technology is the main development branch of the current large model compression technology.

[0004] The structured pruning technology proposed by Xinyin Ma et al.: Although it can selectively remove non-critical coupling structures according to gradient information, it requires secondary training of the pruned model, which undoubtedly increases the training cost; The one-shot pruning technology proposed by Elias Frantar et al.: simplifies the pruning problem into a large-scale sparse regression instance, without fine-tuning and the pruning loss accuracy can be ignored, but the process is cumbersome, it is not an end-to-end method, the sample difference is large, and a special acceleration operator needs to be designed; The Wanda pruning method proposed by Mingjie Sun et al.: prunes according to weights and activations. Although it does not require retraining or weight update, the selection of hyperparameters has a great impact on the model performance, and it is a simple but unstable pruning method.

[0005] The above overall technologies have limitations. In the prior art, unstructured pruning cannot adapt to general acceleration operators, the pruned model often needs fine-tuning to restore performance, and the pruning process is unstable, and the model performance is severely affected by hyperparameters such as the pruning rate.

[0006] In view of this, it is very meaningful to propose a large model compression method and system based on adversarial learning and sparse regularization. Summary of the Invention

[0007] To solve the problems in the prior art that unstructured pruning cannot adapt to general acceleration operators, the pruned model often needs to be fine-tuned to restore performance, the pruning process is unstable, and the model performance is severely affected by hyperparameters such as the pruning rate, etc., the present invention provides a large model compression method and system based on adversarial learning and sparse regularization to solve the above existing technical defect problems.

[0008] In a first aspect, the present invention proposes a large model compression method based on adversarial learning and sparse regularization, and the method includes the following steps:

[0009] S1. Obtain the encoded features of the source model: Obtain text data from a text database, segment and encode it using a tokenizer adapted to the source model to obtain the input sequence of the source model. After fusing the position encoding information with the word vectors, input the feature vector after fusing the position information into the multi-head attention module, and then input the output into the FFN module to obtain the encoded features of the source model;

[0010] S2. Obtain the encoded features of the pruned model: On the basis of obtaining the encoded features of the source model in step S1, introduce a masking mechanism in the multi-head attention module and the residual structure. The input matrix of the multi-head attention module calculates the attention through a linear transformation, and the masking parameters are randomly initialized and optimized along with the parameters of the pruned model; the residual module is calculated according to a preset expression, and finally the encoded feature output of the pruned model is obtained;

[0011] S3. Update the parameters of the discriminator model: Input the obtained encoded features of the source model and the pruned model into the discriminator model. The discriminator model includes 1 gated neural network and N lightweight expert models. The gated neural network is used to calculate the probability distribution. The output features of the expert models are fused and then pass through the multi-head attention module and the SoftMax function to obtain the probability distributions corresponding to the source model and the pruned model; the optimization objective of the discriminator model includes the parameters of the discriminator model, the encoded features and distributions of the source model and the pruned model, and regularization constraints, to prevent the discriminator from dominating in training. Update the discriminator model according to this optimization objective to obtain the updated parameters of the discriminator model;

[0012] S4. Update the parameters of the pruned model: The optimization objective of the pruned model is the same as that of the discriminator model, except for introducing the regularization constraints of the mask and the parameters of the pruned model, and using the FISTA algorithm to iteratively update the mask and the parameters; specifically, first transform the optimization objective function, set the initial values of the FISTA hyperparameters, update the parameters according to a preset strategy, introduce intermediate variables to record the update process, then update the parameters according to the optimization function, and finally fix the mask and update the parameters in a random gradient descent manner; and

[0013] S5. Iterative Training and Model Pruning: Repeat the above steps, synchronously set the model compression rate according to the number of adversarial iteration rounds. When the discriminator cannot distinguish the output feature vectors of the source model and the pruned model, you can choose to continue training or terminate. Finally, prune the masked multi-head attention module and residual structure module in the pruned model according to the updated masked feature matrix to obtain the pruned model.

[0014] Preferably, in obtaining the encoded features of the source model in step S1, it specifically includes:

[0015] Obtain text data T = {t1, t2,..., t n} from the text database, tokenize the text data T using a tokenizer adapted to the source model, and encode the Tokens according to the tokenization results and the dictionary to obtain the input sequence X = {x1, x2,...x n} of the source model;

[0016] Perform vector representation on the model according to the input sequence X to obtain the word vector representation E = {e1, e2,..., e n} corresponding to each Token, where d model is the vector dimension of the model, and seq is the maximum window length supported by the model;

[0017] Then fuse the positional encoding information with the word vectors, input the feature vectors after fusing the positional information into the multi-head attention module, and input the output after passing through the multi-head attention module into the FFN module to obtain the encoded features of the text data by the source model, i.e., the unpruned model

[0018] z = max(0, xW1 + b1)W2 + b2

[0019]

[0020] where W1 and W2 represent the weight matrices of the first layer and the second layer respectively, b1 is the bias term corresponding to W1, b2 is the bias term corresponding to W2, and z represents the intermediate variable; γ is the scaling factor, β is the offset, used to adjust the scaled features, z i is the i-th element in the z vector, μ is the mean of z, σ 2 is the variance of z, and ∈ represents a positive number.

[0021] More preferably, in obtaining the encoded features of the pruned model in step S2, it specifically includes:

[0022] For the multi-head attention module with a masking mechanism, the input matrix is obtained through three different linear transformations:

[0023]

[0024] For each head, compute the attention:

[0025]

[0026] where n is the sequence length, and d model is the dimension of the model, i.e., the dimension of the input vector; represents the key matrix of the i-th attention head, with dimension n×dk, represents the query matrix of the i-th attention head, with the same dimension as the key matrix, represents the value matrix of the i-th attention head, with the same dimension as the key and query matrices; represents the key weight matrix of the i-th attention head, with dimension d model ×d k , represents the query weight matrix of the i-th attention head, with the same dimension as the key weight matrix; represents the value weight matrix of the i-th attention head, with the same dimension as the key and query weight matrices, and d k is the dimension of the key, query, and value of each attention head; represents the attention mask of the i-th attention head, represents the function for computing the attention output of the i-th attention head, where softmax is the softmax function, used to convert the product with the mask into a probability distribution; represents the square root of the dimension d k of the key, used to scale the result;

[0027] At this time, is a randomly initialized value, which will be optimized together with the parameters of the pruned model later to obtain the final matrix, and the model will be pruned according to the matrix;

[0028] For the residual module with the mask mechanism introduced, the expression is as follows:

[0029]

[0030] where i represents the sequence of different sub-layer structures of the same encoder, represents the parameters of the i-th layer of the pruned model, represents the mask parameter, Represents the structure of the residual; other structures are the same as those of the source model, and finally, the encoded feature output of the pruned model with the masking mechanism is obtained.

[0031] Further preferably, in step S3 of updating the discriminator model parameters, the parameters of the source model and the pruned model are fixed, and the parameters of the discriminator model are updated, specifically including:

[0032] The overall discriminator model adopts the structural paradigm of MoE, mainly including 1 gating neural network and n lightweight expert models. Its input includes two parts, namely the encoded features of the source model and the encoded features of the pruned model Among them, the expression of the gating neural network is as follows:

[0033]

[0034] Among them g i Corresponds to the probability distribution of each expert model; the number and size of the expert models are reasonably selected according to the computing resources, and then the features output by each expert model are fused, and the fusion method needs to meet the same dimension;

[0035] After that, the fused features are input into the multi-head attention module, and finally, the probability distributions of the input features corresponding to the source model and the pruned model are obtained through the SoftMax function; the optimization objective of the discriminator is:

[0036]

[0037] Among them, are the parameters of the discriminator model, D represents the discriminator model, represents the encoded features of the source model, p origin represents the distribution of the encoded features of the source model, represents the parameters of the pruned model, represents the encoded features of the pruned model, p pruned represents the distribution of the encoded features of the pruned model, represents the L2 regularization constraint of the discriminator model to prevent the discriminator model from dominating during training, because is randomly initialized, and it is easy for the discriminator model to distinguish the difference in the feature distributions of the source model and the pruned model in the early stage, The mathematical expression of is: Among them, λ is the regularization parameter, represents the square root of the sum of the squares of all elements in the weight matrix, M is the total number of elements in the weight matrix, w j is the j-th element in the matrix;

[0038] Finally, update the discriminator model to obtain the updated parameters of the discriminator model.

[0039] Further preferably, in step S4 of updating the parameters of the discriminator model, fix the source model and the discriminator model, and update the parameters of the pruning model, specifically including:

[0040] The optimization objective of the pruning model is the same as that of the discriminator model, except that the L1 regularization constraint of the introduced mask and the L2 regularization constraint of the pruning model parameters are introduced. The optimization objective of the pruning model is:

[0041]

[0042] After introducing the L1 regularization constraint of the mask and the L2 regularization constraint of the pruning model parameters, the updated parameters of the expression of the optimization objective of the pruning model are and The expression is:

[0043]

[0044] where is in the expression of the optimization objective of the pruning model Considering that updating the mask and simultaneously will lead to the instability of training, the FISTA algorithm is introduced to iteratively update and parameters;

[0045] When only updating the parameter the optimization objective function is: Set the initial value of the hyperparameter β in FISTA to 1, with α(1) = 1, and the update strategy of β is:

[0046] Then introduce an intermediate variable to record the update process of the parameter : where k represents the update step in the process of only updating and then obtain the optimization function of :

[0047] After updating the parameter fix the parameter and update The update method of is stochastic gradient descent.

[0048] Further preferably, in step S5 of iterative training and model pruning, steps S1 to S4 are repeated to iteratively update the parameters of the pruning model and the discriminator model until the maximum number of rounds is reached to stop the adversarial process, which specifically includes:

[0049] Obtain the latest mask of the feature matrix, and use the feature matrix of to prune the multi-head attention module and the residual structure module with in the pruning model;

[0050] After multiple rounds of adversarial iterative training, is a highly sparse vector. Next, perform the following operations on : where τ is a hyperparameter set by humans. According to τ, is changed to a high-dimensional vector with only 0 and 1. 0 corresponds to the module being removed and not participating in the calculation, and 1 corresponds to retaining the original value of the module;

[0051] According to the updated prune the multi-head attention and residual structure of the pruning model to finally obtain the pruned model.

[0052] Preferably, in step S1, different models have different methods of integrating position information, but it does not affect the pruning method.

[0053] In a second aspect, an embodiment of the present invention provides a large model compression system based on adversarial learning and sparse regularization, and the system includes:

[0054] A source model encoded feature acquisition module configured to obtain text data from a text database, tokenize and encode it using a tokenizer adapted to the source model to obtain an input sequence of the source model, fuse the position encoding information with the word vector, input the feature vector after fusing the position information into the multi-head attention module, and then input the output into the FFN module to obtain the encoded features of the source model;

[0055] A pruning model encoded feature acquisition module configured to introduce a mask mechanism in the multi-head attention module and the residual structure on the basis of the acquisition steps of the source model encoded feature acquisition module. The input matrix of the multi-head attention module calculates the attention through a linear transformation, and the mask parameters are randomly initialized and optimized along with the parameters of the pruning model; the residual module is calculated according to a preset expression, and finally the encoded feature output of the pruning model is obtained;

[0056] The discriminator model update module is configured to input the encoded features of the obtained source model and pruned model into the discriminator model. The discriminator model includes 1 gated neural network and N lightweight expert models. The gated neural network is used to calculate the probability distribution. The output features of the expert models are fused and then passed through the multi-head attention module and the SoftMax function to obtain the probability distributions corresponding to the source model and the pruned model. The optimization objective of the discriminator model includes the discriminator model parameters, the encoded features and distributions of the source model and the pruned model, and the regularization constraints, to prevent the discriminator from dominating during training. According to this optimization objective, the discriminator model is updated to obtain the updated parameters of the discriminator model.

[0057] The pruned model update module is configured to achieve the same optimization objective as the discriminator model, except that it introduces the regularization constraints of the mask and the regularization constraints of the pruned model parameters, and uses the FISTA algorithm to iteratively update the mask and the parameters. Specifically, it first transforms the optimization objective function, sets the initial values of the FISTA hyperparameters, updates the parameters according to a preset strategy, introduces intermediate variables to record the update process, then updates the parameters according to the optimization function, and finally fixes the mask and updates the parameters in a stochastic gradient descent manner.

[0058] The iterative training and pruning module is configured to repeat the functional steps of the above modules, synchronously set the model compression rate according to the number of adversarial iterations. When the discriminator cannot distinguish the output feature vectors of the source model and the pruned model, it can choose to continue training or terminate. Finally, according to the updated masked feature matrix, the masked multi-head attention module and residual structure module in the pruned model are pruned to obtain the pruned model.

[0059] Preferably, the source model encoded feature acquisition module, the pruned model encoded feature acquisition module, the discriminator model update module, the pruned model update module, and the iterative training and pruning module cooperate with each other to jointly achieve the compression of the large model.

[0060] In a third aspect, an embodiment of the present invention provides an electronic device, including: one or more processors; a storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the first aspect.

[0061] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the method described in any implementation manner of the first aspect.

[0062] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention applies the paradigm of adversarial learning to the large model pruning task, achieving single-stage pruning of the large model. The pruned model can maintain an inference effect similar to that of the source model without fine-tuning, which greatly reduces the computational burden and cost. In addition, the pruning strategy of the present invention can be classified into the category of structured pruning, which does not impose an additional computational burden on the hardware device and is applicable to various existing acceleration optimization strategies. At the same time, in order to ensure the stability of single-stage pruning, FISTA is introduced to optimize the mask parameters and model parameters step by step, enabling the model to maximize the retention of the inference ability of the source model while losing some parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate the embodiments and, together with the description, are used to explain the principles of the invention. Other embodiments and many of the intended advantages of the embodiments will be readily apparent as they become better understood by reference to the following detailed description. The elements of the drawings are not necessarily to scale relative to each other. Like reference numerals refer to corresponding like parts.

[0064] Figure 1 Schematic flowchart of the large model compression method based on adversarial learning and sparse regularization according to an embodiment of the present invention;

[0065] Figure 2 Schematic structural diagram of the multi-head attention module with a mask mechanism according to an embodiment of the present invention;

[0066] Figure 3 Schematic structural diagram of the residual structure with a mask mechanism according to an embodiment of the present invention;

[0067] Figure 4 Architecture diagram of the discriminator model according to an embodiment of the present invention;

[0068] Figure 5 Schematic diagram of the overall architecture diagram of the large model compression method according to a specific embodiment of the present invention;

[0069] Figure 6 Schematic architecture diagram of the large model compression system based on adversarial learning and sparse regularization according to an embodiment of the present invention;

[0070] Figure 7 Schematic structural diagram of the computer device of the electronic device suitable for implementing the embodiments of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0071] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related invention, rather than limiting the invention. Additionally, it should be noted that for the convenience of description, only the parts related to the relevant invention are shown in the drawings.

[0072] It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0073] In view of the problems existing in the existing methods, such as non-structured pruning being unable to adapt to general acceleration operators (operators need to be optimized for non-structured sparse strategies), the pruned model needs to be fine-tuned to restore performance (computing power resource consumption), and the pruning process is unstable (the model performance is severely affected by hyperparameters such as the pruning rate), the embodiments of the present invention design a brand-new model compression method.

[0074] First, the "less contributing" attention heads are pruned through a masked multi-head attention mechanism to reduce the number of parameters. Secondly, a residual structure pruning with sparse constraints is performed on the FFN structure to optimize the residual structure while avoiding information flow blockage. An adaptive threshold is also introduced to set the parameter to 0 when it infinitely approaches 0 to reduce invalid computations. Finally, by combining the adversarial learning paradigm with the regular constraint of the source model features (sharing the influence of hyperparameters on the pruned model to the discriminator) and using the FISTA optimization method (optimizing the discriminator), the stability of the pruning training is further improved. This method belongs to structured pruning, does not require designing specific acceleration operators, and is simple to operate. It is a pruning in an end-to-end manner and does not require fine-tuning of the pruned model, which can greatly reduce the dependence on computing resources.

[0075] In a first aspect, the embodiments of the present invention disclose a large model compression method based on adversarial learning and sparse regularization, as Figure 1 shown, the method includes the following steps:

[0076] S1. Obtain the encoded features of the source model: Obtain text data from a text database, tokenize and encode it using the tokenizer adapted to the source model to obtain the input sequence of the source model. After fusing the position encoding information with the word vectors, input the feature vector after fusing the position information into the multi-head attention module, and then input the output into the FFN module to obtain the encoded features of the source model;

[0077] Specifically, obtaining the encoded features of the source model specifically includes:

[0078] S11. Obtain text data from the text database as T = {t1, t2,..., t n}, tokenize the text data T using a tokenizer adapted to the source model, and encode the tokens according to the tokenization results and the dictionary to obtain the input sequence X of the source model = {x1, x2,..., x n};

[0079] S12. Perform vector representation on the model according to the input sequence X to obtain the word vector representation E = {e1, e2,..., e n} for each token, where d model is the vector dimension of the model, and seq is the maximum window length supported by the model;

[0080] S13. Then fuse the positional encoding information with the word vectors, input the feature vectors after fusing the positional information into the multi-head attention module, and input the output of the multi-head attention module into the FFN module to obtain the encoding feature of the source model, i.e., the unpruned model, for the text data

[0081] z = max(0, xW1 + b1)W2 + b2

[0082]

[0083] where W1 and W2 represent the weight matrices of the first layer and the second layer respectively, b1 is the bias term corresponding to W1, b2 is the bias term corresponding to W2, and z represents the intermediate variable; γ is the scaling factor, β is the offset, used to adjust the scaled features, z i is the i-th element in the z vector, μ is the mean of z, σ 2 is the variance of z, and ∈ represents a positive number.

[0084] It should be noted that different models have different methods of incorporating positional information, but it does not affect the pruning method of the present invention.

[0085] S2. Obtain the encoding feature of the pruned model: On the basis of obtaining the encoding feature of the source model in step S1, introduce a masking mechanism in the multi-head attention module and the residual structure. The input matrix of the multi-head attention module calculates the attention through linear transformation. The masking parameters are randomly initialized and optimized along with the parameters of the pruned model; the residual module is calculated according to the preset expression, and finally the encoding feature output of the pruned model is obtained;

[0086] Specifically included in obtaining the encoding feature of the pruned model is:[[]]

[0087] S21. For the multi-head attention module with the masking mechanism introduced, the input matrix is obtained through three different linear transformations:[[]]

[0088]

[0089] S22. For each head, calculate the attention:

[0090]

[0091] where n is the sequence length, and d model is the dimension of the model, i.e., the dimension of the input vector; represents the key matrix of the i-th attention head, with dimension n×dk, represents the query matrix of the i-th attention head, with the same dimension as the key matrix, represents the value matrix of the i-th attention head, with the same dimension as the key and query matrices; represents the key weight matrix of the i-th attention head, with dimension d model ×d k , represents the query weight matrix of the i-th attention head, with the same dimension as the key weight matrix; represents the value weight matrix of the i-th attention head, with the same dimension as the key and query weight matrices, and d k is the dimension of the key, query, and value of each attention head; represents the attention mask of the i-th attention head, represents the function for calculating the attention output of the i-th attention head, where softmax is the softmax function, used to convert the product with the mask into a probability distribution; represents the square root of the dimension d k of the key, used to scale the result;

[0092] S23. At this time, is a randomly initialized value, which will be optimized together with the parameters of the pruned model later to obtain the final matrix, and the model will be pruned according to the matrix;

[0093] S24. For the residual module with the mask mechanism, the expression is as follows:

[0094]

[0095] where i represents the sequence of different sub-layer structures of the same encoder, represents the parameters of the i-th layer of the pruned model, represents the mask parameter, represents the structure of the residual; other structures are the same as those of the source model, and finally the encoded feature output of the pruned model with the mask mechanism is obtained

[0096] Figure 2 It is a schematic structural diagram of a multi-head attention module with a masking mechanism; Figure 3 It is a schematic structural diagram of a residual structure with a masking mechanism.

[0097] S3. Update the discriminator model parameters: Input the encoded features of the obtained source model and pruned model into the discriminator model. The discriminator model includes 1 gating neural network and N lightweight expert models. The gating neural network is used to calculate the probability distribution. The features output by the expert models are fused and then passed through the multi-head attention module and the SoftMax function to obtain the probability distributions corresponding to the source model and the pruned model. The optimization objective of the discriminator model includes the discriminator model parameters, the encoded features and distributions of the source model and the pruned model, and the regularization constraint to prevent the discriminator from dominating during training. Update the discriminator model according to this optimization objective to obtain the updated parameters of the discriminator model;

[0098] In the step of updating the discriminator model parameters, fix the parameters of the source model and the pruned model and update the parameters of the discriminator model. Specifically, the overall discriminator model adopts the structural paradigm of MoE, mainly including 1 gating neural network and n lightweight expert models. Its input includes two parts, namely the encoded features of the source model and the encoded features of the pruned model The expression of the gating neural network is as follows:

[0099]

[0100] where g i corresponds to the probability distribution of each expert model; The number and size of the expert models are reasonably selected according to the computing resources, and then the features output by each expert model are fused. The fusion method needs to satisfy the same dimension.

[0101] S31. Then input the fused features into the multi-head attention module, and finally obtain the probability distributions corresponding to the input features of the source model and the pruned model through the SoftMax function; The optimization objective of the discriminator is:

[0102]

[0103] where, are the parameters of the discriminator model, D represents the discriminator model, represents the encoded features of the source model, p origin represents the encoded feature distribution of the source model, represents the parameters of the pruned model, represents the encoded features of the pruned model, ppruned Represents the distribution of the encoded features of the pruned model, represents the L2 regularization constraint of the discriminator model to prevent the discriminator model from dominating during training, because is randomly initialized, and it is easy for the discriminator model to distinguish the difference in the feature distributions of the source model and the pruned model in the early stage. The mathematical expression of is: where λ is the regularization parameter, represents the square root of the sum of the squares of all elements in the weight matrix, M is the total number of elements in the weight matrix, and w j is the j-th element in the matrix;

[0104] S32. Finally, update the discriminator model to obtain the updated parameters of the discriminator model.

[0105] Figure 4 is the architecture diagram of the discriminator model of the embodiment of the present invention.

[0106] S4. Update the parameters of the pruned model: The optimization objective of the pruned model is the same as that of the discriminator model, but the difference is that the L1 regularization constraint of the mask and the L2 regularization constraint of the pruned model parameters are introduced, and the FISTA algorithm is used to iteratively update the mask and the parameters; specifically, it includes first transforming the optimization objective function, setting the initial values of the FISTA hyperparameters, updating the parameters according to a preset strategy, introducing intermediate variables to record the update process, then updating the parameters according to the optimization function, and finally fixing the mask and updating the parameters in the way of stochastic gradient descent; and

[0107] In the step of updating the parameters of the discriminator model, fix the source model and the discriminator model, and update the parameters of the pruned model. The optimization objective of the pruned model is the same as that of the discriminator model, but the difference is that the L1 regularization constraint of the mask and the L2 regularization constraint of the pruned model parameters are introduced. The optimization objective of the pruned model is:

[0108]

[0109] After introducing the L1 regularization constraint of the mask and the L2 regularization constraint of the pruned model parameters, the parameters updated in the expression of the optimization objective of the pruned model are and The expression is:

[0110]

[0111] where is in the optimization objective expression of the pruned model Considering that updating the mask and simultaneously will lead to instability in training, so the FISTA algorithm is introduced to iteratively update and parameters;

[0112] When only updating the parameters the optimization objective function is: Set the initial value of the hyperparameter α in FISTA to 1, so α(1) = 1, and the update strategy of α is:

[0113] Then introduce an intermediate variable to record the update process of the parameters : where k represents the update step number during the process of only updating and then obtain the optimization function: Update the parameters and then fix the parameters The update of is by stochastic gradient descent.

[0114] S5. Iterative training and model pruning: Repeat the above steps, synchronously set the model compression rate according to the number of adversarial iteration rounds. When the discriminator cannot distinguish the output feature vectors of the source model and the pruned model, one can choose to continue training or terminate. Finally, prune the masked multi-head attention module and residual structure module in the pruned model according to the updated masked feature matrix to obtain the pruned model.

[0115] In the iterative training and model pruning of this step, repeat steps S1 to S4, iteratively update the parameters of the pruned model and the discriminator model until the maximum number of rounds is reached to stop the adversarial process, specifically including:

[0116] Obtain the feature matrix of the latest mask and use the feature matrix of to prune the multi-head attention module and residual structure module with in the pruned model; After multiple adversarial iterative trainings, is a highly sparse vector. Next, perform the following operations on : where τ is a hyperparameter set manually. According to τ, is changed to a high-dimensional vector with only 0 and 1. 0 corresponds to the module being removed and not participating in the calculation, and 1 corresponds to retaining the original value of the module; Prune the multi-head attention and residual structure of the pruned model according to the updated to finally obtain the pruned model.

[0117] The present invention applies the paradigm of adversarial learning to the task of large model pruning, achieving single-stage pruning of large models. The pruned model can maintain an inference effect similar to that of the source model without fine-tuning, which greatly reduces the computational burden and cost. In addition, our pruning strategy can be classified into the category of structured pruning, which does not impose an additional computational burden on hardware devices and is applicable to various existing acceleration and optimization strategies. Meanwhile, to ensure the stability of single-stage pruning, we introduce FISTA to optimize the mask parameters and model parameters step by step, enabling the model to maximize the retention of the inference ability of the source model while losing some parameters.

[0118] The algorithms, related functions, and definitions mentioned in the embodiments of the present invention are explained as follows: The Lipschitz constant is a constant used to describe the rate of change of a function in a certain region. It is used to define Lipschitz continuous functions, which have specific smoothness.

[0119] Lipschitz continuous: A function is said to be Lipschitz continuous if there exists a constant L ≥ 0 such that for all x, y ∈ R n , we have: |f(x) - f(y)| ≤ L‖x - y‖;

[0120] Here, ‖x - y‖ represents the distance between x and y, usually the Euclidean distance. This inequality indicates that the value change of the function f does not exceed L times the input change.

[0121] Lipschitz constant: The constant L is called the Lipschitz constant of the function. The smaller the Lipschitz constant, the smoother the change of the function f. The Lipschitz constant provides a means to control the rate of change of the function and is very important in optimization and numerical analysis.

[0122] Lipschitz continuous gradient: In optimization problems, especially in convex optimization, the concept of Lipschitz continuous gradient is also very important. The gradient of a function is said to be continuous if there exists a constant L ≥ 0 such that for all x, y ∈ R n , we have:

[0123] Here, L is also called the Lipschitz constant. This condition means that the rate of change of the gradient is also bounded and does not exceed L times the input change.

[0124] The Lipschitz constant is very important in optimization algorithms. For example, in gradient descent and its variants (such as FISTA), the Lipschitz constant is used to determine the step size to ensure the convergence and stability of the algorithm. If the gradient is Lipschitz continuous, we can choose an appropriate step size to ensure that the algorithm can converge effectively.

[0125] For example, for the function whose gradient is We can obtain: Therefore, the value of the Lipschitz constant L is 1. The Lipschitz constant is a key parameter used to quantify the rate of change of a function or its gradient and plays an important role in many mathematical and optimization algorithms.

[0126] Fista: In FISTA (Fast Iterative Shrinkage-Thresholding Algorithm), the parameter α k+1 is not updated in a fixed manner but is dynamically calculated through a specific formula. This parameter is used to accelerate the convergence rate of the gradient descent algorithm. The update formula is as follows: The initial condition is set as α1 = 1. In each iteration, the next α k is calculated based on the current α k+1 .

[0127] Function: By dynamically updating α, FISTA can adjust the momentum factor in each iteration, thus accelerating the convergence of the algorithm. This update method enables the algorithm to approach the optimal solution faster and has a faster convergence rate compared to the standard gradient descent method.

[0128]

[0129] In EISTA, this dynamically updated process is one of the important mechanisms for the algorithm to accelerate convergence. Multi-Head: For the input matrix It is obtained through three different linear transformations.

[0130]

[0131] where n is the sequence length and d model is the dimension of the model. For each head, the attention is calculated:

[0132]

[0133] h i = Attn(Q i , K i , V i )

[0134] After inputting the concatenated result into a matrix for linear transformation, the dimensional consistency is maintained.

[0135]

[0136] Proximal: The Proximal operator is an important tool in convex optimization for dealing with non-smooth regularization terms. It plays a key role in solving optimization problems with non-smooth regularization terms.

[0137] For a function g(x), its Proximal operator prox tg is defined as follows:

[0138]

[0139] where t is a positive scalar, usually representing the step size or penalty parameter; v is the given vector input; and g(x) is a convex function, which is non-smooth.

[0140] The role of the Proximal operator is to balance two objectives: making x close to v while minimizing g(x). To minimize the sum of these two terms simultaneously, Proximal provides a compromise solution. This is useful when dealing with optimization problems with non-smooth regularization terms.

[0141] For example: For the Proximal operator of L1 regularization:

[0142] For g(x) = λ||x||1, its Proximal operation is:

[0143]

[0144] For its Proximal operation is:

[0145]

[0146] Figure 5 This is a schematic diagram of the overall architecture of the large model compression method for a specific embodiment of the present invention. In this specific embodiment, the specific implementation method of the present invention is as follows:

[0147] In the first step, obtain the encoding features of the source large model (unpruned model) for text data Specifically, it includes obtaining text data T = {t1, t2,... t from the text database n}, and then use a tokenizer adapted to the model to tokenize the text data and encode the Tokens according to the tokenization results and the dictionary. Obtain the input sequence X of the source model = {x1, x2,... x n}, and then perform vector representation on the model according to the input sequence to obtain the word vector representation E = {e1, e2,..., e n}, where d model is the vector dimension of the model, and seq is the maximum window length supported by the model.

[0148] Then fuse the position encoding information with the word vectors. Different models have different methods of incorporating position information, but it does not affect this pruning method. Input the feature vector after fusing the position information into the multi-head attention module, see Multi-Head above, and then input the output after the multi-head attention module into the FFN module to obtain the encoded features of the source model

[0149] z = max(0, xW1 + b1)W2 + b2

[0150]

[0151] Second, obtain the encoded feature output of the pruning model with a masking mechanism The specific steps are the same as those in the first step, except that a masking mechanism is introduced in the multi-head attention module and the residual structure. For the multi-head attention module with a masking mechanism, as Figure 2 shown, the input matrix is obtained through three different linear transformations.

[0152]

[0153] where n is the sequence length, and d model is the dimension of the model. For each head, calculate the attention:

[0154]

[0155] At this time, is a randomly initialized value, and will be optimized together with the parameters of the pruning model later to obtain the final matrix. And prune the model according to the matrix. For the residual module with a masking mechanism, as Figure 4 shown, its expression is as follows:

[0156]

[0157] where i represents different sub-layer structure sequences of the same encoder, denotes the parameters of the i-th layer of the pruned model, denotes the mask parameter, denotes the structure of the residual. Other structures are the same as those of the source model, and finally the encoded feature output is obtained

[0158] In the third step, fix the parameters of the source model and the pruned model, and update the parameters of the discriminator model. The model architecture of the discriminator is as Figure 4 shown: The overall discriminator model adopts the structure paradigm of MoE, mainly including 1 gating neural network and n lightweight expert models. Its input consists of 2 parts, namely the encoded features of the source model and the encoded features of the pruned model

[0159] The expression of the gating neural network is as follows:

[0160]

[0161] where g i corresponds to the probability distribution of each expert model. The number and size of the expert models can be reasonably selected according to the computing resources, and then the features output by each expert model are fused. There is no specific requirement for the fusion method as long as the dimensions are the same. After that, the fused features are input into the multi-head attention module (Appendix Multi-Head). Finally, the probability distributions of the input features corresponding to the source model and the pruned model are obtained through the SoftMax function.

[0162] The optimization objective of the discriminator is:

[0163]

[0164] where, are the parameters of the discriminator model, D represents the discriminator model, denotes the encoded features of the source model, p origin denotes the distribution of the encoded features of the source model, denotes the parameters of the pruned model, denotes the encoded features of the pruned model, p pruned denotes the distribution of the encoded features of the pruned model, denotes the L2 regularization constraint of the discriminator model. The objective of this term is to prevent the discriminator model from dominating during training. Because is randomly initialized, it is easy for the discriminator model to distinguish the difference in the feature distributions of the source model and the pruned model in the early stage. Its mathematical expression is:

[0165]

[0166] Update the discriminator model to obtain the updated parameters of the discriminator model, and then proceed to the fourth step.

[0167] Fourthly, fix the source model and the discriminator model, and update the parameters of the pruning model. The optimization objective of the pruning model is:

[0168]

[0169] The optimization objective of the pruning model is generally the same as that of the discriminator model, except for the introduction of the L1 regularization constraint of the mask and the L2 regularization constraint of the pruning model parameters. The parameters updated by the above expression are and In this embodiment, the above expression can be written in the following form:

[0170]

[0171] where is in the above formula. Considering that updating the mask and simultaneously will lead to training instability, this method introduces the FISTA algorithm to iteratively update and parameters (refer to Appendix FISTA and Lipschitz).

[0172] When only updating the parameter , the optimization objective function can be written as:

[0173]

[0174] In this embodiment, the initial value of the hyperparameter α in Fista is set to 1, so α(1) = 1, and the update strategy of α is as follows:

[0175] We introduce an intermediate variable to record the update process of the parameter : where k represents the update step in the process of only updating , and then we can obtain the optimization function of .

[0176]

[0177] (prox ng Refer to the above Proximal), after updating the parameter , fix the parameter and update The update method is stochastic gradient descent.

[0178] Step 5: Repeat steps 1 to 4, iteratively update the parameters of the pruning model and the discriminator model until the maximum number of rounds is reached to stop the adversarial process. Obtain the latest mask feature matrix of, and utilize feature matrix of to prune the multi-head attention module and the residual structure module with in the pruning model.

[0179] After multiple adversarial iterative trainings, is a highly sparse vector. We perform the following operations on : where τ is a hyperparameter set manually. We change to a high-dimensional vector with only 0 and 1 according to τ. 0 corresponds to the module being removed and not participating in the calculation; 1 corresponds to retaining the original value of the module. We prune the multi-head attention and residual structure of the pruning model according to the updated :

[0180]

[0181] Finally, obtain the pruned model.

[0182] Benefiting from the paradigm of adversarial learning, we can synchronously set the compression rate of the model according to the number of adversarial iterations. When the discriminator can no longer distinguish the feature vectors output by the source model and the pruned model during the adversarial process, we can dynamically choose to continue increasing the compression rate (continue adversarial training) or terminate the training to obtain the compressed model. Given that the discriminator model changes from being able to easily distinguish to being unable to distinguish the feature vectors output by the source model and the pruned model, we believe that the pruned model can maintain almost the same performance as the source model.

[0183] For further reference Figure 6 , as an implementation of the methods shown in the above figures, this application provides an embodiment of a large model compression system based on adversarial learning and sparse regularization. This system embodiment corresponds to Figure 1 the method embodiment shown, and this system can be specifically applied to various electronic devices.

[0184] In a second aspect, an embodiment of the present invention also discloses a large model compression system based on adversarial learning and sparse regularization, as shown in Figure 6 . This system includes: a source model encoded feature acquisition module 61, a pruning model encoded feature acquisition module 62, a discriminator model update module 63, a pruning model update module 64, and an iterative training and pruning module 65.

[0185] In a specific embodiment, the source model encoding feature acquisition module 61 is configured to obtain text data from a text database, segment and encode it using a tokenizer adapted to the source model to obtain an input sequence of the source model. After fusing positional encoding information with word vectors, the feature vectors after fusing the positional information are input into the multi-head attention module, and then the output is input into the FFN module to obtain the encoding features of the source model; the pruning model encoding feature acquisition module 62 is configured to, on the basis of the acquisition steps of the source model encoding feature acquisition module 61, introduce a masking mechanism in the multi-head attention module and the residual structure. The input matrix of the multi-head attention module calculates attention through linear transformation, and the masking parameters are randomly initialized and optimized along with the pruning model parameters; the residual module is calculated according to a preset expression, and finally the encoding feature output of the pruning model is obtained;

[0186] The discriminator model update module 63 is configured to input the obtained encoding features of the source model and the pruning model into the discriminator model. The discriminator model includes 1 gated neural network and N lightweight expert models. The gated neural network is used to calculate the probability distribution. After the output features of the expert models are fused, the corresponding probability distributions of the source model and the pruning model are obtained through the multi-head attention module and the SoftMax function; the optimization objective of the discriminator model includes the discriminator model parameters, the encoding features and distributions of the source model and the pruning model, and regularization constraints to prevent the discriminator from dominating during training. According to this optimization objective, the discriminator model is updated to obtain the updated parameters of the discriminator model; the pruning model update module 64 is configured to achieve the same optimization objective as the discriminator model, except that it introduces regularization constraints on the mask and the parameters of the pruning model, and iteratively updates the mask and the parameters using the FISTA algorithm; specifically, it first transforms the optimization objective function, sets the initial values of the FISTA hyperparameters, updates the parameters according to a preset strategy, introduces intermediate variables to record the update process, then updates the parameters according to the optimization function, and finally fixes the mask and updates the parameters in a stochastic gradient descent manner;

[0187] The iterative training and pruning module 65 is configured to repeat the functional steps of the above modules, synchronously set the model compression rate according to the number of adversarial iteration rounds. When the discriminator cannot distinguish the output feature vectors of the source model and the pruning model, it can choose to continue training or terminate. Finally, the pruning model with a mask in the multi-head attention module and the residual structure module is pruned according to the updated masked feature matrix to obtain the pruned model.

[0188] Preferably, the source model encoding feature acquisition module 61, the pruning model encoding feature acquisition module 62, the discriminator model update module 63, the pruning model update module 64, and the iterative training and pruning module 65 cooperate with each other to jointly achieve the compression of the large model.

[0189] The functions of the above modules correspond to the methods and will not be elaborated here.

[0190] The technology disclosed in the present invention effectively reduces the computational resource requirements, improves the inference speed, reduces the memory occupation and storage space by reducing the number of model parameters and the computational complexity, thereby reducing the energy consumption and operating costs. This enables large models to run efficiently on mobile devices, embedded systems, and edge computing devices, and is applicable to multiple scenarios such as public safety and industrial Internet of Things. In particular, the pruned model in a low-bandwidth environment can be transmitted and deployed more quickly, enhancing the usability and coverage.

[0191] Next, refer to Figure 7 , which shows a schematic structural diagram of a computer device 600 of an electronic device (such as Figure 1 the server or terminal device shown) suitable for implementing the embodiments of the present invention. Figure 7 The electronic device shown is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present invention.

[0192] As Figure 7 shown, the computer device 600 includes a central processing unit (CPU) 601 and a graphics processing unit (GPU) 602, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 603 or the program loaded from the storage section 609 into the random access memory (RAM) 604. In the RAM 604, various programs and data required for the operation of the device 600 are also stored. The CPU 601, GPU 602, ROM 603, and RAM 604 are connected to each other via a bus 605. An input / output (I / O) interface 606 is also connected to the bus 605.

[0193] The following components are connected to the I / O interface 606: an input section 607 including a keyboard, a mouse, etc.; an output section 608 including, for example, a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 609 including a hard disk, etc.; and a communication section 610 including a network interface card such as a LAN card, a modem, etc. The communication section 610 performs communication processing via a network such as the Internet. A drive 611 can also be connected to the I / O interface 606 as needed. A removable medium 612, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 611 as needed so that the computer program read from it can be installed into the storage section 609 as needed.

[0194] In particular, according to the embodiments disclosed in the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present invention include a computer program product that includes a computer program carried on a computer-readable medium, and the computer program contains program codes for executing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from the network through the communication section 610 and / or installed from the removable medium 612. When the computer program is executed by the central processing unit (CPU) 601 and the graphics processing unit (GPU) 602, the above functions defined in the methods of the present invention are executed.

[0195] It should be noted that the computer-readable medium described in the present invention can be a computer-readable signal medium or a computer-readable medium or any combination of the two. The computer-readable medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor device, apparatus, or component, or any combination of the above. More specific examples of the computer-readable medium can include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, the computer-readable medium can be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution device, apparatus, or component. And in the present invention, the computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, in which the computer-readable program code is carried. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable medium, and the computer-readable medium can send, propagate, or transmit a program for use by or in combination with an instruction execution device, apparatus, or component. The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0196] Computer program code for performing the operations of the present invention may be written in one or more programming languages or combinations thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and also including conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0197] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of the apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based device for performing the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0198] The modules described in the embodiments of the present invention may be implemented in software or in hardware. The described modules may also be provided in a processor.

[0199] As another aspect, the present invention also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or may exist separately and not be assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device is caused to perform the method steps described as in the first aspect.

[0200] The above description is only a preferred embodiment of the present invention and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of the invention involved in the present invention is not limited to the technical solutions formed by the specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the technical solutions formed by mutually replacing the above features with the technical features (but not limited to) disclosed in the present invention that have similar functions.

Claims

1. A large model compression method based on adversarial learning and sparse regularization, characterized in that, The method includes the following steps: S1. Obtain the encoded features of the source model: Obtain text data from the text database, tokenize and encode it using the tokenizer adapted to the source model to obtain the input sequence of the source model. After fusing the position encoding information with the word vectors, input the feature vectors after fusing the position information into the multi-head attention module, and then input the output into the FFN module to obtain the encoded features of the source model; S2. Obtain the encoded features of the pruned model: On the basis of obtaining the encoded features of the source model in step S1, introduce a masking mechanism in the multi-head attention module and the residual structure. The input matrix of the multi-head attention module calculates the attention through linear transformation. The masking parameters are randomly initialized and optimized along with the parameters of the pruned model; the residual module is calculated according to a preset expression, and finally the encoded feature output of the pruned model is obtained; S3. Update the parameters of the discriminator model: Input the obtained encoded features of the source model and the pruned model into the discriminator model. The discriminator model includes 1 gated neural network and N lightweight expert models. The gated neural network is used to calculate the probability distribution. The output features of the expert models are fused and then pass through the multi-head attention module and the SoftMax function to obtain the probability distributions corresponding to the source model and the pruned model; the optimization objective of the discriminator model includes the parameters of the discriminator model, the encoded features and distributions of the source model and the pruned model, and the regularization constraints to prevent the discriminator from dominating during training. Update the discriminator model according to this optimization objective to obtain the updated parameters of the discriminator model; S4. Update the parameters of the pruned model: The optimization objective of the pruned model is the same as that of the discriminator model, but the difference is that the regularization constraints of the introduced mask and the parameters of the pruned model are introduced, and the FISTA algorithm is used to iteratively update the mask and the parameters; specifically, it includes first transforming the optimization objective function, setting the initial values of the FISTA hyperparameters, updating the parameters according to a preset strategy, introducing intermediate variables to record the update process, then updating the parameters according to the optimization function, and finally fixing the mask and updating the parameters in the way of stochastic gradient descent; and S5. Iterative training and model pruning: Repeat the above steps, synchronously set the model compression ratio according to the number of adversarial iteration rounds. When the discriminator cannot distinguish the output feature vectors of the source model and the pruned model, you can choose to continue training or terminate. Finally, prune the masked multi-head attention module and the residual structure module in the pruned model according to the updated masked feature matrix to obtain the pruned model.

2. The large model compression method based on adversarial learning and sparse regularization according to claim 1, wherein, In step S1 of obtaining the encoded features of the source model, it specifically includes: Obtain text data T = {t1, t2,..., t n} from the text database, tokenize the text data T using a tokenizer adapted to the source model, and encode the Tokens according to the tokenization results and the dictionary to obtain the input sequence X = {x1, x2,... x n} of the source model; Perform vector representation on the model according to the input sequence X to obtain the word vector representation E = {e1, e2,..., e n}, where d model is the vector dimension of the model, and seq is the maximum window length supported by the model; Then, the positional encoding information is fused with the word vectors, and the feature vectors after fusing the positional information are input into the multi-head attention module. The output after passing through the multi-head attention module is input into the FFN module to obtain the encoding features of the source model, i.e., the unpruned model, for the text data z = max(0, xW1 + b1)W2 + b2 Among them, W1 and W2 respectively represent the weight matrices of the first layer and the second layer, b1 is the bias term corresponding to W1, b2 is the bias term corresponding to W2, and z represents the intermediate variable; γ is the scaling factor, β is the offset, which is used to adjust the scaled features, z i is the i-th element in the z vector, μ is the mean of z, σ 2 is the variance of z, and ∈ represents a positive number.

3. The large model compression method based on adversarial learning and sparse regularization according to claim 2, characterized in that In step S2 of obtaining the encoded features of the pruned model, it specifically includes: For the multi-head attention module with the introduced masking mechanism, the input matrix is obtained through three different linear transformations: For each head, calculate the attention: where n is the sequence length, and d model is the dimension of the model, i.e., the dimension of the input vector; represents the key matrix of the i-th attention head, with dimension n×dk, represents the query matrix of the i-th attention head, with the same dimension as the key matrix, represents the value matrix of the i-th attention head, with the same dimension as the key and query matrices; represents the key Key weight matrix of the i-th attention head, with dimension d model ×d k , represents the query Query weight matrix of the i-th attention head, with the same dimension as the key weight matrix; represents the value Value weight matrix of the i-th attention head, with the same dimension as the key and query weight matrices, d k is the dimension of the key, query, and value of each attention head; represents the attention mask of the i-th attention head, represents the function for calculating the attention output of the i-th attention head, softmax is the softmax function, used to convert the product with the mask into a probability distribution; represents the square root of the dimension d k of the key, used to scale the result; At this time is a randomly initialized value, which will be optimized together with the parameters of the pruning model later to obtain the final matrix, and the model is pruned according to matrix; For the residual module with the introduced masking mechanism, the expression is as follows: where i represents the sequence of different sub-layer structures of the same encoder, denotes the parameters of the i-th layer of the pruned model, denotes the masking parameter, denotes the structure of the residual; other structures are the same as those of the source model, and finally, the encoded feature output of the pruned model with the masking mechanism is obtained 4. The large model compression method based on adversarial learning and sparse regularization according to claim 3, wherein In step S3 of updating the parameters of the discriminator model, fix the parameters of the source model and the pruned model, and update the parameters of the discriminator model, specifically including: The discriminator model as a whole adopts the structural paradigm of MoE, mainly including 1 gating neural network and n lightweight expert models. Its input consists of two parts, namely the encoded features of the source model and the encoded features of the pruning model Among them, the expression of the gating neural network is as follows: Among them g i corresponds to the probability distribution of each expert model; the number and size of the expert models are reasonably selected according to the computing resources, and then the features output by each expert model are fused, and the fusion method needs to satisfy the same dimension; Then input the fused features into the multi-head attention module, and finally obtain the probability distributions of the input features corresponding to the source model and the pruned model through the SoftMax function; the optimization objective of the discriminator is: Among them, are the parameters of the discriminator model, where D represents the discriminator model, represents the encoded features of the source model, and p origin represents the distribution of the encoded features of the source model, represents the parameters of the pruning model, represents the encoded features of the pruning model, and p pruned represents the distribution of the encoded features of the pruning model, represents the L2 regularization constraint of the discriminator model to prevent the discriminator model from dominating during training because is randomly initialized. For the discriminator model, it is easy to distinguish the difference in the feature distributions of the source model and the pruning model in the early stage. The mathematical expression of where λ is the regularization parameter, denotes the square root of the sum of the squares of all elements in the weight matrix, M is the total number of elements in the weight matrix, and w j is the j-th element in the matrix; Finally, update the discriminator model to obtain the updated parameters of the discriminator model.

5. The large model compression method based on adversarial learning and sparse regularization according to claim 3, wherein In step S4 of updating the discriminator model parameters, fix the source model and the discriminator model, and update the parameters of the pruning model, specifically including: The optimization objective of the pruning model is the same as that of the discriminator model, except that it introduces the L1 regularization constraint of the mask and the L2 regularization constraint of the pruning model parameters. The optimization objective of the pruning model is: After introducing the L1 regularization constraint with a mask and the L2 regularization constraint for pruning model parameters, the parameters updated in the expression of the optimization objective of the pruned model are and The expression is: Among them in the optimization objective expression of the pruning model Considering updating the mask simultaneously and will lead to instability in training. Therefore, the FISTA algorithm is introduced to iteratively update and parameters; When only updating the parameters the optimization objective function is as follows: Set the initial value of the hyperparameter α in FISTA to 1, so α(1) = 1, and the update strategy of α is: Reintroduce intermediate variables to record the parameters during the update process: where k represents the number of update steps during only the update process, and then obtain the optimized function: Update parameters After that, fix the parameters Update The update method is stochastic gradient descent.

6. The large model compression method based on adversarial learning and sparse regularization according to claim 5, wherein In step S5 of iterative training and model pruning, repeat steps S1 to S4 to iteratively update the parameters of the pruning model and the discriminator model until the maximum number of rounds is reached to stop the adversarial process, specifically including: Obtain the latest mask of the feature matrix, and use the feature matrix to prune the multi-head attention module and residual structure module with in the pruning model; After multiple adversarial iterative trainings, is a highly sparse vector. Next, perform the following operations on : where τ is a hyperparameter set by humans. According to τ, it is changed into a high-dimensional vector with only 0 and 1. 0 corresponds to the removal of the module, which does not participate in the calculation, and 1 corresponds to retaining the original value of the module; According to the updated prune the multi-head attention and residual structure of the pruned model, and finally obtain the pruned model.

7. The large model compression method based on adversarial learning and sparse regularization according to claim 2, wherein In step S1, different models have different methods of integrating position information, but it does not affect the pruning method.

8. A large model compression system based on adversarial learning and sparse regularization, characterized in that, The system includes: A source model encoding feature acquisition module, configured to obtain text data from a text database, tokenize and encode it using a tokenizer adapted to the source model to obtain the input sequence of the source model. After fusing the position encoding information with the word vectors, the feature vectors after fusing the position information are input into the multi-head attention module, and then the output is input into the FFN module to obtain the encoding features of the source model; A pruning model encoding feature acquisition module, configured to introduce a mask mechanism in the multi-head attention module and the residual structure on the basis of the acquisition steps of the source model encoding feature acquisition module. The input matrix of the multi-head attention module calculates the attention through linear transformation, and the mask parameters are randomly initialized and optimized along with the pruning model parameters; the residual module is calculated according to a preset expression, and finally the encoding feature output of the pruning model is obtained; A discriminator model update module, configured to input the obtained encoding features of the source model and the pruning model into the discriminator model. The discriminator model includes 1 gated neural network and N lightweight expert models. The gated neural network is used to calculate the probability distribution. The output features of the expert models are fused and then passed through the multi-head attention module and the SoftMax function to obtain the probability distributions corresponding to the source model and the pruning model; the optimization objective of the discriminator model includes the discriminator model parameters, the encoding features and distributions of the source model and the pruning model, and the regularization constraints, preventing the discriminator from dominating during training. Update the discriminator model according to this optimization objective to obtain the updated parameters of the discriminator model; A pruning model update module, configured to achieve the same optimization objective as the discriminator model, except that it introduces the regularization constraints of the mask and the pruning model parameters, and uses the FISTA algorithm to iteratively update the mask and the parameters; specifically, it includes first transforming the optimization objective function, setting the initial values of the FISTA hyperparameters, updating the parameters according to a preset strategy, introducing intermediate variables to record the update process, then updating the parameters according to the optimization function, and finally fixing the mask and updating the parameters in a stochastic gradient descent manner; The iterative training and pruning module is configured to repeat the functional steps of the above module, synchronously set the model compression rate according to the number of adversarial iteration rounds, and can choose to continue training or terminate when the discriminator cannot distinguish the output feature vectors of the source model and the pruned model. Finally, prune the masked multi-head attention module and residual structure module in the pruned model according to the updated masked feature matrix to obtain the pruned model.

9. The large model compression system based on adversarial learning and sparse regularization according to claim 8, characterized in that, The source model encoded feature acquisition module, pruned model encoded feature acquisition module, discriminator model update module, pruned model update module, and iterative training and pruning module cooperate with each other to jointly achieve the compression of the large model.

10. An electronic device, comprising: One or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the large model compression method based on adversarial learning and sparse regularization as described in any one of claims 1 to 7.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the large model compression method based on adversarial learning and sparse regularization as described in any one of claims 1 to 7.

Images