Model compression method, apparatus, device, and storage medium

Abstract

One or more embodiments of the present application provide a model compression method, device, equipment and storage medium, the method comprising: obtaining public data samples used for model pre-training of a large language model, and screening public data samples similar to private data samples from the public data samples; performing model compression on the large language model whose pre-training is completed based on the screened public data samples, to obtain a compression model corresponding to the large language model; and performing model fine-tuning on the compression model based on the private data samples, to complete compression processing for the large language model.

Description

Technical Field

[0001] One or more embodiments of this application relate to the field of artificial intelligence technology, and in particular to a model compression method, apparatus, device, and storage medium. Background Technology

[0002] Large Language Models (LLMs) are deep learning models trained on large amounts of text data. They can be used to generate natural language text or understand the meaning of language text. LLMs can handle a variety of natural language tasks, such as text classification, question answering, and dialogue, and are an important pathway to artificial intelligence.

[0003] The emergence of large language models has dramatically changed the way deep learning models are trained in Natural Language Processing (NLP). First, large language models can be pre-trained on extremely large and diverse public datasets. Then, using smaller, task-specific datasets, the pre-trained large language model can be fine-tuned for the specific task. Finally, because large language models typically contain hundreds of millions of parameters, their inference time and memory usage are excessive for many applications. Furthermore, many parameters are redundant and can be removed while maintaining model performance. Therefore, after fine-tuning, large language models are usually further compressed to reduce the number of parameters. The compressed deep learning model can then be deployed directly. Summary of the Invention

[0004] One or more embodiments of this application provide the following technical solutions:

[0005] This application provides a model compression method, the method comprising:

[0006] Obtain public data samples for model pre-training of a large language model, and filter out public data samples similar to private data samples from the public data samples;

[0007] Based on the selected public data samples, the pre-trained large language model is compressed to obtain a compressed model corresponding to the large language model.

[0008] Based on the private data samples, the compression model is fine-tuned to complete the compression processing for the large language model.

[0009] This application also provides a model compression apparatus, the apparatus comprising:

[0010] The sample screening module acquires public data samples for pre-training large language models and filters out public data samples that are similar to private data samples from the public data samples.

[0011] The model compression module compresses the pre-trained large language model based on the selected public data samples to obtain a compressed model corresponding to the large language model.

[0012] The model fine-tuning module performs model fine-tuning on the compressed model based on the private data samples to complete the compression processing for the large language model.

[0013] This application also provides an electronic device, including:

[0014] processor;

[0015] Memory used to store processor-executable instructions;

[0016] The processor executes the executable instructions to implement the steps of the method as described in any of the preceding descriptions.

[0017] This application also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method as described in any of the preceding claims.

[0018] In the above technical solution, firstly, a large language model can be pre-trained based on public data samples. Then, based on public data samples similar to private data samples used for model fine-tuning selected from the public data samples used for model pre-training, the pre-trained large language model can be compressed to obtain a compressed model corresponding to the large language model. Finally, the compressed model can be fine-tuned based on the private data samples, thereby completing the compression process for the large language model and obtaining a compressed model more suitable for deployment.

[0019] By adopting the above approach, on the one hand, by swapping the execution order of the fine-tuning and compression processes, the fine-tuning process no longer needs to be performed on the pre-trained large language model, but rather on the compressed model corresponding to the large language model, thus reducing the computational resource consumption during fine-tuning. On the other hand, since the compression process no longer uses privacy-preserving training but instead uses non-privacy-preserving training, only the fine-tuning process still uses privacy-preserving training; therefore, the performance overhead of the model can be reduced. Moreover, since the compressed model contains fewer model parameters than the large language model, this means that introducing privacy-preserving training on the compressed large language model for fine-tuning has a smaller negative impact on model performance compared to directly introducing privacy-preserving training on the uncompressed large language model. Therefore, by swapping the execution order of the fine-tuning and compression processes, it also helps to reduce the performance overhead of the model and improve its performance. Attached Figure Description

[0020] The accompanying drawings used in the description of the exemplary embodiments will now be explained, wherein:

[0021] Figure 1 This is a schematic diagram of the compression process for large language models in related technologies.

[0022] Figure 2 This is a schematic diagram illustrating a compression process for a large language model, as shown in an exemplary embodiment of this application.

[0023] Figure 3 This is a flowchart illustrating a model compression method in an exemplary embodiment of this application.

[0024] Figure 4 This is a flowchart illustrating another model compression method in an exemplary embodiment of this application.

[0025] Figure 5 This is a schematic diagram of the structure of a device shown in an exemplary embodiment of this application.

[0026] Figure 6 This is a block diagram illustrating a model compression device according to an exemplary embodiment of this application. Detailed Implementation

[0027] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this application. Rather, they are merely examples consistent with some aspects of one or more embodiments of this application.

[0028] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this application in other embodiments. In some other embodiments, the methods may include more or fewer steps than those described in this application. Furthermore, a single step described in this application may be broken down into multiple steps in other embodiments; and multiple steps described in this application may be combined into a single step in other embodiments.

[0029] Large Language Models (LLMs) are deep learning models trained on large amounts of text data. They can be used to generate natural language text or understand the meaning of language text. LLMs can handle a variety of natural language tasks, such as text classification, question answering, and dialogue, and are an important pathway to artificial intelligence.

[0030] Large language models typically employ the Transformer architecture; that is, large language models are usually deep learning models based on the Transformer architecture. Deep learning models based on the Transformer architecture are a class of neural network models that utilize the Transformer architecture, and these models perform exceptionally well in fields such as natural language processing.

[0031] Transformer is a neural network model used for sequence-to-sequence modeling. Transformer does not rely on recursive structures, enabling parallel training and inference, thus accelerating model processing.

[0032] In deep learning models based on the Transformer architecture, multi-layered Transformer encoders are typically used to extract features from the input sequence, and a Transformer decoder is used to transform these features into an output sequence. These models also typically employ self-attention mechanisms to capture long-range dependencies in the input sequence, as well as residual connections and normalization methods to accelerate training and improve model performance.

[0033] Pre-trained models are language models pre-trained on large-scale unlabeled text data. Pre-trained models are general-purpose; they are not designed or optimized for a specific task. To adapt a pre-trained model to a specific task, fine-tuning is required to improve its performance on that task. Large language models, on the other hand, are models that undergo further fine-tuning based on pre-trained models and are learned through supervised learning using labeled text data.

[0034] In other words, the training process of a large language model can generally be divided into two stages: pre-training and fine-tuning. In the pre-training stage, unsupervised learning (e.g., self-supervised learning) can be used to pre-train on public datasets (e.g., online encyclopedias, online articles, books, etc.). The model attempts to predict missing parts or the next word based on context, learning the inherent structure and rules of language, such as semantics and syntax. Optimization algorithms like gradient descent can be used to continuously adjust model parameters, gradually improving the model's performance on the pre-training task. In the fine-tuning stage, a corresponding supervised learning task (e.g., text classification, named entity recognition, question answering systems, etc.) can be selected based on the specific application scenario and task requirements, and a task-specific dataset can be prepared. The pre-trained model serves as the starting point for fine-tuning, using supervised learning on the task-specific dataset. Optimization algorithms like gradient descent can be used to adjust model parameters based on the dataset labels, gradually adapting the model to the requirements of the specific task and dataset. Pre-training and fine-tuning are complementary processes. Pre-training enables the model to have a wide range of language understanding capabilities, while fine-tuning makes the model more specialized and accurate for specific tasks.

[0035] Large language models contain hundreds of millions of parameters. On the one hand, the inference time and memory consumption of these models are too large for many applications. On the other hand, many of these parameters are redundant and can be removed while maintaining model performance. Therefore, in the field of natural language processing, large language models are usually not used directly. Instead, the trained large language models are compressed to reduce the number of model parameters, and the compressed deep learning model can be used when deploying the model.

[0036] Model compression is a technique that reduces storage space and computing resource requirements by decreasing the size and complexity of neural network models. It can improve the deployment efficiency of models in resource-constrained environments such as mobile devices and edge computing, and reduce latency in transmission and inference. Model compression algorithms include pruning, quantization, network architecture optimization, and knowledge distillation. These algorithms can be used individually or in combination for better results. Typically, the appropriate model compression algorithm is selected based on the characteristics of the model, the specific application scenario, and performance requirements.

[0037] While deep learning models based on the Transformer architecture have driven the development of deep learning technology toward designing larger models to achieve better performance, in practical applications, the more model parameters a larger model contains, the more data is needed to train the model. This means that more personal information may be included in the model training.

[0038] If this personal information is not adequately protected (e.g., through encryption or anonymization), it is theoretically possible to recover some or all of the training data by reverse engineering the model parameters. This process of reverse engineering the model parameters is commonly referred to as "model inversion" or "model derivation." Although achieving this reverse engineering is not easy at present, research has shown that by performing reverse data analysis on the model parameters, it is possible to obtain some data features or data distributions related to the training data.

[0039] In related technologies, in order to mitigate the leakage of private data contained in the training dataset of large language models, privacy-preserving training is adopted during the training process of large language models.

[0040] Specifically, for the training process of large language models, the dataset used in the pre-training stage is usually a public dataset, while the dataset used in the fine-tuning stage is a private dataset containing privacy data. Therefore, privacy training is adopted in the fine-tuning stage.

[0041] Privacy-preserving training is a machine learning method designed to protect the privacy of training data during model training. It utilizes techniques such as differential privacy or homomorphic encryption to train and update models without exposing the original data.

[0042] In traditional machine learning, model training is typically performed on a centralized server, where all training data is collected in one place and used to train the model. This approach carries the risk of data privacy breaches because all data is sent to the server for processing.

[0043] In contrast, privacy-preserving training employs various privacy-preserving techniques to protect data privacy. The most common of these is differential privacy, which protects privacy by adding noise to the original data. Specifically, for each sample used in training, differential privacy merges it with another "fake" sample and then uses random techniques to add noise, making the real and fake samples privacy-equivalent while ensuring the model's accuracy.

[0044] Besides differential privacy, homomorphic encryption is another widely used technique for privacy-preserving training. Homomorphic encryption allows computation to be performed on encrypted data, and the output is also encrypted, thus avoiding the problem of data being transmitted anywhere.

[0045] After fine-tuning the large language model, to further reduce the number of model parameters, the trained large language model can be compressed. It should be noted that the compression process typically uses private datasets containing sensitive data, therefore privacy-preserving training methods must be employed.

[0046] Please refer to Figure 1 , Figure 1 This is a schematic diagram of the compression process for large language models in related technologies.

[0047] like Figure 1 As shown, firstly, unsupervised learning can be used to pre-train a large language model using a public dataset. This pre-training process employs a non-privacy training method.

[0048] Then, supervised learning can be used to fine-tune the pre-trained large language model using a private dataset containing privacy-preserving data. This fine-tuning process employs privacy-preserving training methods. This completes the training of the large language model.

[0049] It's important to note that compressing a large language model essentially involves reducing the number of its parameters to obtain a smaller deep learning model. This smaller model is then trained using the same dataset used for pre-training (fine-tuning) the larger language model, allowing its performance to approach that of the larger model. Therefore, the aforementioned private dataset can also be used to compress the trained large language model. This compression process also employs privacy-preserving training methods.

[0050] Therefore, by implementing a training method that includes three steps—pre-training, fine-tuning, and compression—for this large language model, compression processing can be completed, ultimately resulting in a deep learning model with an appropriate number of model parameters, making it more suitable for deployment.

[0051] However, the compression schemes for large language models in related technologies have the following drawbacks:

[0052] First, the fine-tuning process is carried out on the basis of the pre-trained large language model. At this time, the large language model has not been compressed and contains a large number of model parameters, which will result in the training corresponding to the fine-tuning process requiring more computing resources.

[0053] Second, although using privacy-preserving training methods during fine-tuning can reduce the risk of privacy data leakage, current privacy-preserving training methods usually sacrifice model performance to protect privacy data. Therefore, while introducing privacy-preserving training methods can reduce the risk of privacy data leakage to some extent, it will have some negative impact on model performance. Moreover, the larger the number of parameters in the model, the more significant the negative impact on the model's performance will be.

[0054] For example, taking differential privacy algorithms as a privacy training algorithm, these algorithms typically introduce a certain degree of noise or perturbation into the privacy data to mask sensitive information. The presence of this noise or perturbation obviously leads to data distortion in the input model, negatively impacting its performance; for instance, it may reduce the accuracy of the model's output. Furthermore, in practical applications, the amount of noise or perturbation introduced by differential privacy algorithms into the privacy data is usually related to the number of model parameters. The more model parameters a model has, the greater the amount of noise or perturbation introduced. Therefore, it's easy to understand that for models with a larger number of parameters, the amount of noise or perturbation introduced by the differential privacy algorithm will be greater, and consequently, the negative impact of this noise or perturbation on the model's performance will be more significant.

[0055] It is evident that fine-tuning a large language model using privacy-preserving training methods will inevitably result in significant performance overhead. Moreover, this performance overhead is particularly pronounced when privacy-preserving training is employed in both the fine-tuning and compression processes.

[0056] This application provides a technical solution for model compression. First, a large language model can be pre-trained based on publicly available data samples. Then, the pre-trained large language model can be compressed based on publicly available data samples selected from the publicly available data samples used for model pre-training that are similar to private data samples used for model fine-tuning, to obtain a compressed model corresponding to the large language model. Finally, the compressed model can be fine-tuned based on the private data samples, thereby completing the compression process for the large language model and obtaining a compressed model that is more suitable for deployment.

[0057] By adopting the above approach, on the one hand, since the fine-tuning process and the compression process are interchanged in terms of execution order, the fine-tuning process can no longer be performed on the basis of the large language model, but on the basis of the compressed model corresponding to the large language model, thereby reducing the consumption of computational resources during the fine-tuning process.

[0058] On the other hand, since the compression process can now use non-privacy training instead of privacy training, with only the fine-tuning process still employing privacy training, the performance overhead of the model can be reduced. Furthermore, because the compressed model contains fewer model parameters than the large language model, fine-tuning the compressed large language model using privacy training has a smaller negative impact on model performance compared to fine-tuning the uncompressed large language model. Therefore, swapping the execution order of the fine-tuning and compression processes further helps reduce performance overhead and improve model performance.

[0059] Please refer to Figure 2 , Figure 2 This is a schematic diagram illustrating a compression process for a large language model, as shown in an exemplary embodiment of this application.

[0060] and Figure 1 The compression process shown is different, such as Figure 2 The execution order of the fine-tuning process and the compression process in the compression process shown in the diagram has been interchanged.

[0061] After pre-training a large language model using a public dataset and a non-privacy training method, you can temporarily refrain from fine-tuning the pre-trained large language model and instead compress it first.

[0062] Specifically, public data samples similar to the private data samples contained in the private dataset can be selected from the public data samples contained in the aforementioned public dataset, and the selected public data samples can be used to compress the pre-trained large language model.

[0063] It should be noted that since the dataset used in this compression process is essentially a part of the publicly available dataset, the compression process can forgo privacy-preserving training and instead employ non-privacy-preserving training. In this case, the resulting deep learning model can be called a compressed model. However, this compressed model has not yet been optimized for a specific task and is not suitable for any particular task.

[0064] Finally, supervised learning can be used to fine-tune the compressed model using the aforementioned private dataset, optimizing it for a specific task. This fine-tuning process employs privacy-preserving training. Thus, by implementing a training method involving pre-training, fine-tuning, and compression on the aforementioned large language model, compression processing of the large language model is completed, and the fine-tuned compressed model becomes a more suitable deep learning model for deployment.

[0065] Please combine Figure 2 ,refer to Figure 3 , Figure 3 This is a flowchart illustrating a model compression method in an exemplary embodiment of this application.

[0066] In this embodiment, the above-described model compression method can be applied to a server. This server can be a server containing a single independent physical host, or a server cluster consisting of multiple independent physical hosts; alternatively, the server can be a virtual server, cloud server, or similar service hosted by a host cluster.

[0067] Alternatively, the above model compression method can be applied to electronic devices with a certain computing power, such as desktop computers, laptops, PDAs, and tablets.

[0068] The above model compression method may include the following steps:

[0069] Step 302: Obtain public data samples for model pre-training of the large language model, and filter out public data samples similar to private data samples from the public data samples.

[0070] In this embodiment, a pre-trained large language model can be prepared in advance, and publicly available data samples can be obtained for pre-training the large language model.

[0071] In some embodiments, the large language model can be pre-trained based on the publicly available data samples contained in the publicly available dataset.

[0072] Specifically, unsupervised learning can be used to pre-train the aforementioned large language model on the public dataset. This large language model can attempt to predict missing parts or the next word based on context, learning the inherent structure and rules of language, such as semantics and syntax. In this case, optimization algorithms such as gradient descent can be used to continuously adjust the model parameters of the large language model, gradually improving its performance on the pre-training task.

[0073] For the pre-trained large language model, we can temporarily refrain from fine-tuning it and instead compress it first.

[0074] Typically, the dataset used in the compression process is the same dataset used for pre-training the model to be compressed. Therefore, simply swapping the execution order of the fine-tuning and compression processes would result in the compression process using the public dataset while omitting the private data, making the compression process lack specificity and failing to optimize for the private data.

[0075] In practical applications, the closer the distribution of the dataset used in the pre-training process is to that of the dataset used in the fine-tuning process, the better the fine-tuning effect. Since the execution order of the fine-tuning process and the compression process is reversed, in order to ensure that the compression process benefits the private data and guarantees the fine-tuning effect, when compressing the pre-trained large language model, public data samples that are similar to the private data samples in the private dataset used for fine-tuning the large language model can be used. These public data samples are selected from the public data samples in the public dataset.

[0076] In other words, a new dataset can be formed by selecting public data samples similar to the private data samples contained in the aforementioned private dataset from the public data samples contained in the aforementioned public dataset, and then compressing the aforementioned large language model that has been pre-trained. In this case, the new dataset is essentially a part of the public dataset.

[0077] Step 304: Based on the selected public data samples, perform model compression on the pre-trained large language model to obtain a compressed model corresponding to the large language model.

[0078] In this embodiment, in order to further reduce the negative impact of the fine-tuning process on the performance of the pre-trained large language model, the execution order of the fine-tuning process and the compression process can be interchanged. After the pre-training of the large language model is completed, the large language model can be compressed first.

[0079] It should be noted that, to further reduce the risk of privacy data leakage, when compressing the pre-trained large language model, private data samples can be used instead of private data samples. After selecting public data samples similar to the aforementioned private data samples from the public data samples, model compression can be further performed on the pre-trained large language model based on the selected public data samples.

[0080] In some embodiments, model compression is performed on the pre-trained large language model, which can be accomplished by knowledge distillation training on the large language model.

[0081] In this scenario, after selecting public data samples similar to the private data samples from the aforementioned public data samples, knowledge distillation training can be performed on the pre-trained large language model based on the selected public data samples to obtain a compressed model corresponding to the large language model. It should be noted that in this knowledge distillation process, the large language model serves as the teacher model, while the compressed model serves as the student model.

[0082] Knowledge distillation allows the transfer of knowledge from the aforementioned large language model (acting as the teacher) to the aforementioned compressed model (acting as the student), thereby improving the performance of the compressed model. To perform knowledge distillation, a loss function is first defined to measure the difference between the predictions of the student model and the teacher model. This is typically achieved using soft targets, where the output of the teacher model is used as the target of the student model, and cross-entropy loss or other similar loss functions are calculated. Subsequently, the predictions of the teacher model can be used as an auxiliary target, combined with the original target of the student model, for training. The model parameters of the student model are updated by iteratively minimizing the loss function.

[0083] Step 306: Fine-tune the compression model based on the private data sample to complete the compression processing for the large language model.

[0084] In this embodiment, after compressing the large language model, the compressed model has not yet been optimized for a specific task and is not suitable for that task. Therefore, further fine-tuning of the large language model can be performed based on the compressed model obtained from the model compression.

[0085] It should be noted that, unlike model compression of large language models, when fine-tuning the compressed model, the private data samples contained in the private dataset can still be used.

[0086] Specifically, the aforementioned private dataset can be a labeled dataset for a specific task (e.g., text classification, named entity recognition, question answering systems, etc.). The compressed model can be used as a starting point for fine-tuning, and trained on this private dataset using supervised learning. In this case, optimization algorithms such as gradient descent can be used to adjust the model parameters of the compressed model according to the labels of the private dataset, so that the compressed model gradually adapts to the requirements of the specific task and the private dataset.

[0087] Thus, by implementing a training method that includes three steps—pre-training, fine-tuning, and compression—on the aforementioned large language model, the compression process for the large language model was completed. The compressed model after fine-tuning is a more suitable deep learning model for deployment.

[0088] It's important to note that while a single large language model can be pre-trained using the same public dataset, this pre-trained model can be trained on multiple private datasets. Public data samples, selected from the public data samples included in the public dataset and similar to the private data samples in each private dataset, can be used to compress the pre-trained large language model, resulting in multiple compressed models. These compressed models can then be further fine-tuned using the corresponding private datasets. In other words, using the same public dataset and multiple different private datasets, multiple finely tuned compressed models can be obtained based on the same pre-trained large language model.

[0089] In some embodiments, since both the pre-training and compression processes use public datasets, both can be trained using non-privacy methods. However, the fine-tuning process uses a private dataset, therefore it can be trained using privacy methods.

[0090] In other words, when fine-tuning the compressed model based on the aforementioned private data samples, a privacy training algorithm can be used to fine-tune the compressed model based on the private data samples.

[0091] Since the compression process described above can now employ non-privacy training instead of privacy-preserving training, with only the fine-tuning process retaining privacy-preserving training, this approach reduces the model's performance overhead. Furthermore, although the fine-tuning process still uses privacy-preserving training, it no longer fine-tunes the large language model but rather the compressed model corresponding to it. Because the compressed model contains fewer parameters than the large language model, introducing privacy-preserving training into the compressed large language model for fine-tuning has a smaller negative impact on model performance compared to directly introducing privacy-preserving training into the uncompressed large language model. Therefore, by swapping the execution order of the fine-tuning and compression processes, the negative impact of privacy-preserving training on model performance can be significantly reduced, contributing to improved model performance.

[0092] In some embodiments, the above-mentioned privacy training algorithm may include: a training algorithm based on differential privacy (DP).

[0093] Differential privacy is a privacy-preserving technical framework designed to provide privacy protection during data analysis while allowing for meaningful statistical analysis. Differential privacy protects individual privacy by introducing noise or perturbations into the raw data and limits the possibility of inferring sensitive individual information from the analysis results.

[0094] Differential privacy provides a mathematically defined privacy protection mechanism that quantifies the impact on the final result after removing or replacing an individual from a dataset with another. By adding appropriate noise or perturbations, significant privacy protection can be provided while maintaining the usability of the data. The core idea of ​​differential privacy is to minimize the degree to which the output changes due to the participation or non-participation of an individual, thereby protecting the privacy of that individual.

[0095] Differential privacy mechanisms are specific algorithms used to implement differential privacy, including Laplace, Gaussian, Exponential, and noise-adding mechanisms.

[0096] The Laplace mechanism is a differential privacy mechanism based on a probability distribution. In the Laplace mechanism, noise is added to the query results to obscure the original data. The magnitude of the noise is controlled by a Laplace distribution, which has zero mean and a certain scaling parameter.

[0097] The Gaussian mechanism is also a differential privacy mechanism based on probability distribution. The Gaussian mechanism uses a Gaussian distribution to generate noise.

[0098] The Exponential mechanism is a differential privacy mechanism that selects outputs based on data characteristics. It balances privacy protection and data utility by making selections based on data contribution and the sensitivity of the objective function.

[0099] The noise-adding mechanism is a differential privacy mechanism that directly adds noise to the query results. Noise-adding mechanisms can reduce the leakage of individual privacy information by adding noise to the query results; for example, adding noise to the results when calculating the average.

[0100] In some embodiments, the training algorithm based on differential privacy may include: a differentially private stochastic gradient descent (DP-SGD) algorithm.

[0101] DP-SGD combines privacy protection with the Stochastic Gradient Descent (SGD) algorithm. DP-SGD protects the personal privacy information contained in the training samples of the model by adding noise, thus preventing the leakage of sensitive information targeting specific individuals.

[0102] DP-SGD protects privacy based on the concept of differential privacy. During training, DP-SGD introduces noise into the calculated gradients to achieve privacy. Specifically, in each iteration, it perturbs the calculated gradients by adding noise. This can hide the contribution of personal data to a certain extent, thus protecting privacy. To control the degree of privacy leakage, DP-SGD introduces a privacy budget. The privacy budget measures how much noise can be added to protect private information. A smaller privacy budget means stricter privacy protection. DP-SGD trains its model based on the stochastic gradient descent algorithm. In each iteration, it randomly selects a small batch of samples from the training samples, calculates their corresponding gradients, and updates the model parameters according to the gradient direction.

[0103] In the above technical solution, firstly, a large language model can be pre-trained based on public data samples. Then, based on public data samples similar to private data samples used for model fine-tuning selected from the public data samples used for model pre-training, the pre-trained large language model can be compressed to obtain a compressed model corresponding to the large language model. Finally, the compressed model can be fine-tuned based on the private data samples, thereby completing the compression process for the large language model and obtaining a compressed model more suitable for deployment.

[0104] By employing the above method, a training approach based on a large language model, including three steps—pre-training, fine-tuning, and compression—is adopted to obtain a deep learning model more suitable for deployment. Furthermore, by reversing the execution order of the fine-tuning and compression processes, the fine-tuning process can be performed on a compressed model corresponding to the large language model, rather than on the large language model itself, thereby reducing the computational resource consumption during the fine-tuning process.

[0105] exist Figure 3 Based on this, please refer to Figure 4 , Figure 4 This is a flowchart illustrating another model compression method in an exemplary embodiment of this application.

[0106] like Figure 4 As shown, the above model compression method may include the following steps:

[0107] Step 402: Obtain public data samples for pre-training the large language model, and classify the public data samples to obtain at least one data sample classification.

[0108] The specific implementation of a part of step 402 can be referred to the aforementioned step 302, and will not be repeated here.

[0109] As mentioned earlier, the closer the distribution of the dataset used in the pre-training process is to that of the dataset used in the fine-tuning process, the better the fine-tuning effect. Since the execution order of the fine-tuning process and the compression process has been swapped, in order to ensure that the compression process benefits the private data and guarantees the fine-tuning effect, when compressing the pre-trained large language model, public data samples that are similar to the private data samples in the private dataset used for fine-tuning the large language model can be used. These public data samples are selected from the public data samples contained in the public dataset.

[0110] However, when the private data samples are too few or unevenly distributed, simply approximating the distribution of the private dataset can compromise data quality. Therefore, in this embodiment, the knowledge of the pre-trained large language model can be used to divide the public data samples into at least one data sample category. Then, data samples are filtered for each category, ensuring the diversity of public data samples used to compress the pre-trained large language model. This allows for a more comprehensive compression model that is beneficial to the private data through knowledge distillation.

[0111] In other words, before selecting public data samples similar to the private data samples from the aforementioned public data samples, the public data samples can be classified to obtain at least one data sample classification.

[0112] In some embodiments, when classifying the above-mentioned public data samples to obtain at least one data sample classification, the feature vectors generated by the pre-trained large language model corresponding to each public data sample can be obtained, and based on the feature vectors corresponding to each public data sample, these public data samples can be clustered to obtain at least one data sample classification.

[0113] Taking any one of the aforementioned public data samples as an example, this public data sample can be input into the pre-trained large language model, which will then perform calculations based on it. When the large language model performs calculations based on this public data sample, it first performs embedding on it. Embedding converts discrete symbolic or category data into a continuous real-valued vector representation. In machine learning and natural language processing, embedding is commonly used to transform unstructured data such as text, images, and audio into machine-processable numerical forms, thereby supporting various feature representations and model training. In deep learning, embedding is typically used for feature extraction and representation learning. Therefore, the vector obtained by the large language model from embedding this public data sample can be used as the feature vector corresponding to this public data sample.

[0114] Step 404: Select public data samples that are similar to private data samples from the public data samples included in each of the at least one data sample classifications.

[0115] In this embodiment, having obtained at least one data sample classification, public data samples similar to the private data samples can be selected from the public data samples included in each of these at least one data sample classifications.

[0116] In some embodiments, taking any one of the above-mentioned data sample classifications (which may be referred to as the target data sample classification) as an example, when selecting public data samples similar to the private data samples from the public data samples included in the target data sample classification, the public data samples included in the target data sample classification can be input into a classification model trained using a privacy training algorithm, and the probability value of each public data sample output by the classification model belonging to the data sample classification corresponding to the private data sample can be obtained.

[0117] In practical applications, if the probability value of a public data sample is relatively high, it indicates that the classification model cannot distinguish whether the public data sample is a public data sample or a private data sample, meaning that the public data sample is quite similar to the private data sample. Therefore, the N public data samples with the highest probability values ​​can be selected from the public data samples included in the target data sample classification as public data samples similar to the aforementioned private data samples. Here, N represents a preset number; it can be a value preset by a technician or a default value, and this application does not impose any restrictions on this.

[0118] It should be noted that the above classification model can be a binary classification model used to determine whether a data sample is a public data sample or a private data sample. Alternatively, the classification model can be a multi-class classification model used to determine whether a data sample is a public data sample or a specific class of private data samples among multiple classes of private data samples.

[0119] The training samples for the aforementioned classification model can include both public and private data samples. Since private data is involved in the training of this classification model, the training process can employ privacy-preserving methods. However, in the compression process for the aforementioned large language model, this classification model is only used for data sample selection, and its training is pre-completed. Therefore, even if privacy-preserving methods are used in the training process of this classification model, it will not incur additional model performance overhead.

[0120] Step 406: Based on the selected public data samples, perform model compression on the pre-trained large language model to obtain a compressed model corresponding to the large language model.

[0121] The specific implementation of step 406 can be referred to the aforementioned step 304, and will not be repeated here.

[0122] Step 408: Fine-tune the compression model based on the private data sample to complete the compression processing for the large language model.

[0123] The specific implementation of step 408 can be referred to the aforementioned step 306, and will not be repeated here.

[0124] Corresponding to the embodiments of the aforementioned model compression method, this application also provides embodiments of a model compression apparatus.

[0125] Please refer to Figure 5 , Figure 5This is a schematic diagram illustrating the structure of a device according to an exemplary embodiment of this application. At the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, memory 508, and non-volatile memory 510, and may also include other necessary hardware. One or more embodiments of this application can be implemented in software, for example, the processor 502 reads the corresponding computer program from the non-volatile memory 510 into memory 508 and then runs it. Of course, besides software implementation, one or more embodiments of this application do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic modules, but can also be hardware or logic devices.

[0126] Please refer to Figure 6 , Figure 6 This is a block diagram illustrating a model compression device according to an exemplary embodiment of this application.

[0127] The above-mentioned model compression device can be applied to Figure 5 The device shown is used to implement the technical solution of this application. The model compression device may include:

[0128] The sample screening module 602 acquires public data samples for model pre-training of a large language model, and screens out public data samples similar to private data samples from the public data samples.

[0129] The model compression module 604 performs model compression on the pre-trained large language model based on the selected public data samples to obtain a compressed model corresponding to the large language model.

[0130] The model fine-tuning module 606 performs model fine-tuning on the compressed model based on the private data samples to complete the compression processing for the large language model.

[0131] In some embodiments, the apparatus further includes:

[0132] The model pre-training module performs model pre-training on the large language model based on the publicly available data samples.

[0133] In some embodiments, the apparatus further includes:

[0134] The classification module classifies the public data samples before filtering out public data samples similar to private data samples from the public data samples, and obtains at least one data sample classification.

[0135] The sample screening module 602 is specifically used for:

[0136] From the public data samples included in each of the at least one data sample classifications, public data samples similar to private data samples are selected respectively.

[0137] In some embodiments, the classification module is specifically used for:

[0138] Obtain the feature vectors generated by the pre-trained large language model corresponding to the publicly available data samples;

[0139] Clustering is performed on the publicly available data samples based on the feature vectors to obtain at least one data sample classification.

[0140] In some embodiments, the sample screening module 602 selects public data samples similar to private data samples from public data samples included in any target data sample category of the at least one data sample category in the following manner:

[0141] The target data sample classification includes publicly available data samples input into a classification model trained using a privacy-preserving training algorithm;

[0142] Obtain the probability value of the public data sample output by the classification model belonging to the data sample category corresponding to the private data sample;

[0143] From the public data samples included in the target data sample classification, the preset number of public data samples with the highest probability values ​​are selected as public data samples similar to the private data samples.

[0144] In some embodiments, the model compression module 604 is specifically used for:

[0145] Based on the selected public data samples, knowledge distillation training is performed on the pre-trained large language model, which serves as the teacher model, to obtain a compressed model, which serves as the student model, corresponding to the large language model.

[0146] In some embodiments, the model fine-tuning module 606 is specifically used for:

[0147] A privacy-preserving training algorithm is used to fine-tune the compressed model based on the private data samples.

[0148] In some embodiments, the privacy training algorithm includes: a differential privacy-based training algorithm.

[0149] In some embodiments, the differential privacy-based training algorithm includes: a differential privacy-based stochastic gradient descent algorithm.

[0150] For the device embodiments, they basically correspond to the method embodiments; therefore, relevant details can be found in the descriptions of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the technical solution of this application according to actual needs.

[0151] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0152] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0153] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0154] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0155] It should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0156] The foregoing has described specific embodiments of this application. Other embodiments are within the scope of this application. In some cases, the actions or steps described in this application may be performed in a different order than those shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing are also possible or may be advantageous.

[0157] The terminology used in one or more embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this application. The singular forms “a,” “the,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. The term “and / or” refers to and includes any or all possible combinations of one or more associated listed items.

[0158] The terms "an embodiment," "some embodiments," "example," "specific example," or "one implementation," as used in one or more embodiments of this application, refer to specific features or characteristics described in connection with that embodiment, which are included in at least one embodiment of this application. Illustrative descriptions of these terms do not necessarily refer to the same embodiment. Furthermore, the described specific features or characteristics may be combined in a suitable manner in one or more embodiments of this application. In addition, different embodiments and specific features or characteristics from different embodiments may be combined without contradiction.

[0159] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of one or more embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0160] The above description is merely a preferred embodiment of one or more embodiments of this application and is not intended to limit the scope of one or more embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this application should be included within the protection scope of one or more embodiments of this application.

[0161] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

Claims

1. A model compression method, the method comprising: Obtain public data samples for model pre-training of a large language model, and filter out public data samples similar to private data samples from the public data samples; Based on the selected public data samples, the pre-trained large language model is compressed to obtain a compressed model corresponding to the large language model. Based on the private data samples, the compression model is fine-tuned to complete the compression processing for the large language model.

2. The method according to claim 1, further comprising: The large language model is pre-trained based on the publicly available data samples.

3. The method according to claim 1, further comprising, before filtering out public data samples similar to private data samples from the public data samples: The publicly available data samples are classified to obtain at least one data sample classification. The step of filtering public data samples from the public data samples that are similar to private data samples includes: From the public data samples included in each of the at least one data sample classifications, public data samples similar to private data samples are selected respectively.

4. The method according to claim 3, wherein classifying the publicly available data samples to obtain at least one data sample classification includes: Obtain the feature vectors generated by the pre-trained large language model corresponding to the publicly available data samples; Clustering is performed on the publicly available data samples based on the feature vectors to obtain at least one data sample classification.

5. The method according to claim 4, wherein public data samples similar to private data samples are selected from the public data samples included in any target data sample classification of the at least one data sample classification in the following manner: The target data sample classification includes publicly available data samples input into a classification model trained using a privacy-preserving training algorithm; Obtain the probability value of the public data sample output by the classification model belonging to the data sample category corresponding to the private data sample; From the public data samples included in the target data sample classification, the preset number of public data samples with the highest probability values ​​are selected as public data samples similar to the private data samples.

6. The method according to claim 1, wherein the step of compressing the pre-trained large language model based on the selected public data samples to obtain a compressed model corresponding to the large language model includes: Based on the selected public data samples, knowledge distillation training is performed on the pre-trained large language model, which serves as the teacher model, to obtain a compressed model, which serves as the student model, corresponding to the large language model.

7. The method according to claim 1, wherein fine-tuning the compressed model based on the private data sample comprises: A privacy-preserving training algorithm is used to fine-tune the compressed model based on the private data samples.

8. The method according to claim 5 or 7, wherein the privacy training algorithm comprises: Training algorithm based on differential privacy.

9. The method according to claim 8, wherein the differential privacy-based training algorithm comprises: A stochastic gradient descent algorithm based on differential privacy.

10. A model compression apparatus, the apparatus comprising: The sample screening module acquires public data samples for pre-training large language models and filters out public data samples that are similar to private data samples from the public data samples. The model compression module compresses the pre-trained large language model based on the selected public data samples to obtain a compressed model corresponding to the large language model. The model fine-tuning module performs model fine-tuning on the compressed model based on the private data samples to complete the compression processing for the large language model.

11. An electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 1 to 9 by executing the executable instructions.

12. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the method as described in any one of claims 1 to 9.

Images