Model compression methods, apparatus, devices and storage media

Abstract

This application provides one or more embodiments of a model compression method, apparatus, device, and storage medium. The method includes: inputting descriptive information corresponding to at least one private data sample into a first large language model; using the descriptive information as a generation condition to generate at least one generated data sample corresponding to the descriptive information; selecting generated data samples similar to the at least one private data sample from the at least one generated data sample; performing model compression on a pre-trained second large language model based on the selected generated data samples to obtain a compressed model corresponding to the second large language model; wherein the second large language model is pre-trained based on public data samples; and fine-tuning the compressed model based on the private data samples to complete the compression processing for the second large language model.

Description

Technical Field

[0001] One or more embodiments of this application relate to the field of artificial intelligence technology, and in particular to a model compression method, apparatus, device, and storage medium. Background Technology

[0002] Large Language Models (LLMs) are deep learning models trained on large amounts of text data. They can be used to generate natural language text or understand the meaning of language text. LLMs can handle a variety of natural language tasks, such as text classification, question answering, and dialogue, and are an important pathway to artificial intelligence.

[0003] The emergence of large language models has dramatically changed the way deep learning models are trained in Natural Language Processing (NLP). First, large language models can be pre-trained on extremely large and diverse public datasets. Then, using smaller, task-specific datasets, the pre-trained large language model can be fine-tuned for the specific task. Finally, because large language models typically contain hundreds of millions of parameters, their inference time and memory usage are excessive for many applications. Furthermore, many parameters are redundant and can be removed while maintaining model performance. Therefore, after fine-tuning, large language models are usually further compressed to reduce the number of parameters. The compressed deep learning model can then be deployed directly. Summary of the Invention

[0004] One or more embodiments of this application provide the following technical solutions:

[0005] This application provides a model compression method, the method comprising:

[0006] The description information corresponding to at least one private data sample is input into the first large language model, and the first large language model uses the description information as a generation condition to generate at least one generated data sample corresponding to the description information.

[0007] From the at least one generated data sample, select generated data samples that are similar to the at least one private data sample;

[0008] Based on the selected generated data samples, the pre-trained second language model is compressed to obtain a compressed model corresponding to the second language model; wherein, the second language model is pre-trained based on publicly available data samples;

[0009] Based on the private data samples, the compression model is fine-tuned to complete the compression processing for the second major language model.

[0010] This application also provides a model compression apparatus, the apparatus comprising:

[0011] The sample generation module inputs the description information corresponding to at least one private data sample into the first large language model, and the first large language model uses the description information as the generation condition to generate at least one generated data sample corresponding to the description information.

[0012] The sample filtering module filters out generated data samples that are similar to the at least one private data sample from the at least one generated data sample;

[0013] The model compression module compresses the pre-trained second language model based on the selected generated data samples to obtain a compressed model corresponding to the second language model; wherein, the second language model is pre-trained based on publicly available data samples.

[0014] The model fine-tuning module performs model fine-tuning on the compressed model based on the private data samples to complete the compression processing for the second major language model.

[0015] This application also provides an electronic device, including:

[0016] processor;

[0017] Memory used to store processor-executable instructions;

[0018] The processor executes the executable instructions to implement the steps of the method as described in any of the preceding descriptions.

[0019] This application also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method as described in any of the preceding claims.

[0020] In the above technical solution, a first language model can be used to generate generated data samples corresponding to the description information corresponding to the private data samples. Then, generated data samples similar to the private data samples can be further selected from the generated data samples. For the second language model pre-trained based on public data samples, the pre-trained second language model can be compressed based on the selected generated data samples to obtain a compressed model corresponding to the second language model. Finally, the compressed model can be fine-tuned based on the private data samples, thereby completing the compression processing of the second language model and obtaining a compressed model more suitable for deployment.

[0021] By adopting the above approach, on the one hand, since the fine-tuning process and the compression process are interchanged in terms of execution order, the fine-tuning process can no longer be performed on the basis of the large language model, but on the basis of the compressed model corresponding to the large language model, thereby reducing the consumption of computational resources during the fine-tuning process.

[0022] On the other hand, since the compression process can now use non-privacy training instead of privacy training, with only the fine-tuning process still employing privacy training, the performance overhead of the model can be reduced. Furthermore, because the compressed model contains fewer model parameters than the large language model, fine-tuning the compressed large language model using privacy training has a smaller negative impact on model performance compared to fine-tuning the uncompressed large language model. Therefore, swapping the execution order of the fine-tuning and compression processes further helps reduce performance overhead and improve model performance.

[0023] On the other hand, by using the first major language model, corresponding generated data samples are generated based on the descriptive information corresponding to the private data samples. Furthermore, generated data samples similar to the private data samples are selected from the generated data samples. The selected generated data samples are then used to complete the compression processing for the second major language model. This can be considered as expanding the number of private data samples and increasing their diversity. Attached Figure Description

[0024] The accompanying drawings used in the description of the exemplary embodiments will now be explained, wherein:

[0025] Figure 1 This is a schematic diagram of the compression process for large language models in related technologies.

[0026] Figure 2 This is a schematic diagram illustrating a compression process for a large language model, as shown in an exemplary embodiment of this application.

[0027] Figure 3 This is a flowchart illustrating a model compression method in an exemplary embodiment of this application.

[0028] Figure 4 This is a schematic diagram of the structure of a device shown in an exemplary embodiment of this application.

[0029] Figure 5 This is a block diagram illustrating a model compression device according to an exemplary embodiment of this application. Detailed Implementation

[0030] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this application. Rather, they are merely examples consistent with some aspects of one or more embodiments of this application.

[0031] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this application in other embodiments. In some other embodiments, the methods may include more or fewer steps than those described in this application. Furthermore, a single step described in this application may be broken down into multiple steps in other embodiments; and multiple steps described in this application may be combined into a single step in other embodiments.

[0032] Large Language Models (LLMs) are deep learning models trained on large amounts of text data. They can be used to generate natural language text or understand the meaning of language text. LLMs can handle a variety of natural language tasks, such as text classification, question answering, and dialogue, and are an important pathway to artificial intelligence.

[0033] Large language models typically employ the Transformer architecture; that is, large language models are usually deep learning models based on the Transformer architecture. Deep learning models based on the Transformer architecture are a class of neural network models that utilize the Transformer architecture, and these models perform exceptionally well in fields such as natural language processing.

[0034] Transformer is a neural network model used for sequence-to-sequence modeling. Transformer does not rely on recursive structures, enabling parallel training and inference, thus accelerating model processing.

[0035] In deep learning models based on the Transformer architecture, multi-layered Transformer encoders are typically used to extract features from the input sequence, and a Transformer decoder is used to transform these features into an output sequence. These models also typically employ self-attention mechanisms to capture long-range dependencies in the input sequence, as well as residual connections and normalization methods to accelerate training and improve model performance.

[0036] Pre-trained models are language models pre-trained on large-scale unlabeled text data. Pre-trained models are general-purpose; they are not designed or optimized for a specific task. To adapt a pre-trained model to a specific task, fine-tuning is required to improve its performance on that task. Large language models, on the other hand, are models that undergo further fine-tuning based on pre-trained models and are learned through supervised learning using labeled text data.

[0037] In other words, the training process of a large language model can generally be divided into two stages: pre-training and fine-tuning. In the pre-training stage, unsupervised learning (e.g., self-supervised learning) can be used to pre-train on public datasets (e.g., online encyclopedias, online articles, books, etc.). The model attempts to predict missing parts or the next word based on context, learning the inherent structure and rules of language, such as semantics and syntax. Optimization algorithms like gradient descent can be used to continuously adjust model parameters, gradually improving the model's performance on the pre-training task. In the fine-tuning stage, a corresponding supervised learning task (e.g., text classification, named entity recognition, question answering systems, etc.) can be selected based on the specific application scenario and task requirements, and a task-specific dataset can be prepared. The pre-trained model serves as the starting point for fine-tuning, using supervised learning on the task-specific dataset. Optimization algorithms like gradient descent can be used to adjust model parameters based on the dataset labels, gradually adapting the model to the requirements of the specific task and dataset. Pre-training and fine-tuning are complementary processes. Pre-training enables the model to have a wide range of language understanding capabilities, while fine-tuning makes the model more specialized and accurate for specific tasks.

[0038] Large language models contain hundreds of millions of parameters. On the one hand, the inference time and memory consumption of these models are too large for many applications. On the other hand, many of these parameters are redundant and can be removed while maintaining model performance. Therefore, in the field of natural language processing, large language models are usually not used directly. Instead, the trained large language models are compressed to reduce the number of model parameters, and the compressed deep learning model can be used when deploying the model.

[0039] Model compression is a technique that reduces storage space and computing resource requirements by decreasing the size and complexity of neural network models. It can improve the deployment efficiency of models in resource-constrained environments such as mobile devices and edge computing, and reduce latency in transmission and inference. Model compression algorithms include pruning, quantization, network architecture optimization, and knowledge distillation. These algorithms can be used individually or in combination for better results. Typically, the appropriate model compression algorithm is selected based on the characteristics of the model, the specific application scenario, and performance requirements.

[0040] While deep learning models based on the Transformer architecture have driven the development of deep learning technology toward designing larger models to achieve better performance, in practical applications, the more model parameters a larger model contains, the more data is needed to train the model. This means that more personal information may be included in the model training.

[0041] If this personal information is not adequately protected (e.g., through encryption or anonymization), it is theoretically possible to recover some or all of the training data by reverse engineering the model parameters. This process of reverse engineering the model parameters is commonly referred to as "model inversion" or "model derivation." Although achieving this reverse engineering is not easy at present, research has shown that by performing reverse data analysis on the model parameters, it is possible to obtain some data features or data distributions related to the training data.

[0042] In related technologies, in order to mitigate the leakage of private data contained in the training dataset of large language models, privacy-preserving training is adopted during the training process of large language models.

[0043] Specifically, for the training process of large language models, the dataset used in the pre-training stage is usually a public dataset, while the dataset used in the fine-tuning stage is a private dataset containing privacy data. Therefore, privacy training is adopted in the fine-tuning stage.

[0044] Privacy-preserving training is a machine learning method designed to protect the privacy of training data during model training. It utilizes techniques such as differential privacy or homomorphic encryption to train and update models without exposing the original data.

[0045] In traditional machine learning, model training is typically performed on a centralized server, where all training data is collected in one place and used to train the model. This approach carries the risk of data privacy breaches because all data is sent to the server for processing.

[0046] In contrast, privacy-preserving training employs various privacy-preserving techniques to protect data privacy. The most common of these is differential privacy, which protects privacy by adding noise to the original data. Specifically, for each sample used in training, differential privacy merges it with another "fake" sample and then uses random techniques to add noise, making the real and fake samples privacy-equivalent while ensuring the model's accuracy.

[0047] Besides differential privacy, homomorphic encryption is another widely used technique for privacy-preserving training. Homomorphic encryption allows computation to be performed on encrypted data, and the output is also encrypted, thus avoiding the problem of data being transmitted anywhere.

[0048] After fine-tuning the large language model, to further reduce the number of model parameters, the trained large language model can be compressed. It should be noted that the compression process typically uses private datasets containing sensitive data, therefore privacy-preserving training methods must be employed.

[0049] Please refer to Figure 1 , Figure 1 This is a schematic diagram of the compression process for large language models in related technologies.

[0050] like Figure 1 As shown, firstly, unsupervised learning can be used to pre-train a large language model using a public dataset. This pre-training process employs a non-privacy training method.

[0051] Then, supervised learning can be used to fine-tune the pre-trained large language model using a private dataset containing privacy-preserving data. This fine-tuning process employs privacy-preserving training methods. This completes the training of the large language model.

[0052] It's important to note that compressing a large language model essentially involves reducing the number of its parameters to obtain a smaller deep learning model. This smaller model is then trained using the same dataset used for pre-training (fine-tuning) the larger language model, allowing its performance to approach that of the larger model. Therefore, the aforementioned private dataset can also be used to compress the trained large language model. This compression process also employs privacy-preserving training methods.

[0053] Therefore, by implementing a training method that includes three steps—pre-training, fine-tuning, and compression—for this large language model, compression processing can be completed, ultimately resulting in a deep learning model with an appropriate number of model parameters, making it more suitable for deployment.

[0054] However, the compression schemes for large language models in related technologies have the following drawbacks:

[0055] First, the fine-tuning process is carried out on the basis of the pre-trained large language model. At this time, the large language model has not been compressed and contains a large number of model parameters, which will result in the training corresponding to the fine-tuning process requiring more computing resources.

[0056] Second, although using privacy-preserving training methods during fine-tuning can reduce the risk of privacy data leakage, current privacy-preserving training methods usually sacrifice model performance to protect privacy data. Therefore, while introducing privacy-preserving training methods can reduce the risk of privacy data leakage to some extent, it will have some negative impact on model performance. Moreover, the larger the number of parameters in the model, the more significant the negative impact on the model's performance will be.

[0057] For example, taking differential privacy algorithms as a privacy training algorithm, these algorithms typically introduce a certain degree of noise or perturbation into the privacy data to mask sensitive information. The presence of this noise or perturbation obviously leads to data distortion in the input model, negatively impacting its performance; for instance, it may reduce the accuracy of the model's output. Furthermore, in practical applications, the amount of noise or perturbation introduced by differential privacy algorithms into the privacy data is usually related to the number of model parameters. The more model parameters a model has, the greater the amount of noise or perturbation introduced. Therefore, it's easy to understand that for models with a larger number of parameters, the amount of noise or perturbation introduced by the differential privacy algorithm will be greater, and consequently, the negative impact of this noise or perturbation on the model's performance will be more significant.

[0058] It is evident that fine-tuning a large language model using privacy-preserving training methods will inevitably result in significant performance overhead. Moreover, this performance overhead is particularly pronounced when privacy-preserving training is employed in both the fine-tuning and compression processes.

[0059] This application provides a technical solution for model compression. A first language model is used to generate generated data samples corresponding to the description information of the private data samples. These generated data samples are then further filtered to select those similar to the private data samples. For a second language model pre-trained based on public data samples, the pre-trained model is compressed using the filtered generated data samples to obtain a compressed model. Finally, the compressed model is fine-tuned based on the private data samples, thus completing the compression process for the second language model and obtaining a more suitable compressed model for deployment.

[0060] By adopting the above approach, on the one hand, since the fine-tuning process and the compression process are interchanged in terms of execution order, the fine-tuning process can no longer be performed on the basis of the large language model, but on the basis of the compressed model corresponding to the large language model, thereby reducing the consumption of computational resources during the fine-tuning process.

[0061] On the other hand, since the compression process can now use non-privacy training instead of privacy training, with only the fine-tuning process still employing privacy training, the performance overhead of the model can be reduced. Furthermore, because the compressed model contains fewer model parameters than the large language model, fine-tuning the compressed large language model using privacy training has a smaller negative impact on model performance compared to fine-tuning the uncompressed large language model. Therefore, swapping the execution order of the fine-tuning and compression processes further helps reduce performance overhead and improve model performance.

[0062] On the other hand, by using the first major language model, corresponding generated data samples are generated based on the descriptive information corresponding to the private data samples. Furthermore, generated data samples similar to the private data samples are selected from the generated data samples. The selected generated data samples are then used to complete the compression processing for the second major language model. This can be considered as expanding the number of private data samples and increasing their diversity.

[0063] Please refer to Figure 2 , Figure 2This is a schematic diagram illustrating a compression process for a large language model, as shown in an exemplary embodiment of this application.

[0064] and Figure 1 The compression process shown is different, such as Figure 2 The execution order of the fine-tuning process and the compression process in the compression process shown in the diagram has been interchanged.

[0065] After pre-training a large language model using a public dataset and a non-privacy training method, you can temporarily refrain from fine-tuning the pre-trained large language model and instead compress it first.

[0066] Specifically, the first large language model can be used to generate generated data samples corresponding to the description information of the private data samples contained in the private dataset. Then, generated data samples similar to the private data samples can be further selected from the generated data samples. Finally, the selected generated data samples can be used to compress the pre-trained large language model.

[0067] It should be noted that since the dataset used in this compression process is essentially newly generated data samples similar to the private data samples, the compression process can forgo privacy-preserving training and instead employ non-privacy-preserving training. In this case, the deep learning model obtained through compression can be called a compressed model. At this stage, the compressed model has not yet been optimized for a specific task and is not suitable for any particular task.

[0068] Finally, supervised learning can be used to fine-tune the compressed model using the aforementioned private dataset, optimizing it for a specific task. This fine-tuning process employs privacy-preserving training. Thus, by implementing a training method involving pre-training, fine-tuning, and compression on the aforementioned large language model, compression processing of the large language model is completed, and the fine-tuned compressed model becomes a more suitable deep learning model for deployment.

[0069] Please combine Figure 2 ,refer to Figure 3 , Figure 3 This is a flowchart illustrating a model compression method in an exemplary embodiment of this application.

[0070] In this embodiment, the above-described model compression method can be applied to a server. This server can be a server containing a single independent physical host, or a server cluster consisting of multiple independent physical hosts; alternatively, the server can be a virtual server, cloud server, or similar service hosted by a host cluster.

[0071] Alternatively, the above model compression method can be applied to electronic devices with a certain computing power, such as desktop computers, laptops, PDAs, and tablets.

[0072] The above model compression method may include the following steps:

[0073] Step 302: Input the description information corresponding to at least one private data sample into the first large language model, and the first large language model uses the description information as a generation condition to generate at least one generated data sample corresponding to the description information.

[0074] In this embodiment, for the pre-trained large language model (which can be referred to as the second large language model), we can temporarily refrain from fine-tuning the pre-trained second large language model and instead compress it first.

[0075] Simply swapping the execution order of the fine-tuning and compression processes would result in the compression process using the public dataset while the private data is not involved, making the compression process lack specificity and thus not being optimized for the private data.

[0076] In practical applications, the closer the distribution of the dataset used in the pre-training process is to that of the dataset used in the fine-tuning process, the better the fine-tuning effect. Since the execution order of the above fine-tuning process and the above compression process has been swapped, in order to ensure that the compression process benefits the private data and ensures the fine-tuning effect, when compressing the above-mentioned pre-trained large language model, a generated data sample similar to at least one private data sample contained in the private dataset used to fine-tune the large language model can be used. This sample is generated by the fully trained large language model (which can be referred to as the first large language model).

[0077] Specifically, the descriptive information corresponding to at least one private data sample contained in the private dataset can first be input into the first language model. The first language model then uses this descriptive information as a generation condition to generate at least one generated data sample corresponding to the descriptive information.

[0078] Conditional generation refers to influencing the generated text by providing a specific condition during the text generation process. This condition can be a word, a phrase, a question, or a complete sentence. Large language models can make the generated text more accurate and targeted by introducing generation conditions into the generation process.

[0079] When using large language models for conditional generation, methods such as prefix setting, control tags, and context input can be employed. Prefix setting refers to inputting the conditional text as a prefix into the model before generating the text. The model then uses this prefix to influence the output; for example, to generate a description about dogs, "dogs are" can be input as a prefix. Control tags allow adding special control tags to the conditional text to indicate the direction or topic of generation. The model can adjust accordingly based on the presence or absence of these tags; for example, adding a "topic:" tag followed by keywords of the topic can indicate that the generated text should be relevant to that topic. Context input allows the conditional text to be input into the model as part of the context. The model can use this context to understand the meaning of the conditions and generate appropriate responses; for example, in dialogue generation, previous dialogue history can be input into the model as context to generate responses related to previous dialogue content.

[0080] In some embodiments, the data samples described above (including public data samples, private data samples, and generated data samples) may be text data samples. Accordingly, the descriptive information described above may include one or more of the following: text length, text type, text description content, etc.

[0081] Step 304: Select generated data samples that are similar to the at least one private data sample from the at least one generated data sample.

[0082] In this embodiment, the first language model described above can output at least one generated data sample corresponding to the description information, using the description information as the generation condition.

[0083] Furthermore, generated data samples similar to at least one private data sample contained in the aforementioned private dataset can be selected from the at least one generated data sample mentioned above to form a new dataset for compressing the pre-trained second language model mentioned above.

[0084] Step 306: Based on the selected generated data samples, perform model compression on the pre-trained second language model to obtain a compressed model corresponding to the second language model; wherein, the second language model is pre-trained based on publicly available data samples.

[0085] In this embodiment, a pre-trained large language model (which can be referred to as the second large language model) can be prepared in advance. The second large language model can be pre-trained based on publicly available data samples contained in a public dataset.

[0086] In some embodiments, the second language model can be pre-trained based on the publicly available data samples contained in the publicly available dataset.

[0087] Specifically, unsupervised learning can be used to pre-train the second language model on the aforementioned public dataset. The second language model can attempt to predict missing parts or the next word based on context, learning the inherent structure and rules of language, such as semantics and syntax. In this case, optimization algorithms such as gradient descent can be used to continuously adjust the model parameters of the second language model, gradually improving its performance on the pre-training task.

[0088] To further reduce the negative impact of the fine-tuning process on the performance of the pre-trained second language model, the execution order of the fine-tuning and compression processes can be interchanged. After the pre-training of the second language model is completed, the second language model can be compressed first.

[0089] It should be noted that, to further reduce the risk of privacy data leakage, when compressing the pre-trained second language model, private data samples can be used instead. After selecting generated data samples similar to the at least one private data sample from the at least one generated data sample, model compression can be further performed on the pre-trained second language model based on the selected generated data samples.

[0090] In some embodiments, model compression is performed on the pre-trained second language model, which can be accomplished by knowledge distillation training on the second language model.

[0091] In this scenario, after selecting generated data samples similar to the at least one private data sample from the aforementioned generated data samples, knowledge distillation training can be performed on the pre-trained second language model based on the selected generated data samples to obtain a compressed model corresponding to the second language model. In this knowledge distillation process, the second language model serves as the teacher model, while the compressed model serves as the student model.

[0092] Knowledge distillation allows the transfer of knowledge from the aforementioned second-largest language model (acting as the teacher model) to the aforementioned compressed model (acting as the student model), thereby improving the performance of the compressed model. To perform knowledge distillation, a loss function is first defined to measure the difference between the predictions of the student model and the teacher model. This is typically achieved using soft targets, where the output of the teacher model is used as the target of the student model, and cross-entropy loss or other similar loss functions are calculated. Subsequently, the predictions of the teacher model can be used as an auxiliary target, combined with the original target of the student model, for training. The model parameters of the student model are updated by iteratively minimizing the loss function.

[0093] Step 308: Fine-tune the compression model based on the private data sample to complete the compression processing for the second major language model.

[0094] In this embodiment, after the model compression for the second major language model is completed, the compressed model has not yet been optimized for a specific task and is not suitable for any particular task. Therefore, further fine-tuning of the second major language model can be performed based on the compressed model obtained from the model compression.

[0095] It should be noted that, unlike the model compression of the second largest language model mentioned above, when fine-tuning the compressed model, the private data samples contained in the private dataset can still be used.

[0096] Specifically, the aforementioned private dataset can be a labeled dataset for a specific task (e.g., text classification, named entity recognition, question answering systems, etc.). The compressed model can be used as a starting point for fine-tuning, and trained on this private dataset using supervised learning. In this case, optimization algorithms such as gradient descent can be used to adjust the model parameters of the compressed model according to the labels of the private dataset, so that the compressed model gradually adapts to the requirements of the specific task and the private dataset.

[0097] Thus, by implementing a training method that includes three steps—pre-training, fine-tuning, and compression—on the second largest language model, the compression process of the second largest language model was completed. The compressed model after fine-tuning is a deep learning model that is more suitable for deployment.

[0098] In some embodiments, since the pre-training process uses a public dataset and the compression process uses a new dataset composed of generated data samples, both the pre-training and compression processes can be trained in a non-privacy manner. However, the fine-tuning process uses a private dataset, therefore it can be trained in a privacy-preserving manner.

[0099] In other words, when fine-tuning the compressed model based on the aforementioned private data samples, a privacy training algorithm can be used to fine-tune the compressed model based on the private data samples.

[0100] Since the compression process described above can now employ non-privacy training instead of privacy-preserving training, with only the fine-tuning process retaining privacy-preserving training, this approach reduces the model's performance overhead. Furthermore, although the fine-tuning process still uses privacy-preserving training, it no longer fine-tunes the large language model but rather the compressed model corresponding to it. Because the compressed model contains fewer parameters than the large language model, introducing privacy-preserving training into the compressed large language model for fine-tuning has a smaller negative impact on model performance compared to directly introducing privacy-preserving training into the uncompressed large language model. Therefore, by swapping the execution order of the fine-tuning and compression processes, the negative impact of privacy-preserving training on model performance can be significantly reduced, contributing to improved model performance.

[0101] In some embodiments, the above-mentioned privacy training algorithm may include: a training algorithm based on differential privacy (DP).

[0102] Differential privacy is a privacy-preserving technical framework designed to provide privacy protection during data analysis while allowing for meaningful statistical analysis. Differential privacy protects individual privacy by introducing noise or perturbations into the raw data and limits the possibility of inferring sensitive individual information from the analysis results.

[0103] Differential privacy provides a mathematically defined privacy protection mechanism that quantifies the impact on the final result after removing or replacing an individual from a dataset with another. By adding appropriate noise or perturbations, significant privacy protection can be provided while maintaining the usability of the data. The core idea of ​​differential privacy is to minimize the degree to which the output changes due to the participation or non-participation of an individual, thereby protecting the privacy of that individual.

[0104] Differential privacy mechanisms are specific algorithms used to implement differential privacy, including Laplace, Gaussian, Exponential, and noise-adding mechanisms. Laplace is a probability-distribution-based differential privacy mechanism. In Laplace, noise is added to the query results to obscure the original data. The magnitude of the noise is controlled by a Laplace distribution, having zero mean and a certain scaling parameter. Gaussian is also a probability-distribution-based differential privacy mechanism. It uses a Gaussian distribution to generate noise. Exponential is a differential privacy mechanism that selects output results based on data characteristics. It balances the trade-off between privacy protection and data utility by selecting based on data contribution and the sensitivity of the objective function. Noise-adding mechanisms are differential privacy mechanisms that directly add noise to the query results. Noise-adding mechanisms can reduce the leakage of individual privacy information by adding noise to the query results, for example, by adding noise to the results when calculating the average.

[0105] In some embodiments, the training algorithm based on differential privacy may include: a differentially private stochastic gradient descent (DP-SGD) algorithm.

[0106] DP-SGD combines privacy protection with the Stochastic Gradient Descent (SGD) algorithm. DP-SGD protects the personal privacy information contained in the training samples of the model by adding noise, thus preventing the leakage of sensitive information targeting specific individuals.

[0107] DP-SGD protects privacy based on the concept of differential privacy. During training, DP-SGD introduces noise into the calculated gradients to achieve privacy. Specifically, in each iteration, it perturbs the calculated gradients by adding noise. This can hide the contribution of personal data to a certain extent, thus protecting privacy. To control the degree of privacy leakage, DP-SGD introduces a privacy budget. The privacy budget measures how much noise can be added to protect private information. A smaller privacy budget means stricter privacy protection. DP-SGD trains its model based on the stochastic gradient descent algorithm. In each iteration, it randomly selects a small batch of samples from the training samples, calculates their corresponding gradients, and updates the model parameters according to the gradient direction.

[0108] The process of selecting generated data samples similar to the at least one private data sample from the at least one generated data sample in step 304 above will be described in detail below.

[0109] In some embodiments, the following steps may be performed iteratively until the number of iterations reaches a preset threshold (which may be referred to as the first threshold) to filter out generated data samples similar to the at least one private data sample from the at least one generated data sample: performing data transformation processing on the generated data sample to obtain a transformed generated data sample, and determining the generated data sample and the transformed generated data sample as generated data samples to be filtered; filtering out generated data samples similar to the at least one private data sample from the generated data samples to be filtered. The first threshold may be a value manually preset according to actual needs, or it may be a system default value; this application does not limit this.

[0110] It should be noted that the number of at least one generated data sample generated by the first major language model can be equal to the number of at least one private data sample; the number of generated data samples selected from the generated data samples to be screened can also be equal to the number of at least one private data sample.

[0111] For ease of description, at least one generated data sample is represented by N generated data samples, and at least one private data sample is represented by N private data samples.

[0112] The iterative process can be described as follows:

[0113] In the first iteration, the initial N generated data samples (i.e., the N generated data samples output by the first large language model mentioned above) can be subjected to t data transformation processes to obtain N*t transformed generated data samples. These N generated data samples and these N*t transformed generated data samples are determined together as generated data samples to be screened, resulting in N*(t+1) generated data samples to be screened. Thus, N generated data samples similar to the N private data samples can be screened from these N*(t+1) generated data samples to be screened.

[0114] In the second iteration, the N generated data samples obtained after the first iteration can be subjected to t data transformation processes to obtain N*t transformed generated data samples. These N generated data samples and these N*t transformed generated data samples are then determined together as generated data samples to be screened, resulting in N*(t+1) generated data samples to be screened. Thus, N generated data samples similar to the N private data samples can be screened from these N*(t+1) generated data samples to be screened.

[0115] In the third iteration, the N generated data samples obtained after the second iteration can be subjected to t data transformation processes to obtain N*t transformed generated data samples. These N generated data samples and these N*t transformed generated data samples are determined together as generated data samples to be screened, resulting in N*(t+1) generated data samples to be screened. Thus, N generated data samples similar to the N private data samples can be screened from these N*(t+1) generated data samples to be screened.

[0116] This process continues until the number of iterations reaches the first threshold mentioned above.

[0117] In some embodiments, the generated data samples can be subjected to different data transformation processes depending on the number of iterations, so as to obtain a variety of transformed generated data samples. Specifically, it can be first determined whether the number of iterations has reached a preset threshold (which may be referred to as the second threshold). The second threshold can be a value manually preset according to actual needs, or it can be a default value of the system, and this application does not limit it in this regard.

[0118] If the number of iterations does not reach the second threshold, the generated data sample can be input into the first language model, which will then generate data similar to the generated data sample and output the generated data. In this case, the data output by the first language model (i.e., the generated data similar to the generated data sample) can be identified as the transformed generated data sample.

[0119] For example, assuming the generated data sample is text data in sentence form, if the number of iterations mentioned above has not reached the second threshold, the first language model can be used to generate sentences with similar sentence structure or meaning to the generated data sample, which can then be used as the transformed generated data sample. This transformation can be regarded as a transformation of the entire sentence of the generated data sample, and the degree of transformation can be determined by the number of iterations; the higher the number of iterations, the lower the degree of transformation.

[0120] If the number of iterations reaches the second threshold, some data content can be deleted from the generated data sample. The generated data sample with some data content deleted is then input into the first language model. The first language model completes the generated data sample with some data content deleted and outputs the completed generated data sample. In this case, the data output by the first language model (i.e., the completed generated data sample) can be identified as the transformed generated data sample.

[0121] For example, assuming the generated data sample is text data in the form of a sentence, when the above iteration number reaches the second threshold, some words in the generated data sample can be removed (for example, nouns and verbs in the generated data sample can be removed first). For the generated data sample with some words removed, the first language model is used to complete the sentence to obtain the completed generated data sample, which is used as the transformed generated data sample.

[0122] In practical applications, other fully trained large language models can also be used to perform data transformation processing on the generated data samples. This avoids the first large language model from participating in the data transformation processing task and ensures the ability of the first large language model to generate generated data samples that correspond to the descriptive information of the private data samples.

[0123] In some embodiments, when selecting generated data samples similar to at least one private data sample from the generated data samples to be screened, the similarity between each generated data sample in the generated data samples to be screened and each private data sample in at least one private data sample can be calculated first, and then, based on the calculated similarity, generated data samples similar to at least one private data sample can be selected from the generated data samples to be screened.

[0124] For example, assuming the above-mentioned generated data samples to be screened include generated data sample 1, generated data sample 2, generated data sample 3, and generated data sample 4, and the above-mentioned at least one private data sample includes private data sample 1 and private data sample 2, then we can first calculate the similarity between generated data sample 1 and private data sample 1, the similarity between generated data sample 1 and private data sample 2, the similarity between generated data sample 2 and private data sample 1, the similarity between generated data sample 2 and private data sample 2, the similarity between generated data sample 3 and private data sample 1, the similarity between generated data sample 3 and private data sample 2, the similarity between generated data sample 4 and private data sample 1, and the similarity between generated data sample 4 and private data sample 2. Then, based on these 8 similarities, we can select generated data samples that are similar to these 2 private data samples from these 4 generated data samples to be screened.

[0125] In some embodiments, when calculating the similarity between each generated data sample in the above-mentioned generated data samples to be screened and each private data sample in the above-mentioned at least one private data sample, specifically, the feature vectors (referred to as first feature vectors) corresponding to each generated data sample in the generated data samples to be screened can be obtained, and the feature vectors (referred to as second feature vectors) corresponding to each private data sample in the at least one private data sample can be obtained. Thus, the vector similarity between the first feature vectors corresponding to each generated data sample and the second feature vectors corresponding to each private data sample can be calculated, which serves as the similarity between each generated data sample in the generated data samples to be screened and each private data sample in the at least one private data sample. The vector similarity can be Euclidean distance, cosine similarity, or Manhattan distance; this application does not impose any limitations on this.

[0126] In practical applications, the first feature vector corresponding to the generated data sample can be obtained by embedding the generated data sample; similarly, the second feature vector corresponding to the private data sample can be obtained by embedding the private data sample. Embedding transforms discrete symbolic or categorical data into a continuous real-valued vector representation. In machine learning and natural language processing, embedding is commonly used to convert unstructured data such as text, images, and audio into machine-processable numerical forms, thereby supporting various feature representations and model training. In deep learning, embedding is typically used for feature extraction and representation learning.

[0127] It should be noted that a feature extractor can be used to calculate the first feature vector corresponding to the generated data sample and the second feature vector corresponding to the private data sample. The feature extractor can be a BERT (Bidirectional Encoder Representations from Transformers) model, a feature extraction layer in the aforementioned first large language model, or a feature extraction layer in other fully trained large language models; this application does not impose any restrictions on this.

[0128] In some embodiments, when selecting generated data samples similar to at least one private data sample from the generated data samples to be screened based on the similarity between each generated data sample in the generated data samples to be screened and each private data sample in the at least one private data sample, the similarity between each generated data sample and each private data sample can be first converted into the probability distribution of each generated data sample in the generated data samples to be screened. Subsequently, based on the probability distribution of each generated data sample, the generated data samples to be screened can be resampled, and the data obtained through the resampling process can be determined as the generated data samples similar to at least one private data sample selected from the generated data samples to be screened.

[0129] In practical applications, sampling with replacement can be used to resample the generated data samples to be screened based on the probability distribution of each generated data sample.

[0130] Resampling is a statistical method used to generate new samples from an existing dataset. The basic idea of ​​resampling is to generate a new set of samples that are similar to but differ from the original dataset by randomly sampling or transforming the original data. This allows for changes to the probability distribution of the data by increasing the sample size, adjusting sample weights, or generating entirely new samples, thereby meeting specific needs.

[0131] On the one hand, in some embodiments, in order to convert the similarity between each generated data sample in the above-mentioned generated data samples to be screened and each private data sample in the above-mentioned at least one private data sample into the distribution probability of each generated data sample in the generated data samples to be screened, firstly, for each private data sample in the at least one private data sample, the generated data sample with the highest similarity to this private data sample in the generated data samples to be screened can be determined. Then, for each generated data sample in the generated data samples to be screened, the number of private data samples that regard this generated data sample as their highest similarity generated data sample can be counted. Finally, the number counted for each generated data sample can be normalized to obtain the normalized number, and the normalized number can be determined as the distribution probability of each generated data sample.

[0132] Continuing with the example above, suppose the generated data samples to be screened include generated data sample 1, generated data sample 2, generated data sample 3, and generated data sample 4, and the at least one private data sample includes private data sample 1 and private data sample 2. Further suppose that private data sample 1 has the highest similarity to generated data sample 1 among the four generated data samples, and that private data sample 1 has the highest similarity to generated data sample 3 among the four generated data samples. Then, the number of private data samples that use generated data sample 1 as their most similar generated data sample is 1. The number of private data samples for generating data sample 2 with the highest similarity is 0, the number of private data samples for generating data sample 3 with the highest similarity is 1, and the number of private data samples for generating data sample 4 with the highest similarity is 0. Subsequently, these four numbers can be normalized to determine that the probability distribution of generating data sample 1 is 0.5 (i.e., 1 / 2), the probability distribution of generating data sample 2 is 0, the probability distribution of generating data sample 3 is 0.5 (i.e., 1 / 2), and the probability distribution of generating data sample 4 is 0.

[0133] On the other hand, in some embodiments, when resampling the generated data samples to be screened based on the above distribution probability to select generated data samples similar to at least one private data sample from the generated data samples to be screened, a histogram can be generated first based on the distribution probability of each generated data sample in the generated data samples to be screened. The horizontal axis of the histogram can represent each generated data sample in the generated data samples to be screened, and the vertical axis of the histogram can represent the distribution probability of each generated data sample in the generated data samples to be screened. Subsequently, the generated data samples to be screened can be resampled based on the histogram.

[0134] Resampling a histogram representing probabilities refers to generating new samples that conform to the same or similar distribution as the original histogram's probability distribution. Specifically, firstly, the cumulative distribution function (CDF) can be calculated based on the original histogram's probability distribution. The CDF represents the cumulative probability up to a certain value and can be understood as the integral of the probability density function. Then, a random number generator can be used to generate a random number between 0 and 1. Next, the generated random number can be compared with the CDF of the original histogram to find the corresponding interval—that is, the probability distribution interval of the original histogram that makes the random number fall within that interval. Finally, within the determined interval, a new sample value can be generated between the upper and lower limits of the interval using linear interpolation. The interpolation method can be chosen according to the specific situation, such as linear interpolation, multinomial interpolation, etc. By repeating the above steps, the required number of new samples can be generated.

[0135] In some embodiments, when resampling the generated data samples to be screened based on the above histogram, a differential privacy algorithm can be used to resample the generated data samples to be screened based on the histogram to provide a certain level of privacy protection.

[0136] In some embodiments, when selecting generated data samples similar to at least one private data sample from the generated data samples to be screened based on the similarity between each generated data sample in the generated data samples to be screened and each private data sample in the at least one private data sample, these similarities can be normalized to obtain normalized similarities. Then, an exponential mechanism or a singular value thresholding algorithm is used to select generated data samples similar to at least one private data sample from the generated data samples to be screened based on these normalized similarities.

[0137] The exponential mechanism is a privacy-preserving method used to select appropriate data items from datasets with different attributes. First, a "utility" value is calculated for each generated data point, reflecting its similarity to a target data point. The utility value can be calculated using Euclidean distance or other similarity metrics. Then, the exponential mechanism is used to select data points based on the probability distribution of these utility values. The exponential mechanism increases the probability of a data point being selected, making data points with higher utility values ​​more likely to be chosen.

[0138] Singular Value Thresholding (SVT) is a method based on singular value decomposition (SVD) used to reduce the dimensionality of a data matrix while retaining the most important features. In this case, the generated data points are viewed as a data matrix, where each row represents a data point and each column represents an attribute. The SVT algorithm reduces the dimensionality of the generated data matrix, retaining only the most important singular values ​​and their corresponding eigenvectors. By setting an appropriate threshold, the number of singular values ​​to retain can be determined, thus filtering out the most relevant parts of the generated data.

[0139] In some embodiments, when selecting generated data samples similar to at least one private data sample from the generated data samples to be screened, a privacy training algorithm can be used in advance to train a classification model using at least one private data sample and the public data sample. Subsequently, the generated data samples to be screened can be input into the trained classification model, and the probability value of the generated data samples to be screened belonging to the data sample category corresponding to at least one private data sample can be obtained. Thus, a preset number of generated data samples with the highest probability value can be selected from the generated data samples to be screened as generated data samples similar to at least one private data sample.

[0140] Specifically, the above-mentioned data samples to be filtered can be input into a classification model trained using a privacy training algorithm, and the probability value of the data samples to be filtered output by the classification model belonging to the data sample category corresponding to at least one of the above-mentioned private data samples can be obtained.

[0141] In practical applications, if the probability value of a public data sample is relatively high, it indicates that the classification model cannot distinguish whether the public data sample is a public data sample or a private data sample, meaning that the public data sample is quite similar to the private data sample. Therefore, the N public data samples with the highest probability values ​​can be selected from the public data samples included in the target data sample classification as public data samples similar to the aforementioned private data samples. Here, N represents a preset number; it can be a value preset by a technician or a default value, and this application does not impose any restrictions on this.

[0142] It should be noted that the above classification model can be a binary classification model used to determine whether a data sample is a public data sample or a private data sample. Alternatively, the classification model can be a multi-class classification model used to determine whether a data sample is a public data sample or a specific class of private data samples among multiple classes of private data samples.

[0143] The training samples for the aforementioned classification model can include both public and private data samples. Since private data is involved in the training of this classification model, the training process can employ privacy-preserving methods. However, in the compression process for the aforementioned large language model, this classification model is only used for data sample selection, and its training is pre-completed. Therefore, even if privacy-preserving methods are used in the training process of this classification model, it will not incur additional model performance overhead.

[0144] In the above technical solution, a first language model can be used to generate generated data samples corresponding to the description information corresponding to the private data samples. Then, generated data samples similar to the private data samples can be further selected from the generated data samples. For the second language model pre-trained based on public data samples, the pre-trained second language model can be compressed based on the selected generated data samples to obtain a compressed model corresponding to the second language model. Finally, the compressed model can be fine-tuned based on the private data samples, thereby completing the compression processing of the second language model and obtaining a compressed model more suitable for deployment.

[0145] By adopting the above approach, on the one hand, since the fine-tuning process and the compression process are interchanged in terms of execution order, the fine-tuning process can no longer be performed on the basis of the large language model, but on the basis of the compressed model corresponding to the large language model, thereby reducing the consumption of computational resources during the fine-tuning process.

[0146] On the other hand, since the compression process can now use non-privacy training instead of privacy training, with only the fine-tuning process still employing privacy training, the performance overhead of the model can be reduced. Furthermore, because the compressed model contains fewer model parameters than the large language model, fine-tuning the compressed large language model using privacy training has a smaller negative impact on model performance compared to fine-tuning the uncompressed large language model. Therefore, swapping the execution order of the fine-tuning and compression processes further helps reduce performance overhead and improve model performance.

[0147] On the other hand, by using the first major language model, corresponding generated data samples are generated based on the descriptive information corresponding to the private data samples. Furthermore, generated data samples similar to the private data samples are selected from the generated data samples. The selected generated data samples are then used to complete the compression processing for the second major language model. This can be considered as expanding the number of private data samples and increasing their diversity.

[0148] Corresponding to the embodiments of the aforementioned model compression method, this application also provides embodiments of a model compression apparatus.

[0149] Please refer to Figure 4 , Figure 4 This is a schematic diagram illustrating the structure of a device according to an exemplary embodiment of this application. At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, memory 408, and non-volatile memory 410, and may also include other necessary hardware. One or more embodiments of this application can be implemented in software, for example, the processor 402 reads the corresponding computer program from the non-volatile memory 410 into memory 408 and then runs it. Of course, besides software implementation, one or more embodiments of this application do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic modules, but can also be hardware or logic devices.

[0150] Please refer to Figure 5 , Figure 5 This is a block diagram illustrating a model compression device according to an exemplary embodiment of this application.

[0151] The above-mentioned model compression device can be applied to Figure 4 The device shown is used to implement the technical solution of this application. The model compression device may include:

[0152] The sample generation module 502 inputs the description information corresponding to at least one private data sample into the first large language model, and the first large language model uses the description information as the generation condition to generate at least one generated data sample corresponding to the description information.

[0153] The sample screening module 504 filters out generated data samples that are similar to the at least one private data sample from the at least one generated data sample;

[0154] The model compression module 506 compresses the pre-trained second language model based on the selected generated data samples to obtain a compressed model corresponding to the second language model; wherein, the second language model is pre-trained based on publicly available data samples.

[0155] The model fine-tuning module 508 performs model fine-tuning on the compressed model based on the private data samples to complete the compression processing for the second major language model.

[0156] In some embodiments, the apparatus further includes:

[0157] The pre-training module performs model pre-training on the second large language model based on the publicly available data samples.

[0158] In some embodiments, the data sample is a text data sample; the description information includes one or more of the following: text length, text type, and text description content.

[0159] In some embodiments, the following steps are performed iteratively until the number of iterations reaches a preset first threshold, in order to filter out generated data samples similar to the at least one private data sample from the at least one generated data sample:

[0160] The generated data sample is subjected to data transformation processing to obtain the transformed generated data sample, and the generated data sample and the transformed data sample are determined as the generated data sample to be screened.

[0161] Select generated data samples that are similar to the at least one private data sample from the generated data samples to be screened.

[0162] In some embodiments, the step of performing data deformation processing on the generated data sample to obtain the deformed generated data sample includes:

[0163] Determine whether the number of iterations has reached a preset second threshold;

[0164] If the number of iterations does not reach the second threshold, the generated data sample will be input into the first large language model, and the first large language model will generate data similar to the generated data sample. The data output by the first large language model will be determined as the transformed generated data sample.

[0165] If the number of iterations reaches the second threshold, some data content is deleted from the data content of the generated data sample, and the generated data sample with some data content deleted is input into the first large language model. The first large language model performs completion processing on the generated data sample with some data content deleted, and the data output by the first large language model is determined as the transformed generated data sample.

[0166] In some embodiments, the step of selecting generated data samples similar to the at least one private data sample from the generated data samples to be screened includes:

[0167] Calculate the similarity between each generated data sample in the generated data samples to be screened and each private data sample in the at least one private data sample;

[0168] Based on the similarity, generate data samples that are similar to the at least one private data sample are selected from the generated data samples to be screened.

[0169] In some embodiments, calculating the similarity between each generated data sample in the generated data samples to be screened and each private data sample in the at least one private data sample includes:

[0170] Obtain the first feature vector corresponding to each generated data sample in the generated data samples to be filtered, and obtain the second feature vector corresponding to each private data sample in the at least one private data sample;

[0171] Calculate the vector similarity between the first feature vector and the second feature vector, and use it as the similarity between each generated data sample in the data samples to be filtered and each private data sample in the at least one private data sample.

[0172] In some embodiments, the step of selecting generated data samples similar to the at least one private data sample from the generated data samples to be screened based on the similarity includes:

[0173] The similarity is converted into the probability distribution of each generated data sample in the generated data sample to be screened;

[0174] The generated data samples to be screened are resampled based on the distribution probability to select generated data samples that are similar to the at least one private data sample.

[0175] In some embodiments, converting the similarity into the probability distribution of each generated data sample in the generated data samples to be filtered includes:

[0176] For each private data sample in the at least one private data sample, determine the generated data sample with the highest similarity to the private data sample in the generated data samples to be screened, and for each generated data sample in the generated data samples to be screened, count the number of private data samples that take the generated data sample as their highest similarity generated data sample.

[0177] The quantity is normalized to obtain a normalized quantity, and the normalized quantity is determined as the distribution probability of each generated data sample in the generated data sample to be screened.

[0178] In some embodiments, the resampling process of the generated data samples to be screened based on the distribution probability, in order to screen generated data samples similar to the at least one private data sample from the generated data samples to be screened, includes:

[0179] A histogram is generated based on the distribution probability; wherein, the horizontal axis of the histogram represents each generated data sample in the generated data samples to be screened, and the vertical axis of the histogram represents the distribution probability of each generated data sample in the generated data samples to be screened.

[0180] The generated data samples to be screened are resampled based on the histogram.

[0181] In some embodiments, the resampling process of the generated data samples to be filtered based on the histogram includes:

[0182] A differential privacy algorithm is used to resample the generated data samples to be screened based on the histogram.

[0183] In some embodiments, the step of selecting generated data samples similar to the at least one private data sample from the generated data samples to be screened based on the similarity includes:

[0184] The similarity is normalized to obtain the normalized similarity.

[0185] Using an exponential mechanism or a singular value thresholding algorithm, based on the normalized similarity, generated data samples similar to the at least one private data sample are selected from the generated data samples to be screened.

[0186] In some embodiments, the step of selecting generated data samples similar to the at least one private data sample from the generated data samples to be screened includes:

[0187] The data samples to be filtered are input into the classification model; wherein the classification model adopts a privacy training algorithm and is trained based on the at least one private data sample and the public data sample;

[0188] Obtain the probability value of the generated data sample to be filtered, output by the classification model, belonging to the data sample category corresponding to the at least one private data sample;

[0189] Select the preset number of generated data samples with the highest probability value from the generated data samples to be screened, and use them as generated data samples similar to the at least one private data sample.

[0190] In some embodiments, the step of compressing the pre-trained second language model based on the selected generated data samples to obtain a compressed model corresponding to the second language model includes:

[0191] Based on the selected generated data samples, knowledge distillation training is performed on the pre-trained second language model, which serves as the teacher model, to obtain a compressed model that serves as the student model, corresponding to the second language model.

[0192] In some embodiments, fine-tuning the compressed model based on the private data sample includes:

[0193] A privacy-preserving training algorithm is used to fine-tune the compressed model based on the private data samples.

[0194] In some embodiments, the privacy training algorithm includes: a differential privacy-based training algorithm.

[0195] In some embodiments, the differential privacy-based training algorithm includes: a differential privacy-based stochastic gradient descent algorithm.

[0196] For the device embodiments, they basically correspond to the method embodiments; therefore, relevant details can be found in the descriptions of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the technical solution of this application according to actual needs.

[0197] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0198] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0199] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0200] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0201] It should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0202] The foregoing has described specific embodiments of this application. Other embodiments are within the scope of this application. In some cases, the actions or steps described in this application may be performed in a different order than those shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing are also possible or may be advantageous.

[0203] The terminology used in one or more embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this application. The singular forms “a,” “the,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. The term “and / or” refers to and includes any or all possible combinations of one or more associated listed items.

[0204] The terms "an embodiment," "some embodiments," "example," "specific example," or "one implementation," as used in one or more embodiments of this application, refer to specific features or characteristics described in connection with that embodiment, which are included in at least one embodiment of this application. Illustrative descriptions of these terms do not necessarily refer to the same embodiment. Furthermore, the described specific features or characteristics may be combined in a suitable manner in one or more embodiments of this application. In addition, different embodiments and specific features or characteristics from different embodiments may be combined without contradiction.

[0205] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of one or more embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0206] The above description is merely a preferred embodiment of one or more embodiments of this application and is not intended to limit the scope of one or more embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this application should be included within the protection scope of one or more embodiments of this application.

[0207] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

Claims

1. A model compression method, the method comprising: The descriptive information corresponding to at least one private data sample is input into the first large language model. The first large language model uses the descriptive information as a generation condition to generate at least one generated data sample corresponding to the descriptive information. The data sample is a text data sample. The descriptive information includes one or more of the following: text length, text type, and text description content. From the at least one generated data sample, select generated data samples that are similar to the at least one private data sample; Based on the selected generated data samples, the pre-trained second language model is compressed to obtain a compressed model corresponding to the second language model; wherein, the second language model is pre-trained based on publicly available data samples; Based on the private data samples, the compression model is fine-tuned to complete the compression processing for the second major language model; The process involves iteratively executing the following steps until the number of iterations reaches a preset first threshold, in order to select generated data samples similar to the at least one private data sample from the at least one generated data sample: The generated data sample is subjected to data transformation processing to obtain the transformed generated data sample, and the generated data sample and the transformed data sample are determined as the generated data sample to be screened. Select generated data samples that are similar to the at least one private data sample from the generated data samples to be screened; The step of performing data deformation processing on the generated data sample to obtain the deformed generated data sample includes: Determine whether the number of iterations has reached a preset second threshold; If the number of iterations does not reach the second threshold, the generated data sample will be input into the first large language model, and the first large language model will generate data similar to the generated data sample. The data output by the first large language model will be determined as the transformed generated data sample. If the number of iterations reaches the second threshold, some data content is deleted from the data content of the generated data sample, and the generated data sample with some data content deleted is input into the first large language model. The first large language model performs completion processing on the generated data sample with some data content deleted, and the data output by the first large language model is determined as the transformed generated data sample.

2. The method according to claim 1, further comprising: The second language model is pre-trained based on the publicly available data samples.

3. The method according to claim 1, wherein selecting generated data samples similar to the at least one private data sample from the generated data samples to be screened comprises: Calculate the similarity between each generated data sample in the generated data samples to be screened and each private data sample in the at least one private data sample; Based on the similarity, generate data samples that are similar to the at least one private data sample are selected from the generated data samples to be screened.

4. The method according to claim 3, wherein calculating the similarity between each generated data sample in the generated data samples to be screened and each private data sample in the at least one private data sample comprises: Obtain the first feature vector corresponding to each generated data sample in the generated data samples to be filtered, and obtain the second feature vector corresponding to each private data sample in the at least one private data sample; Calculate the vector similarity between the first feature vector and the second feature vector, and use it as the similarity between each generated data sample in the data samples to be filtered and each private data sample in the at least one private data sample.

5. The method according to claim 3, wherein selecting generated data samples similar to the at least one private data sample from the generated data samples to be screened based on the similarity comprises: The similarity is converted into the probability distribution of each generated data sample in the generated data sample to be screened; The generated data samples to be screened are resampled based on the distribution probability to select generated data samples that are similar to the at least one private data sample.

6. The method according to claim 5, wherein converting the similarity into the probability distribution of each generated data sample in the generated data samples to be screened comprises: For each private data sample in the at least one private data sample, determine the generated data sample with the highest similarity to the private data sample in the generated data samples to be screened, and for each generated data sample in the generated data samples to be screened, count the number of private data samples that take the generated data sample as their highest similarity generated data sample. The quantity is normalized to obtain a normalized quantity, and the normalized quantity is determined as the distribution probability of each generated data sample in the generated data sample to be screened.

7. The method according to claim 5, wherein resampling the generated data samples to be screened based on the distribution probability to screen generated data samples similar to the at least one private data sample comprises: A histogram is generated based on the distribution probability; wherein, the horizontal axis of the histogram represents each generated data sample in the generated data samples to be screened, and the vertical axis of the histogram represents the distribution probability of each generated data sample in the generated data samples to be screened. The generated data samples to be screened are resampled based on the histogram.

8. The method according to claim 7, wherein the resampling process of the generated data sample to be screened based on the histogram includes: A differential privacy algorithm is used to resample the generated data samples to be screened based on the histogram.

9. The method according to claim 3, wherein selecting generated data samples similar to the at least one private data sample from the generated data samples to be screened based on the similarity comprises: The similarity is normalized to obtain the normalized similarity. Using an exponential mechanism or a singular value thresholding algorithm, based on the normalized similarity, generated data samples similar to the at least one private data sample are selected from the generated data samples to be screened.

10. The method according to claim 1, wherein selecting generated data samples similar to the at least one private data sample from the generated data samples to be screened comprises: The data samples to be filtered are input into the classification model; wherein the classification model adopts a privacy training algorithm and is trained based on the at least one private data sample and the public data sample; Obtain the probability value of the generated data sample to be filtered, output by the classification model, belonging to the data sample category corresponding to the at least one private data sample; Select the preset number of generated data samples with the highest probability value from the generated data samples to be screened, and use them as generated data samples similar to the at least one private data sample.

11. The method according to claim 1, wherein the step of compressing the pre-trained second language model based on the selected generated data samples to obtain a compressed model corresponding to the second language model includes: Based on the selected generated data samples, knowledge distillation training is performed on the pre-trained second language model, which serves as the teacher model, to obtain a compressed model that serves as the student model, corresponding to the second language model.

12. The method according to claim 1, wherein fine-tuning the compressed model based on the private data sample comprises: A privacy-preserving training algorithm is used to fine-tune the compressed model based on the private data samples.

13. The method according to claim 10 or 12, wherein the privacy training algorithm comprises: Training algorithm based on differential privacy.

14. The method according to claim 13, wherein the differential privacy-based training algorithm comprises: A stochastic gradient descent algorithm based on differential privacy.

15. A model compression apparatus, the apparatus comprising: The sample generation module inputs the description information corresponding to at least one private data sample into the first large language model. The first large language model uses the description information as a generation condition to generate at least one generated data sample corresponding to the description information. The data sample is a text data sample. The description information includes one or more of the following: text length, text type, and text description content. The sample filtering module filters out generated data samples that are similar to the at least one private data sample from the at least one generated data sample; The model compression module compresses the pre-trained second language model based on the selected generated data samples to obtain a compressed model corresponding to the second language model; wherein, the second language model is pre-trained based on publicly available data samples. The model fine-tuning module performs model fine-tuning on the compressed model based on the private data samples to complete the compression processing for the second major language model; The process involves iteratively executing the following steps until the number of iterations reaches a preset first threshold, in order to select generated data samples similar to the at least one private data sample from the at least one generated data sample: The generated data sample is subjected to data transformation processing to obtain the transformed generated data sample, and the generated data sample and the transformed data sample are determined as the generated data sample to be screened. Select generated data samples that are similar to the at least one private data sample from the generated data samples to be screened; The step of performing data deformation processing on the generated data sample to obtain the deformed generated data sample includes: Determine whether the number of iterations has reached a preset second threshold; If the number of iterations does not reach the second threshold, the generated data sample will be input into the first large language model, and the first large language model will generate data similar to the generated data sample. The data output by the first large language model will be determined as the transformed generated data sample. If the number of iterations reaches the second threshold, some data content is deleted from the data content of the generated data sample, and the generated data sample with some data content deleted is input into the first large language model. The first large language model performs completion processing on the generated data sample with some data content deleted, and the data output by the first large language model is determined as the transformed generated data sample.

16. An electronic device comprising: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 1 to 14 by executing the executable instructions.

17. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the method as described in any one of claims 1 to 14.

Images