Language model training method, text processing method, device, equipment and medium

Abstract

The present invention provides a language model training method, a text processing method, an apparatus, a device and a medium, and relates to the technical field of natural language processing. The method comprises: based on a teacher model, predicting a first probability matrix corresponding to each data unit in a sample text; the first probability matrix includes probability values ​​of each data unit belonging to each word element in a first vocabulary; compressing the first probability matrix according to the numerical values ​​of each probability value in the first probability matrix to obtain a second probability matrix corresponding to each data unit; aligning the second vocabulary according to the word elements corresponding to each probability value in the second probability matrix to obtain a third vocabulary; and performing distillation training on a student model according to the third vocabulary and the second probability matrix to obtain a target language model, thereby reducing storage costs, improving distillation training efficiency, and enabling the target language model trained thereby to better adapt to different model architectures and text processing scenarios while maintaining high performance.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology, and in particular to a language model training method, text processing method, device, equipment and medium. Background Art

[0002] Large language models have demonstrated excellent performance and broad application prospects in text processing. However, these large language models typically have large parameter sizes and complex structures, resulting in high computational costs and other issues, making them difficult to deploy directly on resource-constrained devices. Therefore, related technologies use distillation training to transfer the capabilities of large language models to lightweight student models. This allows the trained student model to not only learn how to determine the correct sample category from labeled data at a low computational cost, but also learn inter-class relationships from the teacher model.

[0003] Existing distillation training methods for large language models typically pre-generate soft labels for all tokens (data units) in all training data using a teacher model constructed using the large language model. These soft labels are then stored locally along with the hard labels for each token in the teacher model's vocabulary for direct loading during student model training. However, with the explosive growth of vocabulary size, the number of soft labels required to be stored for each token in a single piece of training data also increases, resulting in excessively high storage costs and reduced model training efficiency. Summary of the Invention

[0004] The present invention provides a language model training method, a text processing method, an apparatus, a device and a medium to solve the defects in the prior art.

[0005] The present invention provides a language model training method, comprising:

[0006] Based on the teacher model, predicting a first probability matrix corresponding to each data unit in the sample text; the first probability matrix includes a probability value of each data unit belonging to each word in a first vocabulary, the first vocabulary being the vocabulary of the teacher model;

[0007] Compressing the first probability matrix according to the numerical values ​​of each probability value in the first probability matrix to obtain a second probability matrix corresponding to each of the data units;

[0008] Performing an alignment operation on the second vocabulary according to the word elements corresponding to the probability values ​​in the second probability matrix to obtain a third vocabulary; the second vocabulary is the vocabulary of the student model;

[0009] The student model is subjected to distillation training according to the third vocabulary and the second probability matrix to obtain a target language model.

[0010] According to a language model training method provided by the present invention, compressing the first probability matrix according to the numerical values ​​of each probability value in the first probability matrix to obtain a second probability matrix corresponding to each data unit includes:

[0011] receiving user input information, and determining a target quantity according to a compression mode in the user input information;

[0012] Sort the probability values ​​in the first probability matrix in descending order according to their numerical values;

[0013] In the first probability matrix, the probability values ​​of the target quantities with higher sorting positions are selected to construct the second probability matrix.

[0014] According to a language model training method provided by the present invention, determining the target number according to the compression mode in the user input information includes:

[0015] When the compression mode is the adaptive compression mode, cumulatively calculating each probability value in the first probability matrix according to the descending sorting results until the cumulative calculated value is greater than or equal to a preset threshold, and determining the target number based on the number of probability values ​​involved in the cumulative calculation;

[0016] When the compression mode is a fixed compression mode, the preset number is determined as the target number.

[0017] According to a language model training method provided by the present invention, performing distillation training on the student model according to the third vocabulary and the second probability matrix to obtain a target language model includes:

[0018] Constructing an empty matrix according to the number of word elements in the third vocabulary;

[0019] Filling the empty matrix with each probability value in the second probability matrix according to the first index of the word element corresponding to each probability value in the second probability matrix, thereby obtaining a first reconstructed probability matrix corresponding to each data unit; the first index is the index of the word element corresponding to each probability value in the second probability matrix in the third vocabulary;

[0020] Normalizing each probability value in the first reconstruction probability matrix to obtain a second reconstruction probability matrix corresponding to each data unit;

[0021] Based on the student model, predicting a third probability matrix corresponding to each of the data units; the third probability matrix includes probability values ​​of each of the data units belonging to each word in the third vocabulary;

[0022] The student model is subjected to distillation training according to the second reconstruction probability matrix, the third probability matrix, and the word-unit label corresponding to each of the data units to obtain the target language model.

[0023] According to a language model training method provided by the present invention, performing an alignment operation on the second vocabulary according to word units corresponding to each probability value in the second probability matrix to obtain a third vocabulary includes:

[0024] In the second probability matrix, obtaining each first probability value and each second probability value; the word element corresponding to the first probability value does not have a mapping relationship with any word element in the second vocabulary; the word element corresponding to the second probability value does have a mapping relationship with a word element in the second vocabulary;

[0025] A word segmenter based on the student model maps and generates a mapping index of each word corresponding to each first probability value;

[0026] Determining the second index of the target word element that has a mapping relationship with the word element corresponding to each second probability value as the mapping index of the word element corresponding to each second probability value; the second index is the index of the target word element in the second vocabulary;

[0027] According to the mapping index of the word-gram corresponding to each first probability value, fill the word-gram corresponding to each first probability value into an empty word list, and according to the mapping index of the word-gram corresponding to each second probability value, fill the word-gram corresponding to each second probability value into the empty word list;

[0028] According to the filling result, the third vocabulary is obtained.

[0029] According to a language model training method provided by the present invention, the method further includes:

[0030] When the second probability matrix is ​​obtained and no training instruction for the student model is received, encoding and storing each of the data units, each probability value in the second probability matrix, and a third index of a word element corresponding to each probability value in the second probability matrix to disk; the third index is an index of each probability value in the second probability matrix in the first vocabulary;

[0031] When the training instruction of the student model is received, each of the data units, each probability value in the second probability matrix, and the third index are parsed and obtained in the disk.

[0032] According to a language model training method provided by the present invention, encoding and storing each of the data units, each probability value in the second probability matrix, and a third index of a word unit corresponding to each probability value in the second probability matrix to a disk includes:

[0033] Performing quantization encoding on each probability value in the second probability matrix to obtain a first encoding result;

[0034] Performing quantization encoding on the third index to obtain a second encoding result;

[0035] Performing hash coding on each of the data units to obtain a third coding result;

[0036] The first encoding result, the second encoding result, and the third encoding result are stored in the disk in the form of a compressed file.

[0037] According to a language model training method provided by the present invention, parsing and obtaining each of the data units, each probability value in the second probability matrix, and the third index in the disk includes:

[0038] Loading the compressed file into the disk;

[0039] Performing inverse quantization decoding on the first encoding result in the compressed file to obtain each probability value in the second probability matrix;

[0040] Performing inverse quantization decoding on the second encoding result in the compressed file to obtain the third index;

[0041] A hash reverse table lookup operation is performed on the third encoding result in the compressed file to obtain each of the data units.

[0042] The present invention also provides a text processing method, comprising:

[0043] Get the text to be processed;

[0044] Based on the target language model, word unit prediction is performed on each data unit in the text to be processed to obtain a word unit prediction result corresponding to the text to be processed;

[0045] Performing text processing on the text to be processed according to the word-unit prediction result;

[0046] The text processing includes text generation, code generation, machine translation or text classification; the target language model is trained based on the language model training method as described in any of the above items.

[0047] The present invention also provides a language model training device, comprising:

[0048] A first prediction unit is configured to predict, based on the teacher model, a first probability matrix corresponding to each data unit in the sample text; the first probability matrix includes a probability value of each data unit belonging to each word in a first vocabulary, the first vocabulary being the vocabulary of the teacher model;

[0049] a compression unit, configured to compress the first probability matrix according to the numerical values ​​of the probability values ​​in the first probability matrix to obtain a second probability matrix corresponding to each of the data units;

[0050] a mapping unit, configured to perform an alignment operation on the second vocabulary according to word units corresponding to the probability values ​​in the second probability matrix to obtain a third vocabulary; the second vocabulary is a vocabulary of the student model;

[0051] A training unit is used to perform distillation training on the student model according to the third vocabulary and the second probability matrix to obtain a target language model.

[0052] The present invention also provides a text processing device, comprising:

[0053] An acquisition unit, used to acquire the text to be processed;

[0054] A second prediction unit is configured to perform word unit prediction on each data unit in the to-be-processed text based on a target language model to obtain a word unit prediction result corresponding to the to-be-processed text;

[0055] a processing unit, configured to perform text processing on the text to be processed according to the word-unit prediction result;

[0056] The text processing includes text generation, code generation, machine translation or text classification; the target language model is trained based on the language model training method as described in any of the above items.

[0057] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the language model training method described above is implemented.

[0058] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described language model training methods.

[0059] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described language model training methods.

[0060] The language model training method, text processing method, device, equipment and medium provided by the present invention predict the first probability matrix corresponding to each data unit in the sample text based on the teacher model to obtain the full probability distribution of each data unit, and adaptively compress the first probability matrix according to the size of each probability value in the first probability matrix to generate a second probability matrix with lower storage capacity, thereby effectively reducing the amount of data and storage requirements; then, using the word elements corresponding to each probability value in the second probability matrix, the vocabulary of the student model is aligned to obtain a third vocabulary, and through dynamic vocabulary mapping, the problem of inconsistency between the vocabulary of the teacher model and the student model is solved, and the problem of knowledge transfer failure caused by vocabulary differences is avoided. Finally, the student model is distilled and trained using the third vocabulary and the second probability matrix to obtain a target language model. Since distillation training is performed by sparse compression of probability matrix and vocabulary alignment, it not only significantly reduces the problem of excessive storage cost caused by the growth of vocabulary size, effectively improves the efficiency of distillation training, but also enables the target language model trained thereby to better adapt to different model architectures and text processing scenarios while maintaining high performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0062] Figure 1 This is one of the flow charts of the language model training method provided by the present invention.

[0063] Figure 2 This is the second flow chart of the language model training method provided by the present invention.

[0064] Figure 3 This is the third flow chart of the language model training method provided by the present invention.

[0065] Figure 4 This is the fourth flow chart of the language model training method provided by the present invention.

[0066] Figure 5 This is the fifth flow chart of the language model training method provided by the present invention.

[0067] Figure 6 It is a flowchart of the text processing method provided by the present invention.

[0068] Figure 7 It is a structural diagram of the language model training device provided by the present invention.

[0069] Figure 8 It is a structural diagram of the text processing device provided by the present invention.

[0070] Figure 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0071] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0072] Large language models have demonstrated excellent performance and broad application prospects in text processing. However, these large language models typically have large parameter sizes and complex structures, resulting in high computational costs and other issues, making them difficult to deploy directly on resource-constrained devices. Therefore, related technologies use distillation training to transfer the capabilities of large language models to lightweight student models. In other words, the student model training is guided by soft labels (probability matrices) generated by the teacher model. This allows the trained student model to not only learn how to determine the correct sample category from the labeled data at a low computational cost, but also learn inter-class relationships from the teacher model.

[0073] Existing distillation training methods for large language models typically pre-generate full soft labels for all tokens in all training data using a teacher model built using the large language model. The full soft labels for each token in the teacher model's vocabulary are then stored locally along with the hard labels for direct loading during student model training. The dimension of these full soft labels is directly related to the size of the vocabulary, meaning they have a dimension of 1×vocab_size, where vocab_size is the number of tokens in the teacher model's vocabulary. Therefore, as the vocabulary size explodes, the storage capacity for the full soft labels of tokens in a single piece of training data also increases. For example, when the teacher model's vocabulary contains 150,000 tokens and a single piece of data contains 3,000 tokens, stored as float32 data, storing the full soft labels for just one piece of data requires approximately 180MB of space. This results in high storage costs and low training efficiency during model training.

[0074] However, if traditional compression methods such as ZIP compression or quantization encoding (e.g., directly converting data from float32 type to float16) are used to compress the full soft labels of all tokens in all training data, not only will the compression rate be limited, but the data must still be fully loaded into memory after decompression. Furthermore, the distribution format of the decompressed data is inconsistent with the data distribution format required for distillation training, requiring additional conversion processing. This results in high storage costs and low training efficiency in model training. Directly using hard labels to replace the full soft labels can avoid the problem of high storage costs, but because hard labels cannot effectively represent inter-class relationships, the relative relationship information between categories in the student model's training data is lost, which can easily lead to poor performance of the student model trained accordingly.

[0075] To address this issue, this application proposes a language model training method. This method can be widely applied to lightweight model deployment in scenarios such as text generation, code generation, machine translation, text classification, or intelligent customer service, providing key technical support for the implementation of large-scale artificial intelligence models.

[0076] Figure 1 This is one of the flow charts of the language model training method provided by the present invention, such as Figure 1 As shown, the method includes step 110 , step 120 , step 130 and step 140 .

[0077] Step 110: predict a first probability matrix corresponding to each data unit in the sample text based on the teacher model; the first probability matrix includes probability values ​​of each data unit belonging to each word in a first vocabulary, and the first vocabulary is the vocabulary of the teacher model.

[0078] The teacher model here is a large-scale model constructed based on a large language model. The large language model (LLM) here, also known as a large model or a general pre-trained large model, refers to a natural language processing (NLP) model with a large parameter scale, and the number of model parameters and / or the complexity of the model structure exceeds a set threshold. The model is pre-trained on large-scale text data and has a high degree of semantic understanding and the ability to generate natural language. The large language model can be a bidirectional encoder representation transformer (BERT), a generative pre-trained transformer, a large language model meta-artificial intelligence, etc., which is not specifically limited in this embodiment.

[0079] The sample text here refers to the text data used to train the student model, which can be text corpus in various text processing tasks, such as news articles in text classification tasks.

[0080] Figure 2 This is the second flow chart of the language model training method provided by the present invention. Figure 3 This is the third flow chart of the language model training method provided by the present invention.

[0081] like Figure 2 and Figure 3 As shown, during the model training process, the sample text can first be input into the teacher model. The teacher model uses forward propagation reasoning to predict the probability value of each data unit (token) in the sample text belonging to all word units in the first vocabulary (hereinafter also referred to as the teacher vocabulary or the teacher model vocabulary) to obtain the first probability matrix corresponding to each data unit (hereinafter also referred to as the full soft label), that is, the full soft label that can be used for student model training. In this process, the sample text can be divided into multiple tokens using the teacher model's word segmenter, and each token processed by word segmentation is forward propagated to obtain the score of each token unit belonging to each word unit in the first vocabulary. Then, a normalization function (such as the softmax function, etc.) is used to normalize the score of each token unit belonging to each word unit in the first vocabulary to obtain the probability value of each token belonging to each word unit in the first vocabulary, thereby forming the first probability matrix corresponding to each data unit.

[0082] Among them, the dimension of the first probability matrix is Among them is the number of word-grams contained in the first vocabulary.

[0083] For example, the reasoning steps of the teacher model can be implemented by the following code:

[0084] {def generate_soft_labels(teacher_model, input_text):

[0085] tokenized_input = teacher_tokenizer(input_text) # word segmentation processing

[0086] logits = teacher_model( tokenized_input).logits #forward propagation

[0087] probs = torch.softmax(logits, dim=-1) #normalization

[0088] return probs}.

[0089] Step 120: compress the first probability matrix according to the numerical values ​​of the probability values ​​in the first probability matrix to obtain a second probability matrix corresponding to each of the data units.

[0090] Optionally, after obtaining the first probability matrix, the first probability matrix can be compressed based on the numerical size of each probability value in the first probability matrix to retain some probability values ​​in the first probability sequence and their corresponding word element information (such as the index of the word element in the first vocabulary), thereby reducing its dimension and further reducing its storage space.

[0091] It should be noted that during the compression process, the second probability sequence may be constructed by selecting probability values ​​in the first probability matrix that are greater than the target value and their corresponding word element information; or the probability values ​​in the first probability sequence may be sorted in descending order, and a target number of probability values ​​and their corresponding word element information that are ranked at the top may be selected to form the second probability sequence, etc. This embodiment does not specifically limit this. The target value and target number here may be fixed according to actual needs, or adaptively determined based on the distribution characteristics of the probability values ​​in the first probability sequence, etc. This embodiment does not specifically limit this.

[0092] Step 130 , performing an alignment operation on the second vocabulary according to the word units corresponding to the probability values ​​in the second probability matrix to obtain a third vocabulary; the second vocabulary is the vocabulary of the student model.

[0093] The student model here is a lightweight network model, which can be a network model that uses the same model architecture as the teacher model, but has a much smaller model parameter scale than the teacher model, or a network model that uses a different model architecture from the teacher model, and has a much smaller model parameter scale than the teacher model, etc. This embodiment does not make specific restrictions on this.

[0094] It should be noted that during the knowledge distillation process, when the vocabulary of the teacher model is inconsistent with that of the student model, the index of the word element in the first vocabulary corresponding to each probability value in the full soft label generated by the teacher model will not be directly aligned with the vocabulary of the student model. Existing solutions force vocabulary alignment by forcing a unified vocabulary, such as unifying the word segmenter, sharing the embedding layer between the student model and the teacher model, or discarding the probability values ​​of tokens in the full soft label generated by the teacher model that do not appear in the student vocabulary, retaining only the probability values ​​of the shared word elements. However, this method of forcing a unified vocabulary cannot effectively solve the index misalignment problem caused by differences in sub-word combinations, which in turn causes the student model to be unable to effectively utilize the soft labels of the teacher model for effective learning, which in turn causes the failure of knowledge transfer. Experiments have shown that this method of forcing a unified vocabulary will increase the perplexity of the student model in generation tasks by 18%-25%, greatly limiting the application of distillation across model architectures or cross-language scenarios.

[0095] To overcome the above problems, this embodiment uses the probability distribution characteristics of the full set of soft labels to dynamically remap the vocabulary of the student model to align it. This solves the probability distribution misalignment caused by word segmentation differences and supports lossless distillation between heterogeneous models. The specific implementation steps are as follows:

[0096] According to the word elements corresponding to each probability value in the second probability matrix and the mapping relationship between the word elements of the first vocabulary and the second vocabulary, the second vocabulary is aligned to obtain a third vocabulary. For example, for any probability value in the second probability matrix, if the word element corresponding to the probability value is mapped to a word element in the second vocabulary, the index of the word element in the second vocabulary that has a mapping relationship with the word element corresponding to the probability value is determined, and the mapping index of the word element corresponding to the probability value is added to the initialized empty vocabulary according to the mapping index of the word element corresponding to the probability value; if the word element corresponding to the probability value is not mapped to any word element in the second vocabulary, the word segmenter of the student model is used to generate an index of an unknown word element for the word element corresponding to the probability value to determine the mapping index of the word element corresponding to the probability value, and the word element corresponding to the probability value is added to the initialized empty vocabulary as an unknown word element according to the index of the unknown word element. Thus, each probability value in the second probability matrix is ​​traversed according to this step to obtain the third vocabulary.

[0097] Step 140: Perform distillation training on the student model based on the third vocabulary and the second probability matrix to obtain a target language model.

[0098] The target language model here is a student model that has been trained through distillation. It can learn the knowledge of the teacher model as much as possible under a smaller model scale, so as to be able to effectively predict word units of the input text, and then use the word unit prediction results obtained to complete the corresponding text processing tasks with high performance, such as text generation, code generation, machine translation or text classification, etc. This embodiment does not make specific limitations on this.

[0099] Optionally, after obtaining the third vocabulary, the sample text can be input into the student model, and the student model predicts the probability value of each data unit in the sample text belonging to each word in the third vocabulary through forward propagation reasoning to form a third probability matrix corresponding to each data unit.

[0100] And according to the structure and quantity of the third vocabulary, the second probability matrix is ​​reconstructed to obtain a reconstructed probability matrix associated with the third probability matrix; for example, according to the structure and quantity of the third vocabulary, the position of each probability value in the second probability matrix is ​​adjusted, and zero-padded at necessary positions to align it with the third vocabulary, thereby obtaining a reconstructed probability matrix associated with the third probability sequence matrix, so as to ensure that the soft label knowledge of the teacher model can match the output of the student model.

[0101] Then, the third probability matrix and the reconstruction probability matrix are used to calculate the difference between the output of the student model and the output of the teacher model. For example, the KL divergence loss can be used to measure the difference between the output of the student model and the output of the teacher model to obtain the first loss; the difference between the third probability matrix and the word-unit label (that is, hard label) corresponding to each data unit is used, such as the cross entropy loss, to measure the difference between the output of the student model and the hard label to obtain the second loss.

[0102] Then, the first loss and the second loss are combined to update the student model through the back-propagation algorithm, so as to minimize the first loss so that the learning output of the student model is as close as possible to the output of the teacher model, thereby effectively transferring the knowledge of the teacher model to the student model, and minimizing the second loss so that the student model can learn the ability to fit the real label. In this way, the student model can learn the soft label knowledge of the teacher model and retain a good fit to the hard label during the distillation process, so as to obtain a target language model that can process text quickly and with high performance.

[0103] The method provided in this embodiment predicts the first probability matrix corresponding to each data unit in the sample text based on the teacher model to obtain the full probability distribution of each data unit, and adaptively compresses the first probability matrix in a sparse manner according to the size of each probability value in the first probability matrix to generate a second probability matrix with lower storage capacity, effectively reducing the amount of data and storage requirements; then, using the word elements corresponding to each probability value in the second probability matrix, the vocabulary of the student model is aligned to obtain a third vocabulary, and through dynamic vocabulary mapping, the problem of inconsistency between the vocabulary of the teacher model and the student model is solved, and the problem of knowledge transfer failure caused by vocabulary differences is avoided. Finally, the student model is distilled and trained using the third vocabulary and the second probability matrix to obtain a target language model. Since distillation training is performed by sparse compression of probability matrices and vocabulary alignment, it not only significantly reduces the problem of excessive storage cost caused by the growth of vocabulary size and effectively improves the efficiency of distillation training, but also enables the target language model trained thereby to better adapt to different model architectures and text processing scenarios while maintaining high performance.

[0104] Figure 4 This is the fourth flow chart of the language model training method provided by the present invention; Figure 4 As shown, the method includes step 410 , step 420 , step 430 and step 440 .

[0105] Step 410: predict a first probability matrix corresponding to each data unit in the sample text based on the teacher model; the first probability matrix includes probability values ​​of each data unit belonging to each word in a first vocabulary, and the first vocabulary is the vocabulary of the teacher model.

[0106] Optionally, during the model training process, the sample text can be first input into the teacher model, and the teacher model uses forward propagation reasoning to predict the probability value of each token in the sample text belonging to all the word elements in the first vocabulary to obtain the first probability matrix corresponding to each data unit, that is, to obtain the full amount of soft labels that can be used for student model training. The specific implementation steps can be found in step 110 and will not be repeated here.

[0107] Step 420: Receive user input information, determine the target quantity based on the compression mode in the user input information, sort the probability values ​​in the first probability matrix in descending order according to the numerical value, and select the probability value of the target quantity with a higher sorting position in the first probability matrix to construct the second probability matrix.

[0108] The user input information here is the information input by the user in the front-end interface to select the compression mode for the full soft labels output by the teacher model. The information can be input through the command line interface, graphical interface input, touch input, drop-down selection input, voice input, gesture input and other input forms. This embodiment does not make specific limitations on this.

[0109] Optionally, after receiving user input information, a compression mode may be parsed from the user input information, and a corresponding number of probability values ​​to be retained may be determined based on the compression mode, thereby obtaining the target number. Compression modes herein include fixed compression modes and adaptive compression modes, and different compression modes correspond to different numbers of probability values ​​to be retained. For example, the number of probability values ​​to be retained for a fixed compression mode is a fixed number, while the number of probability values ​​to be retained for an adaptive compression mode is a number dynamically determined based on probability distribution characteristics.

[0110] In a possible implementation, determining the target quantity according to the compression mode in the user input information includes:

[0111] When the compression mode is the adaptive compression mode, cumulatively calculating each probability value in the first probability matrix according to the descending sorting results until the cumulative calculated value is greater than or equal to a preset threshold, and determining the target number based on the number of probability values ​​involved in the cumulative calculation;

[0112] When the compression mode is a fixed compression mode, the preset number is determined as the target number.

[0113] like Figure 2 and Figure 3 As shown, when the compression mode is an adaptive compression mode, the probability values ​​in the first probability matrix can be sorted in descending order according to the numerical size, and the probability values ​​in the first probability matrix are accumulated and calculated in turn according to the descending sorting results until the accumulated value is greater than or equal to the preset threshold, and the cumulative calculation step is stopped; then, the target number K is dynamically determined based on the number of probability values ​​participating in the cumulative calculation to balance the information retention rate and compression efficiency.

[0114] The preset threshold here can be defined based on actual needs, such as 0.95 or 0.9. It can also be dynamically determined based on the performance requirements of the model. For example, if the student model needs to have high accuracy after distillation training, the preset threshold can be set to a higher value, such as 0.9 or 0.95, to retain more high-probability word information. If higher requirements are placed on the model's inference speed, the preset threshold can be appropriately lowered to reduce the amount of information retained, thereby reducing computational complexity. This embodiment does not specifically limit the method for determining the preset threshold.

[0115] When the compression mode is fixed, the number of retained probability values ​​(i.e., the preset number) directly specified according to the task requirements (such as high-precision distillation requirements or low storage overhead requirements) is determined as the target number K. For example, the target number is set to 50, etc., which is not specifically limited in this embodiment.

[0116] In summary, during the compression process, by flexibly determining the target number according to the compression mode in the user input information to compress the soft labels, it can not only improve the user experience and enhance the flexibility of soft label compression, but also achieve a flexible balance between model accuracy and training efficiency while reducing storage and computing costs.

[0117] After obtaining the target number K through the above steps, the probability values ​​in the first probability matrix can be sorted in descending order according to the numerical size, and according to the descending sorting results, the top K probability values ​​and their corresponding third indexes in the first probability matrix are selected to construct a second probability matrix. In this way, the high probability values ​​and their indexes are retained through the structured compression method of Top-K probability value truncation, so as to reduce the storage capacity of the soft label from O(vocab_size) to O(K), and reduce the storage space of a single data record by more than 99%, thereby significantly reducing storage costs, improving data transmission efficiency, accelerating model training and inference processes, and improving model deployment efficiency and performance.

[0118] For example, the compression step here can be implemented by the following code:

[0119] {def compress_probs(probs: Tensor, mode: str, param: float) ->Tuple[Tensor, Tensor]:

[0120] if mode == "fixed_k":

[0121] k = int(param)

[0122] top_values, top_indices = torch.topk(probs, k)

[0123] elif mode == "adaptive_threshold":

[0124] sorted_probs, _ = torch.sort(probs, descending=True)

[0125] cum_sum = torch.cumsum(sorted_probs, dim=0)

[0126] k = torch.argmax(cum_sum>= param) + 1

[0127] top_values ​​= sorted_probs[:k]

[0128] top_indices = torch.argsort(probs, descending=True)[:k]

[0129] return top_values, top_indices}.

[0130] Step 430 , performing an alignment operation on the second vocabulary according to the word units corresponding to the probability values ​​in the second probability matrix to obtain a third vocabulary; the second vocabulary is the vocabulary of the student model.

[0131] Optionally, after obtaining the second probability matrix, the second vocabulary can be aligned according to the word elements corresponding to each probability value in the second probability matrix and the mapping relationship between the word elements between the first vocabulary and the second vocabulary to obtain a third vocabulary. The specific implementation steps can be found in step 130 and will not be repeated here.

[0132] Step 440: Perform distillation training on the student model based on the third vocabulary and the second probability matrix to obtain a target language model.

[0133] Optionally, after obtaining the third vocabulary, the sample text can be input into the student model, and the student model predicts the probability value of each data unit in the sample text belonging to each word in the third vocabulary through forward propagation reasoning to form a third probability matrix corresponding to each data unit; and according to the structure and quantity of the third vocabulary, the second probability matrix is ​​reconstructed to obtain a reconstructed probability matrix associated with the third probability matrix; then, the difference between the output of the student model and the output of the teacher model is calculated using the third probability matrix and the reconstructed probability matrix to obtain a first loss; the difference between the third probability matrix and the word label corresponding to each data unit is used to obtain a second loss; then, the student model is updated by the backpropagation algorithm in combination with the first loss and the second loss to obtain a target language model that can perform text processing quickly and with high performance. The specific implementation steps can be found in step 140 and will not be repeated here.

[0134] The method provided in this embodiment performs efficient, cross-framework, and cross-vocabulary model distillation training through sparse compression probability matrix and dynamic vocabulary mapping. It not only significantly reduces the problem of excessive storage costs caused by the growth of vocabulary size, effectively improves the model training efficiency, but also enables the target language model trained thereby to better adapt to different model architectures and text processing scenarios while maintaining high performance.

[0135] In some embodiments, performing distillation training on the student model according to the third vocabulary and the second probability matrix to obtain a target language model includes:

[0136] Constructing an empty matrix according to the number of word elements in the third vocabulary;

[0137] Filling the empty matrix with each probability value in the second probability matrix according to the first index of the word element corresponding to each probability value in the second probability matrix, thereby obtaining a first reconstructed probability matrix corresponding to each data unit; the first index is the index of the word element corresponding to each probability value in the second probability matrix in the third vocabulary;

[0138] Normalizing each probability value in the first reconstruction probability matrix to obtain a second reconstruction probability matrix corresponding to each data unit;

[0139] Based on the student model, predicting a third probability matrix corresponding to each of the data units; the third probability matrix includes probability values ​​of each of the data units belonging to each word in the third vocabulary;

[0140] The student model is subjected to distillation training according to the second reconstruction probability matrix, the third probability matrix, and the word-unit label corresponding to each of the data units to obtain the target language model.

[0141] like Figure 2 and Figure 3 As shown, when training the student model, in order to ensure the integrity of the distillation effect, the second probability matrix is ​​first reconstructed. Specifically, according to the number of words in the third vocabulary, an empty matrix is ​​constructed, which can be ,in, is the number of word units in the third vocabulary. Then, based on the index of the word unit corresponding to each probability value in the second probability matrix in the third vocabulary (i.e., the first index), each probability value in the second probability matrix is ​​filled into the empty matrix to obtain the first reconstructed probability matrix corresponding to each data unit, so that the first reconstructed probability matrix reconstructed thereby is compatible with the output of the student model. At the same time, the first reconstructed probability matrix is ​​normalized (such as rescaling the softmax function) to ensure that the sum of all probability values ​​in the second reconstructed probability matrix constructed thereby is 1, avoiding deviations in loss calculations due to truncation of probability values. Experiments have verified that this approach can reduce distillation loss by 32%.

[0142] Exemplarily, the reconstruction step of the second probability matrix can be implemented based on the following code:

[0143] {def reconstruct_probs(top_values, student_indices, vocab_S):

[0144] probs = torch.zeros(vocab_S) # Empty matrix construction

[0145] probs.scatter_(0, student_indices, top_values) # probability value filling

[0146] if config.normalize:

[0147] probs = probs / probs.sum() # Normalization

[0148] return probs}.

[0149] In addition, the sample text is input into the student model, and the student model predicts the probability value of each data unit in the sample text belonging to all the word units in the third vocabulary through forward propagation reasoning to form a third probability matrix corresponding to each data unit.

[0150] Then, the third probability matrix and the second reconstruction probability matrix are used to calculate the difference between the output of the student model and the output of the teacher model. For example, the KL divergence loss can be used to measure the difference between the output of the student model and the output of the teacher model, and the first loss (also called soft label loss) is obtained, so that the learning output of the student model can be minimized as much as possible by minimizing the first loss, thereby effectively transferring the knowledge of the teacher model to the student model; the difference between the third probability matrix and the word-unit label corresponding to each data unit is used, such as the cross entropy loss, to measure the difference between the output of the student model and the hard label, and the second loss (also called hard label loss) is obtained, so that the student model can learn the ability to fit the real label by minimizing the second loss.

[0151] Then, the first loss and the second loss are fused (such as direct addition or weighted addition, etc.) to obtain a mixed loss, and the student model is updated by the backpropagation algorithm through the mixed loss, so as to enable the student model to learn the soft label knowledge of the teacher model while retaining a good fit to the hard labels during the distillation process, and finally obtain a target language model that can process text quickly and with high performance.

[0152] For example, the calculation steps of the hybrid loss can be implemented based on the following code:

[0153] {def distillation_loss(student_logits, hard_labels, soft_probs, alpha=0.7):

[0154] loss_ce = F.cross_entropy(student_logits, hard_labels) #hard label loss

[0155] log_probs = F.log_softmax(student_logits, dim=-1) #soft label loss

[0156] loss_kl = F.kl_div(log_probs, soft_probs, reduction="batchmean")

[0157] total_loss = alpha loss_kl + (1 - alpha) loss_ce # weighted sum

[0158] return total_loss}.

[0159] The method provided in this embodiment reconstructs and normalizes the compressed second probability matrix so that the reconstructed second probability matrix is ​​compatible with the output of the student model, avoiding the loss calculation deviation caused by probability truncation, ensuring that the student model can effectively learn the soft label knowledge of the teacher model, and through the fusion of multiple losses, the student model is enabled to retain the ability to fit the hard labels while learning the teacher model knowledge, effectively realizing the training of a high-performance target language model.

[0160] In some embodiments, performing an alignment operation on the second vocabulary according to word-units corresponding to each probability value in the second probability matrix to obtain a third vocabulary includes:

[0161] In the second probability matrix, obtaining each first probability value and each second probability value; the word element corresponding to the first probability value does not have a mapping relationship with any word element in the second vocabulary; the word element corresponding to the second probability value does have a mapping relationship with a word element in the second vocabulary;

[0162] A word segmenter based on the student model maps and generates a mapping index of each word corresponding to each first probability value;

[0163] Determining the second index of the target word element that has a mapping relationship with the word element corresponding to each second probability value as the mapping index of the word element corresponding to each second probability value; the second index is the index of the target word element in the second vocabulary;

[0164] According to the mapping index of the word-gram corresponding to each first probability value, fill the word-gram corresponding to each first probability value into an empty word list, and according to the mapping index of the word-gram corresponding to each second probability value, fill the word-gram corresponding to each second probability value into the empty word list;

[0165] According to the filling result, the third vocabulary is obtained.

[0166] like Figure 3As shown, when performing vocabulary alignment, it can be determined whether the word element corresponding to each probability value in the second probability matrix has a mapping relationship with each word element in the second vocabulary. For any probability value in the second probability matrix, if the word element corresponding to the probability value is mapped to a word element in the second vocabulary, it is used as the second probability value, and according to the second index of the target word element in the second vocabulary that has a mapping relationship with the word element corresponding to each second probability value, the mapping index of the word element corresponding to the probability value is obtained, and according to the mapping index of the word element corresponding to the probability value, it is added to the initialized empty vocabulary; if the word element corresponding to the probability value does not have a mapping with all the words in the second vocabulary, it is used as the first probability value, and the word segmenter of the student model is used to generate an index of an unknown word element (UNK) for it, and the index of the unknown word element is used as the mapping index of the word element corresponding to the probability value, and according to the index of the unknown word element, the word element corresponding to the probability value is added to the initialized empty vocabulary in the form of an unknown word element. Thus, each probability value in the second probability matrix is ​​traversed according to this step to obtain the third vocabulary.

[0167] For example, the vocabulary alignment operation here can be implemented by the following code:

[0168] {def map_to_student_vocab(raw_tokens, student_tokenizer):

[0169] student_indices = [] # Initialize an empty vocabulary

[0170] for tok in raw_tokens:

[0171] if tok in student_tokenizer.get_vocab():

[0172] idx = student_tokenizer.convert_tokens_to_ids(tok)

[0173] student_indices.append(idx)

[0174] else:

[0175] if config.oov_handle == "discard":

[0176] continue

[0177] else:

[0178] student_indices.append(student_tokenizer.unk_token_id)

[0179] return torch.tensor(student_indices)}.

[0180] In summary, the method provided in this embodiment performs a dynamic mapping alignment operation on the second vocabulary according to the mapping relationship between each word in each probability value in the compressed second probability matrix and each word in the second vocabulary, which effectively alleviates the problem of knowledge transfer failure caused by vocabulary differences and improves the success rate of cross-model distillation. Simulation verification shows that this method can increase the success rate of cross-model distillation to more than 95%.

[0181] Figure 5 This is the fifth flow chart of the language model training method provided by the present invention; Figure 5 As shown, the method includes step 510 , step 520 , step 530 , step 540 and step 550 .

[0182] Step 510: Based on the teacher model, predict a first probability matrix corresponding to each data unit in the sample text; the first probability matrix includes the probability value of each data unit belonging to each word in the first vocabulary, and the first vocabulary is the vocabulary of the teacher model.

[0183] Optionally, during the model training process, the sample text can be first input into the teacher model, and the teacher model uses forward propagation reasoning to predict the probability value of each token in the sample text belonging to all the word elements in the first vocabulary to obtain the first probability matrix corresponding to each data unit, that is, to obtain the full amount of soft labels that can be used for student model training. The specific implementation steps can be found in step 110 and will not be repeated here.

[0184] Step 520: Compress the first probability matrix according to the numerical values ​​of the probability values ​​in the first probability matrix to obtain a second probability matrix corresponding to each of the data units.

[0185] Optionally, after obtaining the first probability matrix, the first probability matrix can be compressed based on the numerical size of each probability value in the first probability matrix to retain some probability values ​​in the first probability sequence and their corresponding word element information, thereby reducing its dimension and further reducing its storage space. The specific implementation steps can be found in step 120 and will not be repeated here.

[0186] Step 530, when the second probability matrix is ​​obtained and the training instruction of the student model is not received, each of the data units, each probability value in the second probability matrix, and the third index of the word corresponding to each probability value in the second probability matrix are encoded and stored on the disk; the third index is the index of each probability value in the second probability matrix in the first vocabulary; when the training instruction of the student model is received, each of the data units, each probability value in the second probability matrix, and the third index are parsed and obtained on the disk.

[0187] It should be noted that in the existing technology, real-time inference is usually used for distillation training, that is, the teacher model is called to perform instant inference on the sample data during each forward propagation to generate soft labels (probability distribution), which are directly used for student model training. This training method relies on the deep coupling of the teacher model and the student model training framework, and requires synchronous processing of model inference and gradient return, requiring the teacher model and the student model to share computing resources. However, the teacher model and the student model are often based on different frameworks, resulting in frequent switching of training equipment between inference and training tasks, resulting in resource scheduling conflicts and a significant decrease in resource utilization. In addition, the data transmission overhead between heterogeneous frameworks significantly reduces training efficiency.

[0188] To address this issue, this embodiment designs a soft label pre-compression and decompression training mechanism to completely decouple teacher model reasoning from student model training. This effectively supports cross-framework hybrid training, eliminates resource competition, and improves training efficiency. The specific implementation steps are as follows:

[0189] like Figure 2 and Figure 3 As shown, when the second probability matrix is ​​obtained and the training instruction of the student model is not received, it means that the training step of the representation learning model is not triggered. At this time, each data unit, each probability value in the second probability matrix, and the third index of the word element corresponding to each probability value in the second probability matrix can be encoded first and then stored offline to the disk, thereby further reducing the storage cost.

[0190] The encoding herein may be encoding each data unit, each probability value in the second probability matrix, and at least one of the third indices of the word units corresponding to each probability value in the second probability matrix. The encoding herein may be hash encoding, quantization encoding, or the like, which is not specifically limited in this embodiment.

[0191] It should be noted that when encoding data units, probability values ​​and the third index, a unified encoding method can be used, or different encoding methods can be used for different data. For example, hash encoding can be used for data units, and quantization encoding can be used for probability values ​​and the third index.

[0192] When a training instruction for the student model is received, the representation learning model training step is triggered. At this time, the encoded data on the disk can be read and decoded accordingly using a decoding method corresponding to the encoding method to obtain each data unit, each probability value in the second probability matrix, and the third index of the word element corresponding to each probability value in the second probability matrix, thereby assisting the student model in performing distillation training based on the data obtained through parsing. For example, if hash coding is used for the data unit during the encoding process, the data unit can be decoded through a hash reverse table lookup operation or the inverse process of the hash algorithm; if quantization coding is used for the probability value and the third index, the probability value and the third index can be decoded through a reverse quantization operation.

[0193] In one possible implementation, encoding and storing each of the data units, each probability value in the second probability matrix, and a third index of a word corresponding to each probability value in the second probability matrix to a disk includes:

[0194] Performing quantization encoding on each probability value in the second probability matrix to obtain a first encoding result;

[0195] Performing quantization encoding on the third index to obtain a second encoding result;

[0196] Performing hash coding on each of the data units to obtain a third coding result;

[0197] The first encoding result, the second encoding result, and the third encoding result are stored in the disk in the form of a compressed file.

[0198] like Figure 3 As shown, during the encoding and storage process, each probability value in the second probability matrix can be quantized and encoded, such as quantizing and encoding each probability value in the second probability matrix from float32 type to float16 type, etc., so as to reduce the storage space requirement of each probability value in the second probability matrix, thereby significantly reducing the storage cost.

[0199] Furthermore, the third index of the word unit corresponding to each probability value in the second probability matrix may be quantized and encoded. For example, differential quantization encoding may be performed on the third index of the word unit corresponding to each probability value of the float16 data type to map the third index to a smaller quantization interval, thereby significantly reducing the storage space required for the third index while maintaining data integrity. Furthermore, hash encoding may be performed on each data unit to encode each data unit into a binary vector, thereby reducing the storage space required for each data unit.

[0200] For example, the encoding step here can be specifically implemented by the following code:

[0201] { def encode_for_storage(top_values, top_indices, raw_tokens):

[0202] values_quantized = top_values.to(torch.float16) # Quantization encoding

[0203] indices_quantized = top_indices.to(torch.int32) # Quantization encoding

[0204] token_hashes = [hash(tok) % 2 32 for tok in raw_tokens] #Hash encoding

[0205] return (values_quantized, indices_quantized, token_hashes)}.

[0206] After completing the multi-level encoding of the compressed data, the encoded data (i.e., the first, second, and third encoding results) can be written to disk as compressed files in formats such as HDF5 / Parquet. This optimized storage of the compressed data through multi-level encoding further reduces storage space requirements and costs, improving model training efficiency.

[0207] Accordingly, in a possible implementation, parsing and obtaining each of the data units, each probability value in the second probability matrix, and the third index in the disk includes:

[0208] Loading the compressed file into the disk;

[0209] Performing inverse quantization decoding on the first encoding result in the compressed file to obtain each probability value in the second probability matrix;

[0210] Performing inverse quantization decoding on the second encoding result in the compressed file to obtain the third index;

[0211] A hash reverse table lookup operation is performed on the third encoding result in the compressed file to obtain each of the data units.

[0212] like Figure 3As shown, when the training instruction of the student model is received, the compressed file can be read from the disk, and the decoding method corresponding to the encoding method of the probability value is used to perform inverse quantization decoding on the first encoding result in the compressed file, such as inverse quantization decoding of the first encoding result from float16 type to float32 type to obtain each probability value in the second probability matrix; in addition, the decoding method corresponding to the encoding method of the third index can be used to perform inverse quantization decoding on the second encoding result in the compressed file, such as decoding the second encoding result from differential encoding to an absolute index of float16 type to obtain the third index corresponding to each probability value in the second probability matrix. In addition, a hash reverse table lookup operation can be performed on the third encoding result in the compressed file to obtain each data unit. Therefore, by reverse decoding the compressed file stored in the disk through this step, the data can be recovered from the compressed file efficiently and completely, and the probability values ​​and their corresponding third indexes and data units in the complete second probability sequence matrix can be obtained, so as to effectively transfer the knowledge of the teacher model to the student model, improve the performance of the student model, and at the same time realize the decoupling of the teacher model reasoning step and the learning model training step, eliminate resource competition, thereby reducing memory requirements, improving training efficiency, and enhancing the training effect of the student model.

[0213] For example, the decoding step here can be implemented by the following code:

[0214] { values ​​= loaded_values.to(torch.float32)#dequantization

[0215] indices = loaded_indices.to(torch.int64) #dequantization

[0216] raw_tokens = decode_hashes(loaded_hashes) # Hash reverse lookup table}.

[0217] Step 540: Perform an alignment operation on the second vocabulary according to the word units corresponding to the probability values ​​in the second probability matrix to obtain a third vocabulary; the second vocabulary is the vocabulary of the student model.

[0218] Optionally, an alignment operation is performed on the second vocabulary according to word units corresponding to each probability value in the second probability matrix to obtain a third vocabulary; the second vocabulary is the vocabulary of the student model.

[0219] Optionally, after obtaining the second probability matrix, the second vocabulary can be aligned according to the word elements corresponding to each probability value in the second probability matrix and the mapping relationship between the word elements between the first vocabulary and the second vocabulary to obtain a third vocabulary. The specific implementation steps can be found in step 130 and will not be repeated here.

[0220] Step 550: Perform distillation training on the student model based on the third vocabulary and the second probability matrix to obtain a target language model.

[0221] Optionally, after obtaining the third vocabulary, the sample text can be input into the student model, and the student model predicts the probability value of each data unit in the sample text belonging to each word in the third vocabulary through forward propagation reasoning to form a third probability matrix corresponding to each data unit; and according to the structure and quantity of the third vocabulary, the second probability matrix is ​​reconstructed to obtain a reconstructed probability matrix associated with the third probability matrix; then, the difference between the output of the student model and the output of the teacher model is calculated using the third probability matrix and the reconstructed probability matrix to obtain a first loss; the difference between the third probability matrix and the word label corresponding to each data unit is used to obtain a second loss; then, the student model is updated by the backpropagation algorithm in combination with the first loss and the second loss to obtain a target language model that can perform text processing quickly and with high performance. The specific implementation steps can be found in step 140 and will not be repeated here.

[0222] The method provided in this embodiment first predicts the first probability matrix of each data unit in the sample text based on the teacher model, and compresses it according to the size of the probability value to obtain the second probability matrix; when the student model training instruction is not received, the data unit, the probability value in the second probability matrix and the third index code are stored on the disk to reduce the storage space requirement; when the student model training instruction is received, the encoded data is read from the disk and the corresponding decoding method is used to restore the data, and then the second vocabulary is aligned according to the third index to obtain the third vocabulary, and finally the student is distilled and trained in combination with the third vocabulary and the second probability matrix to obtain the target language model. The entire training process not only decouples the teacher model reasoning from the student model training, but also reduces memory requirements, improves training efficiency, and enables the target language model trained thereby to better adapt to different model architectures and language processing scenarios while maintaining high performance.

[0223] The effectiveness of the method provided in this embodiment is specifically described below with experimental comparison results.

[0224] To ensure fairness during the experimental comparison, a comprehensive comparison was conducted on text generation tasks between the student model (also known as the target language model) trained using the method provided by this embodiment and the student model trained using traditional hard-label methods, maintaining the same compression ratio. The experimental results showed that, at the same compression ratio, the student model trained using the method provided by this embodiment achieved a 12.7% improvement in the bilingual evaluation replacement metric in the text generation task compared to the student model trained using traditional hard-label methods. This demonstrates that the student model trained using the method provided by this embodiment is capable of generating higher-quality text content.

[0225] At the same time, in terms of storage cost, the storage cost of the method provided by this embodiment can be reduced to 1 / 100 of the traditional hard label training solution.

[0226] As can be seen from the above, compared with the existing technology, the method provided in this embodiment not only improves model performance, but also significantly reduces the space required for model storage and improves the efficiency and applicability of distillation training.

[0227] Figure 6 It is a flowchart of the text processing method provided by the present invention, such as Figure 6 As shown, the method includes step 610 , step 620 and step 630 .

[0228] Step 610, obtaining the text to be processed;

[0229] Step 620: performing word-gram prediction on each data unit in the text to be processed based on the target language model to obtain a word-gram prediction result corresponding to the text to be processed;

[0230] Step 630: performing text processing on the text to be processed according to the word-unit prediction result;

[0231] The text processing includes text generation, code generation, machine translation or text classification; the target language model is trained based on a language model training method.

[0232] The text to be processed here can be the text to be processed, which can be the text corpus required for various text processing tasks, such as news articles in text classification tasks. The text to be processed here can be obtained by reading a file or user input.

[0233] Optionally, when processing text, you can first use Figure 1The training method shown is used to train a target language model, and then use the target language model to analyze each data unit in the text, predict the word unit corresponding to each data unit, and obtain the word unit prediction result; then, the word unit prediction result is used to perform text processing tasks such as text generation, code generation, machine translation or text classification on the text to be processed, so as to realize intelligent processing of the text to be processed.

[0234] For example, in text generation tasks, new text content can be gradually generated based on lemma prediction results. In code generation tasks, lemma prediction results can be used to generate grammatically and logically correct code snippets. In machine translation tasks, lemma prediction results in the source language can be converted into lemmas in the target language to generate translated text. In text classification tasks, lemma prediction results can be used to determine the category of the text, such as news or sentiment analysis.

[0235] The method provided in this embodiment performs word prediction on a lightweight target language model trained by distillation using a sparse compressed probability matrix and vocabulary alignment, and implements intelligent processing of tasks such as text generation, code generation, machine translation, or text classification based on the prediction results, thereby significantly improving the accuracy, efficiency, and adaptability of text processing.

[0236] The language model training device provided by the present invention is described below. The language model training device described below and the language model training method described above can be referenced to each other.

[0237] Figure 7 Schematic diagram of the structure of the language model training device provided by the present invention; Figure 7 As shown, the device includes:

[0238] The first prediction unit 710 is configured to predict a first probability matrix corresponding to each data unit in the sample text based on the teacher model; the first probability matrix includes probability values ​​of each data unit belonging to each word in a first vocabulary, where the first vocabulary is the vocabulary of the teacher model;

[0239] The compression unit 720 is configured to compress the first probability matrix according to the numerical values ​​of the probability values ​​in the first probability matrix to obtain a second probability matrix corresponding to each of the data units;

[0240] The mapping unit 730 is configured to perform an alignment operation on the second vocabulary according to the word elements corresponding to the probability values ​​in the second probability matrix to obtain a third vocabulary; the second vocabulary is the vocabulary of the student model;

[0241] The training unit 740 is used to perform distillation training on the student model according to the third vocabulary and the second probability matrix to obtain a target language model.

[0242] The device provided in this embodiment predicts the first probability matrix corresponding to each data unit in the sample text based on the teacher model to obtain the full probability distribution of each data unit, and adaptively compresses the first probability matrix in a sparse manner according to the size of each probability value in the first probability matrix to generate a second probability matrix with lower storage capacity, effectively reducing the amount of data and storage requirements; then, using the word elements corresponding to each probability value in the second probability matrix, the vocabulary of the student model is aligned to obtain a third vocabulary, and through dynamic vocabulary mapping, the problem of inconsistency between the vocabulary of the teacher model and the student model is solved, and the problem of knowledge transfer failure caused by vocabulary differences is avoided. Finally, the student model is distilled and trained using the third vocabulary and the second probability matrix to obtain a target language model. Since distillation training is performed by sparse compression of probability matrix and vocabulary alignment, it not only significantly reduces the problem of excessive storage cost caused by the growth of vocabulary size and effectively improves the efficiency of distillation training, but also enables the target language model trained thereby to better adapt to different model architectures and text processing scenarios while maintaining high performance.

[0243] In some embodiments, the compression unit is specifically used to: receive user input information and determine the target quantity based on the compression mode in the user input information; sort the probability values ​​in the first probability matrix in descending order according to the numerical value; in the first probability matrix, select the probability value of the target quantity with a higher sorting position to construct the second probability matrix.

[0244] In some embodiments, the compression unit is further used to: when the compression mode is an adaptive compression mode, perform cumulative calculations on each probability value in the first probability matrix in descending order according to the results, until the cumulative calculation value is greater than or equal to a preset threshold, and determine the target number based on the number of probability values ​​participating in the cumulative calculation; when the compression mode is a fixed compression mode, determine the preset number as the target number.

[0245] In some embodiments, the training unit is specifically used to: construct an empty matrix based on the number of word units in the third vocabulary; fill the probability values ​​in the second probability matrix into the empty matrix according to the first index of the word unit corresponding to each probability value in the second probability matrix to obtain a first reconstructed probability matrix corresponding to each data unit; the first index is the index of the word unit corresponding to each probability value in the second probability matrix in the third vocabulary; normalize the probability values ​​in the first reconstructed probability matrix to obtain a second reconstructed probability matrix corresponding to each data unit; predict the third probability matrix corresponding to each data unit based on the student model; the third probability matrix includes the probability value of each data unit belonging to each word unit in the third vocabulary; perform distillation training on the student model according to the second reconstructed probability matrix, the third probability matrix, and the word unit label corresponding to each data unit to obtain the target language model.

[0246] In some embodiments, the mapping unit is specifically used to: obtain each first probability value and each second probability value in the second probability matrix; there is no mapping relationship between the word element corresponding to the first probability value and all the word elements in the second vocabulary; there is a mapping relationship between the word element corresponding to the second probability value and a word element in the second vocabulary; based on the word segmenter of the student model, map and generate the mapping index of the word element corresponding to each first probability value; determine the second index of the target word element that has a mapping relationship with the word element corresponding to each second probability value as the mapping index of the word element corresponding to each second probability value; the second index is the index of the target word element in the second vocabulary; according to the mapping index of the word element corresponding to each first probability value, fill the word element corresponding to each first probability value into the empty vocabulary, and according to the mapping index of the word element corresponding to each second probability value, fill the word element corresponding to each second probability value into the empty vocabulary; according to the filling result, obtain the third vocabulary.

[0247] In some embodiments, the device also includes an encoding unit and a decoding unit; the encoding unit is specifically used to encode and store each of the data units, each probability value in the second probability matrix, and the third index of the word element corresponding to each probability value in the second probability matrix to the disk when the second probability matrix is ​​obtained and the training instruction of the student model is not received; the third index is the index of each probability value in the second probability matrix in the first vocabulary; the decoding unit is specifically used to parse and obtain each of the data units, each probability value in the second probability matrix, and the third index in the disk when the training instruction of the student model is received.

[0248] In some embodiments, the encoding unit is further used to: perform quantization encoding on each probability value in the second probability matrix to obtain a first encoding result; perform quantization encoding on the third index to obtain a second encoding result; perform hash encoding on each of the data units to obtain a third encoding result; and store the first encoding result, the second encoding result and the third encoding result in the form of a compressed file to the disk.

[0249] In some embodiments, the decoding unit is further used to: load the compressed file into the disk; perform inverse quantization decoding on the first encoding result in the compressed file to obtain each probability value in the second probability matrix; perform inverse quantization decoding on the second encoding result in the compressed file to obtain the third index; perform a hash reverse table lookup operation on the third encoding result in the compressed file to obtain each data unit.

[0250] Figure 8 Schematic diagram of the structure of the text processing device provided by the present invention; Figure 8 As shown, the device includes:

[0251] The acquisition unit 810 is used to acquire the text to be processed;

[0252] The second prediction unit 820 is used to perform word unit prediction on each data unit in the text to be processed based on the target language model to obtain a word unit prediction result corresponding to the text to be processed;

[0253] The processing unit 830 is configured to perform text processing on the text to be processed according to the word-unit prediction result;

[0254] The text processing includes text generation, code generation, machine translation or text classification; the target language model is trained based on a language model training method.

[0255] The device provided in this embodiment performs word prediction on a lightweight target language model trained through distillation using a sparse compressed probability matrix and vocabulary alignment, and implements intelligent processing of tasks such as text generation, code generation, machine translation, or text classification based on the prediction results, thereby significantly improving the accuracy, efficiency, and adaptability of text processing.

[0256] The device provided by the present invention is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for the specific processes and detailed contents, which will not be repeated here.

[0257] Figure 9 An example of a physical structure diagram of an electronic device is shown below. Figure 9As shown, the electronic device may include: a processor 910 , a communication interface 920 , a memory 930 and a communication bus 940 , wherein the processor 910 , the communication interface 920 and the memory 930 communicate with each other via the communication bus 940 . The processor 910 can call the logic instructions in the memory 930 to execute the language model training method, which includes: based on the teacher model, predicting a first probability matrix corresponding to each data unit in the sample text; the first probability matrix includes the probability value of each data unit belonging to each word in the first vocabulary, and the first vocabulary is the vocabulary of the teacher model; according to the numerical value of each probability value in the first probability matrix, the first probability matrix is ​​compressed to obtain a second probability matrix corresponding to each data unit; according to the word corresponding to each probability value in the second probability matrix, the second vocabulary is aligned to obtain a third vocabulary; the second vocabulary is the vocabulary of the student model; according to the third vocabulary and the second probability matrix, the student model is distilled and trained to obtain a target language model, or a text processing method, which includes: obtaining a text to be processed; based on the target language model, performing word prediction on each data unit in the text to be processed to obtain a word prediction result corresponding to the text to be processed; performing text processing on the text to be processed based on the word prediction result; wherein the text processing includes text generation, code generation, machine translation or text classification; the target language model is trained based on the language model training method.

[0258] Furthermore, the logic instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0259] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the language model training method provided by the above methods, which includes: based on the teacher model, predicting a first probability matrix corresponding to each data unit in the sample text; the first probability matrix includes the probability values ​​of each data unit belonging to each word in the first vocabulary, and the first vocabulary is the vocabulary of the teacher model; according to the numerical size of each probability value in the first probability matrix, the first probability matrix is ​​compressed to obtain a second probability matrix corresponding to each data unit; according to the second The word elements corresponding to each probability value in the probability matrix are aligned with the second word list to obtain a third word list; the second word list is the word list of the student model; according to the third word list and the second probability matrix, the student model is distilled and trained to obtain a target language model, or a text processing method, the method includes: obtaining a text to be processed; based on the target language model, performing word element prediction on each data unit in the text to be processed to obtain a word element prediction result corresponding to the text to be processed; according to the word element prediction result, text processing is performed on the text to be processed; wherein, the text processing includes text generation, code generation, machine translation or text classification; the target language model is trained based on the language model training method.

[0260] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the language model training method provided by the above-mentioned methods, the method comprising: based on a teacher model, predicting a first probability matrix corresponding to each data unit in a sample text; the first probability matrix includes probability values ​​of each data unit belonging to each word element in a first vocabulary, the first vocabulary being the vocabulary of the teacher model; compressing the first probability matrix according to the numerical value of each probability value in the first probability matrix to obtain a second probability matrix corresponding to each data unit; and compressing the first probability matrix according to the word element corresponding to each probability value in the second probability matrix. , perform an alignment operation on the second vocabulary to obtain a third vocabulary; the second vocabulary is the vocabulary of the student model; according to the third vocabulary and the second probability matrix, perform distillation training on the student model to obtain a target language model, or a text processing method, the method comprising: obtaining a text to be processed; based on the target language model, perform word unit prediction on each data unit in the text to be processed to obtain a word unit prediction result corresponding to the text to be processed; according to the word unit prediction result, perform text processing on the text to be processed; wherein, the text processing includes text generation, code generation, machine translation or text classification; the target language model is trained based on the language model training method.

[0261] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0262] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0263] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A language model training method, characterized in that: include: Based on the teacher model, predict the first probability matrix corresponding to each data unit in the sample text; The first probability matrix includes probability values ​​of each data unit belonging to each word in a first vocabulary, the first vocabulary being the vocabulary of the teacher model; receiving user input information and determining a target quantity according to a compression mode in the user input information; the compression mode includes a fixed compression mode or an adaptive compression mode; Sort the probability values ​​in the first probability matrix in descending order according to their numerical values; In the first probability matrix, the probability values ​​of the target quantities with the highest ranking positions are selected to construct a second probability matrix; In the second probability matrix, obtaining each first probability value and each second probability value; the word element corresponding to the first probability value does not have a mapping relationship with any word element in the second vocabulary; the word element corresponding to the second probability value does have a mapping relationship with a word element in the second vocabulary; A word segmenter based on the student model maps and generates a mapping index of each word corresponding to each first probability value; Determining the second index of the target word element that has a mapping relationship with the word element corresponding to each second probability value as the mapping index of the word element corresponding to each second probability value; the second index is the index of the target word element in the second vocabulary; According to the mapping index of the word-gram corresponding to each first probability value, fill the word-gram corresponding to each first probability value into an empty word list, and according to the mapping index of the word-gram corresponding to each second probability value, fill the word-gram corresponding to each second probability value into the empty word list; According to the filling result, a third vocabulary is obtained; the second vocabulary is the vocabulary of the student model; Constructing an empty matrix according to the number of word elements in the third vocabulary; Filling the empty matrix with each probability value in the second probability matrix according to the first index of the word element corresponding to each probability value in the second probability matrix, thereby obtaining a first reconstructed probability matrix corresponding to each data unit; the first index is the index of the word element corresponding to each probability value in the second probability matrix in the third vocabulary; Normalizing each probability value in the first reconstruction probability matrix to obtain a second reconstruction probability matrix corresponding to each data unit; Based on the student model, predicting a third probability matrix corresponding to each of the data units; the third probability matrix includes probability values ​​of each of the data units belonging to each word in the third vocabulary; The student model is distilled and trained according to the second reconstruction probability matrix, the third probability matrix, and the word-unit label corresponding to each of the data units to obtain a target language model.

2. The language model training method according to claim 1, characterized in that The determining the target quantity according to the compression mode in the user input information includes: When the compression mode is the adaptive compression mode, cumulatively calculating each probability value in the first probability matrix according to the descending sorting results until the cumulative calculated value is greater than or equal to a preset threshold, and determining the target number based on the number of probability values ​​involved in the cumulative calculation; When the compression mode is a fixed compression mode, the preset number is determined as the target number.

3. The language model training method according to any one of claims 1-2, characterized in that The method further comprises: When the second probability matrix is ​​obtained and no training instruction for the student model is received, encoding and storing each of the data units, each probability value in the second probability matrix, and a third index of a word element corresponding to each probability value in the second probability matrix to disk; the third index is an index of each probability value in the second probability matrix in the first vocabulary; When the training instruction of the student model is received, each of the data units, each probability value in the second probability matrix, and the third index are parsed and obtained in the disk.

4. The language model training method according to claim 3, wherein: The encoding and storing each of the data units, each probability value in the second probability matrix, and a third index of a word corresponding to each probability value in the second probability matrix to a disk includes: Performing quantization encoding on each probability value in the second probability matrix to obtain a first encoding result; Performing quantization encoding on the third index to obtain a second encoding result; Performing hash coding on each of the data units to obtain a third coding result; The first encoding result, the second encoding result, and the third encoding result are stored in the disk in the form of a compressed file.

5. The language model training method according to claim 4, characterized in that Parsing and obtaining each of the data units, each probability value in the second probability matrix, and the third index in the disk includes: Loading the compressed file into the disk; Performing inverse quantization decoding on the first encoding result in the compressed file to obtain each probability value in the second probability matrix; Performing inverse quantization decoding on the second encoding result in the compressed file to obtain the third index; A hash reverse table lookup operation is performed on the third encoding result in the compressed file to obtain each of the data units.

6. A text processing method, characterized in that: include: Get the text to be processed; Based on the target language model, word unit prediction is performed on each data unit in the text to be processed to obtain a word unit prediction result corresponding to the text to be processed; Performing text processing on the text to be processed according to the word-unit prediction result; The text processing includes text generation, code generation, machine translation or text classification; the target language model is trained based on the language model training method according to any one of claims 1 to 5.

7. A language model training device, characterized in that: include: A first prediction unit is used to predict a first probability matrix corresponding to each data unit in the sample text based on the teacher model; The first probability matrix includes probability values ​​of each data unit belonging to each word in a first vocabulary, the first vocabulary being the vocabulary of the teacher model; A receiving unit, configured to receive user input information and determine a target quantity according to a compression mode in the user input information; the compression mode includes a fixed compression mode or an adaptive compression mode; a sorting unit, configured to sort the probability values ​​in the first probability matrix in descending order according to their numerical values; a constructing unit, configured to select the probability values ​​of the target quantities with higher sorting positions in the first probability matrix to construct a second probability matrix; an acquiring unit, configured to acquire, in the second probability matrix, each first probability value and each second probability value; wherein a word-gram corresponding to the first probability value has no mapping relationship with any word-gram in the second vocabulary; and a word-gram corresponding to the second probability value has a mapping relationship with a word-gram in the second vocabulary; A mapping unit, configured to generate mapping indexes of word units corresponding to the first probability values ​​based on the word segmenter of the student model; a mapping index determination unit, configured to determine the second index of a target word-gram that has a mapping relationship with the word-gram corresponding to each of the second probability values ​​as the mapping index of the word-gram corresponding to each of the second probability values; The second index is the index of the target word in the second vocabulary; a first filling unit, configured to fill the word-grams corresponding to the first probability values ​​into an empty word list according to the mapping index of the word-grams corresponding to the first probability values, and to fill the word-grams corresponding to the second probability values ​​into the empty word list according to the mapping index of the word-grams corresponding to the second probability values; A third vocabulary acquisition unit is used to acquire a third vocabulary according to the filling result; The second vocabulary is a vocabulary of the student model; Constructing an empty matrix unit, for constructing an empty matrix according to the number of word elements in the third vocabulary; a filling unit, configured to fill the empty matrix with each probability value in the second probability matrix according to a first index of a word element corresponding to each probability value in the second probability matrix, to obtain a first reconstructed probability matrix corresponding to each data unit; the first index being an index in the third vocabulary of the word element corresponding to each probability value in the second probability matrix; a processing unit, configured to normalize each probability value in the first reconstruction probability matrix to obtain a second reconstruction probability matrix corresponding to each of the data units; a prediction unit, configured to predict, based on the student model, a third probability matrix corresponding to each of the data units; the third probability matrix including probability values ​​of each of the data units belonging to each word in the third vocabulary; A training unit is used to perform distillation training on the student model according to the second reconstruction probability matrix, the third probability matrix, and the word-unit label corresponding to each of the data units to obtain a target language model.

8. A text processing device, characterized in that: include: An acquisition unit, used to acquire the text to be processed; A second prediction unit is configured to perform word unit prediction on each data unit in the to-be-processed text based on a target language model to obtain a word unit prediction result corresponding to the to-be-processed text; a processing unit, configured to perform text processing on the text to be processed according to the word-unit prediction result; The text processing includes text generation, code generation, machine translation or text classification; the target language model is trained based on the language model training method according to any one of claims 1 to 5.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it implements the language model training method according to any one of claims 1 to 5, or implements the text processing method according to claim 6.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the language model training method according to any one of claims 1 to 5, or implements the text processing method according to claim 6.

Images