Large language model pre-training method and device, electronic equipment and storage medium

Abstract

The invention discloses a large language model pre-training method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining multiple types of text data, and carrying out the preprocessing of the text data, and obtaining an initial pre-training data set; the method comprises the following steps: constructing a generative large language model based on a decoder architecture of a Transform model; according to the initial pre-training data set, performing multi-stage training on the generative large language model; forming a self-distillation mixed data set according to the teacher model and the initial pre-training data set; the trained generative large language model in each stage is used as a teacher model; training a student model by using the self-distillation mixed data set; the student model is a generative large language model at a distance of k stages after the teacher model; and then carrying out cycle training. According to the method, the comprehensiveness of pre-training learning is improved by using the self-distillation mixed data set, and the problem of repeated learning of low-quality pre-training data can be effectively solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a large language model pre-training method, device, electronic device and storage medium. Background Art

[0002] Large language model pre-training refers to the process of using large-scale text data for preliminary training on large neural network models to learn common feature representations and knowledge. Due to the expansion of the model structure and the surge in the amount of pre-training data, large language model pre-training often encounters the problem of unstable training, which interferes with the pre-training process of the large language model, resulting in poor pre-training results for the large language model, and even causing pre-training crashes. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a large language model pre-training method, device, electronic device and storage medium to improve the stability and accuracy of large language model pre-training.

[0004] To achieve the above purpose, an embodiment of the present application provides a large language model pre-training method, the method comprising the following steps:

[0005] Acquire multiple types of text data, and preprocess the text data to obtain an initial pre-training data set;

[0006] Build a generative large language model based on the decoder architecture of the Transformer model;

[0007] Performing multiple stages of training on the generative large language model according to the initial pre-training dataset;

[0008] A self-distilled mixed dataset is synthesized according to the teacher model and the initial pre-training dataset; wherein the generative large language model trained in each stage is used as the teacher model;

[0009] Using the self-distillation mixed data set to train a student model; wherein the student model is the generative large language model that is k stages behind the teacher model, and k is a positive integer;

[0010] Return to the step of synthesizing the self-distilled mixed data set based on the teacher model and the initial pre-training data set until the preset number of stages of training are completed, and the student model finally obtained by the training is used as the pre-trained generative large language model.

[0011] In some embodiments, the Transformer model-based decoder architecture constructs a generative large language model, comprising the following steps:

[0012] The construction step is performed on the causal decoder architecture of the Transformer model to construct the generative large language model; the construction step includes:

[0013] Normalize each sublayer of the causal decoder architecture of the Transformer model using the pre-root mean square normalization RMSNorm algorithm;

[0014] Setting a group query attention mechanism for the causal decoder architecture of the Transformer model;

[0015] Determining the activation function of the feedforward neural network layer of the causal decoder architecture of the Transformer model as a SwiGLU function;

[0016] The position encoding of the causal decoder architecture of the Transformer model is set to RoPE relative rotation position encoding.

[0017] In some embodiments, the training of the generative large language model in multiple stages according to the initial pre-training dataset comprises the following steps:

[0018] Determining the mixing ratios of different types of text data at different stages in the initial pre-training data set;

[0019] Sampling from the initial pre-training data set according to the mixing ratio to obtain a proportioned mixed pre-training data set;

[0020] The texts in the mixed pre-training data set are concatenated and padded to a set length to obtain multiple texts of the same length as training samples;

[0021] Randomly divide the training samples into a number of batches of data;

[0022] Initializing weight parameters of the embedding layer, the decoder layer, and the normalization layer of the generative large language model using a random method;

[0023] The generative large language model is trained for N stages according to the batch data; wherein each stage of training uses at least B batches of data; and N and B are positive integers.

[0024] In some embodiments, the training of the generative large language model for N stages according to the batch data comprises the following steps:

[0025] Each stage of training performs the following steps:

[0026] Inputting the word unit sequence in each batch of data into the generative large language model, and then forward propagating to obtain the predicted probability distribution of the target word unit;

[0027] Calculating the loss value between the predicted distribution of the generative large language model and the actual annotation result based on the loss function;

[0028] Calculating the gradient of the loss function relative to the parameters of the generative large language model using a back-propagation algorithm;

[0029] Propagate the gradient from the output layer of the generative large language model back to the input layer of the generative large language model in sequence;

[0030] According to the gradient, minimizing the loss value is used as a training goal, and an optimization algorithm is used to update the parameters of the generative large language model; wherein, each time the generative large language model processes a batch of the data, the parameters of the generative large language model are updated accordingly.

[0031] In some embodiments, synthesizing the self-distilled mixed dataset based on the teacher model and the initial pre-training dataset comprises the following steps:

[0032] Initialize an empty data pool;

[0033] During the training of each stage, training samples are sampled from each batch of data of each stage with a first probability, and then the sampled training samples are added to the data pool;

[0034] After each stage of training, the current parameters of the generative large language model are saved to obtain the teacher model;

[0035] After each stage of training, candidate synthetic samples are randomly sampled from the data pool; wherein the data volume of the candidate synthetic samples is the same as the data volume of the batch data;

[0036] Performing text segmentation on each of the training samples in the candidate synthetic samples using a separator to obtain a plurality of original text data;

[0037] Selecting data to be synthesized from each of the original text data with a second probability;

[0038] Selecting characters or words of each piece of the data to be synthesized as a cutoff point with a third probability; wherein the third probability obeys a normal distribution;

[0039] Taking the text at and before the truncation point as a query text, and inputting the query text into the teacher model at the current stage;

[0040] Outputting predicted text corresponding to the query text using the teacher model;

[0041] splicing the query text and the predicted text to obtain a synthesized text;

[0042] The synthesized text and the remaining original data in each of the original data that are not selected as the data to be synthesized are synthesized into the self-distillation mixed data set.

[0043] In some embodiments, the method of training a student model using the self-distillation mixed dataset comprises the following steps:

[0044] Calculating a knowledge transfer interval; wherein k represents the knowledge transfer interval;

[0045] The calculation formula of the knowledge transfer interval includes:

[0046]

[0047] Among them, K and c are hyperparameters; K represents the maximum knowledge interval, c is used to control the growth rate and approximation speed of k as the training stage i grows, c>0; is the rounding symbol; f(i) represents the intermediate function;

[0048] Selecting the generative large language model that is k stages behind the teacher model as the student model;

[0049] The student model is trained using the self-distillation mixed dataset.

[0050] In some embodiments, the method further comprises the following steps:

[0051] Using a text dataset of the target type to train the student model finally trained, thereby obtaining a target large language model;

[0052] The target type of text is input into the target large language model, and then the target large language model is used to output a prediction result corresponding to the target type of text.

[0053] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application provides a large language model pre-training device, the device comprising:

[0054] A data processing unit, used to obtain multiple types of text data and pre-process the text data to obtain an initial pre-training data set;

[0055] A model building unit, which is used to build a generative large language model based on the decoder architecture of the Transformer model;

[0056] A stage training unit, used for performing multiple stages of training on the generative large language model according to the initial pre-training data set;

[0057] A data set synthesis unit, used for synthesizing a self-distilled mixed data set according to the teacher model and the initial pre-trained data set; wherein the generative large language model trained in each stage is used as the teacher model;

[0058] A distillation training unit, configured to train a student model using the self-distillation mixed data set; wherein the student model is the generative large language model that is k stages behind the teacher model, and k is a positive integer;

[0059] A cyclic training unit is used to return to the step of synthesizing a self-distilled mixed data set based on the teacher model and the initial pre-training data set until a preset number of stages of training are completed, and the student model finally obtained by the training is used as the pre-trained generative large language model.

[0060] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the above-mentioned large language model pre-training method is implemented.

[0061] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned large language model pre-training method is implemented.

[0062] The embodiments of the present application include at least the following beneficial effects:

[0063] The present application can obtain multiple types of text data, and pre-process the text data to obtain an initial pre-training data set; construct a generative large language model based on the decoder architecture of the Transformer model; train the generative large language model in multiple stages according to the initial pre-training data set; synthesize a self-distillation mixed data set according to the teacher model and the initial pre-training data set; wherein the generative large language model trained in each stage is used as the teacher model; train the student model using the self-distillation mixed data set; wherein the student model is a generative large language model that is k stages away from the teacher model, and k is a positive integer; return to the step of synthesizing the self-distillation mixed data set according to the teacher model and the initial pre-training data set until the training of the preset number of stages is completed, and the student model finally trained is used as the pre-trained generative large language model. This application uses a self-distilled mixed data set to self-improve the generative large language model, integrating the language representations and knowledge obtained in different training stages into the pre-training steps in the subsequent stages, thereby enhancing the stability of the pre-training process, increasing the breadth and diversity of the pre-training data, and improving the comprehensiveness of pre-training learning. It can effectively solve the problem of repeated learning of low-quality pre-training data and significantly improve the parameter accuracy of the generative large language model during the pre-training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0065] Figure 1 A flowchart of a large language model pre-training method provided in an embodiment of the present application;

[0066] Figure 2 A flowchart of the steps for constructing a generative large language model provided in an embodiment of the present application;

[0067] Figure 3 A flowchart of the steps of S120 provided in an embodiment of the present application;

[0068] Figure 4 A flowchart of the steps of S126 provided in an embodiment of the present application;

[0069] Figure 5 A flowchart of the steps of S140 provided in an embodiment of the present application;

[0070] Figure 6 An example flow chart of a large language model pre-training provided in an embodiment of the present application;

[0071] Figure 7 A flow chart of a self-distillation large language model pre-training method provided in an embodiment of the present application;

[0072] Figure 8 A flowchart for constructing a self-distillation mixed data set provided in an embodiment of the present application;

[0073] Fig. 9 A schematic diagram of the structure of a large language model pre-training device provided in an embodiment of the present application;

[0074] Fig.10 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0076] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0077] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0078] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0079] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations:

[0080] Language Model: An artificial intelligence model that uses machine learning technology to understand and generate human natural language. By analyzing and modeling text data, the language model can learn the usage patterns of human language, and based on this, accurately predict the probability distribution of the masked position or the next word sequence for a given content segment. This probability calculation process can help computers understand and generate natural language, and then be applied to a variety of task scenarios such as intelligent question answering, machine translation, and content analysis. A large language model (LLM) refers to a neural network language model that has been scaled up, usually with a billion or more parameters. When solving complex tasks, it demonstrates capabilities that small pre-trained language models do not have, such as strong contextual learning, reasoning, cross-domain adaptability, and emergence.

[0081] Pre-training: refers to the process of pre-training a model on a large-scale, unsupervised text dataset, which is the initial stage of language model learning. In the pre-training stage, the model extracts and models the patterns, structures, and semantic knowledge of natural language from massive data to learn general feature representations and knowledge, which has the potential to solve downstream tasks.

[0082] Self-distillation: Knowledge distillation refers to transferring the knowledge of the teacher model to the student model, which is usually achieved by minimizing the difference in the output distribution of the teacher model and the student model in the hidden layer or classification layer. Generally speaking, the teacher model has the same or even more complex network capacity and model structure than the student model. Self-distillation is a special case of knowledge distillation. It does not rely on an external large language model as a teacher model, but uses the same model for self-transfer and optimization of knowledge. Common practices include distilling from one module of the model to another, or from one stage of the model to another.

[0083] Large language model pre-training refers to the process of using large-scale text data to perform preliminary training on a large neural network model to learn common feature representations and knowledge. Due to the expansion of the model structure and the surge in the amount of pre-training data, large language model pre-training often encounters training instability problems, which interferes with the learning process and even causes training crashes.

[0084] In order to solve the above problems, the commonly used methods are gradient clipping, dynamic batch training scheduling and regularization technology. Gradient clipping is a method for controlling the norm of the gradient. It constrains the size of the gradient by setting a threshold. When the norm of the calculated gradient exceeds this threshold, it is clipped according to the threshold to avoid gradient explosion. Dynamic batch training scheduling refers to dynamically adjusting the batch size (Batch Size) during the training process, gradually increasing the batch size, and eventually reaching the million level. Regularization technology is a method of controlling model complexity to avoid overfitting, such as weight decay, which suppresses overfitting of parameters by adding a penalty term to the loss function, thereby improving the generalization ability of the model and stabilizing the training process.

[0085] In summary, existing methods mainly maintain the stability of model training by optimizing parameter configuration. Although existing technologies can enhance stability to a certain extent, they still face the limitations of being highly dependent on hyperparameter tuning and lacking universality on data sets from different sources. This application starts from the dimension of data generalization and introduces self-distillation technology in the pre-training stage of large language models. It aims to apply and strengthen knowledge in the language representation space across different training stages, thereby enhancing the stability of the training process and improving the generalization ability of large language models.

[0086] The embodiment of the present application provides a large language model pre-training method, device, electronic device and storage medium. The technical solution of the present application includes: obtaining multiple types of text data, and pre-processing the text data to obtain an initial pre-training data set; constructing a generative large language model based on the decoder architecture of the Transformer model; training the generative large language model in multiple stages according to the initial pre-training data set; synthesizing a self-distilled mixed data set according to the teacher model and the initial pre-training data set; wherein the generative large language model trained in each stage is used as the teacher model; using the self-distilled mixed data set to train the student model; wherein the student model is a generative large language model k stages away from the teacher model, and k is a positive integer; returning to the step of synthesizing the self-distilled mixed data set according to the teacher model and the initial pre-training data set, until the training of the preset number of stages is completed, and the student model finally trained is used as the pre-trained generative large language model. This application uses a self-distilled mixed data set to self-improve the generative large language model, integrating the language representations and knowledge obtained in different training stages into the pre-training steps in the subsequent stages, thereby enhancing the stability of the pre-training process, increasing the breadth and diversity of the pre-training data, and improving the comprehensiveness of pre-training learning. It can effectively solve the problem of repeated learning of low-quality pre-training data and significantly improve the parameter accuracy of the generative large language model during the pre-training process.

[0087] The embodiments of the present application provide a large language model pre-training method, device, electronic device and storage medium, which relate to the field of artificial intelligence technology. The large language model pre-training method, device, electronic device and storage medium provided in the embodiments of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements a knowledge extraction method, etc., but is not limited to the above forms.

[0088] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0089] Reference Figure 1 The present application embodiment provides a large language model pre-training method, which may include but is not limited to S100 to S150, as follows:

[0090] S100: Acquire multiple types of text data, and preprocess the text data to obtain an initial pre-training data set.

[0091] Exemplarily, general and various types of specialized text data are collected and preprocessed to obtain an initial pre-training data set.

[0092] Sources of text data include public pre-trained data and proprietary data collected based on preset application scenarios, covering web pages, books, multiple languages, mathematical codes and other content.

[0093] Data preprocessing operations include quality filtering, sensitive content filtering, data deduplication, and lemmatization.

[0094] S110: Building a generative large language model based on the decoder architecture of the Transformer model.

[0095] Further, S110 may include S111:

[0096] S111: Execute a construction step on the causal decoder architecture of the Transformer model, thereby constructing the generative large language model; refer to Figure 2 , the construction steps in S111 include S1111 to S1114:

[0097] S1111: normalizing each sublayer of the causal decoder architecture of the Transformer model using a pre-implemented root mean square normalization (RMSNorm) algorithm;

[0098] S1112: Setting a group query attention mechanism for the causal decoder architecture of the Transformer model;

[0099] S1113: Determine the activation function of the feedforward neural network layer of the causal decoder architecture of the Transformer model as a SwiGLU function;

[0100] S1114: Setting the position encoding of the causal decoder architecture of the Transformer model to RoPE relative rotation position encoding.

[0101] Exemplarily, a generative large language model is constructed based on the decoder architecture of the Transformer model.

[0102] The decoder architecture can be subdivided into causal decoders, prefix decoders, etc. This embodiment adopts a causal decoder architecture, based on a unidirectional masked attention mechanism, which only focuses on the word element in front of it and itself in the input sequence, and then predicts the next word element autoregressively. The large language model of this embodiment includes at least a GPT series model and any generative large language model implemented by inheriting the GPT series model structure.

[0103] S120: Performing multiple stages of training on the generative large language model according to the initial pre-training dataset.

[0104] Reference Figure 3 , S120 may further include S121 to S126:

[0105] S121: Determine the mixing ratios of different types of text data at different stages in the initial pre-training data set;

[0106] S122: sampling from the initial pre-training data set according to the mixing ratio to obtain a proportioned mixed pre-training data set;

[0107] S123: splicing and padding the texts in the proportioned mixed pre-training data set to a set length to obtain multiple texts of the same length as training samples;

[0108] S124: Randomly divide the training samples into a number of batches of data;

[0109] S125: Initializing weight parameters of the embedding layer, the decoder layer, and the normalization layer of the generative large language model using a random method;

[0110] S126: Performing N stages of training on the generative large language model according to the batch data; wherein each stage of training uses at least B batches of data; N and B are positive integers.

[0111] Furthermore, refer to Figure 4 When executing S126, the training of each stage executes the following steps S1261 to S1265:

[0112] S1261: Inputting the word-unit sequence in each batch of data into the generative large language model, and then forward propagating to obtain the predicted probability distribution of the target word-unit;

[0113] S1262: Calculating the loss value between the predicted distribution of the generative large language model and the actual annotation result based on the loss function;

[0114] S1263: Calculate the gradient of the loss function relative to the parameters of the generative large language model using a back propagation algorithm;

[0115] S1264: Propagate the gradient from the output layer of the generative large language model back to the input layer of the generative large language model in sequence;

[0116] S1265: Based on the gradient, minimizing the loss value is used as a training goal, and an optimization algorithm is used to update the parameters of the generative large language model; wherein, each time the generative large language model processes a batch of the data, the parameters of the generative large language model are updated accordingly.

[0117] Exemplarily, this embodiment can pre-train a large language model based on large-scale unsupervised text data, and the specific steps include:

[0118] (1) Batch dataset preparation. The vocabulary of the large language model is initialized based on the pre-training dataset, and the pre-training dataset is divided into data batches and the learning sequence is arranged through course learning, data matching, and sample splicing strategies to obtain a batch dataset sequence.

[0119] (2) Initialize the weight parameters of the large language model, including the weight parameters of the embedding layer, decoder layer, and normalization layer.

[0120] (3) Training stage division: The pre-training process of the large language model is divided into N training stages, where each training stage contains the learning of at least B batches of data.

[0121] (4) Use batch dataset sequences to iteratively train the large language model. In each round of batch training, the batch data is input, the loss is calculated through forward propagation, the gradient is calculated through backward propagation, and the optimizer updates the parameters of the large language model.

[0122] S130: Synthesize a self-distillation mixed dataset based on the teacher model and the initial pre-training dataset; wherein the generative large language model trained in each stage is used as the teacher model.

[0123] Further, S130 may include S131 to S1311:

[0124] S131: Initialize an empty data pool;

[0125] S132: During training at each stage, sampling training samples from each batch of data at each stage with a first probability, and then adding the sampled training samples to the data pool;

[0126] S133: After each stage of training, save the current parameters of the generative large language model to obtain the teacher model;

[0127] S134: After each stage of training, randomly sampling from the data pool to obtain candidate synthetic samples; wherein the data volume of the candidate synthetic samples is the same as the data volume of the batch data;

[0128] S135: performing text segmentation on each of the training samples in the candidate synthetic samples using a separator to obtain a plurality of original text data;

[0129] S136: Selecting data to be synthesized from each piece of the original data according to the second probability;

[0130] S137: Selecting characters or words of each piece of the data to be synthesized as a truncation point with a third probability; wherein the third probability obeys a normal distribution;

[0131] S138: taking the text at and before the truncation point as a query text, and inputting the query text into the teacher model at the current stage;

[0132] S139: Outputting a predicted text corresponding to the query text using the teacher model;

[0133] S1310: concatenating the query text and the predicted text to obtain a synthesized text;

[0134] S1311: The synthesized text and the remaining original data in each of the original data that are not selected as the data to be synthesized are synthesized into the self-distillation mixed data set.

[0135] Exemplarily, this embodiment can use the generative large language model trained in the i-th stage as a teacher model to synthesize a self-distillation mixed data set.

[0136] This embodiment proposes a self-distillation large language model pre-training method, which aims to promote the self-improvement of the large language model through the distillation iteration mechanism, so as to improve the training stability, model learning effect and generalization ability. Its core technical point is to apply the data generalization scheme based on self-distillation to the iterative training of large language models. Using real-world data as the above, when the i-th stage model training is completed, the model obtained in the i-th training stage is used as the teacher model to generate high-quality synthetic data that is closer to the knowledge distribution of the basic model. These data will be used for the learning of the student model in the i+k-th training stage.

[0137] Specifically, the steps include:

[0138] 1. Maintain a real data pool P, initialized to empty.

[0139] 2. For each training stage i, synthesize a self-distilled mixed dataset, which specifically includes the following steps:

[0140] 2.1 Real data sampling. For each piece of data in each batch of training data, there is a first probability α to be selected into the data pool P. α can be unified as a fixed value for all samples, or it can vary depending on the sample category label.

[0141] This embodiment enables the samples in the data pool P to reflect the distribution characteristics of the original data set, and can adjust the sampling probability according to the representativeness and priority of different categories, thereby optimizing the balance of learning of each category during the model training process, which helps to enhance the generalization ability of the final model and make it more robust when facing new data.

[0142] 2.2 Model archiving. The last training round of each training phase is used as the model archiving point. When the training is completed, the current model parameters are saved.

[0143] 2.3 Obtaining distillation candidate sets.

[0144] After the current training phase is completed, b data samples are randomly sampled from the data pool P as candidate synthetic samples, where b is equal to the batch data size of normal training.

[0145] Using a special identifier that indicates the end of a sentence as a delimiter, the b data samples are segmented to obtain multiple original text data of varying lengths and quantities. Since the pre-training samples are composed of multiple data and fillers with a fixed length and in accordance with certain rules, it is necessary to reversely restore the original text data according to the splicing rules.

[0146] For each piece of original text data, a second probability β is selected for synthesis, and the probability 1-β remains unchanged. The proportion of distilled synthetic data needs to be dynamically adjusted at different training stages. In the early stages of training, the model is not mature enough and needs to learn basic patterns and features from purer real-world data in order to build a robust basic representation capability. As training progresses, the probability β of data synthesis is gradually increased, so that the model gradually adapts to the data form generated by itself, thereby improving its generalization ability and robustness.

[0147] 2.4 Self-distillation synthesis.

[0148] For each source data to be synthesized, the position of the truncation point is selected. The third probability of each character or word in the text being selected as the truncation point follows the normal distribution law to ensure the diversity and randomness of the synthesized data while maintaining a high content of real-world knowledge.

[0149] The text fragment before and after the truncation point is used as a query and input into the model saved after the training of this stage to predict the subsequent content of the sentence.

[0150] The text before the source data cutoff point is concatenated with the generated text of the large language model to form a complete distilled mixed text sentence. This sentence not only retains the authenticity of the original data, but also ensures the distribution consistency of the large language model.

[0151] 2.5 Merge the original text and the distilled mixed text data to obtain a self-distilled mixed dataset D that combines real-world knowledge and self-distillation synthesis guided by real data.

[0152] 3. Delete the b data samples sampled in this stage from the data pool P.

[0153] S140: Train a student model using the self-distillation mixed data set; wherein the student model is the generative large language model that is k stages behind the teacher model, and k is a positive integer.

[0154] Reference Figure 5 , S140 may further include S141 to S143:

[0155] S141: Calculate the knowledge transfer interval; wherein k represents the knowledge transfer interval;

[0156] The calculation formula of the knowledge transfer interval includes:

[0157]

[0158] Among them, K and c are hyperparameters; K represents the maximum knowledge interval, c is used to control the growth rate and approximation speed of k as the training stage i grows, c>0; is the rounding symbol; f(i) represents the intermediate function;

[0159] S142: Select the generative large language model that is k stages behind the teacher model as the student model;

[0160] S143: Train the student model using the self-distillation mixed dataset.

[0161] Exemplarily, this embodiment uses the self-distillation mixed dataset D for training the student model in the i+kth stage.

[0162] The self-distilled mixed data set D obtained in step S130 is further processed into a batch data format that can be used for model training, and is used as a batch data of the i+k stage to participate in the training of the i+k stage. The above process can be regarded as a process of knowledge transfer and generalization from the previous teacher model to the growing student model.

[0163] Dynamically adjust the knowledge transfer interval k according to the training process. In the early stage of pre-training of large language models, the learning rate is high, the loss curve drops quickly, and the learning speed is fast, so the frequency of knowledge transfer (i.e., the knowledge transfer interval k) should be set to a smaller value. As the training progresses, the learning rate of the model gradually slows down. At this time, the knowledge transfer interval k should be increased to adapt to the learning process of the model and improve the training efficiency, thereby enhancing the robustness of the pre-training model.

[0164] S150: Return to the step of synthesizing the self-distilled mixed data set based on the teacher model and the initial pre-training data set until the preset number of stages of training are completed, and the student model finally obtained by the training is used as the pre-trained generative large language model.

[0165] It is understandable that this embodiment can be iterated repeatedly to continuously adjust and optimize the parameters of the large language model until the training loss of the large language model converges or reaches a preset number of training stages to obtain a pre-trained generative large language model.

[0166] As a further implementation, the embodiment of the present application may further include training the pre-trained generative large language model using the target type of text data, so that the generative large language model can generate corresponding prediction results under the guidance of the target type of text data. The embodiment of the present application may further include the following steps S161-S162:

[0167] S161: Using a text data set of a target type to train the student model finally trained, thereby obtaining a target large language model;

[0168] S162: Input the target type of text into the target large language model, and then use the target large language model to output a prediction result corresponding to the target type of text.

[0169] Exemplarily, texts in different scenarios correspond to text datasets of different target types. For example, the text types in different scenarios such as education, daily life or travel are generally different. Using texts from different scenarios as target types of text datasets to train a pre-trained generative large language model can enable the generative large language model to better learn the knowledge in the corresponding scenarios, and then output prediction results with a higher matching degree. The prediction results may include text, links, images, audio and video content.

[0170] Next, the solution of the embodiment of the present application will be introduced and explained in detail with reference to specific application examples.

[0171] The stability of the pre-trained model is the key to ensuring its generalization ability and practical application effect. A pre-trained model with high stability can adapt to downstream tasks more effectively, thereby showing better performance in a variety of task scenarios. In addition, the stability of the model is also directly related to the training efficiency and the demand for hardware resources.

[0172] In order to enhance the stability of training and further improve the generalization ability of large language models, this embodiment proposes to introduce self-distillation technology in the pre-training stage of large language models. The core technical point of this embodiment is to use the previous model as the teacher model and the model currently being trained as the student model. In the process of student model training, in addition to using the original data, a mixed data set synthesized by self-distillation of the teacher model will also be added. The synthetic data is randomly extracted from the previously learned data according to a specific probability distribution, and the truncation point position is determined according to the normal distribution. The content starting from the truncation point is completely generated by the teacher model. The synthetic data formed in this way not only retains the knowledge of the real world, but also fits the knowledge distribution of the basic model.

[0173] This embodiment aims to use distilled mixed data for self-improvement, integrating the language representations and knowledge obtained in different training stages into the subsequent learning process. The above strategy not only enhances the stability of the pre-training process, but also increases the breadth and diversity of the data, ensures the comprehensiveness of learning, and can effectively avoid repeated learning of low-quality data, significantly improving its performance in downstream tasks. Traditional distillation techniques usually require the teacher model to have an equally or even more complex structure, resulting in higher training costs; the self-distillation method simplifies the model compression and knowledge transfer process through a self-supervision mechanism, without relying on complex teacher models or additional computational steps. Although the self-distillation process does not actually add new information, a large number of studies have shown that the student model of self-distillation can achieve better generalization capabilities than the teacher model.

[0174] Reference Figure 6 , this embodiment provides an example flowchart of a large language model pre-training method.

[0175] Specifically, this embodiment may include the following steps:

[0176] S1. Collect general and various types of special text data and preprocess them to obtain the initial pre-training dataset.

[0177] Data sources include public pre-trained data and proprietary data collected based on preset application scenarios, covering web pages, books, multiple languages, mathematical codes and other content.

[0178] Data preprocessing operations include quality filtering, sensitive content filtering, data deduplication, and lemmatization.

[0179] S2. Build a generative large language model based on the decoder architecture of the Transformer model.

[0180] This embodiment adopts the standard, dense Transformer causal decoder architecture to build a generative large language model network structure. The Transformer decoder structure is composed of multiple identical network blocks stacked together, each of which contains a self-attention network layer and a prefix neural network layer.

[0181] The causal decoder architecture is an autoregressive architecture that uses a unidirectional attention mask mechanism. Its characteristic is that when predicting the word at each position, only the word at the previous position is used as the context, and no future word information is used. This mechanism ensures the temporal coherence and logical compliance of the generated text, showing good context learning and generalization capabilities, and is suitable for text generation tasks.

[0182] Based on the original Transformer decoder architecture, the preferred implementation scheme includes: using the pre-root mean square normalization RMSNorm algorithm to normalize each sublayer to enhance the stability of the training process; using the group query attention mechanism to balance the reasoning efficiency and performance; using the SwiGLU function as the activation function of the feedforward neural network layer to improve the model's expressiveness; and using RoPE relative rotation position encoding instead of the traditional trigonometric function absolute position encoding to more efficiently process long sequence data.

[0183] Exemplarily, a large language model structure with 7 billion parameters is constructed, which includes 30 Transformer layers, a hidden layer dimension of 4096, an attention layer query head number of 32, a key and value head number of 8, and a vocabulary size of 12,000.

[0184] S3. Pre-train large language models based on large-scale unsupervised text data.

[0185] For example, Figure 7 A flow chart of a self-distillation large language model pre-training method provided in this embodiment.

[0186] S301: Design a suitable scheduling strategy for the initial pre-training data set, determine the order in which each data source is used for training, and use different data source mixing ratios at different stages.

[0187] S302: According to the data mixing ratio that can obtain the most ideal performance, data is sampled from different data sources to adjust the overall distribution of the pre-training data to obtain a pre-training data set after mixing.

[0188] Preferably, the capabilities of certain corresponding models can be enhanced by increasing the proportion of specific data sources, such as using more math and code data to enhance reasoning capabilities in the later stages of pre-training.

[0189] S303: Based on the proportioned and mixed pre-training dataset, a batch dataset sequence available for training is constructed.

[0190] In order to adapt the context length of the model and save computing resources, multiple pieces of data must first be concatenated and padded to a fixed length using the minimum number of special identifiers to form a training sample. Then a large number of training samples are shuffled and randomly divided into several small batches of data to obtain a batch data set sequence that can be used for training. In the pre-training process of large language models, the batch size is usually set to a higher value.

[0191] S304: Initialize the weight parameters of the embedding layer, decoder layer, and normalization layer of the generative large language model using a random method.

[0192] S305: Divide the pre-training process of the large language model into N training stages, where each training stage includes learning of at least B batches of data.

[0193] S306: Iteratively train the large language model using a batch data set sequence. The core process of large language model pre-training includes operations such as inputting batch data, forward propagation to calculate loss, backward propagation to calculate gradient, and optimizer updating large language model parameters. Specifically:

[0194] The language modeling task is used as the pre-training task. For each word sequence x={x 1 ,…,x n}, the language modeling task is based on the word sequence x before the current position of the sequence <t , using autoregression to predict the next target word x t .

[0195] For each batch of data, the word sequence is input into the large language model, and the forward propagation obtains the predicted probability distribution of the target word. The loss between the model prediction distribution and the actual annotation result is calculated based on the loss function. The back propagation algorithm is used to calculate the gradient of the loss function relative to the model parameters, and the gradient information is propagated from the output layer back to the input layer in sequence. After obtaining the gradient, the optimization algorithm is used to update the large language model parameters with the goal of minimizing the loss. Each time the model processes a batch of data, the model parameters are updated.

[0196] For example, the likelihood function is As the loss function, AdamW optimizer is used to implement parameter update.

[0197] S4. The model trained in each stage is used as the teacher model to synthesize the self-distillation mixed dataset.

[0198] For example, Figure 8 A flowchart for constructing a self-distillation mixed dataset provided for this embodiment.

[0199] S401: Maintain a real data pool P, which is initialized to be empty.

[0200] S402: Real data sampling. For each piece of data in each batch of data, there is a first probability α of being selected to enter the data pool P. α can be unified as a fixed value for all samples, or it can vary depending on the sample category label.

[0201] This embodiment enables the samples in the data pool P to reflect the distribution characteristics of the original data set, and can adjust the sampling probability according to the representativeness and priority of different categories, thereby optimizing the balance of learning of each category during the model training process, which helps to enhance the generalization ability of the final model and make it more robust when facing new data.

[0202] S403: Model archiving. The last training round of each training phase is used as the model archiving point. When the training is completed, the current model parameters are saved.

[0203] The synthesis process of the self-distillation mixed data set includes the following steps (S404-S409):

[0204] S404: After the current training phase is completed, randomly sample b data samples from the data pool P as candidate synthetic samples, where b is equal to the batch data size of normal training.

[0205] S405: Using a special identifier indicating the end of a sentence as a delimiter, perform text segmentation on the b data samples to obtain multiple original text data of varying lengths and quantities. Since the pre-training samples are formed by splicing multiple data and fillers with a fixed length according to a certain rule, it is necessary to reversely restore the original text data according to the splicing rule.

[0206] S406: For each piece of original text data, a second probability β is selected for synthesis, and the probability 1-β remains unchanged.

[0207] The proportion of distilled synthetic data needs to be adjusted dynamically at different training stages. In the early stages of training, the model is not mature enough and needs to learn basic patterns and features from purer real-world data in order to build a robust basic representation capability. As training progresses, the probability of data synthesis β is gradually increased so that the model gradually adapts to the data form generated by itself, thereby improving its generalization ability and robustness.

[0208] S407: For each source data to be synthesized, select the position of the truncation point. The third probability of each character or word in the text being selected as the truncation point obeys the normal distribution law to ensure the diversity and randomness of the synthesized data while maintaining a high content of real-world knowledge.

[0209] S408: The text fragment before and after the truncation point is used as a query and input into the model saved after the training of this stage to predict the subsequent content of the sentence.

[0210] S409: The text before the source data cutoff point is concatenated with the generated text of the large language model to form a complete distilled mixed text sentence. This sentence not only retains the authenticity of the original data, but also ensures the distribution consistency of the large language model.

[0211] S410: Merge the original text and the distilled mixed text data to obtain a self-distilled mixed data set D that combines real-world knowledge and self-distillation synthesis guided by real data.

[0212] S411: Delete b data samples in the self-distillation mixed dataset D from the real data pool P.

[0213] S5. The self-distilled mixed dataset is used to train the student model in the i+k stage.

[0214] S501: The self-distilled mixed dataset D obtained in the previous step is further processed into a batch data format that can be used for model training, and is used as a batch data of the i+kth stage to participate in the training of the i+kth stage. The above process can be regarded as the process of knowledge transfer and generalization from the previous teacher model to the growing student model.

[0215] Dynamically adjust the knowledge transfer interval k according to the training process. In the early stage of pre-training of large language models, the learning rate is high, the loss curve drops quickly, and the learning speed is fast, so the frequency of knowledge transfer (i.e., the knowledge transfer interval k) should be set to a smaller value. As the training progresses, the learning rate of the model gradually slows down. At this time, the knowledge transfer interval k should be increased to adapt to the learning process of the model and improve the training efficiency, thereby enhancing the robustness of the pre-training model.

[0216] Preferably, this embodiment provides an adaptive selection scheme for the knowledge transfer interval k:

[0217]

[0218] Among them, K and c are hyperparameters. K represents the maximum knowledge gap, and c controls the growth rate and approximation speed of k as the training stage i grows, and c>0. It is the round-up symbol, indicating the smallest integer greater than this value.

[0219] S6. After each training phase is completed, execute steps S4 and S5.

[0220] S7. Repeat step S4 to continuously adjust and optimize the parameters of the large language model until the model loss converges or the preset number of training rounds is reached to obtain the pre-trained large language model.

[0221] In summary, the technical features and corresponding beneficial effects of this embodiment include:

[0222] 1. This embodiment proposes a self-distillation method for pre-training a large language model, which promotes the self-improvement of the large language model through a distillation iteration mechanism during its growth. During pre-training, not only new training data is used, but also real data from the previous teacher model and synthetic data generated by the teacher model under the guidance of real data. This self-distillation method of pre-training through self-supervision improves training stability, simplifies model compression and knowledge transfer processes, and can better generalize to downstream tasks, improving the reliability and generalization of pre-trained models.

[0223] 2. This embodiment proposes a novel distilled mixed data synthesis method. First, a truncation point is selected, and the text before the truncation point remains unchanged, while the text after the truncation point is re-synthesized using the teacher model. Compared with using the teacher model to synthesize complete data, the method of this embodiment uses real-world data to provide positive guidance in the generation process, which reduces the hallucination substitution and error accumulation of synthetic data and reduces the distribution deviation from real-world data. In addition, by dynamically adjusting the proportion of synthetic data in the mixed data set at different training stages, overfitting of the feature distribution of a previous specific period is avoided. At the same time, when faced with data drift in practical applications, it can show stronger adaptability and generalization.

[0224] 3. This embodiment proposes a dynamic knowledge transfer interval selection strategy. The distilled mixed data set synthesized by the teacher model will be adaptively transferred to the training of the student model in the subsequent stage according to the model training process. In the initial stage of pre-training, the knowledge growth rate is fast and the knowledge transfer interval is set small; as the training progresses, the learning rate of the model gradually slows down, and the time interval of knowledge transfer is correspondingly increased. This dynamic adjustment strategy can more effectively adapt to the growth trend of the model and enhance the robustness of the pre-trained model.

[0225] Reference Fig. 9 The embodiment of the present application also provides a large language model pre-training device, which can implement the above-mentioned large language model pre-training method, and the device includes:

[0226] A data processing unit, used to obtain multiple types of text data and pre-process the text data to obtain an initial pre-training data set;

[0227] A model building unit, which is used to build a generative large language model based on the decoder architecture of the Transformer model;

[0228] A stage training unit, used for performing multiple stages of training on the generative large language model according to the initial pre-training data set;

[0229] A data set synthesis unit, used for synthesizing a self-distilled mixed data set according to the teacher model and the initial pre-trained data set; wherein the generative large language model trained in each stage is used as the teacher model;

[0230] A distillation training unit, configured to train a student model using the self-distillation mixed data set; wherein the student model is the generative large language model that is k stages behind the teacher model, and k is a positive integer;

[0231] A cyclic training unit is used to return to the step of synthesizing a self-distilled mixed data set based on the teacher model and the initial pre-training data set until a preset number of stages of training are completed, and the student model finally obtained by training is used as the pre-trained generative large language model.

[0232] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0233] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned large language model pre-training method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0234] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0235] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0236] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0237] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes a large language model pre-training method of the embodiment of this application;

[0238] Input / output interface 1003, used to implement information input and output;

[0239] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0240] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0241] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0242] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned large language model pre-training method.

[0243] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0244] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0245] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0246] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0247] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0248] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0249] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0250] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0251] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0252] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0253] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0254] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0255] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A large language model pre-training method, characterized in that: The method comprises the following steps: Acquire multiple types of text data, and preprocess the text data to obtain an initial pre-training data set; Build a generative large language model based on the decoder architecture of the Transformer model; Performing multiple stages of training on the generative large language model according to the initial pre-training dataset; A self-distilled mixed dataset is synthesized according to the teacher model and the initial pre-training dataset; wherein the generative large language model trained in each stage is used as the teacher model; Using the self-distillation mixed data set to train a student model; wherein the student model is the generative large language model that is k stages behind the teacher model, and k is a positive integer; Return to the step of synthesizing the self-distilled mixed data set based on the teacher model and the initial pre-training data set until the preset number of stages of training are completed, and the student model finally obtained by the training is used as the pre-trained generative large language model.

2. A large language model pre-training method according to claim 1, characterized in that: The decoder architecture based on the Transformer model constructs a generative large language model, including the following steps: The construction step is performed on the causal decoder architecture of the Transformer model to construct the generative large language model; the construction step includes: Normalize each sublayer of the causal decoder architecture of the Transformer model using the pre-root mean square normalization RMSNorm algorithm; Setting a group query attention mechanism for the causal decoder architecture of the Transformer model; Determining the activation function of the feedforward neural network layer of the causal decoder architecture of the Transformer model as a SwiGLU function; The position encoding of the causal decoder architecture of the Transformer model is set to RoPE relative rotation position encoding.

3. A large language model pre-training method according to claim 1, characterized in that: The step of training the generative large language model in multiple stages according to the initial pre-training data set comprises the following steps: Determining the mixing ratios of different types of text data at different stages in the initial pre-training data set; Sampling from the initial pre-training data set according to the mixing ratio to obtain a proportioned mixed pre-training data set; The texts in the mixed pre-training data set are concatenated and padded to a set length to obtain multiple texts of the same length as training samples; Randomly divide the training samples into a number of batches of data; Initializing weight parameters of the embedding layer, the decoder layer, and the normalization layer of the generative large language model using a random method; The generative large language model is trained for N stages according to the batch data; wherein each stage of training uses at least B batches of data; and N and B are positive integers.

4. A large language model pre-training method according to claim 3, characterized in that: The training of the generative large language model for N stages according to the batch data comprises the following steps: Each stage of training performs the following steps: Inputting the word unit sequence in each batch of data into the generative large language model, and then forward propagating to obtain the predicted probability distribution of the target word unit; Calculating the loss value between the predicted distribution of the generative large language model and the actual annotation result based on the loss function; Calculating the gradient of the loss function relative to the parameters of the generative large language model using a back-propagation algorithm; Propagate the gradient from the output layer of the generative large language model back to the input layer of the generative large language model in sequence; According to the gradient, minimizing the loss value is used as a training goal, and an optimization algorithm is used to update the parameters of the generative large language model; wherein, each time the generative large language model processes a batch of the data, the parameters of the generative large language model are updated accordingly.

5. A large language model pre-training method according to claim 1, characterized in that: The step of synthesizing a self-distilled mixed dataset based on the teacher model and the initial pre-training dataset comprises the following steps: Initialize an empty data pool; During the training of each stage, training samples are sampled from each batch of data of each stage with a first probability, and then the sampled training samples are added to the data pool; After each stage of training, the current parameters of the generative large language model are saved to obtain the teacher model; After each stage of training, candidate synthetic samples are randomly sampled from the data pool; wherein the data volume of the candidate synthetic samples is the same as the data volume of the batch data; Performing text segmentation on each of the training samples in the candidate synthetic samples using a separator to obtain a plurality of original text data; Selecting data to be synthesized from each of the original text data with a second probability; Selecting characters or words of each piece of the data to be synthesized as a cutoff point with a third probability; wherein the third probability obeys a normal distribution; Taking the text at and before the truncation point as a query text, and inputting the query text into the teacher model at the current stage; Outputting predicted text corresponding to the query text using the teacher model; splicing the query text and the predicted text to obtain a synthesized text; The synthesized text and the remaining original data in each of the original data that are not selected as the data to be synthesized are synthesized into the self-distillation mixed data set.

6. A large language model pre-training method according to claim 1, characterized in that: The method of training a student model using the self-distillation mixed data set comprises the following steps: Calculating a knowledge transfer interval; wherein k represents the knowledge transfer interval; The calculation formula of the knowledge transfer interval includes: Among them, K and c are hyperparameters; K represents the maximum knowledge interval, c is used to control the growth rate and approximation speed of k as the training stage i grows, c>0; is the rounding symbol; f(i) represents the intermediate function; Selecting the generative large language model that is k stages behind the teacher model as the student model; The student model is trained using the self-distillation mixed dataset.

7. A large language model pre-training method according to any one of claims 1 to 6, characterized in that: The method further comprises the following steps: Using a text dataset of the target type to train the student model finally trained, thereby obtaining a target large language model; The target type of text is input into the target large language model, and then the target large language model is used to output a prediction result corresponding to the target type of text.

8. A large language model pre-training device, characterized in that: The device comprises: A data processing unit, used to obtain multiple types of text data and pre-process the text data to obtain an initial pre-training data set; A model building unit, which is used to build a generative large language model based on the decoder architecture of the Transformer model; A stage training unit, used for performing multiple stages of training on the generative large language model according to the initial pre-training data set; A data set synthesis unit, used for synthesizing a self-distilled mixed data set according to the teacher model and the initial pre-trained data set; wherein the generative large language model trained in each stage is used as the teacher model; A distillation training unit, configured to train a student model using the self-distillation mixed data set; wherein the student model is the generative large language model that is k stages behind the teacher model, and k is a positive integer; A cyclic training unit is used to return to the step of synthesizing a self-distilled mixed data set based on the teacher model and the initial pre-training data set until a preset number of stages of training are completed, and the student model finally obtained by the training is used as the pre-trained generative large language model.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements a large language model pre-training method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, a large language model pre-training method as described in any one of claims 1 to 7 is implemented.

Images