Methods, devices, electronic equipment, and storage media for generating large language models

Abstract

This invention provides a method, apparatus, electronic device, and storage medium for generating a large language model. The generation method includes the following steps: acquiring a text training dataset and generating a Bin file and an Idx file; creating a large language model, which contains a 30-layer Transformer Encoder-Only architecture. The large language model receives a vocabulary and multiple index values ​​corresponding to the vocabulary. It inputs all received index values ​​into an embedding layer, which converts any index value i into a word embedding vector based on a preset word embedding matrix. All word embedding vectors are input into the first layer of the Encoder-Only architecture. Subsequent layers receive word embedding vectors output from the previous layer and pass them through a multi-head attention mechanism. The large language model is then trained based on the Bin and Idx files. This generation method can generate a large language model.

Description

Technical Field

[0001] This invention relates to the field of large language model technology, and in particular to a method, apparatus, electronic device and storage medium for generating large language models. Background Technology

[0002] Machine learning is a subfield of artificial intelligence (AI). Its core is to enable computer systems to improve performance by learning from data. In machine learning, instead of directly programming the computer to complete tasks, you provide it with a large amount of data and let it find hidden patterns or rules, and then use these rules to predict new and unknown data.

[0003] Deep learning is a subfield of machine learning that attempts to mimic the way the human brain works, creating so-called artificial neural networks to process data. These neural networks contain multiple processing layers, hence the term "deep" learning. Deep learning models are able to learn and represent a large number of complex patterns, making them highly effective in tasks such as image recognition, speech recognition, and natural language processing.

[0004] Large language models are one application of deep learning, especially in the field of Natural Language Processing (NLP). These models aim to understand and generate human language. To achieve this, the models need to be trained on large amounts of text data to learn various patterns and structures of language.

[0005] Therefore, how to generate a large language model has become an urgent problem to be solved. Summary of the Invention

[0006] The purpose of this invention is to provide a method, apparatus, electronic device, and storage medium for generating large language models.

[0007] To achieve one of the aforementioned objectives, an embodiment of the present invention provides a method for generating a large language model, comprising the following steps: acquiring a text training dataset; using a word segmenter to segment and encode the text in the text training dataset; then writing the encoded data into a Bin file; and writing the starting position of each line of text data in the Bin file into an Idx file; creating a large language model, wherein the large language model contains a 30-layer Transformer Encoder-Only architecture; the large language model is used to receive a vocabulary and multiple index values ​​corresponding to the vocabulary; the large language model is used to input all received index values ​​into an embedding layer; and the embedding layer converts any index value i into a word embedding vector based on a preset word embedding matrix E(V,d). Where V is the size of the vocabulary, d is the dimension of the word embedding vector, and each row in the word embedding matrix E(V,d) corresponds to the embedding representation of a word. The word with index i is a word in the vocabulary; all word embedding vectors are input into the first layer Encoder-Only architecture, and layers 2 to 30 are used to receive word embedding vectors output from the previous layer, and pass through a multi-head attention mechanism to capture long-distance dependencies; the large language model is trained based on the Bin file and Idx file.

[0008] As a further improvement to one embodiment of the present invention, the Encoder-Only architecture is used to obtain the embedding vector of any input word. The corresponding Q matrix, K matrix, and V matrix, , , ,in, , and All are trainable parameter matrices; in all Encoder-Only architectures, any word embedding vector The corresponding attention score is V, where, Let K be the dimension value of matrix K.

[0009] As a further improvement of one embodiment of the present invention, the text training dataset includes at least: The Pile data, Skypile data, and Redpajama data.

[0010] As a further improvement to one embodiment of the present invention, the step of training the large language model based on the Bin file and Idx file specifically includes: training the large language model based on the Bin file, Idx file and Megatron-LM framework.

[0011] As a further improvement of one embodiment of the present invention, the step of obtaining the text training dataset specifically includes: obtaining the text training dataset, and performing data cleaning and deduplication processing on the text training dataset.

[0012] As a further improvement to one embodiment of the present invention, the Encoder-Only architecture includes an input encoding part and a feature encoding part; the prefix tokens of the input encoding part adopt a random masking strategy and are used to provide context information, and the masking probability of the prefix tokens is... The suffix tokens in the feature encoding part adopt a uniform mask strategy, and the probability of the uniform mask used for the suffix tokens is... ; .

[0013] As a further improvement of one embodiment of the present invention, the uniform masking strategy includes: a uniform sampling part and a secondary masking part.

[0014] This invention also provides a large language model generation device, comprising the following modules: a data acquisition module, used to acquire a text training dataset, segment and encode the text in the training dataset using a word segmenter, then write the encoded data into a Bin file, and write the starting position of each line of text data in the Bin file into an Idx file; a model creation module, used to create a large language model, the large language model containing a 30-layer Transformer Encoder-Only architecture, the large language model receiving a vocabulary and multiple index values ​​corresponding to the vocabulary, the large language model inputting all received index values ​​into an embedding layer, the embedding layer converting any index value i into a word embedding vector based on a preset word embedding matrix E(V,d). Where V is the size of the vocabulary, d is the dimension of the word embedding vector, and each row in the word embedding matrix E(V,d) corresponds to the embedding representation of a word. The word with index i is a word in the vocabulary; all word embedding vectors are input into the first layer Encoder-Only architecture, and layers 2 to 30 are used to receive word embedding vectors output from the previous layer and pass through a multi-head attention mechanism to capture long-distance dependencies; the training module is used to train the large language model based on the Bin file and Idx file.

[0015] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps in the above-described method for generating a large language model.

[0016] This invention also provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described method for generating a large language model.

[0017] Compared to existing technologies, the technical advantages of this invention are as follows: This invention provides a method, apparatus, electronic device, and storage medium for generating a large language model. The generation method includes the following steps: acquiring a text training dataset and generating a Bin file and an Idx file; creating a large language model, which contains a 30-layer Transformer Encoder-Only architecture. The large language model receives a vocabulary and multiple index values ​​corresponding to the vocabulary. It inputs all received index values ​​into an embedding layer. The embedding layer converts any index value i into a word embedding vector based on a preset word embedding matrix. All word embedding vectors are input into the first layer of the Encoder-Only architecture. Subsequent layers receive word embedding vectors output from the previous layer and pass them through a multi-head attention mechanism. The large language model is trained based on the Bin file and the Idx file. This generation method can generate a large language model. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the method for generating a large language model in an embodiment of the present invention;

[0019] Figure 2 , Figure 3 , Figure 4 and Figure 5 This is a schematic diagram of the large language model in an embodiment of the present invention. Detailed Implementation

[0020] The present invention will now be described in detail with reference to the embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are included within the scope of protection of the present invention.

[0021] The terms used herein, such as “above,” “over,” “below,” and “under,” indicating spatial relative position, are for illustrative purposes to describe the relationship of one unit or feature relative to another unit or feature as shown in the accompanying drawings. These terms may be intended to include different orientations of the device in use or operation other than those shown in the figures. For example, if the device in the figures is flipped, a unit described as being “below” or “under” another unit or feature would be “above” that unit or feature. Therefore, the exemplary term “below” can encompass both above and below orientations. The device may be oriented in other ways (rotated 90 degrees or otherwise), and the spatially related descriptive terms used herein will be interpreted accordingly.

[0022] Furthermore, it should be understood that although the terms "first," "second," etc., may be used herein to describe various elements or structures, the objects being described should not be limited by these terms. These terms are only used to distinguish these objects from one another. For example, a first rack may be referred to as a second rack, and similarly, a second rack may be referred to as a first rack, without departing from the scope of protection of this application.

[0023] This invention provides a method for generating large language models, such as... Figure 1 As shown, it includes the following steps:

[0024] Step 101: Obtain the text training dataset, use a word segmenter to segment and encode the text in the text training dataset, then write the encoded data into the Bin file, and write the starting position of each line of text data in the Bin file into the Idx file;

[0025] Here, the text training dataset can contain both Chinese and English data. First, a word segmenter is used to segment and encode the text. The encoded data is written into a Bin file, and the starting position (offset) of each line of text data is written into an Idx file. The Bin file contains the actual data needed for model training, stored in binary format, while the Idx file contains indices of the data in the Bin file, indicating the data's position within the Bin file. This combination of file formats makes the training process more efficient and flexible when handling large-scale datasets. Finally, the scattered Bin and Idx files are merged into a single unified Bin and Idx file for model training.

[0026] Step 102: Create a large language model containing a 30-layer Transformer Encoder-Only architecture. This large language model receives a vocabulary and multiple index values ​​corresponding to that vocabulary. It then inputs all received index values ​​into an embedding layer. The embedding layer converts any index value i into a word embedding vector based on a predefined word embedding matrix E(V,d). Where V is the size of the vocabulary, d is the dimension of the word embedding vector, and each row in the word embedding matrix E(V,d) corresponds to the embedding representation of a word. The word with index i is a word in the vocabulary; all word embedding vectors are input into the first layer of the Encoder-Only architecture, and layers 2 to 30 are used to receive word embedding vectors output from the previous layer and pass through a multi-head attention mechanism to capture long-distance dependencies.

[0027] Since large language models need to know the target sequence length before inference, an additional length prediction module can be added during the model fine-tuning stage. This module predicts the target sequence length based on the input text sequence. During inference, the model utilizes multiple decoding steps to predict the final target sequence. Specifically, given the total number of decoding steps T and the target sequence length L, the model starts decoding from a fully masked sequence. In subsequent decoding steps, each decoded sequence is masked with a relatively lower probability than the previous step. The number of masked tokens can be calculated using the following formula: N = (1 - t / T) × L, where N represents the number of masked tokens and t represents the number of decoding steps in the current iteration. These multiple masking and re-prediction decoding steps help the model better optimize the target sequence. Optionally, T = 10.

[0028] The masking operation on the input text is divided into two parts. For a text sequence with a maximum length of Len1, we decompose it into two parts with lengths of Len2 and Len3, where Len2 + Len3 = Len1. The former represents prefix tokens, which provide contextual information to help the model understand the entire text sequence; this part uses a random masking strategy. The latter represents suffix tokens, which help the model better learn its generation capabilities; this part uses a uniform masking strategy.

[0029] Instead of using a fixed mask probability for the prefix tokens mask, it correlates this probability with the uniform mask probability used for the suffix tokens. Specifically, β X =0.3−β Y ×0.2, where β X β represents the probability of using a mask for the prefix tokens. Y This represents the uniform masking probability used for the suffix tokens. The purpose of this is to ensure that the more words in the suffix tokens are masked, i.e., the higher the probability of uniform masking. Y The larger the prefix, the fewer words are masked in the prefix tokens, providing more contextual information.

[0030] Step 103: Train the large language model based on the Bin file and Idx file.

[0031] Here, for each word index i in the input sequence, its corresponding word embedding vector is obtained by looking up the word embedding matrix E. This process can be represented as: Since the Transformer model does not have built-in order information, the output vector of the embedding layer is positionally encoded to form the final input representation.

[0032] In this embodiment, the Encoder-Only architecture is used to obtain the embedding vector of any input word. The corresponding Q matrix, K matrix, and V matrix, , , ,in, , and All are trainable parameter matrices; in all Encoder-Only architectures, any word embedding vector The corresponding attention score is V, where, Let K be the dimension value of matrix K.

[0033] like Figure 2 As shown, each word vector in the input sequence is first multiplied by three matrices to obtain the Q, K, and V matrices.

[0034] When calculating the attention score, the Q matrix corresponding to the vector is multiplied by the K matrix. To prevent the inner product from becoming too large, the result of the multiplication is divided by the square root of the word vector dimension. Finally, this result is processed by Softmax and multiplied by the corresponding V matrix to obtain the self-attention score between this word vector and other word vectors. The calculation formula is as follows: V.

[0035] A word vector is compared with all other word vectors, including itself, to calculate the aforementioned self-attention score. These scores are then summed to obtain the word vector's output at the self-attention layer. Due to the multi-head self-attention mechanism, each word vector has multiple outputs; these outputs are concatenated to obtain the final output. Figure 3 As can be seen, the self-attention output passes through a feedforward neural network. This network layer processes the input through weights and activation functions, gradually extracting complex features. The vector processed by the Transformer layer then undergoes linear transformation by the model's output layer and softmax calculation to obtain the final prediction result.

[0036] In this embodiment, the text training dataset includes at least: The Pile data, Skypile data, and Redpajama data.

[0037] The Pile dataset is not a single dataset, but rather consists of more than 20 different subsets, including books, news articles, Wikipedia entries, GitHub code, scientific papers, and conversation logs. Each subset has its own unique corpus characteristics, covering a wide range of text types and topics, which helps train models capable of understanding diverse language scenarios.

[0038] The SkyPile dataset is a large-scale dataset containing a wealth of textual information and is widely used in various natural language processing tasks, such as machine translation, text classification, and sentiment analysis.

[0039] The RedPajama dataset is primarily used for training large-scale language models. This dataset contains over 1.2 trillion tokens, collected strictly according to the methods described in the LLaMA paper, aiming to create a high-quality pre-training dataset. The purpose of the RedPajama dataset is to provide a high-quality, large-scale pre-training dataset to support the infrastructure development of large-scale generative models.12

[0040] These datasets cover web pages, books, scientific literature, finance, and more.

[0041] In this embodiment, training the large language model based on the Bin file and Idx file specifically includes: training the large language model based on the Bin file, Idx file, and Megatron-LM framework.

[0042] In this embodiment, obtaining the text training dataset specifically includes: obtaining the text training dataset and performing data cleaning and deduplication on the text training dataset.

[0043] In this embodiment, the Encoder-Only architecture includes an input encoding part and a feature encoding part; the prefix tokens of the input encoding part adopt a random masking strategy and are used to provide context information, and the masking probability of the prefix tokens is... The suffix tokens in the feature encoding part adopt a uniform mask strategy, and the probability of the uniform mask used for the suffix tokens is... ; .

[0044] In this embodiment, the uniform masking strategy includes: a uniform sampling part and a secondary masking part.

[0045] The Uniform Sampling (US) part specifically involves strictly randomizing one patch from each 2×2 grid. This method preserves equivalent elements in multiple non-overlapping local windows and supports pyramid-based Vit. The Secondary Masking (SM) part specifically involves randomly masking a portion of the already sampled region as learnable Tokens.

[0046] Here, the large language model includes a Transformer Encoder-Only architecture, which comprises an input encoding part and a feature encoding part; the prefix tokens in the input encoding part employ a random masking strategy to provide contextual information, and the masking probability of the prefix tokens is... The suffix tokens in the feature encoding part adopt a uniform mask strategy, and the probability of the uniform mask used for the suffix tokens is... ; .

[0047] Here, the Encoder-Only architecture only selects the encoder part of the Transformer, which is used to receive the input text and generate context-sensitive features. The Encoder-Only architecture consists of three parts, such as... Figure 5 As shown, these are the input encoding part, the feature encoding part, and the task processing part, respectively.

[0048] The input encoding part includes three processes: word segmentation, vectorization, and adding positional encoding.

[0049] The feature encoding part consists of multiple identical encoder blocks stacked together, each of which contains a self-attention module and a fully connected feedforward module. The task processing module is specifically designed for task requirements and can be designed by the user to meet those requirements.

[0050] In the pre-training and inference phases of the Encoder-Only architecture model, the input encoding and feature encoding parts are consistent, while the task processing part needs to be customized according to the different characteristics of the task.

[0051] In the input encoding section, the original input text is broken down into a token sequence by a tokenizer, and then mapped into a vector sequence through a vocabulary and word embedding matrix to ensure that the text information is digitally represented. Next, in order to preserve the word order information in the text, each vector sequence is assigned a positional encoding.

[0052] In the feature encoding part, the previously obtained vector sequence will pass through a series of encoding modules in sequence. These modules will further extract and deepen the text features through self-attention mechanism and feedforward network.

[0053] The task processing differs between the pre-training and downstream task adaptation phases. During pre-training, the model typically uses fully connected layers as output heads for tasks such as mask prediction. In the downstream task adaptation phase, the output heads are customized according to the specific task requirements.

[0054] Pseudo Random Binary Sequence (PRBS) possesses the "random" property because the binary numbers "0" and "1" appear randomly in the PRBS bitstream. However, it differs from truly random code in that this "randomness" is only local; within a single period, "0" and "1" appear randomly, but the bitstream remains identical across all periods. PPRBS provides contextual information and helps the model understand the entire text sequence. Uniform Masking is a technique used in Masked AutoEncoder (MAE) pre-training to help the model better learn its generative capabilities.

[0055] The random mask probability of the prefix tokens is not a fixed value, but is related to the uniform mask probability of the suffix tokens. Specifically, the relationship between the two is as follows: ,in, This indicates the probability of using the mask for the prefix tokens. This represents the probability of using a uniform mask for the suffix tokens. From the formula above, it can be seen that when the uniform mask probability of the suffix tokens... The larger the value, the higher the mask probability of the prefix tokens. The smaller the number of words masked, the fewer words masked in the prefix tokens. This means that the more words masked in the suffix tokens, the fewer words masked in the prefix tokens, which provides more contextual information and helps the model better predict the entire text sequence.

[0056] from Figure 4As can be seen, during model training, the Transformer layer is divided into an input encoding part and a feature encoding part. The Attention mechanism helps the model capture the dependencies between different positions in the input sequence when generating or processing sequences. By calculating the correlations between words in the input sequence, the model can better understand the context, thereby generating more coherent outputs. Since the number of words masked in the input encoding part is not large, the dependencies between different positions in the input sequence obtained by the model are relatively rich, so the Attention mechanism only needs to focus on itself. However, since the input of the feature encoding part masks more words than the input encoding part, the positional dependencies between the sequences obtained by the model are fewer. Therefore, the Attention mechanism needs to pay attention to the content of the input encoding part, i.e., the contextual information, in addition to focusing on itself. The training of this large language model utilizes data parallelism, model parallelism, and pipeline parallelism provided by Megatron-LM to accelerate the training process.

[0057] In large language models, the attention mechanism helps the model capture dependencies between different positions in the input sequence when generating or processing sequences. Attention is a common feature of the Transformer layer in large models. This mechanism calculates the influence (i.e., weight) of each input element on the current output and dynamically assigns different weights to focus on different parts of the input data. By calculating the correlations between words in the input sequence, the model can better understand the context, thus generating more coherent outputs. Because the masking probability of prefix tokens is low, the number of masked words is not large, and the dependencies between different positions in the input sequence obtained by the model are relatively rich. Therefore, the attention mechanism only needs to focus on the prefix tokens themselves. However, because suffix tokens mask more words than the first part, the positional dependencies between the sequences obtained by the model are fewer. Therefore, the attention mechanism needs to pay attention to the first part, i.e., the contextual information, in addition to focusing on the prefix tokens themselves.

[0058] In the experiment of this large language model, multiple GPUs were prepared: , ... First, use , ... To train the large language model in step 102, and then use , ... Execute step 103 to ensure that all graphics cards are fully utilized, thereby improving the utilization efficiency of the computing server, where N is a natural number and N≥2.

[0059] This parallel generative model training breaks the serial order of sequence generation, and can decode the entire prediction sequence simultaneously or generate prediction sequences in batches, which greatly improves the inference speed of the model while ensuring the quality of the model generation results.

[0060] Embodiment 2 of the present invention provides a large language model generation device, comprising the following modules:

[0061] The data acquisition module is used to acquire the text training dataset, use a word segmenter to segment and encode the text in the text training dataset, and then write the encoded data into the Bin file, and write the starting position of each line of text data in the Bin file into the Idx file.

[0062] The model creation module is used to create a large language model, which contains a 30-layer Transformer Encoder-Only architecture. The large language model receives a vocabulary and multiple index values ​​corresponding to the vocabulary. It then inputs all received index values ​​into an embedding layer, which converts any index value i into a word embedding vector based on a preset word embedding matrix E(V,d). Where V is the size of the vocabulary, d is the dimension of the word embedding vector, and each row in the word embedding matrix E(V,d) corresponds to the embedding representation of a word. The word with index value i is a word in the vocabulary; all word embedding vectors are input into the first layer of the Encoder-Only architecture, and layers 2 to 30 are used to receive word embedding vectors output from the previous layer and pass through a multi-head attention mechanism to capture long-distance dependencies;

[0063] The training module is used to train the large language model based on the Bin file and Idx file.

[0064] Embodiment 3 of the present invention provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps in the method for generating a large language model in Embodiment 1.

[0065] Embodiment 3 of the present invention provides a storage medium storing a computer program. When the computer program is executed by a processor, it implements the steps of the method for generating a large language model in Embodiment 1.

[0066] It should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This way of describing the specification is only for clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0067] The detailed descriptions listed above are merely specific descriptions of feasible embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. All equivalent embodiments or modifications made without departing from the spirit of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating a large language model, characterized in that, Includes the following steps: Obtain the text training dataset, use a word segmenter to segment and encode the text in the text training dataset, then write the encoded data into a Bin file, and write the starting position of each line of text data in the Bin file into an Idx file; A large language model is created, comprising a 30-layer Transformer Encoder-Only architecture. This model receives a vocabulary and multiple index values ​​corresponding to the vocabulary. It then inputs all received index values ​​into an embedding layer, which converts any index value i into a word embedding vector based on a predefined word embedding matrix E(V,d). Where V is the size of the vocabulary, d is the dimension of the word embedding vector, and each row in the word embedding matrix E(V,d) corresponds to the embedding representation of a word. Let i be the word in the vocabulary with index i. All word embedding vectors are input to the first layer of the Encoder-Only architecture. Layers 2 through 30 receive word embedding vectors output from the previous layer and pass them through a multi-head attention mechanism to capture long-range dependencies. The Encoder-Only architecture includes an input encoding part and a feature encoding part. The prefix tokens in the input encoding part employ a random masking strategy to provide contextual information, and the masking probability of the prefix tokens is [value missing]. The suffix tokens in the feature encoding part adopt a uniform mask strategy, and the probability of the uniform mask used for the suffix tokens is... ; ; The large language model is trained based on the Bin and Idx files.

2. The method for generating a large language model according to claim 1, characterized in that, The Encoder-Only architecture is used to obtain the embedding vector of any input word. The corresponding Q matrix, K matrix, and V matrix, , , ,in, , and All of these are trainable parameter matrices; In all Encoder-Only architectures, any word embedding vector The corresponding attention score is V, where, Let K be the dimension value of matrix K.

3. The generation method according to claim 1, characterized in that, The text training dataset includes at least the following: The Pile data, Skypile data, and Redpajama data.

4. The generation method according to claim 1, characterized in that, The training of the large language model based on the Bin file and Idx file specifically includes: The large language model is trained based on the Bin file, Idx file, and Megatron-LM framework.

5. The generation method according to claim 4, characterized in that, The acquisition of the text training dataset specifically includes: Obtain the text training dataset, and perform data cleaning and deduplication on the text training dataset.

6. The generation method according to claim 5, characterized in that, The uniform masking strategy includes: Uniform sampling portion and secondary mask portion.

7. A device for generating a large language model, characterized in that, Includes the following modules: The data acquisition module is used to acquire the text training dataset, use a word segmenter to segment and encode the text in the text training dataset, and then write the encoded data into the Bin file, and write the starting position of each line of text data in the Bin file into the Idx file. The model creation module is used to create a large language model, which contains a 30-layer Transformer Encoder-Only architecture. The large language model receives a vocabulary and multiple index values ​​corresponding to the vocabulary. It then inputs all received index values ​​into an embedding layer, which converts any index value i into a word embedding vector based on a preset word embedding matrix E(V,d). Where V is the size of the vocabulary, d is the dimension of the word embedding vector, and each row in the word embedding matrix E(V,d) corresponds to the embedding representation of a word. Let i be the word in the vocabulary with index i. All word embedding vectors are input to the first layer of the Encoder-Only architecture. Layers 2 through 30 receive word embedding vectors output from the previous layer and pass them through a multi-head attention mechanism to capture long-range dependencies. The Encoder-Only architecture includes an input encoding part and a feature encoding part. The prefix tokens in the input encoding part employ a random masking strategy to provide contextual information, and the masking probability of the prefix tokens is [value missing]. The suffix tokens in the feature encoding part adopt a uniform mask strategy, and the probability of the uniform mask used for the suffix tokens is... ; ; The training module is used to train the large language model based on the Bin file and Idx file.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps in the method for generating a large language model according to any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for generating a large language model according to any one of claims 1 to 6.

Images