Generation method and device of large language model, electronic equipment and storage medium

Abstract

The invention provides a large language model generation method and device, electronic equipment and a storage medium, and the generation method comprises the following steps: obtaining a text training data set, and generating a Bin file and an Idx file; a big language model is created, the big language model comprises an Encoder-Only framework with 30 layers of Transformers, the big language model is used for receiving the vocabulary and a plurality of index values corresponding to the vocabulary, and the big language model is used for inputting all the received index values into the embedding layer; the embedding layer converts any index value i into word embedding vectors based on a preset word embedding matrix, all the word embedding vectors are input into the first layer Encoder-Only architecture, and the next layer is used for receiving the word embedding vectors output by the upper layer and passes through a multi-head attention mechanism; and training the large language model based on the Bin file and the Idx file. The generation method can generate a large language model.

Description

Technical Field

[0001] The present invention relates to the technical field of large language models, and in particular to a method, device, electronic device and storage medium for generating a large language model. Background Art

[0002] Machine learning is a subfield of artificial intelligence (AI), and its core is to enable computer systems to improve their performance by learning from data. In machine learning, instead of directly programming computers to complete tasks, large amounts of data are provided, allowing the machine to find hidden patterns or laws through the data, and then use these laws to predict new, unknown data.

[0003] Deep learning is a subfield of machine learning that attempts to mimic the way the human brain works, creating so-called artificial neural networks to process data. These neural networks contain multiple processing layers, hence the name "deep" learning. Deep learning models are able to learn and represent large numbers of complex patterns, which makes them very effective in tasks such as image recognition, speech recognition, and natural language processing.

[0004] Large language models are one of the applications of deep learning, especially in the field of Natural Language Processing (NLP). The goal of these models is to understand and generate human language. To achieve this goal, the model needs to be trained on a large amount of text data to learn various patterns and structures of the language.

[0005] Therefore, how to generate a large language model becomes a problem that needs to be solved urgently. Summary of the invention

[0006] The object of the present invention is to provide a method, device, electronic device and storage medium for generating a large language model.

[0007] In order to achieve one of the above-mentioned purposes of the invention, an embodiment of the present invention provides a method for generating a large language model, comprising the following steps: obtaining a text training data set, using a word segmenter to segment and encode the text in the text training data set, and then writing the encoded data into a Bin file, and writing the starting position of each line of text data in the Bin file into an Idx file; creating a large language model, the large language model includes an Encoder-Only architecture of a 30-layer Transformer, the large language model is used to receive a vocabulary and multiple index values ​​corresponding to the vocabulary, the large language model is used to input all received index values ​​into an embedding layer, and the embedding layer converts any index value i into a word embedding vector based on a preset word embedding matrix E(V,d) , where V is the size of the vocabulary, d is the dimension of the word embedding vector, and each row in the word embedding matrix E(V,d) corresponds to the embedding representation of a word. is the word with index value i in the vocabulary; all word embedding vectors are input into the first layer of Encoder-Only architecture, and the 2nd to 30th layers are used to receive the word embedding vector output by the previous layer, and pass through the multi-head attention mechanism to capture long-distance dependencies; based on the Bin file and Idx file, the large language model is trained.

[0008] As a further improvement of an embodiment of the present invention, the Encoder-Only architecture is used to obtain the embedding vector of any input word The corresponding Q matrix, K matrix and V matrix, , , ,in, , and are all trainable parameter matrices; in all Encoder-Only architectures, any word embedding vector The corresponding attention score is V, where is the dimension value of matrix K.

[0009] As a further improvement of an embodiment of the present invention, the text training data set at least includes: The Pile data, Skypile data and Redpajama data.

[0010] As a further improvement of an implementation manner of the present invention, the training of the large language model based on the Bin file and the Idx file specifically includes: training the large language model based on the Bin file, the Idx file and the Megatron-LM framework.

[0011] As a further improvement of an implementation manner of the present invention, the obtaining of a text training data set specifically includes: obtaining a text training data set, and performing data cleaning and data deduplication processing on the text training data set.

[0012] As a further improvement of an embodiment of the present invention, the Encoder-Only architecture includes an input encoding part and a feature encoding part; the prefix Tokens of the input encoding part adopt a random masking strategy and are used to provide context information, and the masking probability adopted by the prefix Tokens is ; The suffix tokens of the feature encoding part adopt a uniform masking strategy and the uniform masking probability adopted by the suffix tokens is ; .

[0013] As a further improvement of an implementation manner of the present invention, the uniform masking strategy includes: a uniform sampling part and a secondary masking part.

[0014] The embodiment of the present invention also provides a device for generating a large language model, including the following modules: a data acquisition module, used to acquire a text training data set, use a word segmenter to segment and encode the text in the text training data set, and then write the encoded data into a Bin file, and write the starting position of each line of text data in the Bin file into an Idx file; a model creation module, used to create a large language model, the large language model includes an Encoder-Only architecture of a 30-layer Transformer, the large language model is used to receive a vocabulary and multiple index values ​​corresponding to the vocabulary, the large language model is used to input all received index values ​​into an embedding layer, and the embedding layer converts any index value i into a word embedding vector based on a preset word embedding matrix E(V,d) , where V is the size of the vocabulary, d is the dimension of the word embedding vector, and each row in the word embedding matrix E(V,d) corresponds to the embedding representation of a word. is the word with index value i in the vocabulary; all word embedding vectors are input into the first layer of Encoder-Only architecture, and the 2nd to 30th layers are used to receive the word embedding vector output by the previous layer and pass through the multi-head attention mechanism to capture long-distance dependencies; a training module is used to train the large language model based on the Bin file and the Idx file.

[0015] An embodiment of the present invention further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in the above-mentioned method for generating a large language model are implemented.

[0016] An embodiment of the present invention further provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned method for generating a large language model are implemented.

[0017] Compared with the prior art, the technical effect of the present invention is that: the embodiment of the present invention provides a method, device, electronic device and storage medium for generating a large language model, the generation method includes the following steps: obtaining a text training data set, generating a Bin file and an Idx file; creating a large language model, the large language model includes an Encoder-Only architecture of 30 layers of Transformer, the large language model is used to receive a vocabulary and multiple index values ​​corresponding to the vocabulary, the large language model is used to input all received index values ​​into an embedding layer, the embedding layer converts any index value i into a word embedding vector based on a preset word embedding matrix, all word embedding vectors are input into the first layer of the Encoder-Only architecture, the next layer is used to receive the word embedding vector output by the previous layer, and passes through a multi-head attention mechanism; based on the Bin file and the Idx file, the large language model is trained. The generation method can generate a large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flow chart of a method for generating a large language model in an embodiment of the present invention; Figure 2 , Figure 3 , Figure 4 and Figure 5 4 is a schematic diagram of a large language model in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The present invention will be described in detail below in conjunction with the various embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional changes made by a person skilled in the art based on these embodiments are all within the scope of protection of the present invention.

[0020] As used herein, terms indicating spatial relative positions such as "upper," "above," "lower," and "below" are used for the purpose of convenience to describe the relationship of one unit or feature relative to another unit or feature as shown in the accompanying drawings. Terms of spatial relative position may be intended to include different orientations of the device in use or operation in addition to the orientation shown in the drawings. For example, if the device in the figure is turned over, the units described as being "below" or "beneath" other units or features will be located "above" the other units or features. Thus, the exemplary term "below" can encompass both the above and below orientations. The device may be oriented in other ways (rotated 90 degrees or in other orientations), and the spatially relative descriptors used herein interpreted accordingly.

[0021] Furthermore, it should be understood that although the terms first, second, etc. may be used herein to describe various elements or structures, these described objects should not be limited by these terms. These terms are only used to distinguish these described objects from each other. For example, a first rack may be referred to as a second rack, and similarly, a second rack may also be referred to as a first rack, without departing from the scope of protection of this application.

[0022] The embodiment of the present invention provides a method for generating a large language model, such as Figure 1 As shown, the following steps are included: Step 101: Obtain a text training data set, use a word segmenter to segment and encode the text in the text training data set, then write the encoded data into a Bin file, and write the starting position of each line of text data in the Bin file into an Idx file; Here, the text training data set can contain Chinese data and English data. The text can be first segmented and encoded using a word segmenter, and the encoded data is written into a Bin file, and the starting position of each line of text data, that is, the offset, is written into an Idx file. The Bin file here contains the actual data required for model training, stored in binary format, and the Idx file contains the index of the data in the Bin file, indicating the location of the data in the Bin file. The combination of these two file formats makes the training process more efficient and flexible when processing large-scale data sets. Finally, the scattered Bin files and Idx files are merged into a total Bin file and Idx file for model training.

[0023] Step 102: Create a large language model, the large language model includes an Encoder-Only architecture with 30 layers of Transformers, the large language model is used to receive a vocabulary and multiple index values ​​corresponding to the vocabulary, the large language model is used to input all received index values ​​into an embedding layer, and the embedding layer converts any index value i into a word embedding vector based on a preset word embedding matrix E(V,d) , where V is the size of the vocabulary, d is the dimension of the word embedding vector, and each row in the word embedding matrix E(V,d) corresponds to the embedding representation of a word. is the word with index value i in the vocabulary; all word embedding vectors are input into the first layer of Encoder-Only architecture, and the 2nd to 30th layers are used to receive the word embedding vector output by the previous layer and pass through the multi-head attention mechanism to capture long-distance dependencies.

[0024] Since the large language model needs to know the target sequence length before inference, an additional length prediction module can be added during the model fine-tuning phase, which predicts the target sequence length based on the input text sequence. During inference, the model will use multiple decoding steps to predict the final target sequence. Specifically, given the total decoding steps T and the target sequence length L, the model will start decoding from a fully masked sequence. In the following decoding steps, each decoded sequence will be masked with a lower probability than the previous step. The specific number of masked tokens can be calculated by the following formula: N=(1-t / T)×L, where N represents the number of masked tokens and t represents the number of decoding steps in the current iteration. Such multiple masking and re-predicted decoding steps can help the model better optimize the target sequence. Optionally, T=10.

[0025] The masking operation on the input text is divided into two parts. For a text sequence with a maximum length of Len1, we decompose it into two parts with lengths of Len2 and Len3, where Len2+Len3=Len1. The former represents prefix tokens, which are used to provide contextual information to help the model understand the entire text sequence. The masking operation of this part adopts a random masking strategy. The latter represents suffix tokens, which helps the model better learn generation capabilities. The masking operation of this part adopts a uniform masking strategy.

[0026] Instead of using a fixed masking probability for prefix tokens, we link it to a uniform masking probability for suffix tokens. Specifically, β X =0.3−β Y ×0.2, where β X represents the mask probability adopted by prefix tokens, β Y Represents the uniform masking probability of suffix tokens. The purpose of this is that when there are more masked words in the suffix tokens, that is, β Y When it is larger, fewer words are masked in prefix tokens, which can provide more context information.

[0027] Step 103: Based on the Bin file and the Idx file, the large language model is trained.

[0028] Here, for each word index i in the input sequence, its corresponding word embedding vector is obtained by looking up the word embedding matrix E , this process can be expressed as: Since the Transformer model has no built-in sequential information, the output vector of the embedding layer is positionally encoded to form the final input representation.

[0029] In this embodiment, the Encoder-Only architecture is used to obtain the embedding vector of any input word The corresponding Q matrix, K matrix and V matrix, , , ,in, , and are all trainable parameter matrices; in all Encoder-Only architectures, any word embedding vector The corresponding attention score is V, where is the dimension value of matrix K.

[0030] like Figure 2 As shown in the figure, each word vector in the input sequence is first multiplied by three matrices to obtain Q, K, and V matrices.

[0031] When calculating the attention score, the Q matrix corresponding to the vector will be multiplied by the K matrix. In order to prevent the inner product from being too large, the multiplication result will be divided by the square root of the word vector dimension. Finally, the result will be multiplied by the corresponding V matrix after the Softmax calculation to obtain the self-attention score of the word vector and other word vectors. The calculation formula is V.

[0032] A word vector will calculate the above self-attention score with all word vectors including itself, and add up the self-attention scores to get an output of the word vector in the self-attention layer. Since the multi-head self-attention mechanism is used, each word vector will have multiple outputs, and these outputs are concatenated to get the final output. Figure 3 As you can see, the self-attention output will pass through the feedforward neural network, which processes the input through weights and activation functions to gradually extract complex features. The vector processed by the Transformer layer is then subjected to linear changes in the model output layer and Softmax calculation to obtain the final prediction result. In this embodiment, the text training data set includes at least: The Pile data, Skypile data and Redpajama data.

[0033] The Pile data is not just a single dataset, but consists of more than 20 different subsets, including books, news, Wikipedia, GitHub code, scientific papers, conversation records, etc. Each subset has its own unique corpus characteristics, covering a variety of text types and topics, which helps to train models that can understand multiple language scenarios.

[0034] SkyPile data is a large-scale dataset that contains a large amount of text information and is widely used in various natural language processing tasks such as machine translation, text classification, and sentiment analysis.

[0035] The ‌RedPajama dataset‌ is primarily used for training large language models. The dataset contains more than 1.2 trillion tokens and was collected strictly according to the method described in the LLaMA paper to create a high-quality pre-training dataset. The purpose of the RedPajama dataset is to provide a high-quality, large-scale pre-training dataset to support the construction of infrastructure for large-scale generative models‌ 12.

[0036] These datasets cover web pages, books, scientific literature, financial fields, etc.

[0037] In this embodiment, the training of the large language model based on the Bin file and the Idx file specifically includes: training the large language model based on the Bin file, the Idx file and the Megatron-LM framework.

[0038] In this embodiment, the acquiring of the text training data set specifically includes: acquiring the text training data set, and performing data cleaning and data deduplication processing on the text training data set.

[0039] In this embodiment, the Encoder-Only architecture includes an input encoding part and a feature encoding part; the prefix Tokens of the input encoding part adopt a random masking strategy and are used to provide context information, and the masking probability adopted by the prefix Tokens is ; The suffix tokens of the feature encoding part adopt a uniform masking strategy and the uniform masking probability adopted by the suffix tokens is ; .

[0040] In this embodiment, the uniform masking strategy includes: a uniform sampling part and a secondary masking part.

[0041] The Uniform Sampling (US) part is as follows: 1 patch is strictly randomly sampled from each 2×2 grid. This method retains equivalent elements in multiple non-overlapping local windows and supports pyramid-based Vit. The Secondary Masking (SM) part is as follows: ‌ Randomly mask a part of the sampled area as learnable Tokens.

[0042] Here, the large language model includes the Transformer Encoder-Only architecture, which includes an input encoding part and a feature encoding part; the prefix Tokens of the input encoding part adopt a random masking strategy and are used to provide context information, and the masking probability of the prefix Tokens is ; The suffix tokens of the feature encoding part adopt a uniform masking strategy and the uniform masking probability adopted by the suffix tokens is ; .

[0043] Here, the Encoder-Only architecture only selects the encoder part of the Transformer to receive input text and generate context-related features. The Encoder-Only architecture consists of three parts, such as Figure 5 As shown, they are the input encoding part, feature encoding part and task processing part respectively.

[0044] The input encoding part includes three processes: word segmentation, vectorization, and adding position encoding.

[0045] The feature encoding part is composed of multiple identical encoder blocks stacked together, each of which contains a self-attention module and a fully connected feedforward module. The task processing module is a module specially designed for task requirements, which can be designed by the user according to task requirements.

[0046] The pre-training and inference stages of the Encoder-Only architecture model are consistent in terms of input encoding and feature encoding, while the task processing part needs to be customized according to the different characteristics of the task.

[0047] In the input encoding part, the original input text will be broken down into a sequence of tokens by the tokenizer, and then mapped into a vector sequence through the vocabulary and word embedding matrix to ensure that the text information can be expressed digitally. Then, in order to preserve the order information of the words in the text, each vector sequence will be given a positional encoding.

[0048] In the feature encoding part, the previously obtained vector sequence will pass through a series of encoding modules in turn, which further extract and deepen the text features through the self-attention mechanism and feedforward network.

[0049] The task processing part is generally different in the pre-training stage and the downstream task adaptation stage. In the pre-training stage, the model usually uses a fully connected layer as the output head to complete tasks such as mask prediction. In the downstream task adaptation stage, the output head will be customized according to the specific task requirements.

[0050] The random masking strategy (PRBS, Pseudo Random Binary Sequence) has a "random" characteristic because in the PRBS code stream, the binary numbers "0" and "1" appear randomly, but it is different from the real random code. This "random" characteristic is only local, that is, within the cycle, "0" and "1" appear randomly, but the code stream in each cycle is exactly the same. The random masking strategy is used to provide contextual information and help the model understand the entire text sequence. The uniform masking strategy is a technology used in Masked AutoEncoder (MAE) pre-training. The uniform masking strategy is used to help the model better learn generation capabilities.

[0051] The random masking probability of prefix tokens is not a fixed value, but is linked to the uniform masking probability of suffix tokens. Specifically, the relationship between the two is ,in, Indicates the mask probability used by prefix Tokens, It indicates the uniform masking probability used by suffix tokens. As can be seen from the above formula, when the uniform masking probability of suffix tokens The larger the value, the higher the probability of masking the prefix tokens. That is to say, if there are more masked words in the suffix tokens, there will be fewer masked words in the prefix tokens, which can provide more context information and help the model better predict the entire text sequence.

[0052] from Figure 4It can be seen that during the model training process, the Transformer layer of the model is divided into the input encoding part and the feature encoding part. The Attention mechanism helps the model capture the dependencies between different positions in the input sequence when generating or processing the sequence. By calculating the correlation between each word in the input sequence, the model can better understand the context and generate more coherent output. Since the number of masked words in the input encoding part is not large, the model obtains rich dependencies between different positions in the input sequence, so the Attention mechanism only needs to focus on itself. Since the input of the feature encoding part masks more words than the input encoding part, the positional dependencies between the sequences obtained by the model are less, so in addition to paying attention to itself, the Attention mechanism also needs to pay attention to the content of the input encoding part, that is, the context information. The training of this large language model uses data parallelism, model parallelism, pipeline parallelism and other technologies provided by Megatron-LM to accelerate the training process.

[0053] In large language models, the Attention mechanism helps the model capture the dependencies between different positions in the input sequence when generating or processing sequences. The Attention mechanism is a common feature of the Transformer layer of large models. The mechanism calculates the degree of influence (i.e., weight) of each input element on the current output and dynamically assigns different weights to focus on different parts of the input data. By calculating the correlation between words in the input sequence, the model can better understand the context and generate more coherent output. Since the masking probability of the prefix tokens used is low, the number of masked words is not large, and the model obtains rich dependencies between different positions in the input sequence. Therefore, the Attention mechanism only needs to focus on itself. Since the suffix tokens mask more words than the first part, the positional dependencies between the sequences obtained by the model are less. Therefore, in addition to paying attention to itself, the Attention mechanism also needs to pay attention to the content of the first part, that is, the contextual information.

[0054] In the experiment of this large language model, multiple GPUs were prepared: , ,..., First, use , ,..., To train the large language model in step 102, then use , ,..., Execute step 103 to ensure that all graphics cards are fully utilized and improve the utilization efficiency of the computing server, where N is a natural number, N≥2.

[0055] The parallel generation model training breaks the serial order of sequence generation and can decode the entire prediction sequence at the same time or generate prediction sequences in batches, which greatly improves the reasoning speed of the model while ensuring the quality of the model generation results.

[0056] Embodiment 2 of the present invention provides a device for generating a large language model, comprising the following modules: A data acquisition module is used to acquire a text training data set, segment and encode the text in the text training data set using a word segmenter, and then write the encoded data into a Bin file, and write the starting position of each line of text data in the Bin file into an Idx file; A model creation module is used to create a large language model, wherein the large language model includes an Encoder-Only architecture with 30 layers of Transformers, wherein the large language model is used to receive a vocabulary and multiple index values ​​corresponding to the vocabulary, wherein the large language model is used to input all received index values ​​into an embedding layer, wherein the embedding layer converts any index value i into a word embedding vector based on a preset word embedding matrix E(V, d) , where V is the size of the vocabulary, d is the dimension of the word embedding vector, and each row in the word embedding matrix E(V,d) corresponds to the embedding representation of a word. is the word with index value i in the vocabulary; all word embedding vectors are input into the first layer of Encoder-Only architecture, and the 2nd to 30th layers are used to receive the word embedding vector output by the previous layer and pass through the multi-head attention mechanism to capture long-distance dependencies; A training module is used to train the large language model based on the Bin file and the Idx file.

[0057] Embodiment 3 of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in the method for generating a large language model in embodiment 1 are implemented.

[0058] Embodiment 3 of the present invention provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for generating a large language model in Embodiment 1 are implemented.

[0059] It should be understood that although this specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each implementation mode may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

[0060] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. Any equivalent implementation methods or changes that do not deviate from the technical spirit of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for generating a large language model, characterized in that: The following steps are involved: Obtain a text training data set, use a word segmenter to segment and encode the text in the text training data set, then write the encoded data into a Bin file, and write the starting position of each line of text data in the Bin file into an Idx file; Create a large language model, the large language model includes an Encoder-Only architecture with 30 layers of Transformers, the large language model is used to receive a vocabulary and multiple index values ​​corresponding to the vocabulary, the large language model is used to input all received index values ​​into an embedding layer, and the embedding layer converts any index value i into a word embedding vector based on a preset word embedding matrix E(V,d) , where V is the size of the vocabulary, d is the dimension of the word embedding vector, and each row in the word embedding matrix E(V,d) corresponds to the embedding representation of a word. is the word with index value i in the vocabulary; all word embedding vectors are input into the first layer of Encoder-Only architecture, and the 2nd to 30th layers are used to receive the word embedding vector output by the previous layer and pass through the multi-head attention mechanism to capture long-distance dependencies; The large language model is trained based on the Bin file and the Idx file.

2. The method for generating a large language model according to claim 1, characterized in that: The Encoder-Only architecture is used to obtain the embedding vector of any input word The corresponding Q matrix, K matrix and V matrix, , , ,in, , and All are parameter matrices that can be trained; In all Encoder-Only architectures, any word embedding vector The corresponding attention score is V, where is the dimension value of matrix K.

3. The generation method according to claim 1, characterized in that: The text training data set includes at least: The Pile data, Skypile data and Redpajama data.

4. The generation method according to claim 1, characterized in that: The training of the large language model based on the Bin file and the Idx file specifically includes: Based on the Bin file, the Idx file and the Megatron-LM framework, the large language model is trained.

5. The generation method according to claim 4, characterized in that: The obtaining of the text training data set specifically includes: A text training data set is obtained, and data cleaning and data deduplication processing are performed on the text training data set.

6. The generation method according to claim 1, characterized in that: The Encoder-Only architecture includes an input encoding part and a feature encoding part; the prefix Tokens of the input encoding part adopt a random masking strategy and are used to provide context information, and the masking probability adopted by the prefix Tokens is ; The suffix tokens of the feature encoding part adopt a uniform masking strategy and the uniform masking probability adopted by the suffix tokens is ; .

7. The generation method according to claim 6, characterized in that: The uniform mask strategy includes: Uniform sampling part and secondary mask part.

8. A device for generating a large language model, characterized in that: Includes the following modules: A data acquisition module is used to acquire a text training data set, segment and encode the text in the text training data set using a word segmenter, and then write the encoded data into a Bin file, and write the starting position of each line of text data in the Bin file into an Idx file; A model creation module is used to create a large language model, wherein the large language model includes an Encoder-Only architecture with 30 layers of Transformers, wherein the large language model is used to receive a vocabulary and multiple index values ​​corresponding to the vocabulary, wherein the large language model is used to input all received index values ​​into an embedding layer, wherein the embedding layer converts any index value i into a word embedding vector based on a preset word embedding matrix E(V, d) , where V is the size of the vocabulary, d is the dimension of the word embedding vector, and each row in the word embedding matrix E(V,d) corresponds to the embedding representation of a word. is the word with index value i in the vocabulary; all word embedding vectors are input into the first layer of Encoder-Only architecture, and the 2nd to 30th layers are used to receive the word embedding vector output by the previous layer and pass through the multi-head attention mechanism to capture long-distance dependencies; A training module is used to train the large language model based on the Bin file and the Idx file.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the steps in the method for generating a large language model according to any one of claims 1 to 7 are implemented.

10. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the method for generating a large language model according to any one of claims 1 to 7 are implemented.

Images