General large model-based national language large model construction method and device for operating system

Abstract

The application discloses a method and device for constructing a national language large model based on a general large model for an operating system, which is suitable for deployment to an operating system and belongs to the technical field of information. First, a pre-training language model is used to convert words in an input low-resource national language into word embedding representations, obtaining word embedding vectors, and then the word embedding vectors are input into a projection layer to form alignable embedding representation vectors. In the word table of the training corpus, the appropriate word embedding vectors of the embedding representation vectors are obtained and mapped onto the semantic space of the large model to complete the mapping of the low-resource national language to the high-resource national language. Second, the training corpus corresponding to the mapped high-resource national language and low-resource national language is input into the general large model, and the output is the corresponding word embedding representation. To improve the national language output capability of the model, the corresponding word embedding representation is input into a multilingual generation layer to obtain a final embedding representation, which is iteratively updated by the pre-training language model according to the final embedding representation, realizing the mapping from the high-resource national language to the low-resource national language. In addition, the overall model is fine-tuned for instructions, realizing the instructions of the user and enhancing the capability and controllability of the large model.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of information technology, in particular to a method and device for constructing a national language large model based on a general large model for an operating system. BACKGROUND

[0002] With the development of globalization, exchanges between countries and nations have increased, and language differences have led to communication barriers. Therefore, constructing a national language large model based on a general large language model (LLM) has become an important way to solve this problem. General large models exhibit a variety of processing capabilities by training on large-scale text corpora, but constructing models suitable for national languages still faces challenges, especially data scarcity and insufficient corpus size.

[0003] Mongolian language is relatively less data and complex structure, it is easy to appear over fitting phenomenon in the process of constructing model. In addition, the sentence structure of Mongolian (subject-object-predicate) is different from that of Chinese (subject-predicate-object), which affects the accuracy of machine translation. Currently, there are limited research documents on Mongolian-Chinese translation, mostly using neural machine translation methods. Although there are some basic researches such as encoder-decoder structure, BERT model, etc., the research on low-resource national languages is still not sufficient.

[0004] This paper proposes a large model construction technology for national languages such as Mongolian, Tibetan and Uyghur, using a Mongolian corpus as a dataset and implementing it based on the general large model of the Institute of Advanced Nationalities. Next, this paper will discuss the construction technology of the national language large model in detail. SUMMARY

[0005] Embodiments of the present application provide a method and device for constructing a national language large model based on a general large model for an operating system, to solve the problem of lack of real-time intervention capability in the alignment process of the large language model when facing complex adversarial attacks.

[0006] In one aspect, a method for constructing a national language large model based on a general large model for an operating system is provided, the national language large model comprising a multilingual encoding layer, an LLM understanding and reasoning layer, and a multilingual generation layer, wherein the multilingual encoding layer is composed of a pre-trained language model, a projection layer, and a general large model, and the method comprises:

[0007] generating a word list of multilingual training corpus;

[0008] converting words in the input low-resource national language into word embedding representation using the pre-trained language model, to obtain a word embedding vector;

[0009] sending the word embedding vector to the projection layer to form an alignable embedding representation vector;

[0010] In the vocabulary of the training corpus, the appropriate word embedding vector of the embedding representation vector is obtained and mapped to the semantic space of the large model to complete the mapping of the low-resource national language to the high-resource national language;

[0011] The training corpus corresponding to the mapped high-resource national language and low-resource national language is input into the general large model, and the output is the corresponding word embedding representation;

[0012] The corresponding word embedding representation is input into the multilingual generation layer to obtain the final embedding representation;

[0013] The pre-training language model is iteratively updated according to the final embedding representation to realize the mapping from the high-resource national language to the low-resource national language.

[0014] Optionally, the pre-training language model uses a BERT architecture, and the pre-training language model is used to convert an input word in the low-resource national language into a word embedding representation to obtain a word embedding vector, including:

[0015] Each word in the input low-resource national language is converted into a fixed-size vector;

[0016] Each word is converted into a fixed-size vector;

[0017] The pre-training language model of the BERT architecture adds position encoding for each word in the language sequence, and the position encoding is a set of learnable vectors;

[0018] In the pre-training language model of the BERT architecture, a Segment Embedding identifier is added to distinguish between sentences;

[0019] The initial word embedding vector, the position encoding, and the Segment Embedding identifier are added to obtain the word embedding representation of each word in the low-resource national language.

[0020] Optionally, after obtaining the word embedding representation of each word in the low-resource national language, the method further includes:

[0021] The word embedding representation of each word is input into a self-attention layer for processing, and the self-attention layer introduces a multi-head attention mechanism for processing the word embedding representation of each word, and the multi-head attention mechanism is used to capture the context information of the word;

[0022] The processed word embedding representation of each word is input into a feedforward network for further processing, and the feedforward network uses linear transformation and a nonlinear activation function;

[0023] Layer normalization is performed after the self-attention layer and the feed-forward network, and the input and output of the self-attention layer and the feed-forward network are added through a residual connection;

[0024] In the pre-training language model of the BERT architecture, the final word embedding representation of each word in the low-resource ethnic language is output.

[0025] Optionally, the projection layer is composed of a weight matrix;

[0026] The input word embedding vector is multiplied by the weight matrix in the projection layer to obtain a transformed alignable word embedding vector.

[0027] The input word embedding vector is multiplied by the weight matrix in the projection layer to obtain a transformed alignable word embedding vector.

[0028] Optionally, the training corpus corresponding to the high-resource ethnic language and the low-resource ethnic language after completing the mapping is input into the general large model, and the output is the corresponding word embedding representation, which includes:

[0029] Each word in the training corpus is mapped into a vector form by a word embedding matrix;

[0030] Each word in the training corpus in the vector form is input into the self-attention layer for processing, and the self-attention layer is applied through a multi-head attention mechanism;

[0031] The processed word embedding representation of each word is input into the feed-forward network for further processing, and the feed-forward network uses linear transformation and nonlinear activation function;

[0032] Layer normalization and residual connection help stabilize and accelerate the training of deep networks;

[0033] The general large model is updated by further minimizing the loss function, and the word embedding representation is updated using an optimization algorithm;

[0034] The final word embedding representation is output.

[0035] Optionally, after the mapping from the high-resource ethnic language to the low-resource ethnic language is implemented, the method further includes:

[0036] Part of the parameters in the ethnic language large model are fine-tuned using LoRA, and the fine-tuning is used to realize the simultaneous updating of the projection layers at both ends.

[0037] In another aspect, a device for constructing an ethnic language large model based on a general large model for an operating system is provided, the ethnic language large model comprising a multilingual encoding layer, an LLM understanding and reasoning layer, and a multilingual generation layer, wherein the multilingual encoding layer is composed of a pre-trained language model, a projection layer, and a general large model, and the device comprises:

[0038] a vocabulary generation module configured to generate a vocabulary of the multilingual training corpus;

[0039] a representation conversion module configured to convert words in an input low-resource ethnic language into word embedding representations using the pre-trained language model to obtain word embedding vectors;

[0040] a vector input module configured to input the word embedding vectors into the projection layer to form alignable embedding representation vectors;

[0041] a high-resource language mapping module configured to obtain suitable word embedding vectors of the embedding representation vectors in the vocabulary of the training corpus and map the low-resource ethnic language to the high-resource ethnic language in a large model semantic space;

[0042] a general large model module configured to input the training corpus corresponding to the mapped high-resource ethnic language and low-resource ethnic language into the general large model to output corresponding word embedding representations;

[0043] a final representation module configured to input the corresponding word embedding representations into the multilingual generation layer to obtain final embedding representations;

[0044] a low-resource language mapping module configured to update iteratively according to the final embedding representations through the pre-trained language model to realize mapping from the high-resource ethnic language to the low-resource ethnic language.

[0045] In another aspect, a computer-readable storage medium is provided, the storage medium storing at least one instruction, the at least one instruction being used to be executed by a processor to implement the method for constructing an ethnic language large model based on a general large model for an operating system as described in the above aspect.

[0046] In another aspect, a computer program product is also provided, the computer program product storing at least one instruction, the at least one instruction being loaded and executed by the processor to implement the method for constructing an ethnic language large model based on a general large model for an operating system as described in the above aspect.

[0047] In the present application, a general large model oriented to an operating system based on a national language large model construction method and device are disclosed, which belong to the field of information technology. First, the pre-training language model is used to convert the input words in the low-resource national language into word embedding representation, and the word embedding vector is obtained. Then, the word embedding vector is sent to the projection layer to form an alignable embedding representation vector. In the word table of the training corpus, the appropriate word embedding vector of the embedding representation vector is obtained and mapped to the large model semantic space to complete the mapping of the low-resource national language to the high-resource national language. Second, the training corpus corresponding to the high-resource national language and the low-resource national language after the mapping is input into the general large model, and the output is the corresponding word embedding representation. In order to improve the output ability of the model in the national language, the corresponding word embedding representation is input into the multilingual generation layer to obtain the final embedding representation. Through the pre-training language model, the final embedding representation is iteratively updated to realize the mapping from the high-resource national language to the low-resource national language. In addition, the overall model is fine-tuned for instructions to realize the instructions of the user and enhance the ability and controllability of the large model. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 A flowchart of the general large model oriented to an operating system based on a national language large model construction method provided by an example embodiment of the present application is shown;

[0049] Figure 2 A general model diagram of the national language large model is shown;

[0050] Figure 3 A semantic alignment model diagram of the multilingual encoding layer is shown;

[0051] Figure 4 A semantic alignment model diagram of the multilingual generation layer is shown;

[0052] Figure 5 A fine-tuning instruction model diagram is shown;

[0053] Figure 6 A flowchart of the general large model oriented to an operating system based on a national language large model construction method provided by another example embodiment of the present application is shown. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical scheme and advantages of the present application clearer, the embodiments of the present application will be described in further detail below with reference to the drawings.

[0055] In this paper, "a plurality of" refers to two or more. "And / or" describes the association between the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the existence of A alone, the existence of A and B, and the existence of B alone. The character " / " generally represents that the associated objects before and after it are in an "or" relationship.

[0056] First, the terms related to the field of the present application are explained.

[0057] National language: National language refers to the language used by a specific ethnic or cultural group, usually with unique grammatical, lexical, and phonetic characteristics. Compared to major languages such as Chinese and English, national languages are often considered low-resource national languages, meaning there is relatively less data and research in the field of natural language processing and machine learning.

[0058] Semantic alignment: Semantic alignment refers to the process of finding the same or similar meanings between different languages. This process involves mapping words, phrases, or sentences in one language to corresponding expressions in another language to achieve cross-language understanding and translation.

[0059] Word order: Word order refers to the arrangement of words in a sentence. Different languages may have different basic word orders, such as Chinese, which is usually "subject-verb-object" (SVO), while Mongolian and Tibetan languages may be "subject-object-verb" (SOV). This difference affects the structural adaptability of language models when processing different languages.

[0060] Data scarcity: Data scarcity refers to the lack of sufficient high-quality labeled data in a particular field or language for model training. This phenomenon is particularly common in low-resource national languages, limiting the development of related algorithms and models.

[0061] Training data: Training data refers to sample data used to train machine learning models. These data usually need to be labeled to help the model learn the relationship between input and output, improving the model's prediction and generation capabilities.

[0062] Generalization: Generalization refers to the ability of a model to perform well when faced with new data. A model with good generalization ability can not only perform well on training data, but also maintain relatively stable performance on unseen data.

[0063] Cultural differences: Cultural differences refer to the differences in expression, idiomatic expressions, and thinking patterns in different cultural backgrounds. This difference affects the accurate understanding and expression of semantics in natural language processing, especially in cross-cultural communication.

[0064] Polysemy: Polysemy refers to the characteristic of a word or expression having multiple meanings in different contexts. This requires careful consideration of the specific context when performing semantic alignment to ensure correct understanding and translation.

[0065] Therefore, it can be understood that, compared with Chinese and English, as low-resource ethnic languages, ethnic languages form sentences with complete semantics, strong order and logic according to their strict grammatical rules. The phonetics, grammar, vocabulary and semantics of ethnic languages are very different from those of Chinese, for example, the basic word order of Mongolian and Tibetan languages is subject-object-verb, so ethnic languages cannot directly use the Chinese-English method and theory to adapt to large models.

[0066] In the embodiments of the present application, the core of adapting large models to ethnic languages is to solve the difficulties faced by semantic alignment technology. The difficulties of this research in semantic alignment technology are as follows: first, the complexity and diversity of ethnic language semantics make it extremely difficult to accurately map the meanings expressed in different languages. Second, the flexibility and polysemy of ethnic languages increase the requirements on algorithms, which need to be able to handle contextual information and semantic meanings in different contexts. In addition, cultural differences in ethnic languages are also a challenge, because expression methods and idioms in different cultural backgrounds may differ greatly, and it is necessary to consider how to handle such differences. Finally, data scarcity and quality inconsistency are also a problem, because semantic alignment algorithms usually need a large amount of labeled data to train the model, and it is necessary to ensure the quality and representativeness of the data to improve the generalization ability of the model.

[0067] In addition, how to enhance the generation and understanding ability of large models is a complex and challenging task. One of the difficulties is how to process large-scale data and extract effective information from it. Large models need sufficient training data to learn various language expressions, semantic associations and contextual information, but processing such large-scale data also requires powerful computing resources and efficient algorithms. In addition, how to design the model structure to maximize its generation and understanding ability is also an important problem. It is necessary to balance the complexity and trainability of the model, as well as the universality and professionalism of the model in different tasks and fields. Another challenge is how to handle the generalization ability of the model, that is, to make the model perform well when facing new and unseen data.

[0068] In summary, the present application provides a method for constructing an ethnic language large model based on a general large model for an operating system, to enhance the generation and understanding ability of large models, which is illustrated by the following embodiments.

[0069] Please refer to Figure 1 which shows a flowchart of the method for constructing an ethnic language large model based on a general large model for an operating system according to an example embodiment of the present application.

[0070] First, as Figure 2The ethnic language large model structure diagram shown, the ethnic language large model can solve the problem of realizing the input and output compatibility of ethnic languages, we can process the ethnic language as an independent modal information problem, the ethnic language large model includes a multilingual encoding layer, a LLM understanding and reasoning layer and a multilingual generation layer.

[0071] In the embodiment of the present application, the multilingual encoding layer maps the input ethnic language into an embedding representation through a trained encoder, and then maps it to the large model semantic space through a projection layer. The LLM understanding and reasoning layer takes the open-source nine-grid large model as the core agent, and the LLM takes the representation from different languages as input and performs semantic understanding and reasoning on the input, thereby outputting corresponding information and signal markers of each language. The signal marker of each language refers to an instruction that can guide the multilingual generation layer to generate corresponding mode content. To improve the model's output capability of ethnic languages, a 4-layer transformer is used in the multilingual generation layer to capture the input N-bit special tokens to guide the large model to output the embedding representation of the answer, obtain the embedding representation features of each input sequence, and then use the decoder to output the ethnic language.

[0072] In order to bridge the gap between the feature spaces of different languages and achieve the purpose of semantic understanding of different languages by the ethnic language large model, it is essential to perform alignment learning on the model. Considering that the model level we designed has three layers, the model only needs to update the two projection layers of the encoding end and the decoding end to achieve different semantic alignment. In addition, in order to implement the user's instructions and enhance the capability and controllability of the large model, it is necessary to fine-tune the whole model.

[0073] Embodiment 1

[0074] In the following, taking the nine-grid large model as a representative of the general large model and taking Mongolian as a typical case of ethnic language, a Mongolian large model (Mo-LLM) is constructed, and the ethnic language large model is studied in depth. As shown in Figure 1 The method comprises the following steps.

[0075] Step 101, generating a word table of multilingual training corpus.

[0076] To realize the input compatibility of ethnic languages, we construct a multilingual encoding layer, which consists of three main parts: a pre-trained language model, a projection layer and a nine-grid large model. The specific process is as shown in Figure 3 .

[0077] The multilingual encoding layer of the model maps the input ethnic language sequence onto the large model semantic space. When the sentence of the ethnic language is input into the neural network, it is embedded as the smallest unit of words. Taking Mongolian as an example in the multilingual encoding layer, first, a word list of a Mongolian training corpus is generated.

[0078] Step 102, using a pre-trained language model to convert the input word in the low-resource ethnic language into a word embedding representation, obtaining a word embedding vector.

[0079] First, the vocabulary encoding is performed. Each word in the input low-resource ethnic language is converted into a fixed-size vector, and the index in the vocabulary table is searched to obtain an initialized word embedding vector corresponding to each word, thereby completing the word embedding initialization.

[0080] Second, the word embedding initialization is performed. The input low-resource ethnic language is an input word sequence, wherein each word in the input sequence is converted into a fixed-size vector, such as by a vocabulary table. Specifically, the word in the input low-resource ethnic language is converted into a word embedding representation by using a pre-trained language model of the BERT architecture to obtain a word embedding vector. WordPiece is used as the vocabulary encoding method under the BERT architecture, which means that it divides the words in the vocabulary table into smaller fragments and uses these fragments as basic units, thereby searching for an index in the vocabulary table to obtain an initialized word embedding vector corresponding to each word.

[0081] Second, the position encoding is performed. The pre-trained language model of the BERT architecture adds position encoding for each word in the language sequence. The position encoding is a set of learnable vectors that are added to the word embedding vectors to provide position information of the word in the sequence.

[0082] Second, in the pre-trained language model of the BERT architecture, a Segment Embedding identifier is added to distinguish sentence pairs. In an example, when processing sentence pairs (such as question and answer tasks), BERT uses different embeddings to distinguish the two sentences, which is achieved by adding a Segment Embedding.

[0083] The initialized word embedding vector, the position encoding, and the Segment Embedding identifier are added to obtain the word embedding representation of each word in the low-resource ethnic language.

[0084] Optionally, after obtaining the word embedding representation of each word in the low-resource ethnic language, the method further comprises:

[0085] In addition, multi-head self-attention mechanism is also included. The word embedding representation of each word is input into the self-attention layer for processing, and the self-attention layer introduces a multi-head attention mechanism for processing the word embedding representation of each word, which is used to capture the context information of the word.

[0086] Introduction of feedforward network. The processed word embedding representation of each word is input into the feedforward network for further processing, and the feedforward network uses linear transformation and nonlinear activation function such as ReLU.

[0087] In addition, layer normalization and residual connection are also included. After the self-attention layer and the feedforward network, layer normalization processing is performed, and the input and output of the self-attention layer and the feedforward network are added through the residual connection. In order to stabilize the training process and allow the training of deeper networks, BERT applies layer normalization after each sub-layer (self-attention layer and feedforward network), and adds the input of the sub-layer back to the output through the residual connection.

[0088] In the pre-training language model of the BERT architecture, the final word embedding representation of each word in the low-resource ethnic language is output.

[0089] Step 103, the word embedding vector is input into the projection layer to form an alignable embedding representation vector.

[0090] The projection layer is composed of a weight matrix, and the input word embedding vector is multiplied by the weight matrix in the projection layer to obtain a transformed alignable word embedding vector.

[0091] First, linear transformation. The projection layer is composed of a learnable weight matrix, and each word embedding vector is transformed by multiplying it with this weight matrix. Suppose the dimension of the original word embedding vector is d model , the shape of the weight matrix is d model ×d new , where d new is the target dimension. For a word embedding vector e, the linear transformation can be represented as: e' = eW, where e' is the transformed embedding representation vector and W is the weight matrix.

[0092] Second, alignment. Through linear transformation, the word embedding vector is converted to a new dimensional space, which may be different from the original space, but through the design of the weight matrix, it can be ensured that the transformed vector still retains the important information of the original vector in the new dimensional space, and is aligned with other parts of the model.

[0093] Second, the weight matrix W is learnable, and during the training process, the weight matrix will be continuously updated through the backpropagation algorithm to minimize the loss function of the model.

[0094] Secondly, optimization is performed. During the training process, the weight matrix W is optimized together with other model parameters to ensure that the converted embedding representation vector can effectively support the model to complete the pre-training task. Through the projection layer, the dimension and representation of the word embedding vector can be flexibly adjusted to better adapt to the model.

[0095] Step 104, in the vocabulary of the training corpus, the appropriate word embedding vector of the embedding representation vector is obtained and mapped to the semantic space of the large model to complete the mapping from the low-resource national language to the high-resource national language.

[0096] In the above example, the mapping from Mongolian to Chinese is completed. Among them, the general large model is used for data processing call of each layer of the multilingual general large model.

[0097] In the vocabulary of the training corpus, the word embedding vector of the appropriate word input sequence is found, so as to be mapped to the semantic space of the general large model, thereby realizing the mapping from the low-resource national language to the high-resource national language.

[0098] In one possible implementation, a pre-trained general large model is selected, a data set containing input text and labels is prepared, the model is modified as needed, and then in the fine-tuning process, the pre-trained general large model parameters are frozen in the initial stage, a small learning rate is set, a suitable loss function is selected, and iterative training is performed. At the same time, the model performance is evaluated regularly, and the model structure and hyperparameters are adjusted as needed until the general large model can realize the mapping from the low-resource national language to the high-resource national language.

[0099] Step 105, input the training corpus corresponding to the mapped high-resource national language and low-resource national language into the general large model, and output the corresponding word embedding representation.

[0100] Taking Mongolian as an example, the training corpus corresponding to Chinese (high-resource national language) and Mongolian (low-resource national language) is input into the nine-case large model (general large model), and the output is the corresponding word embedding representation.

[0101] In one possible implementation, step 105 includes the following flow.

[0102] Each word in the training corpus is mapped into a vector form through a word embedding matrix. Among them, word embedding initialization is performed, that is, the input of the model is a series of words, each word is mapped to a vector through a word embedding matrix W. If pre-trained word embedding is used, these vectors can provide position information of words in the sequence through position encoding as initial values.

[0103] Each word in the training corpus in vector form is input into the self-attention layer for processing, and the self-attention layer is applied by stacking through the multi-head attention mechanism. Among them, through the self-attention mechanism (Self-Attention), the model is allowed to consider other words in the entire input sequence when processing each word, and the multi-head attention (Multi-Head Attention) further applies the self-attention mechanism multiple times, each head learns a different representation subspace.

[0104] The processed word embedding representation of each word is input into the feedforward network for further processing, and the feedforward network uses linear transformation and nonlinear activation function. Subsequently, the output of the self-attention is further nonlinearly transformed by the feedforward network: FFN(x) = max(0, xW1 + b1)W2 + b2, where W1 and W2 are weight matrices, and b1 and b2 are bias terms.

[0105] Through layer normalization and residual connection, it helps to stabilize and accelerate the training of deep network. Further, through layer normalization and residual connection, it helps to stabilize and accelerate the training of deep network.

[0106] The general large model is updated by further minimizing the loss function, and the word embedding representation is updated using the optimization algorithm. Among them, the parameters of the further model are updated by minimizing the loss function, the gradient of the loss function with respect to the model parameters is calculated by the back propagation algorithm, and the word embedding and other parameters are updated using the optimization algorithm.

[0107] Finally, the final word embedding representation is output.

[0108] Step 106, input the corresponding word embedding representation into the multilingual generation layer to obtain the final embedding representation.

[0109] The multilingual generation layer needs to be semantically aligned, which means aligning the Mongolian generation model (GPT architecture) with the output instructions of the nine-case large model. The specific process is as follows Figure 4 .

[0110] The multilingual generation layer is a 4-layer transformer architecture. The transformer layer guides the output embedding representation through special tokens, mainly relying on the following core principles: first, special tokens such as [CLS] and [SEP] mark the beginning, end or classification task in the input sequence, and learn the representation related to the task. Second, the multi-head self-attention mechanism allows the model to capture different subspace information of each word in the sequence, and outputs by concatenation and linear transformation. Third, the position encoding provides the position information of the word, using the linear combination of sine and cosine functions. Then, each Transformer layer contains a feedforward network to further process the output of the self-attention layer. In addition, layer normalization and residual connection help stabilize the training process and allow gradient flow. The optional encoder-decoder architecture is used for the generation task, and the decoder layer generates the target sequence. Attention mask prevents information leakage and ensures that the model only focuses on positions before the current step. Finally, the linear layer and activation function of the output layer convert the output of the Transformer layer into the final embedding representation. Through optimization and training, the loss function is minimized to update the model parameters. The guidance of special tokens enables the model to generate embedding representations of task answers based on the context information of the input, suitable for downstream tasks such as text classification, question answering, and text generation.

[0111] In step 107, the final embedding representation is iteratively updated by the pre-trained language model to realize the mapping from high-resource national language to low-resource national language.

[0112] That is, the mapping from Chinese to Mongolian is finally realized.

[0113] The pre-trained model of the GPT architecture is pre-trained on a large amount of high-resource language data to learn the general representation of the language. It is based on the Transformer architecture, which uses self-attention mechanisms to capture long-distance dependencies between words. The core formula of self-attention is to calculate the interaction between query (Q), key (K) and value (V), and generate a weighted sum through the SoftMax function. In addition, the model uses multi-head self-attention to learn different representations of the data in parallel, and uses position encoding to provide word order information to the model. Each Transformer layer also contains a feedforward network to further process the output of the self-attention layer. Residual connection and layer normalization help the model train deep networks to prevent gradient vanishing or explosion. Optimization algorithms such as Adam or SGD update the model parameters according to the gradient of the loss function, and the commonly used loss function is cross-entropy loss.

[0114] After pre-training, the pre-trained model of GPT architecture can be fine-tuned on specific tasks of low-resource languages by adjusting model parameters through a small amount of target language data, realizing the mapping from high-resource to low-resource languages. In addition, GPT can also be trained through a multilingual pre-training data set to learn a cross-language general representation, so as to better migrate to low-resource languages. In summary, the pre-trained model of GPT architecture learns the general representation of language through pre-training, captures the rich features of language through the self-attention and multi-head attention mechanism of the Transformer architecture, and stabilizes the training process through position encoding, feedforward network, residual connection and layer normalization. Through optimization algorithm and loss function to guide the model learning, finally through fine-tuning and multi-language representation learning to realize the effective mapping from high-resource to low-resource languages.

[0115] Embodiment 2

[0116] In addition, as Figure 6 indicated, on the method shown in Figure 1 , the application also includes instruction fine-tuning content, that is, adding step 108.

[0117] Step 108, using LoRA to fine-tune part of the parameters in the national language large model, fine-tuning is used to realize the simultaneous update of the projection layer of both ends.

[0118] As Figure 1 indicated, the word embedding representation is input into a 4-layer transformer, and then processed through the encoder and decoder. The parameters in it will be continuously updated with the training of the network, and finally realize the mapping from Chinese to Mongolian. Although the encoding end and the decoding end are semantically aligned with the nine-grid large model, the national language large model cannot generate the corresponding target according to the user's instructions. Therefore, instruction fine-tuning is needed to solve this problem, and the instruction fine-tuning process is as Figure 5 indicated.

[0119] Taking Mongolian as an example, LoRA is used to enable a small part of the parameters in the national language large model to be efficiently fine-tuned through parameters, so as to realize the simultaneous update of the projection layer of both ends. When a Mongolian prompt dialogue sample is input into the model, the nine-grid large model reconstructs and generates the content of the text input, and marks the content of the Mongolian language with a corresponding signal. In addition to the adjustment of the nine-grid large model, we also fine-tune the decoding end of the national language large model. We use a 4-layer transformer architecture to map the Mongolian embedding representation output by the LLM to the semantic space of the language generation model of the GPT architecture, and then use the pre-trained GPT model to decode the embedding representation into understandable Mongolian text. Thus, parameter instruction fine-tuning enhances the interaction ability and controllability of the large model and the user.

[0120] In summary, the application provides a method and device for constructing a large model of a national language based on semantic alignment technology, which is based on a large model of a general system. First, a pre-trained language model is used to convert words in an input low-resource national language into word embedding representations, obtaining word embedding vectors. Then, the word embedding vectors are input into a projection layer to form alignable embedding representation vectors. In the word table of the training corpus, the appropriate word embedding vectors of the embedding representation vectors are obtained and mapped onto the semantic space of the large model to complete the mapping of the low-resource national language to the high-resource national language. Second, the mapped high-resource national language and the corresponding training corpus of the low-resource national language are input into the general large model, and the output is the corresponding word embedding representation. To improve the output capability of the model in the national language, the corresponding word embedding representation is input into a multilingual generation layer to obtain a final embedding representation. The pre-trained language model is iteratively updated according to the final embedding representation to realize the mapping from the high-resource national language to the low-resource national language. In addition, the overall model is fine-tuned for instructions to realize the instructions of the user and enhance the capability and controllability of the large model.

[0121] In detail, the application has the following beneficial effects.

[0122] (1) The pre-trained language model (PLM) based on the BERT architecture constructs word embedding representations. This preprocessing makes it easier to accurately map the meanings expressed by different national languages.

[0123] (2) The linear projection layer converts the word embedding vectors into an embedding representation vector that can be aligned with the general large model. On the one hand, linear projection is used for dimensionality reduction to reduce the complexity of the data while preserving the most important information. On the other hand, the input data is mapped from the original feature space to a new feature space, which prepares for adaptation to the general large model.

[0124] (3) By fine-tuning the general large model and continuously iterating the training, the national language sequence is mapped to the semantic space of the general large model, efficiently solving the problem of adapting the large model to the national language.

[0125] (4) The high-resource language and low-resource language training corpus is input into the general large model, so that the model can learn the commonalities and differences between different languages and enhance the cross-language understanding capability. In addition, the low-resource language can obtain more rich context information from the high-resource language, thereby obtaining better word embedding representations.

[0126] (5) The special token-based national language embedding representation enables the model to learn more extensive language phenomena, improving the model's generalization ability on different tasks. The four-layer Transformer model guides the output embedding representation through special tokens, providing a powerful architecture to handle complex language tasks and generating more accurate and rich language representations suitable for various natural language processing tasks.

[0127] (6) The method improves the generation and understanding capabilities of general large models in national languages, promotes linguistic research, improves language accessibility, text analysis, and drives innovation and development in language technology.

[0128] It should be noted that the above method not only shows excellent adaptability in Mongolian language research, but also is applicable to other national language research including Tibetan and Uyghur languages due to its design basis for common challenges faced by national languages.

[0129] Embodiment 3

[0130] On the basis of the above embodiments, the present application also provides Mongolian-related data as the data set of the model. After word segmentation, dependency syntax analysis, spelling check and manual correction of Mongolian raw corpus, Mongolian (traditional Mongolian) Chinese parallel corpus data D1 and traditional Mongolian question and answer data D2 are obtained, which follow the standard data format and have high quality. The composition and category of experimental data are mainly daily and legal texts. In addition, we collect data set D3 for classification evaluation benchmark model, data set D4 for named entity recognition benchmark model and data set D5 for translation benchmark model of large model, and the specific conditions are shown in Table 1.

[0131] Table 1 Data set

[0132]

[0133] In one possible implementation, the experimental environment used in the experiment herein is Linux, the operating system deployed is CentOs7.9 (64bit), the CPU is Intel 6330*2, the GPU is NVIDIA A800, the integrated development environment is Vscode (64bit), the programming language is Python 3.6 (64bit), and the deep learning framework is PyTorch 1.13. In the training of national language large model, the maximum length limit of Mongolian single sentence input is 512 tokens, the maximum output length of Mongolian single sentence is 512 tokens, and the total length of the model is 4096 tokens. After repeated debugging and optimization, when the experimental parameters are set as shown in Table 2, the experimental results are better.

[0134] Table 2 National language large model parameters

[0135]

[0136] In addition, the language understanding ability of the large model is evaluated. The evaluation mainly includes sentence-level semantic understanding and token-level semantic understanding, and the accuracy (P), recall (R) and (F1) value are used as the evaluation index of the model, and the specific definition is shown in formula (1) to formula (3).

[0137]

[0138] The evaluation of the language generation ability of the large model mainly includes a part of translation task, and the BLEU evaluation method is used, and the specific calculation formula is shown in formula (4) and (5).

[0139]

[0140] BP = e min(1-r / c,0) (5

[0141] In the formula, BP represents the sentence length penalty factor, (w n ) and N are the weight of n-gram and the maximum base element number of n-gram, respectively. n The accuracy of n-gram is represented by r, and c is the length of the candidate translation in machine translation. The higher the BLEU evaluation value, the better the translation performance of the model, the better the readability and the stronger the understandability of the translation.

[0142] There is no public Mongolian corpus of large models of ethnic languages at present. In addition, the research problems, corpus, experimental environment and research ideas of large model related literatures are different, and the comparability of experimental research results is not strong. Therefore, in order to verify the universality, effectiveness and practicality of the ethnic language large model in this paper, nine classical models commonly used in the research field are selected as baseline models to realize text classification task, entity extraction task and large model translation task, and compared with Mo-LLM proposed in this paper. The comparison of experimental results is shown in table 3, table 4 and table 5.

[0143] From the classification evaluation experimental results of the large model in table 3, we can analyze that: the baseline group model experiment realizes sentence classification by using a multilingual masked language model based on Transformer, and obtains relatively good results, but the XLM-R model may encounter the problem of insufficient representation learning when processing low-resource languages, and the CINO model is affected by the relatively small amount of ethnic language corpus, thereby affecting the accuracy of the pre-training model, so we can see that the F1 value of these models is lower than that of the Mo-LLM proposed in this paper.

[0144] Table 3 classification evaluation experimental results of large model

[0145]

[0146] From the evaluation results of the named entity recognition of the large model in Table 4, we can analyze that the CRF baseline model is a probabilistic graph model for predicting label sequences in sequence data, but has the problem of weak modeling ability for long-distance dependencies. The other two LSTM series baseline models (LSTM-CRF, BiLSTM-CRF) implement named entity recognition and achieve relatively good results, but LSTM cannot capture long-distance context feature information and only considers local features of the sentence. Therefore, compared with the Mo-LLM model in this paper, the P, R, and F1 values of these baseline models are lower.

[0147] Table 4 Evaluation results of large model named entity recognition

[0148]

[0149] From the evaluation results of the large model language generation capability in Table 5, we can analyze that the model in this paper fully integrates the advantages of context awareness of the BERT model and controllable generation of GPT, so Mo-LLM obtains the highest score in the BLEU index.

[0150] Table 5 Evaluation results of large model language generation capability

[0151]

[0152] Finally, in order to adapt other national languages through the Mongolian language large model and achieve the purpose of solving the language understanding ability and language generation ability of national language large model, according to the characteristics of national languages, first, the Mongolian-to-Chinese semantic alignment pre-training model is fused on the basis of the nine-case large model, then the model realizes the Mongolian-to-Chinese semantic output through the fusion of the nine-case large model and four-layer Transformer, and finally the PEFT and LoRA fine-tune the nine-case large model to realize the function of national language intercommunication. From the experimental results in Table 3, Table 4 and Table 5, the F1 value and BLEU value on various benchmark datasets are relatively the best, which verifies the adaptability and effectiveness of the model proposed in this paper.

[0153] The application is based on the morphological form of national languages, and proposes an end-to-end general national language large model on the basis of analyzing the problems of complex morphology, many out-of-vocabulary words and scarce corpus. The model takes Mongolian as the starting point, aiming to provide a new solution for the processing and analysis of national languages. The model first uses the established encoder to encode the input of each national language, and embeds the information into the representation projected by the projection layer into the representation of the language that the general large model can understand. Then the general large model takes the representation from different national languages as input and performs semantic understanding and reasoning on the input. Finally, the embedded information representation with specific instructions from the general large model is received, and the semantic information representation is mapped into the language representation that the decoder behind the output projection layer can understand, thereby the national language large model with generation ability is obtained. Through the language understanding ability evaluation and the language generation ability evaluation experiment of the Mongolian large model, the adaptability and effectiveness of the model are verified.

[0154] It should be noted that the embodiments are described in this paper taking the Linux system as an example, and those skilled in the art can also deploy it on other operating systems according to actual conditions, such as OpenKylin and Windows systems, and the present application does not limit this.

[0155] In addition, an exemplary embodiment of the present application also provides a national language large model construction device based on a general large model for an operating system. The national language large model comprises a multi-language coding layer, an LLM understanding and reasoning layer and a multi-language generation layer, wherein the multi-language coding layer is composed of a pre-training language model, a projection layer and a general large model, and the device comprises:

[0156] A word table generation module is configured to generate a word table of the multi-language training corpus;

[0157] A representation conversion module is configured to convert words in the input low-resource national language into word embedding representations using the pre-training language model to obtain word embedding vectors;

[0158] A vector feeding module is configured to feed the word embedding vectors into the projection layer to form alignable embedding representation vectors;

[0159] A high-resource mapping module is configured to obtain suitable word embedding vectors of the embedding representation vectors in the word table of the training corpus and map them to the semantic space of the large model to complete the mapping of the low-resource national language to the high-resource national language;

[0160] A general large model module is configured to input the training corpus corresponding to the mapped high-resource national language and low-resource national language into the general large model, and output corresponding word embedding representations;

[0161] a final representation module configured to input the corresponding word embedding representation into the multilingual generation layer to obtain a final embedding representation;

[0162] a low-resource language mapping module configured to implement mapping from a high-resource national language to a low-resource national language by iteratively updating the final embedding representation according to the pre-trained language model.

[0163] The above-mentioned sequence numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0164] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or can be instructed by a program to complete the related hardware, and the program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0165] The above-mentioned is only the optional embodiment of the present application, and does not limit the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for constructing a national language large model based on a general large model facing an operating system, suitable for deployment to an operating system, characterized in that, The national language large model comprises a multilingual coding layer, an LLM understanding and reasoning layer, and a multilingual generation layer, wherein the multilingual coding layer is composed of a pre-training language model, a projection layer, and a general large model, and the method comprises: generating a vocabulary of a multilingual training corpus; using the pre-training language model to convert words in an input low-resource national language into word embedding representations, obtaining word embedding vectors; sending the word embedding vectors to the projection layer to form alignable embedding representation vectors; in the vocabulary of the training corpus, obtaining suitable word embedding vectors of the embedding representation vectors and mapping them onto the semantic space of the large model to complete the mapping of the low-resource national language to the high-resource national language; inputting the mapped high-resource national language and low-resource national language corresponding training corpus into the general large model, and outputting the corresponding word embedding representation; the step of inputting the mapped high-resource national language and low-resource national language corresponding training corpus into the general large model, and outputting the corresponding word embedding representation, comprises: mapping each word in the training corpus into a vector form through a word embedding matrix; inputting each word in the vector form of the training corpus into a self-attention layer for processing, and applying the self-attention layer through a multi-head attention mechanism; inputting the word embedding representation of each processed word into a feedforward network for further processing, wherein the feedforward network uses linear transformation and nonlinear activation function; through layer normalization and residual connection, the training of the deep network is stabilized and accelerated; the general large model is updated by further minimizing the loss function, and the word embedding representation is updated using an optimization algorithm; and outputting the final word embedding representation; inputting the corresponding word embedding representation into the multilingual generation layer to obtain a final embedding representation; iteratively updating the pre-training language model according to the final embedding representation to realize the mapping from the high-resource national language to the low-resource national language.

2. The method of claim 1, wherein, The pre-training language model uses the BERT architecture, and the step of using the pre-training language model to convert words in an input low-resource national language into word embedding representations, obtaining word embedding vectors, comprises: converting each word in the input low-resource national language into a fixed-size vector; finding the index in the vocabulary to obtain an initialization word embedding vector corresponding to each word; adding position encoding in the language sequence for each word through the pre-training language model of the BERT architecture, wherein the position encoding is a set of learnable vectors; in the pre-training language model of the BERT architecture, a segment embedding identifier is added to distinguish sentences; adding the initialization word embedding vector, the position encoding, and the segment embedding identifier to obtain the word embedding representation of each word in the low-resource national language.

3. The method of claim 2, wherein, After obtaining the word embedding representation of each word in the low-resource national language, the method further comprises: The word embedding representation of each word is input into a self-attention layer for processing, the self-attention layer introduces a multi-head attention mechanism for processing the word embedding representation of each word, and the multi-head attention mechanism is used to capture the context information of the word. The processed word embedding representation of each word is input into a feedforward network for further processing, and the feedforward network uses linear transformation and nonlinear activation function. The self-attention layer and the feedforward network are processed by layer normalization, and the input and output of the self-attention layer and the feedforward network are added through a residual connection. In the pre-training language model of the BERT architecture, the final word embedding representation of each word in the low-resource ethnic language is output.

4. The method of claim 1, wherein, The projection layer is composed of a weight matrix; The word embedding vector is input into the projection layer to form an alignable embedding representation vector, which includes: The input word embedding vector is multiplied by the weight matrix in the projection layer to obtain a transformed alignable word embedding vector.

5. The method according to any one of claims 1 to 4, characterized in that, After realizing the mapping from the high-resource ethnic language to the low-resource ethnic language, the method further includes: Using LoRA to fine-tune part of the parameters in the ethnic language large model, and the fine-tuning is used to realize the simultaneous update of the projection layers on both ends. 6.A device for constructing an ethnic language large model oriented to an operating system based on a general large model, characterized in that, The ethnic language large model includes a multi-language coding layer, an LLM understanding and reasoning layer, and a multi-language generation layer, wherein the multi-language coding layer is composed of a pre-training language model, a projection layer, and a general large model, and the device includes: A vocabulary generation module for generating a vocabulary of a multi-language training corpus; An expression conversion module for converting an input word in a low-resource ethnic language into a word embedding representation using the pre-training language model to obtain a word embedding vector; A vector input module for inputting the word embedding vector into the projection layer to form an alignable embedding representation vector; A high-resource mapping module for obtaining a suitable word embedding vector of the embedding representation vector in the vocabulary of the training corpus and mapping it to the semantic space of the large model to complete the mapping of the low-resource ethnic language to the high-resource ethnic language; A general large model module for inputting the mapped high-resource ethnic language and low-resource ethnic language corresponding training corpus into the general large model, and outputting the corresponding word embedding representation; The general large model module is also used to: map each word in the training corpus into a vector form through a word embedding matrix; process each word in the training corpus through a self-attention layer, and apply the self-attention layer through a multi-head attention mechanism; input the processed word embedding representation of each word into a feedforward network for further processing, and the feedforward network uses linear transformation and nonlinear activation function; through layer normalization and residual connection, help to stabilize and accelerate the training of deep network; update the general large model by further minimizing the loss function, and update the word embedding representation using an optimization algorithm; output the final word embedding representation; A final representation module for inputting the corresponding word embedding representation into the multi-language generation layer to obtain a final embedding representation. A low-resource language mapping module is configured to implement mapping from a high-resource national language to a low-resource national language by iteratively updating the pre-trained language model according to the final embedding representation.

7. A computer readable storage medium characterized in that, The computer readable storage medium stores a computer program, and the computer program is loaded and executed by the processor to implement the method for constructing a national language large model based on a general large model and facing an operating system according to any one of claims 1 to 5.

Images