Operating system-oriented national language large model construction method and device based on general large model

Abstract

The invention discloses an operating system-oriented national language large model construction method and device based on a general large model, is suitable for being deployed to an operating system, and belongs to the technical field of information. The method comprises the following steps: firstly, converting words in an input low-resource national language into word embedding representation by using a pre-training language model to obtain a word embedding vector, then sending the word embedding vector into a projection layer to form an alignable embedding representation vector, and in a word list of a training corpus, carrying out word embedding representation on the word embedding representation vector; the method comprises the following steps: firstly, obtaining word embedding vectors with proper embedding expression vectors, mapping the word embedding vectors to a large model semantic space to complete mapping from a low-resource national language to a high-resource national language, and then inputting training corpora corresponding to the mapped high-resource national language and low-resource national language into a general large model, and outputting the training corpora as corresponding word embedding expression. And in order to improve the model national language output capability, inputting the corresponding word embedding representation into the multi-language generation layer to obtain a final embedding representation, and performing iterative updating according to the final embedding representation through the pre-training language model to realize mapping from a high-resource national language to a low-resource national language. In addition, instruction fine adjustment is carried out on the whole model, the instruction of the user is achieved, and the capacity and controllability of the large model are enhanced.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of information technology, and in particular to a method and device for constructing a national language big model based on a general big model and oriented to an operating system. Background Art

[0002] With the development of globalization, the exchanges between countries and ethnic groups have increased, and language differences have led to communication barriers. Therefore, building a national language model based on a universal large language model (LLM) has become an important way to solve this problem. Universal large models have demonstrated a variety of processing capabilities by training on large-scale text corpora, but building models suitable for national languages ​​still faces challenges, especially data scarcity and insufficient corpus size.

[0003] Mongolian is prone to overfitting when building models due to its relatively small amount of data and complex structure. In addition, the sentence structure of Mongolian (subject-object-verb) is different from that of Chinese (subject-verb-object), which affects the accuracy of machine translation. Currently, there is limited research literature on Mongolian-Chinese translation, and neural machine translation methods are mostly used. Although there are some basic research, such as encoder-decoder structure and BERT model, research on low-resource ethnic languages ​​is still insufficient.

[0004] This paper proposes a large model construction technology for ethnic languages ​​such as Mongolian, Tibetan and Uyghur, using the Mongolian corpus as a dataset and based on the general large model of Qiyuan National Laboratory. Next, this paper will discuss the construction technology of the ethnic language large model in detail. Summary of the invention

[0005] The embodiments of the present application provide a method and device for constructing a large model of ethnic languages ​​based on a general large model for an operating system, so as to solve the problem of lack of real-time intervention capability in dealing with complex adversarial attacks in the large language model alignment process.

[0006] On the one hand, a method for constructing a national language big model based on a general big model for an operating system is provided, wherein the national language big model includes a multilingual encoding layer, an LLM understanding and reasoning layer, and a multilingual generation layer, wherein the multilingual encoding layer is composed of a pre-trained language model, a projection layer, and a general big model, and the method includes:

[0007] Generate vocabulary for multilingual training corpus;

[0008] Using the pre-trained language model, converting input words in a low-resource ethnic language into word embedding representations to obtain word embedding vectors;

[0009] Sending the word embedding vector to the projection layer to form an alignable embedding representation vector;

[0010] In the vocabulary of the training corpus, a word embedding vector suitable for the embedding representation vector is obtained and mapped to the semantic space of the large model to complete the mapping of the low-resource ethnic language to the high-resource ethnic language;

[0011] Input the training corpus corresponding to the mapped high-resource ethnic language and low-resource ethnic language into the universal large model, and output the corresponding word embedding representation;

[0012] Inputting the corresponding word embedding representation into the multilingual generation layer to obtain a final embedding representation;

[0013] The pre-trained language model is iteratively updated according to the final embedding representation to achieve mapping from high-resource ethnic languages ​​to low-resource ethnic languages.

[0014] Optionally, the pre-trained language model uses a BERT architecture, and the pre-trained language model is used to convert the input words in the low-resource ethnic language into word embedding representations to obtain word embedding vectors, including:

[0015] Each word in the input low-resource ethnic language is converted into a vector of fixed size;

[0016] Look up the index in the vocabulary to get an initialized word embedding vector for each word;

[0017] The pre-trained language model of the BERT architecture is used to add position codes in the language sequence to each word, and the position codes are a set of learnable vectors.

[0018] In the pre-trained language model of the BERT architecture, sentence pairs are distinguished by adding Segment Embedding identifiers;

[0019] The initialized word embedding vector, the position code and the Segment Embedding identifier are added to obtain a word embedding representation of each word in the low-resource ethnic language.

[0020] Optionally, after obtaining the word embedding representation of each word in the low-resource ethnic language, the method further includes:

[0021] Inputting the word embedding representation of each word into a self-attention layer for processing, the self-attention layer introduces a multi-head attention mechanism for processing the word embedding representation of each word, and the multi-head attention mechanism is used to capture the context information of the word;

[0022] The processed word embedding representations of each word are input into a feed-forward network for further processing, wherein the feed-forward network uses a linear transformation and a nonlinear activation function;

[0023] Performing layer normalization on the self-attention layer and the feed-forward network, and adding the input and output of the self-attention layer and the feed-forward network through a residual connection;

[0024] In the pre-trained language model of the BERT architecture, the final word embedding representation of each word in the low-resource ethnic language is output.

[0025] Optionally, the projection layer is composed of a weight matrix;

[0026] The step of sending the word embedding vector to the projection layer to form an alignable embedding representation vector includes:

[0027] The input word embedding vector is multiplied by the weight matrix in the projection layer to obtain a transformed and alignable word embedding vector.

[0028] Optionally, the step of inputting the training corpus corresponding to the mapped high-resource ethnic language and the low-resource ethnic language into the universal large model and outputting the corresponding word embedding representation includes:

[0029] Mapping each word in the training corpus into a vector form through a word embedding matrix;

[0030] Input each word in the training corpus in vector form into a self-attention layer for processing, and superimpose the self-attention layer through a multi-head attention mechanism;

[0031] The processed word embedding representations of each word are input into a feed-forward network for further processing, wherein the feed-forward network uses a linear transformation and a nonlinear activation function;

[0032] Helps stabilize and accelerate the training of deep networks through layer normalization and residual connections;

[0033] Update the general large model by further minimizing the loss function and use the optimization algorithm to update the word embedding representation;

[0034] Output the final word embedding representation.

[0035] Optionally, after the mapping from the high-resource ethnic language to the low-resource ethnic language is achieved, the method further includes:

[0036] LoRA is used to fine-tune some parameters in the national language model, and the fine-tuning is used to achieve simultaneous updating of the projection layers at both ends.

[0037] On the other hand, a device for constructing a national language big model based on a general big model and oriented to an operating system is provided, wherein the national language big model includes a multilingual encoding layer, an LLM understanding and reasoning layer, and a multilingual generation layer, wherein the multilingual encoding layer is composed of a pre-trained language model, a projection layer, and a general big model, and the device includes:

[0038] A vocabulary generation module, used to generate a vocabulary for multilingual training corpus;

[0039] A representation conversion module, used to convert the input words in the low-resource ethnic language into word embedding representations using the pre-trained language model to obtain word embedding vectors;

[0040] A vector input module, used for inputting the word embedding vector into the projection layer to form an alignable embedding representation vector;

[0041] A high-resource mapping module is used to obtain a word embedding vector suitable for the embedding representation vector in the vocabulary of the training corpus and map it to the large model semantic space to complete the mapping of the low-resource ethnic language to the high-resource ethnic language;

[0042] A general large model module, used to input the training corpus corresponding to the mapped high-resource ethnic language and low-resource ethnic language into the general large model, and output the corresponding word embedding representation;

[0043] A final representation module, used for inputting the corresponding word embedding representation into the multilingual generation layer to obtain a final embedding representation;

[0044] A low-resource mapping module is used to iteratively update the pre-trained language model according to the final embedding representation to achieve mapping from a high-resource ethnic language to a low-resource ethnic language.

[0045] On the other hand, a computer-readable storage medium is provided, which stores at least one instruction, and the at least one instruction is used to be executed by a processor to implement the method for constructing a national language big model based on a general big model and oriented to an operating system as described in the above aspects.

[0046] On the other hand, a computer program product is also provided, which stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the method for constructing a national language big model based on a general big model and oriented to an operating system as described in the above aspect.

[0047] In the present application, a method and device for constructing a national language large model based on a general large model for an operating system are disclosed, and belong to the field of information technology. First, a pre-trained language model is used to convert the input words in the low-resource national language into word embedding representations to obtain word embedding vectors, and then the word embedding vectors are sent to the projection layer to form an alignable embedding representation vector. In the vocabulary of the training corpus, the appropriate word embedding vectors are obtained and mapped to the semantic space of the large model to complete the mapping of low-resource national languages ​​to high-resource national languages. Secondly, the training corpus corresponding to the high-resource national languages ​​and low-resource national languages ​​that have completed the mapping is input into the general large model, and the output is the corresponding word embedding representation. In order to improve the output capability of the model's national language, the corresponding word embedding representation is input into the multilingual generation layer to obtain the final embedding representation, and the pre-trained language model is iteratively updated according to the final embedding representation to achieve the mapping from high-resource national languages ​​to low-resource national languages. In addition, the overall model is fine-tuned to implement the user's instructions and enhance the capability and controllability of the large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A flowchart of a method for constructing a national language big model based on a general big model and oriented to an operating system provided by an exemplary embodiment of the present application is shown;

[0049] Figure 2 The overall model diagram of the national language model is shown;

[0050] Figure 3 A diagram of the semantic alignment model of the multilingual encoding layer is shown;

[0051] Figure 4 A diagram of the semantic alignment model of the multilingual generation layer is shown;

[0052] Figure 5 A diagram of the instruction fine-tuning model is shown;

[0053] Figure 6 A flowchart of a method for constructing a national language big model based on a general big model and oriented to an operating system provided by another exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0054] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0055] The term "multiple" as used herein refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0056] First, the terms in the art involved in this application are explained.

[0057] Ethnic languages: Ethnic languages ​​are languages ​​used by a specific ethnic group or cultural group, and usually have unique grammar, vocabulary, and phonetic features. Compared to major languages ​​(such as Chinese and English), ethnic languages ​​are often considered low-resource ethnic languages, meaning that there is relatively little data and research on them in the fields of natural language processing and machine learning.

[0058] Semantic alignment: Semantic alignment refers to the process of finding the same or similar meanings between different languages. This process involves mapping words, phrases or sentences in one language with the corresponding expressions in another language to achieve cross-language understanding and translation.

[0059] Word order: Word order refers to the order in which words are arranged in a sentence. Different languages ​​may have different basic word orders. For example, Chinese is usually "subject-verb-object" (SVO), while Mongolian and Tibetan languages ​​may be "subject-object-verb" (SOV). This difference affects the structural adaptability of the language model when processing different languages.

[0060] Data scarcity: Data scarcity refers to the lack of sufficient high-quality annotated data for model training in a particular field or language. This phenomenon is particularly common in low-resource ethnic languages, limiting the development of related algorithms and models.

[0061] Training data: Training data refers to sample data used to train machine learning models. This data usually needs to be labeled to help the model learn the relationship between input and output and improve the model's prediction and generation capabilities.

[0062] Generalization: Generalization ability refers to the ability of a model to perform well when faced with new data. A model with good generalization ability can not only achieve good results on the training data, but also maintain relatively stable performance on unseen data.

[0063] Cultural differences: Cultural differences refer to the differences in expressions, idioms, and thinking patterns in different cultural backgrounds. Such differences will affect the accurate understanding and expression of semantics in natural language processing, especially in cross-cultural communication.

[0064] Polysemy: Polysemy refers to the property that a word or expression can have multiple meanings in different contexts. This requires careful consideration of the specific context when performing semantic alignment to ensure correct understanding and translation.

[0065] From this, we can understand that compared with Chinese and English, ethnic languages, as low-resource ethnic languages, follow their strict grammatical rules to form sentences with complete semantics and strong order and logic. The phonetics, grammar, vocabulary, and semantics of ethnic languages ​​are very different from those of Chinese. For example, the basic word order of Mongolian and Tibetan languages ​​is subject-object-predicate. Therefore, ethnic languages ​​cannot directly use Chinese and English methods and theories to adapt to the large model.

[0066] In the embodiments of the present application, the core of the problem of adapting large models to ethnic languages ​​is to solve the difficulties faced by semantic alignment technology. The difficulties of this study in semantic alignment technology: First, the complexity and diversity of the semantics of ethnic languages ​​make it extremely difficult to accurately map the meanings of expressions in different languages. Secondly, the flexibility and ambiguity of ethnic languages ​​increase the requirements for algorithms, which need to be able to process contextual information and semantic meanings in different contexts. In addition, cultural differences in ethnic languages ​​are also a challenge, because there may be large differences in expressions and idioms in different cultural backgrounds, and it is necessary to consider how to deal with such differences. Finally, data scarcity and uneven quality are also a problem, because semantic alignment algorithms usually require a large amount of labeled data to train the model, and it is necessary to ensure the quality and representativeness of these data to improve the generalization ability of the model.

[0067] In addition, how to enhance the generation and understanding capabilities of large models is a complex and challenging task. One of the difficulties is how to process large-scale data and extract effective information from it. Large models require sufficient training data to learn various language expressions, semantic associations, and contextual information, but processing such large amounts of data also requires powerful computing resources and efficient algorithms. In addition, how to design the model structure to maximize its generation and understanding capabilities is also an important problem. It is necessary to balance the complexity and trainability of the model, as well as the versatility and specialization of the model in different tasks and fields. Another challenge is how to deal with the generalization ability of the model, that is, to enable the model to perform well when faced with new and unseen data.

[0068] In summary, the present application provides a method for constructing a large model of ethnic languages ​​based on a general large model for an operating system to enhance the generation and understanding capabilities of large models, which will be further illustrated through the following embodiments.

[0069] Please refer to Figure 1 , which shows a flowchart of a method for constructing a national language big model based on a general big model and oriented to an operating system provided by an exemplary embodiment of the present application.

[0070] First, if Figure 2The structural diagram of the national language big model is shown. The national language big model can solve the problem of achieving input and output compatibility of national languages. We can treat national languages ​​as an independent modal information problem. The national language big model includes a multilingual encoding layer, an LLM understanding and reasoning layer, and a multilingual generation layer.

[0071] In an embodiment of the present application, the multilingual encoding layer maps the input ethnic language into an embedded representation through a trained encoder, and then maps it to the semantic space of the large model through a projection layer. The LLM understanding and reasoning layer uses the open source nine-grid large model as the core agent. LLM takes representations from different languages ​​as input, and performs semantic understanding and reasoning on the input, thereby outputting corresponding information and signal markers for each language. The signal marker for each language refers to an instruction that can guide the multilingual generation layer to generate corresponding pattern content. In order to improve the model's ethnic language output capability, a 4-layer transformer is used in the multilingual generation layer to capture the input N-bit special tokens to guide the large model to output the embedded representation of the answer, obtain the embedded representation features of each input sequence, and then use a decoder to output it as the ethnic language.

[0072] In order to bridge the gap between the feature spaces of different languages ​​and achieve the goal of the national language model to understand the semantics of different languages, it is crucial to align the model in this paper. Considering that the model we designed has three layers, the model only needs to update the two projection layers at the encoding and decoding ends to achieve different semantic alignment. In addition, in order to implement user instructions and enhance the ability and controllability of the large model, the overall model needs to be fine-tuned.

[0073] Example 1

[0074] The following article uses the nine-frame large model as a representative of the general large model, and takes Mongolian as a typical case of a national language to construct the Mongolian large language model (Mo-LLM), and conducts an in-depth study of the national language large model. Figure 1 As shown, the method includes the following steps.

[0075] Step 101: Generate a vocabulary of multilingual training corpus.

[0076] To achieve input compatibility with ethnic languages, we built a multilingual encoding layer, which consists of three main parts: pre-trained language model, projection layer and nine-grid model. The specific process is as follows: Figure 3 .

[0077] The multilingual encoding layer of the model maps the input ethnic language sequence to the semantic space of the large model. When the ethnic language sentences are input into the neural network, they are embedded as the smallest unit. Taking Mongolian as an example in the multilingual encoding layer, a vocabulary of Mongolian training corpus is first generated.

[0078] Step 102: Use a pre-trained language model to convert the input words in the low-resource ethnic language into word embedding representations to obtain word embedding vectors.

[0079] First, vocabulary encoding is performed. Each word in the input low-resource ethnic language is converted into a vector of fixed size, and the index in the vocabulary is found to obtain an initialization word embedding vector corresponding to each word, thereby completing the word embedding initialization.

[0080] Secondly, the word embedding is initialized. The input low-resource ethnic language is an input word sequence, where each word in the input sequence is converted into a vector of a fixed size, such as through a vocabulary. Specifically, the words in the input low-resource ethnic language are converted into word embedding representations through the pre-trained language model of the BERT architecture to obtain word embedding vectors. The BERT architecture uses WordPiece as its vocabulary encoding method, which means that it decomposes the words in the vocabulary into smaller fragments and uses these fragments as basic units to find the index in the vocabulary to obtain an initialized word embedding vector corresponding to each word.

[0081] Secondly, position encoding is performed. The pre-trained language model of the BERT architecture is used to add position encodings to each word in the language sequence. The position encodings are a set of learnable vectors, which are added to the word embedding vectors to provide the position information of the word in the sequence.

[0082] Secondly, in the pre-trained language model of the BERT architecture, sentence pairs are distinguished by adding Segment Embedding identifiers. In one example, when processing sentence pairs (such as question-answering tasks), BERT uses different embeddings to distinguish between two sentences, which is achieved by adding a Segment Embedding.

[0083] The initialized word embedding vector, position encoding, and Segment Embedding identifier are added together to obtain the word embedding representation of each word in the low-resource ethnic language.

[0084] Optionally, after obtaining the word embedding representation of each word in the low-resource ethnic language, the method further includes:

[0085] In addition, it also includes a multi-head self-attention mechanism. The word embedding representation of each word is input into the self-attention layer for processing. The self-attention layer introduces a multi-head attention mechanism to process the word embedding representation of each word. The multi-head attention mechanism is used to capture the contextual information of the word.

[0086] Introduction of feedforward network. The word embedding representation of each word after processing is input into the feedforward network for further processing. The feedforward network uses linear transformation and nonlinear activation function, such as ReLU.

[0087] In addition, layer normalization and residual connections are included. Layer normalization is performed after the self-attention layer and the feedforward network, and the input and output of the self-attention layer and the feedforward network are added through residual connections. In order to stabilize the training process and allow training of deeper networks, BERT applies layer normalization after each sub-layer (self-attention layer and feedforward network) and adds the input of the sub-layer back to the output through residual connections.

[0088] In the pre-trained language model of the BERT architecture, the final word embedding representation of each word in the low-resource ethnic language is output.

[0089] Step 103, the word embedding vector is sent to the projection layer to form an alignable embedding representation vector.

[0090] The projection layer consists of a weight matrix. The input word embedding vector is multiplied by the weight matrix in the projection layer to obtain the transformed and aligned word embedding vector.

[0091] First, linear transformation. The projection layer consists of a learnable weight matrix, and each word embedding vector is transformed by multiplying it with this weight matrix. Assume that the dimension of the original word embedding vector is d model , the shape of the weight matrix is ​​d model ×d new , where d new is the target dimension. For a word embedding vector e, the linear transformation can be expressed as: e′=eW, where e′ is the transformed embedding representation vector and W is the weight matrix.

[0092] Second, alignment. Through linear transformation, the word embedding vector is transformed into a new dimensional space, which may be different from the original space, but through the design of the weight matrix, it can be ensured that the transformed vector still retains the important information of the original vector in the new dimensional space and is aligned with other parts of the model.

[0093] Secondly, the weight matrix W is learnable. During the training process, the weight matrix will be continuously updated through the back-propagation algorithm to minimize the loss function of the model.

[0094] Secondly, optimization is performed. During the training process, the weight matrix W is optimized together with other model parameters to ensure that the converted embedding representation vector can effectively support the model to complete the pre-training task. Through the projection layer, the dimension and representation of the word embedding vector can be flexibly adjusted to better adapt it to the model.

[0095] Step 104, in the vocabulary of the training corpus, obtain the word embedding vector with a suitable embedding representation vector and map it to the semantic space of the large model to complete the mapping of the low-resource ethnic language to the high-resource ethnic language.

[0096] In the above example, the mapping from Mongolian to Chinese is completed. The general big model is used for data processing calls at each layer of the multi-language general big model.

[0097] In the vocabulary of the training corpus, we find the word embedding vector of the appropriate word input sequence and map it to the semantic space of the general large model, thus realizing the mapping of low-resource ethnic languages ​​to high-resource languages.

[0098] In one possible implementation, a pre-trained general-purpose large model is selected, a data set containing input text and labels is prepared, the model is modified as needed, and then during the fine-tuning process, the parameters of the pre-trained general-purpose large model are frozen in the initial stage, a small learning rate is set, a suitable loss function is selected, and iterative training is performed. At the same time, the model performance is regularly evaluated, and the model structure and hyperparameters are adjusted as needed until the general-purpose large model can achieve the mapping of low-resource ethnic languages ​​to high-resource languages.

[0099] Step 105, input the training corpus corresponding to the mapped high-resource ethnic languages ​​and low-resource ethnic languages ​​into the universal large model, and output the corresponding word embedding representation.

[0100] Taking Mongolian as an example, the training corpus corresponding to Chinese (a high-resource ethnic language) and Mongolian (a low-resource ethnic language) is input into the nine-grid large model (universal large model), and the output is the corresponding word embedding representation.

[0101] In a possible implementation, step 105 includes the following process.

[0102] Each word in the training corpus is mapped into a vector form through a word embedding matrix. In this process, the word embedding initialization is performed, that is, the input of the model is a series of words, and each word is mapped to a vector through a word embedding matrix W. If pre-trained word embedding is used, these vectors can be used as initial values ​​to provide the position information of the word in the sequence through position encoding.

[0103] Each word in the training corpus in vector form is input into the self-attention layer for processing, and the self-attention layer is superimposed and applied through the multi-head attention mechanism. Among them, the self-attention mechanism (Self-Attention) allows the model to consider other words in the entire input sequence when processing each word, and the multi-head attention (Multi-HeadAttention) further applies the self-attention mechanism multiple times, and each head learns a different representation subspace.

[0104] The word embedding representation of each word is input into the feedforward network for further processing. The feedforward network uses linear transformation and nonlinear activation function. Subsequently, the output of self-attention is further nonlinearly transformed by the feedforward network: FFN(x) = max(0, xW1+b1)W2+b2, where W1 and W2 are weight matrices, and b1 and b2 are bias terms.

[0105] It helps stabilize and accelerate the training of deep networks through layer normalization and residual connections. Furthermore, it helps stabilize and accelerate the training of deep networks through layer normalization and residual connections.

[0106] The general large model is updated by further minimizing the loss function, and an optimization algorithm is used to update the word embedding representation. The parameters of the model are further updated by minimizing the loss function, the gradient of the loss function with respect to the model parameters is further calculated by the back propagation algorithm, and the optimization algorithm is used to update the word embedding and other parameters.

[0107] Finally, the final word embedding representation is output.

[0108] Step 106: Input the corresponding word embedding representation into the multilingual generation layer to obtain the final embedding representation.

[0109] The multi-language generation layer needs to be semantically aligned, which means aligning the output instructions of the Mongolian generation model (GPT architecture) with the nine-grid model. The specific process is as follows: Figure 4 .

[0110] Among them, the multilingual generation layer is a 4-layer transformer architecture. The Transformer layer guides the output embedding representation through special tokens, which mainly relies on the following core principles: First, special tokens such as [CLS] and [SEP] mark the start, end or classification task in the input sequence and learn task-related representations. Second, the multi-head self-attention mechanism allows the model to capture different subspace information of each word in the sequence and output it through concatenation and linear transformation. Third, the position encoding provides the position information of the word, using a linear combination of sine and cosine functions. Next, each Transformer layer contains a feedforward network to further process the output of the self-attention layer. In addition, layer normalization and residual connections help stabilize the training process and allow gradients to flow. The optional encoder-decoder architecture is used to generate tasks, and the decoder layer generates the target sequence. The attention mask prevents information leakage and ensures that the model only pays attention to the position before the current step. Finally, the linear layer and activation function of the output layer transform the output of the Transformer layer into the final embedding representation. Through optimization and training, the loss function is minimized and the model parameters are updated. The guidance of special tokens enables the model to generate an embedded representation of the task answer based on the input context information, which is suitable for downstream tasks such as text classification, question answering, and text generation.

[0111] Step 107, iteratively updating the pre-trained language model according to the final embedding representation to achieve mapping from high-resource ethnic languages ​​to low-resource ethnic languages.

[0112] That is, the final mapping from Chinese to Mongolian is achieved.

[0113] Among them, the pre-trained model of the GPT architecture learns the general representation of language by pre-training on a large amount of high-resource language data. It is based on the Transformer architecture and uses the self-attention mechanism to capture the long-distance dependencies between words. The core formula of self-attention is to calculate the interaction between the query (Q), key (K), and value (V), and generate the weighted sum through the SoftMax function. In addition, the model uses multi-head self-attention to learn different representations of the data in parallel, and uses position encoding to provide word order information to the model. Each Transformer layer also contains a feedforward network to further process the output of the self-attention layer. Residual connections and layer normalization help the model train deep networks to prevent gradients from disappearing or exploding. Optimization algorithms such as Adam or SGD update model parameters according to the gradient of the loss function. The commonly used loss function is cross entropy loss.

[0114] After pre-training is completed, the pre-trained model of the GPT architecture can be fine-tuned on specific tasks in low-resource languages. By adapting a small amount of target language data to adjust model parameters, mapping from high-resource to low-resource languages ​​can be achieved. In addition, GPT can also be trained with multilingual pre-training datasets to learn universal representations across languages, thereby better migrating to low-resource languages. In short, the pre-trained model of the GPT architecture learns universal representations of languages ​​through pre-training, uses the self-attention and multi-head attention mechanisms of the Transformer architecture to capture the rich features of languages, and stabilizes the training process through techniques such as position encoding, feedforward networks, residual connections, and layer normalization. Model learning is guided by optimization algorithms and loss functions, and finally, effective mapping from high-resource to low-resource languages ​​is achieved through fine-tuning and multilingual representation learning.

[0115] Example 2

[0116] In addition, if Figure 6 As shown, in Figure 1 In the method shown, the present application also includes instruction fine-tuning content, that is, adding step 108.

[0117] Step 108, using LoRA to fine-tune some parameters in the national language model, the fine-tuning is used to achieve simultaneous updating of the projection layers at both ends.

[0118] like Figure 1 As shown in the figure, the word embedding representation is input into the 4-layer transformer, and then processed by the encoder and decoder. The parameters will be iteratively updated as the network is trained, and finally the mapping from Chinese to Mongolian is achieved. Although the encoding and decoding ends are semantically aligned with the nine-grid model, the national language model cannot generate the corresponding target according to the user's instructions. Therefore, it is necessary to solve this problem through instruction fine-tuning. The instruction fine-tuning process is as follows: Figure 5 shown.

[0119] Taking Mongolian as an example, LoRA is used to enable a small part of the parameters in the national language large model to be efficiently fine-tuned through parameters to achieve simultaneous updates of the projection layers at both ends. When a Mongolian prompt dialogue sample is input into the model, the nine-grid large model reconstructs and generates the content of the text input, and uses corresponding signal markers to represent the content of the Mongolian language. In addition to the adjustment of the nine-grid large model, we also fine-tuned the decoding end of the national language large model. We use a 4-layer transformer architecture to map the Mongolian embedding representation output by the LLM to the semantic space of the language generation model of the GPT architecture, and then use the pre-trained GPT model to decode the embedding representation into understandable Mongolian text. As a result, parameter instruction fine-tuning enhances the interaction and controllability of the large model and users.

[0120] In summary, the present application provides a method and device for constructing a large model of ethnic languages ​​based on semantic alignment technology. A method for constructing a large model of ethnic languages ​​based on a general large model for an operating system. First, a pre-trained language model is used to convert the input words in the low-resource ethnic language into word embedding representations to obtain word embedding vectors, and then the word embedding vectors are sent to the projection layer to form an alignable embedding representation vector. In the vocabulary of the training corpus, the appropriate word embedding vectors are obtained and mapped to the semantic space of the large model to complete the mapping of low-resource ethnic languages ​​to high-resource ethnic languages. Secondly, the training corpus corresponding to the high-resource ethnic languages ​​and low-resource ethnic languages ​​that have completed the mapping is input into the general large model, and the output is the corresponding word embedding representation. In order to improve the output capability of the model's ethnic languages, the corresponding word embedding representation is input into the multilingual generation layer to obtain the final embedding representation, and the pre-trained language model is iteratively updated according to the final embedding representation to achieve the mapping from high-resource ethnic languages ​​to low-resource ethnic languages. In addition, the overall model is also fine-tuned to implement the user's instructions and enhance the capabilities and controllability of the large model.

[0121] In detail, the present application has the following beneficial effects.

[0122] (1) Construct word embedding representation based on the pre-trained language model (PLM) of the BERT architecture. This pre-processing step makes it easier to accurately map the meanings of expressions in different ethnic languages.

[0123] (2) The linear projection layer converts the word embedding vector into an embedding representation vector that can be aligned with the general large model. On the one hand, linear projection is used to reduce the dimensionality and complexity of the data while retaining the most important information as much as possible. On the other hand, it maps the input data from the original feature space to the new feature space, paving the way for adaptation to the general large model.

[0124] (3) By fine-tuning the universal big model and performing continuous iterative training, the ethnic language sequences are mapped to the semantic space of the universal big model, thus efficiently solving the problem of ethnic language adaptation to the big model.

[0125] (4) The training corpus of high-resource and low-resource languages ​​is fed into a common large model, which enables the model to learn the commonalities and differences between different languages ​​and enhance cross-language understanding capabilities. In addition, low-resource languages ​​can obtain richer contextual information from high-resource languages, thereby obtaining better word embedding representations.

[0126] (5) The specially marked ethnic language embedding representation enables the model to learn a wider range of language phenomena and improve the model's generalization ability on different tasks. The four-layer Transformer model guides the output embedding representation through special tokens, providing a powerful architecture to handle complex language tasks. It can generate more accurate and rich language representations and is suitable for a variety of natural language processing tasks.

[0127] (6) This method enables universal large-scale models to improve the ability to generate and understand national languages, promote linguistic research, improve language accessibility, text analysis, and promote the innovation and development of language technology.

[0128] It should be noted that the above method not only shows excellent adaptability in Mongolian research, but also because its design is based on the common challenges faced by ethnic languages, it is also applicable to the research of other ethnic languages ​​including Tibetan and Uyghur.

[0129] Example 3

[0130] On the basis of the above embodiment, the present application also provides Mongolian-related data as a data set for the model. After word segmentation, dependency syntax analysis, spell checking and manual proofreading of Mongolian raw corpus, the Mongolian (traditional Mongolian) Chinese parallel corpus data D1 and traditional Mongolian question-and-answer data D2 that follow the standard data format and have high quality are obtained. The composition and category of the experimental data are mainly based on daily and legal texts. In addition, we collected the data set D3 of the classification evaluation benchmark model, the data set D4 of the named entity recognition benchmark model and the data set D5 of the translation benchmark model of the large model, as shown in Table 1.

[0131] Table 1 Dataset

[0132]

[0133] In a possible implementation, the experimental environment used in this paper is Linux, the operating system CentOs7.9 (64bit) is deployed, the CPU is Intel 6330*2, the GPU is NVIDIAA800, the integrated development environment is Vscode (64bit), the programming language is Python3.6 (64bit), and the deep learning framework is PyTorch1.13. In the training of the national language model, the maximum input length of a Mongolian sentence is limited to 512 tokens, the maximum output length of a single Mongolian sentence is 512 tokens, and the total length of the model is 4096 tokens. After repeated debugging and optimization, when the experimental parameters are set as shown in Table 2, the experimental results are better.

[0134] Table 2 Parameters of the national language model

[0135]

[0136] In addition, this application evaluates the language understanding ability of the large model. The evaluation mainly includes sentence-level semantic understanding and token-level semantic understanding. Precision (P), recall (R), and (F1) values ​​are used as evaluation indicators of the model. The specific definitions are shown in formulas (1 to (3).

[0137]

[0138] This paper evaluates the language generation ability of large models mainly including the translation task, and adopts the BLEU evaluation method. The specific calculation formulas are shown in (4 and (5).

[0139]

[0140] BP=e min(1-r / c,0 ) (5

[0141] In the formula, BP represents the sentence length penalty factor, (w n ) and N are the weight of the n-gram and the maximum number of primitives in the n-gram, respectively. n Represents the accuracy of n-grams. r represents the length of valid reference translations in the corpus test set, and c is the length of candidate translations in machine translation. The higher the BLEU evaluation value, the better the model translation performance, the better the readability of the translation, and the stronger the comprehensibility.

[0142] There is no Mongolian corpus of the national language big model publicly available. In addition, the research questions, corpora, experimental environments and research ideas of the big model related literature are different, and the experimental research results are not very comparable. Therefore, in order to verify the universality, effectiveness and practicality of the national language big model in this paper, 9 classic models commonly used in the research field are selected as baseline models to implement text classification tasks, entity extraction tasks and big model translation tasks, and compared with the Mo-LLM proposed in this paper. The comparison of the experimental results is listed in Table 3, Table 4 and Table 5.

[0143] From the classification evaluation experimental results of the large model in Table 3, we can analyze that the baseline group model experiments all use a Transformer-based multilingual masked language model to implement sentence classification and obtain relatively good results. However, the XLM-R model may encounter the problem of insufficient representation learning when dealing with low-resource languages. The CINO model has relatively few corpora of minority languages, which affects the accuracy of the pre-trained model. Therefore, it can be seen that the F1 values ​​of these models are lower than the evaluation values ​​of the Mo-LLM we proposed.

[0144] Table 3 Classification evaluation experimental results of large models

[0145]

[0146] From the named entity recognition evaluation experimental results of the large model in Table 4, we can analyze that the CRF baseline model is a probabilistic graph model used to predict label sequences in sequence data, but it has a weak modeling ability for long-distance dependencies. The other two LSTM series baseline models (LSTM-CRF and BiLSTM-CRF) implement named entity recognition and obtain relatively good results, but LSTM cannot capture long-distance context feature information and only considers the local features of the sentence. Therefore, it can be seen that compared with the Mo-LLM model in this article, the P, R, and F1 values ​​of these baseline models are lower.

[0147] Table 4 Experimental results of named entity recognition evaluation of large models

[0148]

[0149] From the experimental results of the language generation ability evaluation results of the large models in Table 5, we can analyze that the model in this paper fully integrates the context-aware advantages of the BERT model and the controllable generation advantages of GPT, so it can be seen that Mo-LLM has achieved the highest score in the BLEU indicator.

[0150] Table 5 Evaluation results of language generation ability of large models

[0151]

[0152] Finally, in order to adapt the Mongolian language model to other ethnic languages ​​and achieve the purpose of solving the language comprehension and language generation capabilities of the ethnic language model, this paper first integrates the pre-trained model of Mongolian to Chinese semantic alignment based on the nine-grid model according to the characteristics of ethnic languages, and then the model realizes Chinese to Mongolian semantic output through the fusion of the nine-grid model and the four-layer Transformer, and finally fine-tunes the nine-grid model through PEFT and LoRA to realize the function of ethnic language intercommunication. The experimental results in Tables 3, 4 and 5 show that the F1 value and BLEU value on various benchmark data sets are relatively the best, verifying the adaptability and effectiveness of the model proposed in this paper.

[0153] Based on the word-forming morphology of ethnic languages, this application proposes an end-to-end universal ethnic language model based on the analysis of its complex morphology, many unregistered words, and scarce corpora. It takes Mongolian as the starting point of the research and aims to provide a new solution for the processing and analysis of ethnic languages. The model first uses the established encoder to encode the input of each ethnic language, and projects the information embedding representation into the representation of the language understandable by the universal large model through the projection layer. Then the universal large model takes the representations from different ethnic languages ​​as input, and performs semantic understanding and reasoning on the input. Finally, it receives the embedded information representation with specific instructions from the universal large model, and maps the semantic information representation to the language representation that can be understood by the subsequent decoder based on the output projection layer of the Transformer, thereby having a large model of ethnic languages ​​with generation capabilities. The adaptability and effectiveness of this model were verified through the Mongolian large model language comprehension ability evaluation and large model language generation ability evaluation experiments.

[0154] It should be noted that this article uses the Linux system as an example to illustrate the embodiment. Technical personnel in this field can also deploy it on other operating systems according to actual conditions, such as on OpenKylin and Windows systems, etc. This application is not limited to this.

[0155] In addition, an exemplary embodiment of the present application further provides a national language big model construction device for an operating system based on a general big model, wherein the national language big model includes a multilingual encoding layer, an LLM understanding and reasoning layer, and a multilingual generation layer, wherein the multilingual encoding layer is composed of a pre-trained language model, a projection layer, and a general big model, and the device includes:

[0156] A vocabulary generation module, used to generate a vocabulary for multilingual training corpus;

[0157] A representation conversion module, used to convert the input words in the low-resource ethnic language into word embedding representations using the pre-trained language model to obtain word embedding vectors;

[0158] A vector input module, used for inputting the word embedding vector into the projection layer to form an alignable embedding representation vector;

[0159] A high-resource mapping module is used to obtain a word embedding vector suitable for the embedding representation vector in the vocabulary of the training corpus and map it to the large model semantic space to complete the mapping of the low-resource ethnic language to the high-resource ethnic language;

[0160] A general large model module, used to input the training corpus corresponding to the mapped high-resource ethnic language and low-resource ethnic language into the general large model, and output the corresponding word embedding representation;

[0161] A final representation module, used for inputting the corresponding word embedding representation into the multilingual generation layer to obtain a final embedding representation;

[0162] A low-resource mapping module is used to iteratively update the pre-trained language model according to the final embedding representation to achieve mapping from a high-resource ethnic language to a low-resource ethnic language.

[0163] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0164] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0165] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for constructing a national language big model based on a general big model for an operating system, suitable for deployment to an operating system, characterized in that: The national language macro model includes a multilingual encoding layer, an LLM understanding and reasoning layer, and a multilingual generation layer, wherein the multilingual encoding layer is composed of a pre-trained language model, a projection layer, and a general macro model, and the method includes: Generate vocabulary for multilingual training corpus; Using the pre-trained language model, converting input words in a low-resource ethnic language into word embedding representations to obtain word embedding vectors; Sending the word embedding vector to the projection layer to form an alignable embedding representation vector; In the vocabulary of the training corpus, a word embedding vector suitable for the embedding representation vector is obtained and mapped to the semantic space of the large model to complete the mapping of the low-resource ethnic language to the high-resource ethnic language; Input the training corpus corresponding to the mapped high-resource ethnic language and low-resource ethnic language into the universal large model, and output the corresponding word embedding representation; Inputting the corresponding word embedding representation into the multilingual generation layer to obtain a final embedding representation; The pre-trained language model is iteratively updated according to the final embedding representation to achieve mapping from high-resource ethnic languages ​​to low-resource ethnic languages.

2. The method according to claim 1, characterized in that The pre-trained language model uses the BERT architecture, and the pre-trained language model is used to convert the input words in the low-resource ethnic language into word embedding representations to obtain word embedding vectors, including: Each word in the input low-resource ethnic language is converted into a vector of fixed size; Look up the index in the vocabulary to get an initialized word embedding vector for each word; The pre-trained language model of the BERT architecture is used to add position codes in the language sequence to each word, and the position codes are a set of learnable vectors. In the pre-trained language model of the BERT architecture, sentence pairs are distinguished by adding Segment Embedding identifiers; The initialized word embedding vector, the position code and the Segment Embedding identifier are added to obtain a word embedding representation of each word in the low-resource ethnic language.

3. The method according to claim 2, characterized in that After obtaining the word embedding representation of each word in the low-resource ethnic language, the method further includes: Inputting the word embedding representation of each word into a self-attention layer for processing, the self-attention layer introduces a multi-head attention mechanism for processing the word embedding representation of each word, and the multi-head attention mechanism is used to capture the context information of the word; The processed word embedding representations of each word are input into a feed-forward network for further processing, wherein the feed-forward network uses a linear transformation and a nonlinear activation function; Performing layer normalization on the self-attention layer and the feed-forward network, and adding the input and output of the self-attention layer and the feed-forward network through a residual connection; In the pre-trained language model of the BERT architecture, the final word embedding representation of each word in the low-resource ethnic language is output.

4. The method according to claim 1, characterized in that The projection layer is composed of a weight matrix; The step of sending the word embedding vector to the projection layer to form an alignable embedding representation vector includes: The input word embedding vector is multiplied by the weight matrix in the projection layer to obtain a transformed and alignable word embedding vector.

5. The method according to claim 1, characterized in that The training corpus corresponding to the mapped high-resource ethnic language and low-resource ethnic language is input into the universal large model, and the output is the corresponding word embedding representation, including: Mapping each word in the training corpus into a vector form through a word embedding matrix; Input each word in the training corpus in vector form into a self-attention layer for processing, and superimpose the self-attention layer through a multi-head attention mechanism; The processed word embedding representations of each word are input into a feed-forward network for further processing, wherein the feed-forward network uses a linear transformation and a nonlinear activation function; Helps stabilize and accelerate the training of deep networks through layer normalization and residual connections; Update the general large model by further minimizing the loss function and use the optimization algorithm to update the word embedding representation; Output the final word embedding representation.

6. The method according to any one of claims 1 to 5, characterized in that: After the mapping from the high-resource ethnic language to the low-resource ethnic language is achieved, the method further includes: LoRA is used to fine-tune some parameters in the national language model, and the fine-tuning is used to achieve simultaneous updating of the projection layers at both ends.

7. A device for constructing a national language big model based on a general big model and oriented to an operating system, characterized in that: The national language large model includes a multilingual encoding layer, an LLM understanding and reasoning layer, and a multilingual generation layer, wherein the multilingual encoding layer is composed of a pre-trained language model, a projection layer, and a general large model, and the device includes: A vocabulary generation module, used to generate a vocabulary for multilingual training corpus; A representation conversion module, used to convert the input words in the low-resource ethnic language into word embedding representations using the pre-trained language model to obtain word embedding vectors; A vector input module, used for inputting the word embedding vector into the projection layer to form an alignable embedding representation vector; A high-resource mapping module is used to obtain a word embedding vector suitable for the embedding representation vector in the vocabulary of the training corpus and map it to the large model semantic space to complete the mapping of the low-resource ethnic language to the high-resource ethnic language; A general large model module, used to input the training corpus corresponding to the mapped high-resource ethnic language and low-resource ethnic language into the general large model, and output the corresponding word embedding representation; A final representation module, used for inputting the corresponding word embedding representation into the multilingual generation layer to obtain a final embedding representation; A low-resource mapping module is used to iteratively update the pre-trained language model according to the final embedding representation to achieve mapping from a high-resource ethnic language to a low-resource ethnic language.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the method for constructing a national language big model based on a general big model and oriented to an operating system as described in any one of claims 1 to 6 above.

Images