A multi-language large model training method, device and related equipment

Abstract

The present invention discloses a multilingual large model training method, apparatus, and related equipment. The method comprises: obtaining parallel corpus data and obtaining monolingual corpus data of low-resource languages ​​from the monolingual corpus data; synchronously inputting the monolingual corpus data and the parallel corpus data into a shared parameter autoencoder and a target multilingual large model, wherein the target multilingual large model includes a knowledge transfer layer for enhancing attention to corpora of similar meaning in different languages; constructing a ternary data set and inputting it into the model to obtain a ternary loss; and updating the model using the ternary loss. The present invention can improve the utilization efficiency of parallel corpus data, enhance model training efficiency and output quality, and improve the multilingual large model's ability to understand low-resource languages ​​without affecting its capabilities.

Description

Technical Field

[0001] The present invention relates to the technical field of language large model knowledge migration, and in particular to a multi-language large model training method, apparatus and related equipment. Background Art

[0002] In the field of artificial intelligence, language models, especially large-scale language models, have become a core force driving technological progress. These models, such as GPT (generative pre-trained transformer) and BERT (bidirectional encoder representations from transformers), are pre-trained on large amounts of data, using masked language models as their training targets. They understand and generate human language and are applied to various downstream natural language processing tasks, such as machine translation and text classification.

[0003] Large multilingual models are an extension of this approach, designed to process and understand multiple languages. These models are typically trained on text in multiple languages ​​to learn the similarities and differences between languages, enabling effective information transfer and communication across them.

[0004] Currently, large multilingual models have poor understanding of low-resource languages. Since the amount of available data for low-resource languages ​​is far less than that for high-resource languages, it is difficult for the models to capture the complexity and diversity of the language through sufficient training data. In addition, the vocabulary of different languages ​​has its own uniqueness based on specific cultural backgrounds, and a single training goal is difficult to learn this uniqueness of different languages.

[0005] Therefore, how to improve the utilization efficiency of parallel corpus data, enhance model training efficiency and output quality, and improve the ability of large multilingual models to understand low-resource languages ​​while maintaining their original capabilities is a technical problem that needs to be solved urgently. Summary of the Invention

[0006] The embodiments of the present invention provide a multilingual large-scale model training method, apparatus, and related equipment. By introducing a parallel training structure, training parameters are shared between different tasks, thereby improving model training efficiency and feature extraction capabilities. In addition, the method utilizes a strong understanding of high-resource languages ​​to transfer high-resource knowledge to low-resource languages ​​through a knowledge transfer layer. The training logic of contrastive learning is used to help the model deeply understand the expression of the same semantics in different languages, thereby enhancing the model's understanding of low-resource languages.

[0007] To solve the above technical problems, an embodiment of the present application provides a multilingual large model training method, comprising the following steps: obtaining parallel corpus data and extracting monolingual corpus data of low-resource languages ​​from the parallel corpus data, wherein the parallel corpus data includes corpus data of high-resource languages ​​and corpus data of low-resource languages; synchronously inputting the monolingual corpus data of low-resource languages ​​and the parallel corpus data into an autoencoder and a target multilingual large model, respectively, wherein some decoder layers of the autoencoder and the target multilingual large model share parameters, and the target multilingual large model includes a knowledge transfer layer, wherein the knowledge transfer layer includes a multi-head attention layer, and the multi-head attention layer is used to enhance attention between corpora with similar meanings in different languages; constructing a ternary data set based on the parallel corpus data, wherein the ternary data set includes positive examples of high-resource languages, positive examples of low-resource languages, and negative examples of low-resource languages; inputting the ternary data set into the target multilingual large model to calculate a ternary loss; and updating the target multilingual large model according to the ternary loss to obtain an updated multilingual large model.

[0008] In one possible implementation, the embedding of the low-resource language corpus data is used as a query, and the embedding of the high-resource language corpus data is used as a key and value to calculate the attention of each head; all heads are connected in series and projected through a linear transformation to obtain the attention of the merged multiple heads.

[0009] In another possible implementation, the attention algorithm of each head is as follows:

[0010]

[0011] Among them, softmax is an activation function that converts vector information into probability distribution. is the dimension of the embedding vector for high-resource language data.

[0012] In another possible implementation, the data features extracted by the decoder layer of the target multilingual large model are input into a feedforward network, and a dynamic weight matrix is ​​output, where the dynamic weight matrix includes a query dynamic weight matrix, a key dynamic weight matrix, and a value dynamic weight matrix.

[0013] In another possible implementation, an adapter is added for each of the low-resource languages, and a gating logic is used to control the weighted sum of shared parameters and each language adapter parameter, where the gating function includes a sigmoid function and a ReLU function.

[0014] In another possible implementation, the gating function is as follows:

[0015] g(z)=σ(ReLU(zW1+b)W2)

[0016] Wherein, z is the data feature before inputting the gating function, and W1, W2 and b are the parameters of the gating mechanism.

[0017] In another possible implementation, the ternary data in the ternary data set are mapped to the same vector space, and the ternary loss is calculated. The ternary loss calculation formula is as follows:

[0018] L(a,p,n)=max(d(a,p)-d(a,n)+margin,0)

[0019] Among them, a is the positive example of the high-resource language, p is the positive example of the low-resource language, n is the negative example of the low-resource language, d is the distance function, and margin is a positive interval hyperparameter.

[0020] In order to solve the above technical problems, the embodiment of the present application also provides a multilingual large model training device, including: an acquisition module, used to acquire parallel corpus data, and extract monolingual corpus data of low-resource languages ​​from the parallel corpus data, wherein the parallel corpus data includes high-resource language corpus data and low-resource language corpus data; a training module, used to synchronously input the monolingual corpus data of low-resource languages ​​and the parallel corpus data into an autoencoder and a target multilingual large model respectively, wherein the autoencoder and some decoder layers of the target multilingual large model share parameters, and the target multilingual The large model includes a knowledge transfer layer, which includes a multi-head attention layer, and the multi-head attention layer is used to enhance the attention between corpora with similar meanings in different languages; a construction module is used to construct a ternary data set based on the parallel corpus data, and the ternary data set includes positive examples in high-resource languages, positive examples in low-resource languages, and negative examples in low-resource languages; a calculation module is used to input the ternary data set into the target multilingual large model and calculate the ternary loss; and an update module is used to update the target multilingual large model according to the ternary loss to obtain an updated multilingual large model.

[0021] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0022] In order to solve the above technical problems, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0023] The multilingual large model training method provided by the embodiment of the present invention integrates different data forms, such as low-resource monolingual data and tri-gram comparative learning datasets, while ensuring data quality, and efficiently utilizes scarce low-resource parallel corpus data. Through the additional structures of parameter sharing, knowledge transfer layer and adapter layer, the target multilingual large model is fine-tuned without increasing excessive computational complexity, thereby improving model training efficiency. In addition, through multi-stage feature learning, the target multilingual large model can understand and learn low-resource languages ​​from multiple perspectives, thereby enhancing the robustness of the model and improving the output quality of the model. Through the structure of the adapter, a certain degree of flexibility is given to the fine-tuning of the multilingual large model. The data and parameter size of the adapter can be flexibly adjusted according to the language type of the existing data, and computing resource requirements can be dynamically arranged. Through different model training objectives, the model can learn the unique information of low-resource languages ​​under different tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0025] Figure 1 is a schematic diagram of an exemplary scenario in which the present application can be applied.

[0026] Figure 2 This is the overall flow chart of the multilingual large model training method of this application.

[0027] Figure 3 It is a structural diagram of the autoencoder of this application.

[0028] Figure 4 This is a schematic diagram of parameter sharing training in this application.

[0029] Figure 5 It is a schematic diagram of the adapter control logic of the present application.

[0030] Figure 6 It is a structural diagram of the multi-language large model training device of the present application.

[0031] Figure 7 It is a structural diagram of the computer device of this application. DETAILED DESCRIPTION

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of the application are for the purpose of describing specific embodiments only and are not intended to limit this application. The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0033] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0034] In order to better understand the contents of the embodiments of the present application, some terms and related technologies involved in the embodiments of the present application are first explained to facilitate understanding by those skilled in the art.

[0035] Large language models (LLMs) are deep learning models trained using large amounts of text data. They can generate natural language text or understand the meaning of text. Large language models can handle a variety of natural language tasks, such as text classification, question-answering, and conversational conversation. They are a key path to artificial intelligence. Currently, large language models use similar transformer architectures and pre-training objectives (such as language modeling) as small models. The difference between large models and small models lies in the increased model size, training data, and computing resources required.

[0036] High-resource languages ​​refer to languages ​​with high frequency of use and more training data, such as English and French. In contrast, low-resource languages ​​refer to languages ​​with low frequency of use and less training data, such as Inuktitut and Arabic.

[0037] Fine-tuning involves retraining a pre-trained model using task-specific datasets to adjust model parameters for a specific language or task. Common fine-tuning methods include: task-specific fine-tuning, which uses labeled data from a specific task (such as named entity recognition or sentiment analysis) to fine-tune the model; and language-specific fine-tuning, which uses data from a specific language to enhance the model's understanding and generation capabilities.

[0038] Knowledge transfer refers to the process of transferring knowledge learned by a model from one task or domain to another new task or domain. Knowledge transfer is widely used in large language models. One approach involves pre-training the model on a large amount of general text and then fine-tuning it on domain-specific data. This approach not only improves the model's performance on specific tasks but also significantly reduces development time and costs.

[0039] Feature learning is a key concept in machine learning. It involves using specific algorithms to automatically discover and learn important features in input data, thereby facilitating the subsequent classification, regression, or other specific tasks. Current feature learning algorithms are typically implemented using unsupervised or semi-supervised methods.

[0040] Currently, large language models learn the general and specific features of language through the masked language modeling task. In the masked language modeling task, the model randomly masks some words in the input sentence and then attempts to predict these masked words. The formula is as follows:

[0041] P(word|context)=softmax(W×h masked )

[0042] Among them, h masked is the hidden state of the masked word, and W is the weight matrix. This task helps the model infer the meaning of words through context and enhance its contextual understanding ability.

[0043] In addition, to effectively process multiple languages, byte pair encoding (BPE) can be used. BPE is a data compression technique used in natural language processing to handle unknown or rare vocabulary. BPE merges common characters or character sequences into a single unit based on frequency. Its operation can be iteratively represented as: merge(x, y)→z, where x and y are the most common pairs of adjacent symbols in the input, and they are merged into a new symbol z. In this way, BPE allows the model to dynamically represent vocabulary in different languages, including long-tail vocabulary, using a fixed-size vocabulary. Although the use of technologies such as BPE can improve the processing capabilities of large language models for multiple languages, their performance is still insufficient when processing low-resource languages. The main reasons for this are imbalanced training data and insufficient representation of low-resource languages.

[0044] Model training relies on large amounts of text data. For high-resource languages ​​(such as English and other major languages), the available training data is far more than that for other low-resource languages ​​(such as Arabic). This imbalance causes the model to perform well on high-resource languages ​​and poorly on low-resource languages, failing to learn sufficient language-specific features. In addition, even using subword segmentation representations such as BPE, the unique language structures and usages in low-resource languages ​​may still not be fully represented because most of the model's capabilities are used for more frequently occurring languages, rather than equally learning the nuances of all languages.

[0045] To improve the performance of multilingual models on specific tasks or low-resource languages, the current general solution is to adopt a fine-tuning strategy. Although fine-tuning can improve model performance on specific tasks and languages, it still has certain drawbacks when dealing with low-resource languages. For example: 1. It is highly dependent on high-quality annotated data (which is often difficult to obtain for low-resource languages); 2. In low-resource languages, the limited available data causes the model to over-adapt to the specific characteristics of the training data, which affects its generalization ability and poses the risk of overfitting; 3. Fine-tuning requires additional computing resources and time, which is a significant challenge for applications that need to support multiple languages; 4. Fine-tuning on new datasets may cause catastrophic forgetting of the original large model's capabilities, causing the model to lose most of its original semantic understanding capabilities.

[0046] Existing approaches have proposed solutions to the aforementioned problems of fine-tuning. For example, they adjust the weights of training samples in different languages ​​based on the difficulty or rarity of the input corpus, focusing on more challenging or less common words and phrases to improve the model's understanding of low-resource languages. This approach enhances model learning where it is most needed by focusing on those parts that are difficult to predict correctly during training. While this focused learning approach has potential for improving model understanding of low-resource languages, it also has some drawbacks. This approach may overemphasize difficult-to-predict samples, leading to overfitting during training. Furthermore, the success of this approach depends heavily on proper initialization and parameter tuning; inappropriate parameter settings can weaken model performance. Another approach is to expand the low-resource language training sample through back-translation (using a machine translation model based on a parallel corpus pair to translate text from a low-resource language into a high-resource language and then back into the original low-resource language). Because the back-translation process often introduces noise due to performance issues with the machine translation model, the new low-resource language data obtained after back-translation can be used as a new batch of data for subsequent model fine-tuning. During the training phase, the embedding layer of the input text can be dynamically adjusted to ensure that text in different languages ​​has different embedding information. Model distillation is also considered to transfer the capabilities of pre-trained multilingual models to lightweight models. However, the addition of a dynamic embedding layer in this method may significantly increase computational costs, and the use of back-translation for data augmentation carries certain risks. This process is heavily dependent on the quality of the machine translation model. If the quality is too poor, the back-translation results will contain too many errors, which will in turn reduce the final model capabilities. In addition, the back-translation output is also affected by the inherent biases and context loss of the machine translation model itself, which will ultimately affect the fine-tuning training quality of the large multilingual model.

[0047] It can be seen that the existing solutions can to some extent solve the problems of low fine-tuning efficiency of large multilingual models and insufficient understanding of low-resource languages. However, there are still problems such as the inability to efficiently utilize existing corpus data to improve the model's understanding of low-resource languages, and the inability to understand and learn the unique information of low-resource languages ​​in different tasks from multiple perspectives.

[0048] Based on the above, the embodiments of the present application provide a multilingual large model training method, apparatus and related equipment, which uses low-resource language monolingual data to synchronously train an autoencoder with the goal of reconstructing the original information, and shares the parameters of this autoencoder with the target multilingual large model, thereby improving training efficiency and enhancing the model's ability to understand the common grammar and semantics of low-resource languages. In the fine-tuning stage, an additional knowledge transfer layer is established, with the attention mechanism as the core, to transfer the target large model's understanding ability of high-resource languages ​​to low-resource languages, and synchronously construct a ternary data set to further migrate and align the semantics of the same vocabulary in different languages ​​through comparative learning.

[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0050] See also Figure 1 ,like Figure 1 As shown, in this scenario, system 100 may include a model training system 110, a network 120, and a corpus database 130. Network 120 is used as a medium for providing a communication link between model training system 110 and corpus database 130. Network 120 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0051] The user can use the model training system 110 to interact with the corpus database 130 through the network 120 to receive or send information, etc.

[0052] The model training system 110 can be deployed on a server that provides large model applications, such as a backend server that provides model training for parallel corpus data stored in a server that deploys the corpus database 130 .

[0053] The corpus database 130 can be deployed in an edge computing device or a central computing device, for example, in a database server.

[0054] It should be noted that the multilingual large model training method provided in the embodiment of the present application is executed by a server, and accordingly, the multilingual large model training device is set in the server.

[0055] It should be understood that Figure 1 The number of model training systems, networks, and corpus databases in the example is merely illustrative. Any number of model training systems, networks, and corpus databases may be used depending on the implementation requirements.

[0056] See also Figure 2 , Figure 2 The following is a general flow chart of a multilingual large model training method provided by an embodiment of the present invention, which is described in detail below.

[0057] S201. Obtain parallel corpus data.

[0058] Specifically, the parallel corpus data may be parallel corpus data stored in a corpus database, or parallel corpus data obtained through a web page, or parallel corpus data obtained through other means. This application does not limit the method for obtaining the parallel corpus data.

[0059] It should be noted that parallel corpus data refers to the corpus data formed by the correspondence between high-resource languages ​​and low-resource languages. For example, English is regarded as a high-resource language and Chinese is regarded as a low-resource language. For the same corpus sample, there are two ways of expression, one in a high-resource language and the other in a low-resource language. These two expressions together constitute parallel corpus data. For example, "I love natural language processing I love natural language processing", which represents a parallel corpus sample.

[0060] Furthermore, the monolingual corpus data of the low-resource language is extracted separately from the original high-resource to low-resource parallel corpus data. For example, only "I love natural language processing" is extracted, and then preprocessing operations such as data cleaning are performed on it.

[0061] S202: Synchronously input the monolingual corpus data and the parallel corpus data of the low-resource language into the autoencoder and the target multilingual large model respectively.

[0062] Specifically, an autoencoder with a multi-layer transformer framework is established, and parameters are shared between the autoencoder and the transformer blocks of the first few layers of the target multilingual large model. Through parameter sharing, the target multilingual large model can learn the monolingual features of different low-resource languages ​​from the autoencoder while retaining its own capture of the common grammatical and semantic features of the language, thereby enhancing the model's generalization ability to understand different low-resource languages ​​and preventing the model from overfitting to a certain language.

[0063] For example, see Figure 3 , Figure 3 This is a schematic diagram of the structure of an autoencoder provided in an embodiment of the present application. Figure 3 As shown in the figure, the training goal of the autoencoder is to reconstruct the input data and minimize the difference between the input data and the reconstructed output data. The encoder learns the hidden features of the input data, and then the decoder reconstructs the original input data from the hidden features.

[0064] Furthermore, when training the target multilingual large model and the autoencoder in parallel, the parameters of the autoencoder are shared with the first few decoder layers of the target multilingual large model. Generally, the multilingual large model framework is decoder-only, so a relatively simple and direct parameter sharing method is to share parameters between the decoding part of the autoencoder and the large model, such as Figure 4 As shown, fine-tuning the target multilingual model is performed simultaneously with the autoencoder training. In addition to the decoder layer, which shares parameters, the target multilingual model also has an independent decoder layer whose parameters are not shared with the autoencoder. This sharing of parameters effectively improves the training efficiency of the target multilingual model.

[0065] In one possible implementation, an additional knowledge transfer layer is established within the existing framework of the target multilingual model. This knowledge transfer layer is based on a multi-head attention layer and is used to enhance the attention between corpora with similar meanings in different languages.

[0066] Optionally, the embedding of the low-resource language corpus data is used as the query (Q), and the embedding of the high-resource language corpus data is used as the key (K) and value (V), so that the interaction between the two languages ​​can be taken into account when calculating the attention output.

[0067] Specifically, the calculation of each head can be obtained by the following formula:

[0068]

[0069] Among them, softmax is an activation function that converts vector information into probability distribution. is the dimension of the embedding vector for high-resource language data. After calculating the attention of each head, all heads are concatenated and projected through a linear transformation (such as another randomly initialized weight matrix W) to merge the information of multiple heads. The specific process can be calculated using the following formula:

[0070] MultiHead(Q,K,V)=Concat(Attention1,Attention2,…,Attention h )×W

[0071] After calculation using the above formula, we can obtain the information of the merged bulls.

[0072] In one possible implementation, before inputting the data into the knowledge transfer layer, the data features extracted by the decoder layer of the target multilingual large model are input into the feedforward network, and a dynamic weight matrix is ​​output, which includes a query dynamic weight matrix, a key dynamic weight matrix, and a value dynamic weight matrix.

[0073] Specifically, in the attention mechanism, the three elements of query (Q), key (K), and value (V) need to be calculated from the product of the input data and the corresponding weights. The specific calculation method is as follows:

[0074] Q=W Q ×X,K=W K ×X,V=W V ×X

[0075] Among them, X is the current input data. Under the standard attention training logic, these three weight matrices W Q , W K , W V Automatically adjusted through backpropagation during the overall learning process, these weights are primarily dependent on the model's overall learning objectives and training data. Therefore, these weights are fixed after the model training process is complete. During the inference phase, they are fixed for any given input and do not dynamically adjust based on the specific content of the input data. However, as a language model, for different input corpora, based on differences in grammar, voice, and sentence type, the model should pay different attention to the same position in different sentences, thus requiring adaptive and dynamic attention weights.

[0076] Therefore, in order to generate adaptive weights, a feedforward network is added before the original knowledge transfer layer. The input of this network is the currently extracted data features, and the output is W Q , W K , W V Three weight matrices. Under this structure, the model's response at the knowledge transfer layer is adjusted according to the specific characteristics of the input, allowing the model to perform personalized processing for each unique input.

[0077] It can be understood that this approach enables the model to effectively handle diverse input data, such as preferences of different users, different types of text, or applications in multilingual environments.

[0078] In one possible implementation, an adapter is added for each low-resource language, and the weighted sum of shared parameters and each language adapter parameter is controlled by gating logic. The gating function includes a sigmoid function and a ReLU function.

[0079] Specifically, instead of using a unified alignment layer, an additional adapter is designed for each low-resource language in the dataset. Gating logic is used to control the weighted sum of shared parameters and the parameters of each language adapter. In this implementation, the gating function can be composed of a sigmoid function and a ReLU function. The sigmoid function controls the output of the gating logic between [0, 1], while the ReLU function provides some nonlinear features.

[0080] Furthermore, the specific gating function can be defined by the following formula:

[0081] g(z)=σ(ReLU(zW1+b)W2)

[0082] Here, z represents the data features before entering the gating function (i.e., input data), and W1, W2, and b are the parameters of the gating mechanism. During training, the adapter also learns the corresponding language through the gating mechanism, allowing the model to obtain information from the corresponding adapter when inputting corresponding low-resource languages.

[0083] For example, see Figure 5 , Figure 5 This is a schematic diagram of an adapter control logic provided by an embodiment of the present application. Figure 5 As shown, after obtaining the input data, it is input into the shared decoder layer and the adapter layer corresponding to each low-resource language, such as the Thai adapter layer, the Chinese adapter layer, etc. For each adapter layer, its output and the output of the shared decoder layer are input into the gating function together to obtain the data output after each network layer. For example, the shared decoder layer and the Chinese adapter layer are passed through the gating logic to obtain the data output after the network layer.

[0084] It should be understood that the above structure allows the model to be more flexible and effective when processing multilingual data, and can adjust the contribution of different language information according to actual conditions, thereby improving overall performance and adaptability. This is especially beneficial for large multilingual models because they maintain wide language coverage while also optimizing the specific performance of each language.

[0085] S203. Construct a ternary data set based on the parallel corpus data.

[0086] Specifically, in order to enable the target multilingual large model to perform comparative learning, a ternary dataset is sampled and constructed from the original parallel corpus dataset. The ternary dataset consists of positive examples in high-resource languages, positive examples in low-resource languages, and negative examples in low-resource languages.

[0087] For example, consider English as a high-resource language and Chinese as a low-resource language. Then, process the original parallel corpus data into the following format: English positive example: "she enjoys reading books and playing tennis", Chinese positive example: "She likes reading books and playing tennis", Chinese negative example: "He often goes to the library to study". This set of triple data is used as part of the training data for fine-tuning the target multilingual large model.

[0088] S204: Input the ternary dataset into the target multilingual model and calculate the ternary loss.

[0089] Specifically, after the ternary data set is input into the target multilingual large model, during the contrastive learning process, after feature extraction by the previous network layer, a set of ternary data is mapped to the same vector space, and then the ternary loss is calculated. Through training, the distance between the positive and negative examples is increased as much as possible while shortening the distance between the two positive examples.

[0090] In one possible implementation, the ternary loss can be calculated using the following formula:

[0091] L(a,p,n)=max(d(a,p)-d(a,n)+margin,0)

[0092] Where a is a positive example in a high-resource language, p is a positive example in a low-resource language, n is a negative example in a low-resource language, d is a distance function, such as the Euclidean distance function, and margin is an optional positive hyperparameter with a value between [0, 2]. The purpose of margin is to provide an optimization goal so that the model can separate positive and negative examples by at least the margin value. Setting margin also prevents large gaps between positive and negative examples, which can lead to poor generalization and overfitting of the model.

[0093] S205. Update the target multilingual large model based on the ternary loss.

[0094] Specifically, after obtaining the ternary loss L, the ternary loss L can be used to update the parameters of the target multilingual large model so that the same language expressions (i.e., positive examples) are closer than different expressions (i.e., negative examples) in the embedding space.

[0095] It should be understood that the updated target multilingual large model has improved its ability to understand the vocabulary of low-resource languages ​​and can be deployed and applied in relevant scenarios, thereby expanding its application scenarios and enhancing its processing effect on low-resource languages.

[0096] In this embodiment, while ensuring data quality, different data forms are integrated, such as low-resource monolingual data and tri-gram contrastive learning datasets, and scarce low-resource parallel corpus data is efficiently utilized. Through the additional structures of parameter sharing, knowledge transfer layer and adapter layer, the target multilingual large model is fine-tuned without increasing excessive computational complexity, thereby improving model training efficiency. In addition, through multi-stage feature learning, the target multilingual large model is able to understand and learn low-resource languages ​​from multiple perspectives, thereby enhancing the robustness of the model and improving the output quality of the model. Through the structure of the adapter, a certain degree of flexibility is given to the fine-tuning of the multilingual large model. The data and parameter size of the adapter can be flexibly adjusted according to the language type of the existing data, and the computing resource requirements can be dynamically arranged. Through different model training objectives, the model can learn the unique information of low-resource languages ​​under different tasks.

[0097] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0098] Figure 6 FIG. 2 shows a block diagram of a multi-language large model training device 200 corresponding to the above embodiment. Figure 6 As shown, the system 200 includes an acquisition module 210, a training module 220, a construction module 230, a calculation module 240, and an update module 250. Each functional module is described in detail as follows.

[0099] The acquisition module 210 is configured to acquire parallel corpus data and extract monolingual corpus data of low-resource languages ​​from the parallel corpus data. The parallel corpus data includes high-resource language corpus data and low-resource language corpus data.

[0100] The training module 220 is used to synchronously input the monolingual corpus data of the low-resource language and the parallel corpus data into the autoencoder and the target multilingual large model respectively. The autoencoder and some decoder layers of the target multilingual large model share parameters. The target multilingual large model includes a knowledge transfer layer, and the knowledge transfer layer includes a multi-head attention layer. The multi-head attention layer is used to enhance the attention between corpora with similar meanings in different languages.

[0101] The construction module 230 is configured to construct a ternary data set based on the parallel corpus data, wherein the ternary data set includes positive examples in high-resource languages, positive examples in low-resource languages, and negative examples in low-resource languages.

[0102] The calculation module 240 is used to input the triple data set into the target multilingual large model to calculate the triple loss.

[0103] The updating module 250 is configured to update the target multilingual large model according to the ternary loss to obtain an updated multilingual large model.

[0104] For the specific definition of the multi-language large model training device, please refer to the definition of the multi-language large model training method above, which will not be repeated here. The various modules in the above-mentioned multi-language large model training device can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0105] To solve the above technical problems, the present application also provides a computer device. Figure 7 , Figure 7 This is a basic structural block diagram of the computer device in this embodiment.

[0106] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 300 having components connected to the memory 310, the processor 320, and the network interface 330, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc.

[0107] The computer device may be a desktop computer, notebook computer, PDA, cloud server, etc. The computer device may interact with the user via a keyboard, mouse, remote control, touchpad, or voice control device.

[0108] The memory 310 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or D interface display memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, magnetic disk, optical disk, etc. In some embodiments, the memory 310 may be an internal storage unit of the computer device 300, such as the hard disk or memory of the computer device 300. In other embodiments, the memory 310 may also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Of course, the memory 310 may also include both the internal storage unit of the computer device 300 and its external storage device. In this embodiment, the memory 310 is generally used to store the operating system and various application software installed on the computer device 300, such as program codes for controlling electronic files, etc. In addition, the memory 310 can also be used to temporarily store various data that has been output or is about to be output.

[0109] In some embodiments, the processor 320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 320 is generally used to control the overall operation of the computer device 300. In this embodiment, the processor 320 is used to execute program code stored in the memory 310 or process data, such as executing program code for controlling electronic files.

[0110] The network interface 330 may include a wireless network interface or a wired network interface. The network interface 330 is generally used to establish a communication connection between the computer device 300 and other electronic devices.

[0111] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores an interface display program, and the interface display program can be executed by at least one processor to enable the at least one processor to perform the steps of the above method.

[0112] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0113] Obviously, the embodiments described above are only some of the embodiments of the present application, rather than all of the embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions described in the aforementioned specific embodiments, or to make equivalent replacements for some of the technical features therein. Any equivalent structure made using the contents of the present application specification and the accompanying drawings, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present application.

Claims

1. A multilingual large model training method, characterized in that: The method comprises: Acquiring parallel corpus data, and extracting monolingual corpus data of low-resource languages ​​from the parallel corpus data, wherein the parallel corpus data includes corpus data of high-resource languages ​​and corpus data of low-resource languages; The monolingual corpus data of the low-resource language and the parallel corpus data are synchronously input into an autoencoder and a target multilingual large model, respectively. The autoencoder and some decoder layers of the target multilingual large model share parameters. The target multilingual large model includes a knowledge transfer layer, and the knowledge transfer layer includes a multi-head attention layer. The multi-head attention layer is used to use the embedding of the low-resource language corpus data as a query and the embedding of the high-resource language corpus data as a key and value, calculate the attention of each head, concatenate all heads, and project them through a linear transformation to obtain a merged multi-head attention. Based on the parallel corpus data, construct a ternary data set, wherein the ternary data set includes positive examples in high-resource languages, positive examples in low-resource languages, and negative examples in low-resource languages; Inputting the ternary data set into the target multilingual large model to calculate the ternary loss; The target multilingual large model is updated according to the ternary loss to obtain an updated multilingual large model.

2. The method according to claim 1, wherein The attention algorithm for each head is as follows: Among them, softmax is an activation function that converts vector information into probability distribution. is the dimension of the embedding vector for high-resource language data.

3. The method according to claim 1, wherein Before inputting the data into the knowledge transfer layer, the method further includes: The data features extracted by the decoder layer of the target multilingual large model are input into a feedforward network, and a dynamic weight matrix is ​​output. The dynamic weight matrix includes a query dynamic weight matrix, a key dynamic weight matrix, and a value dynamic weight matrix.

4. The method according to claim 1, wherein The method further comprises: An adapter is added for each of the low-resource languages, and the weighted sum of shared parameters and each language adapter parameter is controlled by gating logic, wherein the gating function includes a sigmoid function and a ReLU function.

5. The method according to claim 4, wherein The gating function is as follows: g(z)=σ(ReLU(zW1+b)W2) Wherein, z is the data feature before inputting the gating function, and W1, W2 and b are the parameters of the gating mechanism.

6. The method according to claim 1, wherein Inputting the ternary data set into the target multilingual large model and calculating the ternary loss includes: The ternary data in the ternary data set are mapped to the same vector space, and the ternary loss is calculated. The ternary loss calculation formula is as follows: L(a,p,n)=max(d(a,p)-d(a,n)+margin,0) Among them, a is the positive example of the high-resource language, p is the positive example of the low-resource language, n is the negative example of the low-resource language, d is the distance function, and margin is a positive interval hyperparameter.

7. A multi-language large model training device, characterized in that: The device comprises: An acquisition module, configured to acquire parallel corpus data and extract monolingual corpus data of low-resource languages ​​from the parallel corpus data, wherein the parallel corpus data includes corpus data of high-resource languages ​​and corpus data of low-resource languages; A training module is configured to synchronously input the low-resource language monolingual corpus data and the parallel corpus data into an autoencoder and a target multilingual large model, respectively. The autoencoder and some decoder layers of the target multilingual large model share parameters. The target multilingual large model includes a knowledge transfer layer, which includes a multi-head attention layer. The multi-head attention layer is configured to use the embedding of the low-resource language corpus data as a query and the embedding of the high-resource language corpus data as a key and value, calculate the attention of each head, concatenate all heads, and project them through a linear transformation to obtain a merged multi-head attention. A construction module, configured to construct a ternary data set based on the parallel corpus data, wherein the ternary data set includes positive examples in high-resource languages, positive examples in low-resource languages, and negative examples in low-resource languages; a calculation module, configured to input the ternary data set into the target multilingual large model and calculate the ternary loss; An updating module is used to update the target multilingual large model according to the ternary loss to obtain an updated multilingual large model.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images