Multi-language large model training method and device and related equipment

Abstract

The invention discloses a multi-language large model training method and device and related equipment, and the method comprises the steps: obtaining parallel corpus data, and obtaining monolingual corpus data of a low-resource language from the parallel corpus data; the corpus data and parallel corpus data are synchronously input into an auto-encoder sharing parameters and a target multi-language large model, the target multi-language large model comprises a knowledge migration layer, and the knowledge migration layer is used for enhancing attention of corpora with similar meanings in different languages; constructing a ternary data set and inputting the ternary data set into the model to obtain ternary loss; and updating the model by using the ternary loss. By adopting the method, the utilization efficiency of the parallel corpus data can be improved, the model training efficiency and the output quality are improved, and the understanding ability of the multi-language large model to low-resource languages is improved on the basis of not influencing the ability of the multi-language large model.

Description

Technical Field

[0001] The present invention relates to the technical field of language large model knowledge migration, and in particular to a multi-language large model training method, device and related equipment. Background Art

[0002] In the field of artificial intelligence, language models, especially large language models, have become the core force driving technological progress. These models, such as GPT (generative pre-trained transformer) and BERT (bidirectional encoder representations from transformers, BERT), use large-scale data pre-training to mask language models as training targets, understand and generate human language, and are applied to various natural language processing downstream tasks, such as machine translation and text classification.

[0003] Multilingual large models are an extension of this, and they are designed to process and understand multiple languages. Such models are usually trained on texts in multiple languages ​​to learn the commonalities and differences between different languages, so that they can effectively transfer information and communicate between different languages.

[0004] Currently, large multilingual models have poor understanding of low-resource languages. Since the amount of available data for low-resource languages ​​is far less than that for high-resource languages, it is difficult for the models to capture the complexity and diversity of the language through sufficient training data. In addition, the vocabulary of different languages ​​has its own uniqueness based on specific cultural backgrounds, and a single training goal is difficult to learn the uniqueness of different languages.

[0005] Therefore, how to improve the utilization efficiency of parallel corpus data, enhance model training efficiency and output quality, and improve the ability of multilingual large models to understand low-resource languages ​​while ensuring their original capabilities is a technical problem that needs to be solved urgently. Summary of the invention

[0006] The embodiments of the present invention provide a multi-language large model training method, apparatus and related equipment, which introduce a parallel training structure to share training parameters between different tasks, improve model training efficiency and feature extraction capabilities, and utilize a strong understanding of high-resource languages ​​to transfer high-resource knowledge to low-resource languages ​​through a knowledge transfer layer. The training logic of contrastive learning is used to help the model deeply understand the expression of the same semantics in different languages, thereby enhancing the model's understanding of low-resource languages.

[0007] In order to solve the above technical problems, an embodiment of the present application provides a multilingual large model training method, comprising the following steps: obtaining parallel corpus data, and extracting monolingual corpus data of low-resource languages ​​from the parallel corpus data, wherein the parallel corpus data includes high-resource language corpus data and low-resource language corpus data; synchronously inputting the monolingual corpus data of low-resource languages ​​and the parallel corpus data into an autoencoder and a target multilingual large model respectively, wherein some decoder layers of the autoencoder and the target multilingual large model share parameters, and the target multilingual large model includes a knowledge transfer layer, wherein the knowledge transfer layer includes a multi-head attention layer, and the multi-head attention layer is used to enhance the attention between corpora with similar meanings in different languages; constructing a ternary data set based on the parallel corpus data, wherein the ternary data set includes positive examples of high-resource languages, positive examples of low-resource languages, and negative examples of low-resource languages; inputting the ternary data set into the target multilingual large model, and calculating the ternary loss; and updating the target multilingual large model according to the ternary loss to obtain an updated multilingual large model.

[0008] In one possible implementation, the embedding of the low-resource language corpus data is used as a query, and the embedding of the high-resource language corpus data is used as a key and value to calculate the attention of each head; all heads are connected in series and projected through linear transformation to obtain the attention of the merged multiple heads.

[0009] In another possible implementation, the attention algorithm of each head is as follows:

[0010]

[0011] Among them, softmax is an activation function that converts vector information into probability distribution. is the dimension of the embedding vector for high-resource language data.

[0012] In another possible implementation, the data features extracted by the decoder layer of the target multilingual large model are input into a feedforward network, and a dynamic weight matrix is ​​output, where the dynamic weight matrix includes a query dynamic weight matrix, a key dynamic weight matrix, and a value dynamic weight matrix.

[0013] In another possible implementation, an adapter is added for each of the low-resource languages, and the weighted sum of the shared parameters and the adapter parameters of each language is controlled by gating logic, wherein the gating function includes a sigmoid function and a ReLU function.

[0014] In another possible implementation, the gating function is as follows:

[0015] g(z)=σ(ReLU(zW1+b)W2)

[0016] Among them, z is the data feature before inputting the gating function, and W1, W2 and b are the parameters of the gating mechanism.

[0017] In another possible implementation, the ternary data in the ternary data set are mapped to the same vector space, and the ternary loss is calculated. The ternary loss calculation formula is as follows:

[0018] L(a,p,n)=max(d(a,p)-d(a,n)+margin,0)

[0019] Among them, a is a positive example in a high-resource language, p is a positive example in a low-resource language, n is a negative example in a low-resource language, d is a distance function, and margin is an interval hyperparameter with a positive value.

[0020] In order to solve the above technical problems, the embodiment of the present application also provides a multilingual large model training device, including: an acquisition module, used to acquire parallel corpus data, and extract monolingual corpus data of low-resource languages ​​from the parallel corpus data, wherein the parallel corpus data includes high-resource language corpus data and low-resource language corpus data; a training module, used to synchronously input the monolingual corpus data of the low-resource language and the parallel corpus data into an autoencoder and a target multilingual large model respectively, wherein the autoencoder and some decoder layers of the target multilingual large model share parameters, and the target multilingual The large model includes a knowledge transfer layer, which includes a multi-head attention layer, and the multi-head attention layer is used to enhance the attention between corpora with similar meanings in different languages; a construction module is used to construct a ternary data set based on the parallel corpus data, and the ternary data set includes positive examples of high-resource languages, positive examples of low-resource languages, and negative examples of low-resource languages; a calculation module is used to input the ternary data set into the target multilingual large model to calculate the ternary loss; an update module is used to update the target multilingual large model according to the ternary loss to obtain an updated multilingual large model.

[0021] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0022] In order to solve the above technical problems, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0023] The multilingual large model training method provided in the embodiment of the present invention integrates different data forms, such as low-resource monolingual data and tri-gram contrastive learning data sets, while ensuring data quality, and efficiently utilizes scarce low-resource parallel corpus data. Through parameter sharing, additional structures of knowledge transfer layer and adapter layer, the target multilingual large model is fine-tuned without increasing excessive computational complexity, thereby improving model training efficiency. In addition, through multi-stage feature learning, the target multilingual large model can understand and learn low-resource languages ​​from multiple perspectives, thereby enhancing the robustness of the model and improving the output quality of the model. Through the structure of the adapter, a certain degree of flexibility is given to the fine-tuning of the multilingual large model, and the data and parameter size of the adapter can be flexibly adjusted according to the language type of the existing data, and computing resource requirements can be dynamically arranged. Through different model training objectives, the model can learn the unique information of low-resource languages ​​under different tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0025] Figure 1 It is a schematic diagram of an exemplary scenario in which the present application can be applied.

[0026] Figure 2 This is the overall flow chart of the multi-language large model training method of this application.

[0027] Figure 3 It is a schematic diagram of the structure of the autoencoder of this application.

[0028] Figure 4 It is a schematic diagram of parameter sharing training of this application.

[0029] Figure 5 It is a schematic diagram of the adapter control logic of the present application.

[0030] Figure 6 It is a structural diagram of the multi-language large model training device of the present application.

[0031] Figure 7 It is a schematic diagram of the structure of the computer device of the present application. DETAILED DESCRIPTION

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0033] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0034] In order to better understand the contents of the embodiments of the present application, some terms and related technologies involved in the embodiments of the present application are first explained to facilitate understanding by those skilled in the art.

[0035] A large language model (LLM) is a deep learning model trained with a large amount of text data that can generate natural language text or understand the meaning of language text. Large language models can handle a variety of natural language tasks, such as text classification, question answering, and dialogue, and are an important path to artificial intelligence. Currently, large language models use a transformer architecture and pre-training objectives (such as language modeling) similar to small models. The difference from small models is the increase in model size, training data, and computing resources.

[0036] High-resource languages ​​refer to languages ​​with high usage frequency and more training data, such as English and French. In contrast, low-resource languages ​​refer to languages ​​with low usage frequency and less training data, such as Inuit and Arabic.

[0037] Fine-tuning refers to retraining a pre-trained model using a task-specific dataset to adjust model parameters to adapt to a specific language or task. General methods of fine-tuning include: task-specific fine-tuning, which uses labeled data for specific tasks (such as named entity recognition, sentiment analysis, etc.) to fine-tune the model; language-specific fine-tuning, which uses data from a specific language to enhance the model's understanding and generation capabilities for that language.

[0038] Knowledge transfer technology refers to the process of transferring the knowledge learned by the model in one task or field to another new task or field. In large language models, knowledge transfer is widely used. One of the uses is to pre-train the model on a large amount of general text and then fine-tune it on data in a specific field. This method not only improves the performance of the model on specific tasks, but also greatly reduces development time and cost.

[0039] Feature learning is an important concept in machine learning. It uses a specific algorithm to automatically discover and learn important features in input data in order to better perform subsequent classification, regression or other specific tasks. Current feature learning algorithms are usually implemented through unsupervised or semi-supervised methods.

[0040] The current language model learns the general and specific features of language through the masked language modeling task. In the masked language modeling task, the model randomly masks part of the words in the input sentence and then tries to predict these masked words. The formula is as follows:

[0041] P(word|context)=softmax(W×h masked )

[0042] Among them, h masked is the hidden state of the masked vocabulary, W is the weight matrix, this task helps the model to infer the meaning of words through context and enhance its context understanding ability.

[0043] In addition, in order to effectively process multiple languages, byte pair encoding (BPE) can be used. BPE is a data compression technology used in natural language processing to process unknown or rare words. BPE merges common characters or character sequences into one unit by frequency. Its operation can be iteratively represented as: merge(x, y)→z, where x and y are the most common pairs of adjacent symbols in the input, and they are merged into a new symbol z. In this way, BPE allows the model to use a fixed-size vocabulary to dynamically represent the vocabulary of different languages, including long-tail vocabulary. Although the use of technologies such as BPE can improve the processing capabilities of large language models for multiple languages, it still appears to be insufficient when processing low-resource languages. The main reason is the imbalance of training data and insufficient representation of low-resource languages.

[0044] The training of the model relies on a large amount of text data. For high-resource languages ​​(such as English and other major languages), the available training data is far more than that of other low-resource languages ​​(such as Arabic). This imbalance causes the model to perform well on high-resource languages ​​and poorly on low-resource languages, failing to learn enough language-specific features. In addition, even using subword segmentation representations such as BPE, the unique language structure and usage in low-resource languages ​​may still not be fully represented because most of the model's capabilities are used for more frequently occurring languages, rather than equally learning the nuances of all languages.

[0045] In order to improve the performance of multilingual models on specific tasks or low-resource languages, the current general solution is to adopt a fine-tuning strategy. Although fine-tuning can improve the performance of models on specific tasks and languages, it still has certain defects when dealing with low-resource languages, such as: 1. It is highly dependent on high-quality labeled data (which is often difficult to obtain for low-resource languages); 2. In low-resource languages, due to the small amount of available data, the model over-adapts to the specific characteristics of the training data, thereby affecting its generalization ability and causing the risk of overfitting; 3. Fine-tuning requires additional computing resources and time. For applications that need to support multiple languages, its resource consumption is a considerable challenge; 4. Fine-tuning on a new dataset may cause catastrophic forgetting of the original large model capabilities, causing the model to lose most of its original semantic understanding capabilities.

[0046] For the problems that may be caused by the above-mentioned fine-tuning, existing solutions have proposed certain solutions, such as adjusting the weights of different language training samples according to the difficulty or rarity of the input corpus data, focusing on more challenging or less common words and phrases to improve the model's understanding of low-resource languages. This method enhances the model's learning effect where it is most needed by focusing on those parts that are difficult to predict correctly during training. Although the learning-focused method has potential in improving the model's ability to understand low-resource languages, it also has some disadvantages. This method may over-focus on samples that are difficult to predict, resulting in overfitting of the model during training. In addition, the success of this method depends largely on the appropriateness of initialization and parameter adjustment. Inappropriate parameter settings may weaken the performance of the model. Another way is to use back-translation (using a machine translation model with a parallel corpus pair to translate low-resource language text into another high-resource language and then translate it back to the original low-resource language) to expand the low-resource language data training samples. Because in the back-translation process, some noise information will be introduced due to the performance problems of the machine translation model used, so the new low-resource language data obtained after back-translation can be used as a batch of new data to participate in the subsequent model fine-tuning training process. During the training phase, the embedding layer of the input text can be adjusted dynamically so that texts in different languages ​​have different embedding information. Model distillation is also considered to transfer the capabilities of the pre-trained multilingual model to a lightweight model. However, the addition of a dynamic embedding layer in this method may lead to a significant increase in computational costs, and there are certain risks in using back-translation for data enhancement. This process is heavily dependent on the quality of the machine translation model. If the quality is too poor, the back-translation results contain too many errors, which will lead to a reduction in the final model capabilities. In addition, the output of the back-translation will also be affected by the inherent bias and context loss of the machine translation model itself, which will ultimately affect the fine-tuning training quality of the large multilingual model.

[0047] It can be seen that the existing solutions can solve the problems of low efficiency of fine-tuning large multilingual models and insufficient understanding of low-resource languages ​​to a certain extent. However, there are still problems such as the inability to efficiently utilize existing corpus data to improve the model's understanding of low-resource languages, and the inability to understand and learn the unique information of low-resource languages ​​in different tasks from multiple perspectives.

[0048] Based on the above, the embodiments of the present application provide a multilingual large model training method, apparatus and related equipment, which uses low-resource language monolingual data to synchronously train an autoencoder aimed at reconstructing the original information, and shares the parameters of this autoencoder with the target multilingual large model, so that the training efficiency is improved and the model's understanding ability of low-resource languages ​​in general grammar and semantics is enhanced. In the fine-tuning stage, an additional knowledge transfer layer is established, with the attention mechanism as the core, to transfer the target large model's understanding ability of high-resource languages ​​to low-resource languages, and synchronously construct a ternary data set, and further migrate and align the semantics of the same vocabulary in different languages ​​through comparative learning.

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] See also Figure 1 ,like Figure 1 As shown, in this scenario, the system 100 may include a model training system 110, a network 120, and a corpus database 130. The network 120 is used to provide a medium for a communication link between the model training system 110 and the corpus database 130. The network 120 may include various connection types, such as wired, wireless communication links, or optical fiber cables.

[0051] The user can use the model training system 110 to interact with the corpus database 130 through the network 120 to receive or send information, etc.

[0052] The model training system 110 can be deployed on a server that provides large model applications, such as a backend server that provides model training for parallel corpus data stored in a server that deploys the corpus database 130 .

[0053] The corpus database 130 may be deployed in an edge computing device or a central computing device, for example, in a database server.

[0054] It should be noted that the multi-language large model training method provided in the embodiment of the present application is executed by a server, and accordingly, the multi-language large model training device is arranged in the server.

[0055] It should be understood that Figure 1 The number of model training systems, networks, and corpus databases in the example is only illustrative. Any number of model training systems, networks, and corpus databases may be provided as required.

[0056] See also Figure 2 , Figure 2 An overall flow chart of a multi-language large model training method provided by an embodiment of the present invention is shown and described in detail as follows.

[0057] S201. Obtain parallel corpus data.

[0058] Specifically, the parallel corpus data may be parallel corpus data stored in a corpus database, or parallel corpus data obtained through a web page, or parallel corpus data obtained through other means. This application does not limit the method for obtaining the parallel corpus data.

[0059] It should be noted that parallel corpus data refers to the corpus data formed by the correspondence between high-resource languages ​​and low-resource languages. For example, English is regarded as a high-resource language and Chinese is regarded as a low-resource language. For the same corpus sample, there are two ways of expression, namely the expression of the high-resource language and the expression of the low-resource language. These two expressions together constitute parallel corpus data. For example, It represents a parallel corpus sample.

[0060] Furthermore, the monolingual corpus data of the low-resource language is extracted separately from the original high-resource to low-resource parallel corpus data. For example, only "I love natural language processing" is extracted, and then preprocessing operations such as data cleaning are performed on it.

[0061] S202. The monolingual corpus data and the parallel corpus data of the low-resource language are synchronously input into the autoencoder and the target multilingual large model respectively.

[0062] Specifically, an autoencoder with a multi-layer transformer framework is established, and the parameters of the autoencoder and the transformer blocks of the first few layers of the target multilingual large model are shared. Through parameter sharing, the target multilingual large model can learn the monolingual features of different low-resource languages ​​from the autoencoder, while also retaining its own capture of the common grammatical and semantic features of the language, thereby enhancing the model's generalization ability to understand different low-resource languages ​​and preventing the model from overfitting to a certain language.

[0063] For example, see Figure 3 , Figure 3 Schematic diagram of the structure of an autoencoder provided in an embodiment of the present application. Figure 3 As shown in the figure, the training goal of the autoencoder is to reconstruct the input data and minimize the difference between the input data and the reconstructed output data. The encoder learns the hidden features of the input data, and then the decoder reconstructs the original input data from the hidden features.

[0064] Furthermore, when training the target multilingual large model and the autoencoder in parallel, the parameters of the autoencoder are shared with the first few decoder layers of the target multilingual large model. Generally, the multilingual large model framework is only a decoder, so a relatively simple and direct parameter sharing method is to share parameters between the decoding part of the autoencoder and the large model, such as Figure 4 As shown in the figure, the fine-tuning training of the target multilingual model is carried out synchronously with the training of the autoencoder. In addition to the decoder layer with shared parameters, the target multilingual model also has an independent decoder layer, and the parameters of this part are not shared with the autoencoder. It can be seen that the training efficiency of the target multilingual model can be effectively improved by sharing parameters.

[0065] In one possible implementation, an additional knowledge transfer layer is established under the original framework of the target multilingual large model. The knowledge transfer layer is based on a multi-head attention layer and is used to enhance the attention between corpora with similar meanings in different languages.

[0066] Optionally, the embedding of the low-resource language corpus data is used as the query (Q), and the embedding of the high-resource language corpus data is used as the key (K) and value (V), so that the interaction of the two languages ​​can be taken into account when calculating the attention output.

[0067] Specifically, the calculation of each head can be obtained by the following formula:

[0068]

[0069] Among them, softmax is an activation function that converts vector information into probability distribution. is the dimension of the embedding vector of high-resource language data. After calculating the attention of each head, all heads are connected in series and projected through a linear transformation (such as another randomly initialized weight matrix W) to merge the information of multiple heads. The specific process can be calculated by the following formula:

[0070] MultiHead(Q,K,V)=Concat(Attention1,Attention2,…,Attention h )×W

[0071] After calculation using the above formula, we can obtain the information of the merged longs.

[0072] In one possible implementation, before the data is input into the knowledge transfer layer, the data features extracted by the decoder layer of the target multilingual large model are input into the feedforward network, and a dynamic weight matrix is ​​output, which includes a query dynamic weight matrix, a key dynamic weight matrix, and a value dynamic weight matrix.

[0073] Specifically, in the attention mechanism, the three elements of query (Q), key (K) and value (V) need to be calculated from the product of input data and corresponding weights respectively. The specific calculation method is as follows:

[0074] Q=W Q ×X,K=W K ×X,V=W V ×X

[0075] Among them, X is the current input data. Under the standard attention training logic, these three weight matrices W Q , W K , W V Automatic adjustment through back propagation during the overall learning process mainly depends on the overall learning objectives and training data of the model, so these weights will be determined after the model training process is completed. In the inference stage, they are fixed for any given input and will not be dynamically adjusted according to the specific content of the input data. However, as a language model, for different input corpora, based on the differences in grammar, voice, and sentence type, the model should have different attention or attention to the same position in different sentences, so adaptive dynamic attention weights are required.

[0076] Therefore, in order to generate adaptive weights, a feedforward network is added before the original knowledge transfer layer. The input of this network is the currently extracted data features, and the output is W Q , W K , W V Three weight matrices. Under this structure, the model's response at the knowledge transfer layer is adjusted according to the specific characteristics of the input, allowing the model to perform personalized processing for each unique input.

[0077] It can be understood that in this way, the model can effectively handle variable input data, such as preferences of different users, different types of texts, or applications in multilingual environments.

[0078] In one possible implementation, an adapter is added for each low-resource language, and the weight sum of shared parameters and adapter parameters for each language is controlled by gating logic, where the gating function includes a sigmoid function and a ReLU function.

[0079] Specifically, instead of using a unified alignment layer, an additional adapter is designed for each low-resource language in the dataset, and the weights of the shared parameters and the adapter parameters of each language are controlled through the gating logic. In this implementation, the gating function can be composed of a sigmoid function and a ReLU function. The sigmoid function controls the output of the gating logic between [0,1], and the ReLU function provides some nonlinear features.

[0080] Furthermore, the specific gating function can be defined by the following formula:

[0081] g(z)=σ(ReLU(zW1+b)W2)

[0082] Among them, z is the data feature before entering the gating function (i.e., input data), and W1, W2, and b are the parameters of the gating mechanism. During the training process, the adapter can also learn the corresponding language through the gating mechanism, so that when the corresponding low-resource language is input, the model can obtain information from the corresponding adapter.

[0083] For example, see Figure 5 , Figure 5 Schematic diagram of an adapter control logic provided by an embodiment of the present application. Figure 5 As shown, after obtaining the input data, it is respectively input into the shared decoder layer and the adapter layer corresponding to each low-resource language, such as the Thai adapter layer, the Chinese adapter layer, etc. For each adapter layer, its output and the output of the shared decoder layer are input into the gating function together to obtain the data output after each network layer. For example, the shared decoder layer and the Chinese adapter layer are connected through the gating logic to obtain the data output after the network layer.

[0084] It should be understood that the above structure allows the model to be more flexible and effective when processing multilingual data, and can adjust the contribution of different language information according to actual conditions, thereby improving overall performance and adaptability. This is especially beneficial for large multilingual models because they maintain wide language coverage while also optimizing the specific performance of each language.

[0085] S203. Construct a ternary data set based on the parallel corpus data.

[0086] Specifically, in order to allow the target multilingual large model to perform comparative learning, a ternary dataset is sampled and constructed from the original parallel corpus dataset. The ternary dataset consists of positive examples in high-resource languages, positive examples in low-resource languages, and negative examples in low-resource languages.

[0087] For example, English is considered as a high-resource language and Chinese is considered as a low-resource language, and then the original partial parallel corpus data is processed into the following format, for example: English positive example - "she enjoys reading books and playing tennis", Chinese positive example - "she likes reading books and playing tennis", Chinese negative example - "he often goes to the library to study". Such a set of triple data is used as part of the training data for fine-tuning the target multilingual large model.

[0088] S204, inputting the ternary data set into the target multilingual large model, and calculating the ternary loss.

[0089] Specifically, after the ternary data set is input into the target multilingual large model, in the process of contrastive learning, after the feature extraction of the previous network layer, a set of ternary data is mapped to the same vector space, and then the ternary loss is calculated. Through training, the distance between the positive example and the negative example is increased as much as possible while the distance between the two positive examples is shortened.

[0090] In one possible implementation, the ternary loss can be calculated by the following formula:

[0091] L(a,p,n)=max(d(a,p)-d(a,n)+margin,0)

[0092] Among them, a is a positive example in a high-resource language, p is a positive example in a low-resource language, n is a negative example in a low-resource language, d is a distance function, such as a Euclidean distance function, and margin is an optional interval hyperparameter with a positive value, which can be between [0,2]. The purpose of the margin is to provide an optimization goal so that the model can separate positive examples and negative examples by at least the distance of the margin value, and the setting of the margin also prevents the large gap between positive examples and negative examples, which leads to poor generalization ability and overfitting of the model.

[0093] S205. Update the target multilingual large model according to the ternary loss.

[0094] Specifically, after obtaining the ternary loss L, the ternary loss L can be used to update the parameters of the target multilingual large model so that the same language expressions (i.e., positive examples) are closer than different expressions (i.e., negative examples) in the embedding space.

[0095] It should be understood that the updated target multilingual large model has improved its ability to understand the vocabulary of low-resource languages, and can be deployed and applied in relevant scenarios, thereby expanding its application scenarios and enhancing its processing effect on low-resource languages.

[0096] In this embodiment, different data forms are integrated while ensuring data quality, such as low-resource monolingual data and ternary contrastive learning data sets, and scarce low-resource parallel corpus data is efficiently utilized. Through parameter sharing, additional structures of knowledge transfer layer and adapter layer, the target multilingual large model is fine-tuned without increasing excessive computational workload, thereby improving model training efficiency. In addition, through multi-stage feature learning, the target multilingual large model can understand and learn low-resource languages ​​from multiple perspectives, thereby enhancing the robustness of the model and improving the output quality of the model. Through the structure of the adapter, a certain degree of flexibility is given to the fine-tuning of the multilingual large model, and the data and parameter size of the adapter can be flexibly adjusted according to the language type of the existing data, and computing resource requirements can be dynamically arranged. Through different model training objectives, the model can learn the unique information of low-resource languages ​​under different tasks.

[0097] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0098] Figure 6 FIG. 2 is a block diagram showing a multi-language large model training device 200 corresponding to the above embodiment. Figure 6 As shown, the system 200 includes an acquisition module 210, a training module 220, a construction module 230, a calculation module 240 and an update module 250. Each functional module is described in detail as follows.

[0099] The acquisition module 210 is used to acquire parallel corpus data and extract monolingual corpus data of low-resource languages ​​from the parallel corpus data. The parallel corpus data includes high-resource language corpus data and low-resource language corpus data.

[0100] The training module 220 is used to synchronously input the monolingual corpus data of the low-resource language and the parallel corpus data into the autoencoder and the target multilingual large model respectively, the autoencoder and some decoder layers of the target multilingual large model share parameters, the target multilingual large model includes a knowledge transfer layer, the knowledge transfer layer includes a multi-head attention layer, and the multi-head attention layer is used to enhance the attention between corpora with similar meanings in different languages.

[0101] The construction module 230 is used to construct a ternary data set based on the parallel corpus data, wherein the ternary data set includes positive examples in high-resource languages, positive examples in low-resource languages, and negative examples in low-resource languages.

[0102] The calculation module 240 is used to input the triple data set into the target multilingual large model to calculate the triple loss.

[0103] The updating module 250 is used to update the target multilingual large model according to the ternary loss to obtain an updated multilingual large model.

[0104] For the specific definition of the multi-language large model training device, please refer to the definition of the multi-language large model training method above, which will not be repeated here. Each module in the above-mentioned multi-language large model training device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0105] To solve the above technical problems, the present application also provides a computer device. Figure 7 , Figure 7 This is a basic structural block diagram of the computer device in this embodiment.

[0106] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 300 having components connected to the memory 310, the processor 320, and the network interface 330, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASIC), programmable gate arrays (FPGA), digital signal processors (DSP), embedded devices, etc.

[0107] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0108] The memory 310 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or D interface display memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 can be an internal storage unit of the computer device 300, such as a hard disk or memory of the computer device 300. In other embodiments, the memory 310 can also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 310 can also include both the internal storage unit of the computer device 300 and its external storage device. In this embodiment, the memory 310 is generally used to store the operating system and various application software installed in the computer device 300, such as program codes for controlling electronic files, etc. In addition, the memory 310 can also be used to temporarily store various data that have been output or will be output.

[0109] The processor 320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 320 is generally used to control the overall operation of the computer device 300. In this embodiment, the processor 320 is used to run the program code stored in the memory 310 or process data, such as running the program code for controlling the electronic file.

[0110] The network interface 330 may include a wireless network interface or a wired network interface. The network interface 330 is generally used to establish a communication connection between the computer device 300 and other electronic devices.

[0111] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores an interface display program, and the interface display program can be executed by at least one processor so that the at least one processor performs the steps of the above method.

[0112] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0113] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A multi-language large model training method, characterized in that: The method comprises: Acquire parallel corpus data, and extract monolingual corpus data of low-resource languages ​​from the parallel corpus data, wherein the parallel corpus data includes high-resource language corpus data and low-resource language corpus data; The monolingual corpus data of the low-resource language and the parallel corpus data are synchronously input into the autoencoder and the target multilingual large model respectively, the autoencoder and some decoder layers of the target multilingual large model share parameters, the target multilingual large model includes a knowledge transfer layer, the knowledge transfer layer includes a multi-head attention layer, and the multi-head attention layer is used to enhance the attention between corpora with similar meanings in different languages; Based on the parallel corpus data, construct a ternary data set, wherein the ternary data set includes a positive example in a high-resource language, a positive example in a low-resource language, and a negative example in a low-resource language; Inputting the ternary data set into the target multilingual large model to calculate the ternary loss; The target multilingual large model is updated according to the ternary loss to obtain an updated multilingual large model.

2. The method according to claim 1, characterized in that The multi-head attention layer is used to enhance the attention between corpora with similar meanings in different languages, including: Using the embedding of the low-resource language corpus data as a query and the embedding of the high-resource language corpus data as a key and a value, calculating the attention of each head; All heads are concatenated and projected through linear transformation to obtain the merged multi-head attention.

3. The method according to claim 2, characterized in that The attention algorithm for each head is as follows: Among them, softmax is an activation function that converts vector information into probability distribution. is the dimension of the embedding vector of high-resource language data.

4. The method according to claim 1, characterized in that Before inputting the data into the knowledge transfer layer, the method further includes: The data features extracted by the decoder layer of the target multilingual large model are input into a feedforward network, and a dynamic weight matrix is ​​output, wherein the dynamic weight matrix includes a query dynamic weight matrix, a key dynamic weight matrix, and a value dynamic weight matrix.

5. The method according to claim 1 or 2, characterized in that: The method further comprises: An adapter is added for each of the low-resource languages, and the weight sum of the shared parameters and the adapter parameters of each language is controlled by gating logic, wherein the gating function includes a sigmoid function and a ReLU function.

6. The method according to claim 5, characterized in that The gating function is as follows: g(z)=σ(ReLU(zW1+b)W2) Among them, z is the data feature before inputting the gating function, and W1, W2 and b are the parameters of the gating mechanism.

7. The method according to claim 1 or 2, characterized in that: The step of inputting the ternary data set into the target multilingual large model and calculating the ternary loss comprises: The ternary data in the ternary data set are mapped to the same vector space, and the ternary loss is calculated. The ternary loss calculation formula is as follows: L(a,p,n)=max(d(a,p)-d(a,n)+margin,0) Among them, a is a positive example in a high-resource language, p is a positive example in a low-resource language, n is a negative example in a low-resource language, d is a distance function, and margin is an interval hyperparameter with a positive value.

8. A multi-language large model training device, characterized in that: The device comprises: An acquisition module, used to acquire parallel corpus data, and extract monolingual corpus data of low-resource languages ​​from the parallel corpus data, wherein the parallel corpus data includes high-resource language corpus data and low-resource language corpus data; A training module, used for synchronously inputting the monolingual corpus data of the low-resource language and the parallel corpus data into an autoencoder and a target multilingual large model, respectively, wherein the autoencoder and some decoder layers of the target multilingual large model share parameters, and the target multilingual large model includes a knowledge transfer layer, and the knowledge transfer layer includes a multi-head attention layer, and the multi-head attention layer is used to enhance the attention between corpora with similar meanings in different languages; A construction module, configured to construct a ternary data set based on the parallel corpus data, wherein the ternary data set includes a positive example in a high-resource language, a positive example in a low-resource language, and a negative example in a low-resource language; A calculation module, used for inputting the ternary data set into the target multilingual large model to calculate the ternary loss; An updating module is used to update the target multilingual large model according to the ternary loss to obtain an updated multilingual large model.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images