Construction method and device of cross-language question answering system based on generative multilingual model

Abstract

The application provides a cross-language question answering system construction method based on a generative multilingual model, and the method comprises the following steps: acquiring multilingual text data, pre-training a general language model based on the multilingual text data to obtain a multilingual pre-training model; acquiring a single-corpus question and answer data set, processing the question and answer data set by using a placeholder to construct a training sample for prompt learning; iteratively fine-tuning the multilingual pre-training model based on prompt learning according to the training sample to obtain a generative multilingual model; acquiring to-be-recognized text and question text, inputting the to-be-recognized text and the question text into the generative multilingual model, and generating answers in different languages corresponding to the question text based on beam search. The application constructs a question and answer system capable of realizing cross-language question answering based on a multilingual model with a transfer learning capability and easily obtained single-language corpus, and can obtain answers in a generative manner, thereby increasing the diversity of answers.

Description

Technical Field

[0001] This application relates to the field of question-answering system technology, and in particular to a method and apparatus for constructing a cross-language question-answering system based on a generative multilingual model. Background Technology

[0002] Current automated question answering systems mostly focus on a single language. Even multilingual question answering solutions often only implement question answering functions within the text or knowledge of different languages, failing to achieve cross-language sharing of text and knowledge for question answering.

[0003] Current language model-based question-answering systems use language models that cannot perform zero-shot transfer learning across domains or languages. The model's question-answering ability is limited to the domains covered by the training corpus. This requires providing a training corpus encompassing all domains, which is impractical. Furthermore, the model's question-answering ability is restricted to the language range defined by the training corpus, necessitating the provision of sufficient question-answering data for each language, which is also challenging.

[0004] Currently, many question-answering systems are based on language or knowledge retrieval models. The answers generated by these non-generative models will inevitably appear in the original text or knowledge base, which leads to a lack of diversity in the answers and makes it difficult to achieve answers with complex logic. Summary of the Invention

[0005] This application aims to at least partially address one of the technical problems in the related art.

[0006] Therefore, the first objective of this application is to propose a method for constructing a cross-language question answering system based on a generative multilingual model. This method addresses the technical problems of existing question answering systems being unable to perform zero-shot transfer learning across domains and languages, and the lack of diversity in the generated answers. Based on a multilingual model with transfer learning capabilities and readily available monolingual corpora, this method constructs a question answering system capable of cross-language question answering, generating answers in a generative manner, increasing the diversity of answers, and achieving high-quality question answering results.

[0007] The second objective of this application is to propose a cross-language question-answering system construction device based on a generative multilingual model.

[0008] The third objective of this application is to propose a computer device.

[0009] The fourth objective of this application is to provide a non-transitory computer-readable storage medium.

[0010] To achieve the above objectives, the first aspect of this application proposes a method for constructing a cross-language question-answering system based on a generative multilingual model, comprising: acquiring multilingual text data; pre-training a general language model based on the multilingual text data to obtain a multilingual pre-trained model; acquiring a question-answering dataset of a single corpus; processing the question-answering dataset using placeholders to construct training samples for prompt learning; iteratively fine-tuning the multilingual pre-trained model based on prompt learning according to the training samples to obtain a generative multilingual model; acquiring the text to be identified and the question text; inputting the text to be identified and the question text into the generative multilingual model; and generating answers in different languages ​​corresponding to the question text based on beam retrieval.

[0011] Optionally, in one embodiment of this application, acquiring multilingual text data and pre-training a general language model based on the multilingual text data to obtain a multilingual pre-trained model includes:

[0012] Acquire multilingual text data;

[0013] For each language of the multilingual text data, multiple text segments are randomly sampled. Each sampled segment in the text sequence is replaced with a mask mark to obtain a corrupted text sequence, where each segment corresponds to a series of consecutive characters.

[0014] The corrupted text sequence is used as the first sample data, and the fragment replaced by the mask mark is used as the second sample data;

[0015] Based on the first and second sample data, the general language model is pre-trained to obtain a multilingual pre-trained model.

[0016] Optionally, in one embodiment of this application, pre-training a general language model based on multilingual text data to obtain a multilingual pre-trained model further includes:

[0017] By changing the length and number of sampled text fragments, pre-training targets adapted to different tasks are generated to pre-train a general language model, resulting in a multilingual pre-trained model adapted to different tasks.

[0018] Optionally, in one embodiment of this application, fine-tuning the multilingual pre-trained model includes:

[0019] Obtain the learning rate and batch size;

[0020] By employing a learning rate and batch size, and based on a loss scaling mechanism, the parameters of the multilingual pre-trained model are updated through backpropagation, thereby training the multilingual pre-trained model.

[0021] Optionally, in one embodiment of this application, the text to be identified and the question text are input into a generative multilingual model, and answers in different languages ​​corresponding to the question text are generated based on beam retrieval, including:

[0022] Based on the text to be identified and the question text, candidate words and their probabilities are generated using a generative multilingual model;

[0023] Set the bundle size to k, and select the k candidate words with the highest probabilities from the candidate words as the word examples for the first output position based on the probability of the candidate words;

[0024] Based on the probability of the word example and the candidate word combination example at the first output position, select the k combination examples with the highest probability as the word examples at the second output position;

[0025] Based on the probability of the word example and the combination of candidate words at the previous output position, select the k combination words with the highest probability as the word examples at the current output position, and so on until the word examples at all output positions are obtained, and use the word example with the highest probability as the generated answer.

[0026] To achieve the above objectives, a second aspect of this application proposes an apparatus for constructing a cross-language question-answering system based on a generative multilingual model, comprising:

[0027] The pre-training module is used to acquire multilingual text data and pre-train the general language model based on the multilingual text data to obtain a multilingual pre-trained model.

[0028] The training sample construction module is used to obtain a question-and-answer dataset from a single corpus, process the question-and-answer dataset using placeholders, and construct training samples for prompt learning.

[0029] The training module is used to iteratively fine-tune the multilingual pre-trained model based on training samples and prompt-based learning to obtain a generative multilingual model.

[0030] The generation module is used to obtain the text to be identified and the question text, input the text to be identified and the question text into the generative multilingual model, and generate answers in different languages ​​corresponding to the question text based on beam retrieval.

[0031] Optionally, in one embodiment of this application, the pre-training module is specifically used for:

[0032] Acquire multilingual text data;

[0033] For each language of the multilingual text data, multiple text segments are randomly sampled. Each sampled segment in the text sequence is replaced with a mask mark to obtain a corrupted text sequence, where each segment corresponds to a series of consecutive characters.

[0034] The corrupted text sequence is used as the first sample data, and the fragment replaced by the mask mark is used as the second sample data;

[0035] Based on the first and second sample data, the general language model is pre-trained to obtain a multilingual pre-trained model.

[0036] Optionally, in one embodiment of this application, the pre-training module is further configured to:

[0037] By changing the length and number of sampled text fragments, pre-training targets adapted to different tasks are generated to pre-train a general language model, resulting in a multilingual pre-trained model adapted to different tasks.

[0038] To achieve the above objectives, a third aspect of this application provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the cross-language question-answering system construction method based on a generative multilingual model described in the above embodiments.

[0039] To achieve the above objectives, a fourth aspect of this application proposes a non-transitory computer-readable storage medium that, when the instructions in the storage medium are executed by a processor, enables the execution of a method for constructing a cross-language question-answering system based on a generative multilingual model.

[0040] The present application's embodiments of the cross-language question-answering system construction method, apparatus, computer equipment, and non-transitory computer-readable storage medium based on a generative multilingual model solve the technical problems of existing question-answering systems being unable to complete zero-shot transfer learning across domains and languages, and the lack of diversity in generated answers. Based on a multilingual model with transfer learning capabilities and easily obtainable monolingual corpora, a question-answering system capable of cross-language question answering is constructed, which can obtain answers in a generative manner, increase the diversity of answers, and achieve high-quality question-answering results.

[0041] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0042] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0043] Figure 1 A flowchart illustrating a method for constructing a cross-language question-answering system based on a generative multilingual model, as provided in Embodiment 1 of this application;

[0044] Figure 2This is an example diagram of the pre-trained architecture of the GLM model in the method for constructing a cross-language question answering system based on a generative multilingual model, as described in an embodiment of this application.

[0045] Figure 3 This is an example diagram of the floating-point data format for the method of constructing a cross-language question-answering system based on a generative multilingual model according to an embodiment of this application;

[0046] Figure 4 This is an example diagram illustrating loss scaling for a cross-language question answering system construction method based on a generative multilingual model, as described in an embodiment of this application.

[0047] Figure 5 This is an example diagram of the bundle retrieval process of the cross-language question answering system construction method based on a generative multilingual model according to an embodiment of this application;

[0048] Figure 6 This is another flowchart illustrating the method for constructing a cross-language question-answering system based on a generative multilingual model, as described in this application.

[0049] Figure 7 This is a schematic diagram of the structure of a cross-language question-answering system construction device based on a generative multilingual model provided in Embodiment 2 of this application. Detailed Implementation

[0050] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0051] Question answering systems aim to automatically answer users' natural language questions using natural language. Typically, users input contextual information and a question description into the system, which then generates an answer based on the information provided. A question answering system supporting only one language is called a monolingual question answering system; one supporting multiple languages ​​is called a multilingual question answering system; and further, if the contextual language providing the background information differs from the question's language, the system can still correctly answer and provide an answer in the target language. This type of cross-language question answering system can be called a cross-lingual question answering system. This application primarily relates to a cross-lingual question answering system based on a multilingual pre-trained model.

[0052] Based on the differences in the organization of knowledge data, existing question-answering systems can be divided into three categories: question-answering systems based on structured data, question-answering systems based on free text, and question-answering systems based on question-answer samples. Systems based on structured data use structured data to answer questions; systems based on free text learn knowledge and skills from free text through large-scale training to answer questions; and systems based on question-answer samples learn the mapping from questions to answers by training and mastering question-answer samples.

[0053] In addition, based on the method of generating answers, question answering system solutions can be divided into retrieval-based question answering and generative question answering. Retrieval-based question answering provides answers to questions by retrieving text ranges (for text) or knowledge points in a knowledge base (for structured knowledge) where the target answer may appear; generative question answering, on the other hand, directly generates answers to questions after the model has been fully trained.

[0054] Currently, a large body of work and inventions focuses on various fields such as retrieval-based question answering systems, monolingual question answering systems, zero-shot or few-shot learning question answering systems based on large models, and generative question answering systems. However, in the field of generative cross-lingual question answering, no one has yet proposed any related work or inventions.

[0055] Current experimental cases of zero-shot learning for question-answering tasks using various multilingual models are not real-world question-answering systems, but merely evaluation dimensions in the testing phase of various multilingual models. This application proposes for the first time a novel technique for constructing a generative multilingual question-answering system: based on a multilingual model with transfer learning capabilities, it trains on readily available monolingual corpora, thereby constructing a question-answering system capable of cross-language question answering.

[0056] In recent years, pre-training frameworks have been proposed, and this method of self-supervised pre-training of models on unlabeled internet corpora has greatly improved the benchmark performance of various natural language processing tasks. The various pre-trained models that have emerged subsequently continue to break the best performance records for each task.

[0057] Among them, the GLM framework is a type of model trained and generated based on autoregression. Unlike BERT, which randomly masks text regions and allows the model to reconstruct these regions from context, and unlike GPT, which uses a purely autoregressive training method from left to right, GLM combines the advantages of both—randomly masking text regions in the preceding text and then autoregressively generating these masked regions at the end of the input text. This training method is called a general pre-training framework based on autoregressive fill-in, which can simultaneously learn bidirectional and unidirectional attention mechanisms within a unified training framework. Furthermore, this training method makes the model suitable for various natural language processing tasks, including natural language understanding (NLU) and conditional / unconditional natural language generation (NLU).

[0058] Besides models trained on a single language, researchers have also attempted to use large-scale multilingual pre-training corpora to train a pre-trained language model that supports multiple languages. The multilingual pre-trained model in this application possesses many characteristics, one of which is its ability to achieve cross-domain and cross-language transfer learning. For example, a model fine-tuned on an encyclopedic corpus can be directly applied to NLP needs in other domains such as news and academia; a model fine-tuned on an English corpus can perform well on the same tasks in other languages. This characteristic makes it easier to build various multilingual applications based on large multilingual models.

[0059] The following describes, with reference to the accompanying drawings, a method and apparatus for constructing a cross-language question-answering system based on a generative multilingual model, according to embodiments of this application.

[0060] Figure 1 This is a flowchart illustrating a method for constructing a cross-language question-answering system based on a generative multilingual model, as provided in Embodiment 1 of this application.

[0061] like Figure 1 As shown, the method for constructing a cross-language question-answering system based on a generative multilingual model includes the following steps:

[0062] Step 101: Obtain multilingual text data, and pre-train the general language model based on the multilingual text data to obtain a multilingual pre-trained model;

[0063] Step 102: Obtain a question-and-answer dataset of a single corpus, process the question-and-answer dataset using placeholders, and construct training samples for prompt learning;

[0064] Step 103: Based on the training samples, the multilingual pre-trained model is iteratively fine-tuned and trained using prompting learning to obtain a generative multilingual model;

[0065] Step 104: Obtain the text to be identified and the question text, input the text to be identified and the question text into the generative multilingual model, and generate answers in different languages ​​corresponding to the question text based on beam retrieval.

[0066] The method for constructing a cross-language question-answering system based on a generative multilingual model in this application involves: acquiring multilingual text data; pre-training a general language model based on the multilingual text data to obtain a multilingual pre-trained model; acquiring a question-answering dataset of a single corpus; processing the question-answering dataset using placeholders to construct training samples for prompt learning; iteratively fine-tuning the multilingual pre-trained model based on prompt learning using the training samples to obtain a generative multilingual model; acquiring the text to be identified and the question text; inputting the text to be identified and the question text into the generative multilingual model; and generating answers in different languages ​​corresponding to the question text based on beam retrieval. This method addresses the technical problems of existing question-answering systems being unable to achieve zero-shot transfer learning across domains and languages, and the lack of diversity in generated answers. Based on a multilingual model with transfer learning capabilities and easily obtainable single-language corpora, a question-answering system capable of cross-language question answering is constructed, which can generate answers in a generative manner, increasing answer diversity and achieving high-quality question-answering results.

[0067] This application utilizes a multilingual pre-trained model to construct a question-answering system that enables transfer learning. This makes building a multilingual question-answering system more convenient and faster, eliminating the need to acquire as many question-answering data samples as possible from different corpora. The cross-lingual question-answering system built using this application's multilingual pre-trained model through fine-tuning achieves high-quality question-answering results, and its evaluation results on the evaluation dataset are on par with or even better than the current state-of-the-art models.

[0068] Furthermore, in this embodiment of the application, acquiring multilingual text data and pre-training a general language model based on the multilingual text data to obtain a multilingual pre-trained model includes:

[0069] Acquire multilingual text data;

[0070] For each language of the multilingual text data, multiple text segments are randomly sampled. Each sampled segment in the text sequence is replaced with a mask mark to obtain a corrupted text sequence, where each segment corresponds to a series of consecutive characters.

[0071] The corrupted text sequence is used as the first sample data, and the fragment replaced by the mask mark is used as the second sample data;

[0072] Based on the first and second sample data, the general language model is pre-trained to obtain a multilingual pre-trained model.

[0073] For example, given a text sequence, multiple text segments are randomly sampled, where each segment corresponds to a series of consecutive characters. Each segment is replaced by a mask marker, eventually forming a corrupted text sequence. The corrupted text sequence is used as the first sample data, and the segments replaced by the mask markers are used as the second sample data.

[0074] GLM proposes a general pre-training framework based on autoregressive cloze tests. By simultaneously learning bidirectional and unidirectional attention mechanisms within a unified framework, the general language model learns both contextual representations and autoregressive generation during the pre-training phase. In the fine-tuning phase for downstream tasks, different types of downstream tasks can be unified through cloze tests, thus achieving a pre-trained model applicable to all natural language processing tasks.

[0075] GLM is pre-trained by optimizing an autoregressive fill-in-the-blank objective. The GLM model first predicts missing segments from a corrupted text sequence in an autoregressive manner, and then continuously updates the model parameters to reduce the difference between the predicted and actual results, thus pre-training the model parameters. When predicting missing segments, the GLM model can access both the corrupted text and previously predicted segments. To fully capture the interdependencies between different segments, the order of the segments is randomly arranged.

[0076] like Figure 2 As shown, the first sample data and the second sample data are input into GLM, where the first sample data is a corrupted text sequence and the second sample data is a segment replaced by a mask mark. The GLM model predicts the missing segments from the corrupted text sequence in an autoregressive manner and reduces the difference between the predicted results and the true results by continuously updating the model parameters, thereby achieving pre-training of the model parameters.

[0077] In this embodiment, first sample data and second sample data are input into a GLM (Geometric Logic Model). The first sample data is a corrupted text sequence, and the second sample data is a fragment replaced by masked markers. The GLM model learns to generate the second sample data using the first sample data and continuously updates its parameters to make the generated result more closely resemble the second sample data, ultimately resulting in a pre-trained GLM model, i.e., a multilingual pre-trained model. During encoding, characters in the first sample data can be associated with all characters in the first sample data, but cannot be associated with any characters in the second sample data. Characters in the second sample data can be associated with characters in the first sample data and characters preceding them in the second sample data, but cannot be associated with any subsequent characters in the second sample data.

[0078] During the pre-training phase, the GLM model uses a bidirectional encoder to perform bidirectional associative encoding on the text in the first sample data; and a unidirectional encoder to perform unidirectional associative encoding on the text in the second sample data on both the first sample data and the preceding parts of the text in the second sample data. The GLM model updates the weight parameters in the bidirectional and unidirectional encoders by learning the task of generating the second sample data from the first sample data.

[0079] To perform autoregressive generation, a special marker character is padded at the beginning of each segment of the second sample data for input to the general language model, and a special marker character is padded at the end of each segment of the second sample data for output to the general language model.

[0080] Furthermore, in this embodiment of the application, the process of pre-training a general language model based on multilingual text data to obtain a multilingual pre-trained model further includes:

[0081] By changing the length and number of sampled text fragments, pre-training targets adapted to different tasks are generated to pre-train a general language model, resulting in a multilingual pre-trained model adapted to different tasks.

[0082] To develop a question-answering system, the multilingual pre-trained model mGLM needs to be fine-tuned on question-answering corpora (datasets). The chosen question-answering datasets can be XQuAD, MLQA, and TyDiQA. XQuAD is a multilingual question-answering dataset derived from the SQuAD English question-answering dataset through machine translation. MLQA is a cross-lingual question-answering dataset based on parallel corpora, which allows for the selection of text passages and questions from different languages ​​during training, forming different language pair combinations. TyDiQA first provides multilingual questions, then uses reverse retrieval of the internet to find relevant multilingual context or answers, thus forming the dataset.

[0083] Since this application chooses to build a cross-language question answering system based on single-language corpora and the transfer learning capability of the model, it can select English corpora from the above three datasets for training and test the model's performance on samples of the remaining languages.

[0084] This application's fine-tuning method for the multilingual pre-trained model mGLM is based on cue learning, that is, providing the model with cue words to construct question-and-answer data samples, and allowing the model to learn to generate the most appropriate output results. Therefore, for question-and-answer tasks, this application processes the obtained question-and-answer data corpus to construct training samples for cue learning, for example: "Context:[context text]Question:[question text]Answer:[sMASK]".

[0085] Here, [sMASK] is a special placeholder that guides the model to complete the actual content on this placeholder, thereby achieving an answer to the question. It is important to note that this application chooses to use English prompts throughout the inference process, regardless of the language. This step ensures that the model's tasks during generation and inference are completely consistent with those during fine-tuning training.

[0086] Furthermore, in this embodiment of the application, fine-tuning the multilingual pre-trained model includes:

[0087] Obtain the learning rate and batch size;

[0088] By employing a learning rate and batch size, and based on a loss scaling mechanism, the parameters of the multilingual pre-trained model are updated through backpropagation, thereby training the multilingual pre-trained model.

[0089] This application leverages the transfer learning capabilities of a multilingual pre-trained model, selecting only the English corpus from the dataset for fine-tuning. When developing the fine-tuning training program, this application selected appropriate learning rates, batch sizes, training iterations, and hardware environments to achieve sufficient learning from the training corpus. Specific configurations are shown in Table 1.

[0090] Configuration items Specific parameters Learning rate 1e-5 Learning rate update method Adaptive adjustment Batch size 16 Hardware environment for fine-tuning training One A100 80G x 8 machine

[0091] Table 1

[0092] In this application, languages ​​other than English in the dataset will be used as the test set to test the effectiveness of the model in multilingual question-answering scenarios and to compare it with data from other multilingual models. During fine-tuning training and testing, the F1 score and Exact Match (EM) metrics will be used to measure the model's accuracy in answering question-answering tasks. The F1 score is defined as follows: It balances accuracy and recall, providing a comprehensive measure of model performance. The perfect match metric, on the other hand, evaluates the model's ability to match the answer word for word.

[0093] This application uses a 16-bit floating-point data format to represent model parameters in order to compress video memory space. Figure 3 As shown, a 16-bit floating-point number consists of 1 sign bit, 5 exponent bits, and 10 mantissa bits, with the exponent representing a range between [-14, 15]. Therefore, 16-bit floating-point numbers are prone to overflow errors due to their narrow representation range.

[0094] To ensure that the gradient values ​​for each parameter do not cause floating-point overflow and lead to parameter update failure during the backpropagation gradient calculation, this application uses a loss scaling mechanism. For example... Figure 4As shown, the loss scaling mechanism refers to multiplying the loss value (gradient value) by a scaling factor during training to scale the gradient, ensuring that the gradient falls within the range that can be represented by 16-bit floating-point precision, thereby minimizing overflow issues that may occur in floating-point calculations. After parameter gradient aggregation and before the optimizer updates the parameters, the aggregated parameter gradient value is divided by this scaling factor to restore the true gradient.

[0095] A suitable scaling mechanism plays a crucial role in training performance. Too small a scaling factor can cause the minimum gradient value to underflow to 0, while too large a scaling factor can cause the maximum gradient value to overflow to NaN or Inf. Therefore, this application chooses to use a dynamic scaling mechanism to ensure training stability. Specifically, a large initial scaling factor is selected. If overflow occurs, the update is skipped and the scaling factor is reduced; otherwise, gradient updates proceed normally. If no overflow occurs for N consecutive iterations, the scaling factor is increased again.

[0096] In addition, during the fine-tuning training process, in order to ensure the effectiveness of the selected prompt words, in addition to English prompt word template samples such as "Context:[context text]Question:[question text]Answer:[sMASK]", this application also tested samples of various other languages.

[0097] However, using English prompts for training and testing allows the model to achieve its best performance on multilingual test sets. Prompt templates constructed in other languages ​​such as Chinese, French, and Japanese do not perform as well as those in English. Therefore, this application ultimately chooses to use English prompts to construct training templates and training data.

[0098] Furthermore, to achieve distributed training, this application uses the DeepSpeed ​​distributed training framework; to meet the requirements of high customizability, this application uses the SwissArmyTransformer model framework.

[0099] This application uses a bundle retrieval method to construct the question-answering system. Furthermore, if the generative model generates too much text, the topics of later texts may gradually deviate from the original meaning. Therefore, this application limits the maximum length of the generated sequence of the generative model; the specific interaction scheme configuration is shown in Table 2.

[0100]

[0101]

[0102] Table 2

[0103] Further, in the embodiments of the present application, the text to be recognized and the question text are input into a generative multilingual model, and answers in different languages corresponding to the question text are generated based on beam search, including:

[0104] Generate candidate words and their probabilities through the generative multilingual model according to the text to be recognized and the question text;

[0105] Set the beam size to k, and select the k candidate words with the highest probabilities among the candidate words as the word examples at the first output position;

[0106] Based on the probabilities of the combined word examples of the word examples at the first output position and the candidate words, select the k combined word examples with the highest probabilities as the word examples at the second output position;

[0107] Based on the probabilities of the combined word examples of the word examples at the previous output position and the candidate words, select the k combined word examples with the highest probabilities as the word examples at the current output position until the word examples at all output positions are obtained, and take the word example with the highest probability as the generated answer.

[0108] The beam search used in the present application is essentially a type of improved algorithm for the greedy algorithm, which expands the search space. At the first output position, the beam search algorithm will select the k words with the highest current conditional probabilities (k is the beam size); at each subsequent output position, based on the output sequence of the previous step, select the k with the highest conditional probabilities among all combinations as the best candidate sequence up to that position. The effect of the beam search algorithm is usually better than that of the greedy algorithm, and the greedy algorithm can be regarded as a beam search algorithm with beam size = 1. As Figure 5 shown, the English translation "I hate you" of "我恨你" is generated through beam search (using I, H, and U to represent respectively), and the beam size is selected as 2. Then after generating three candidate words at the first position, retain the two highest ones (I and H); use these two candidate words as conditions to generate candidate words at the second position, there are six combinations in total, and also retain the two highest ones (IH and HI); and so on until the reasoning is completed, and the combination with the highest probability is the reasoning result.

[0109] The present application first proposes a generative and cross - language question - answering system by introducing the mGLM general multilingual model into the cross - language question - answering system; and for the first time, it is proposed that multi - language question - answering tasks can be achieved based on the cross - language transfer learning ability of a single - language corpus and model. The present application performs very well in multiple multi - language question - answering evaluation metrics, as shown in Table III.

[0110]

[0111] Table III

[0112] Figure 6 This is another flowchart illustrating the method for constructing a cross-language question-answering system based on a generative multilingual model, as described in this application.

[0113] like Figure 6 As shown, the method for constructing a cross-language question answering system based on a generative multilingual model includes: obtaining a multilingual pre-trained model mGLM; obtaining a publicly available question answering dataset; processing the obtained dataset to obtain high-quality training samples; developing a fine-tuning training program to perform zero-shot transfer fine-tuning training on the multilingual pre-trained model mGLM based on the high-quality training samples, thereby constructing the question answering system.

[0114] Figure 7 This is a schematic diagram of the structure of a cross-language question-answering system construction device based on a generative multilingual model provided in Embodiment 2 of this application.

[0115] like Figure 7 As shown, the apparatus for building a cross-language question-answering system based on a generative multilingual model includes:

[0116] Pre-training module 10 is used to acquire multilingual text data and pre-train a general language model based on the multilingual text data to obtain a multilingual pre-trained model.

[0117] The training sample construction module 20 is used to obtain a question-and-answer dataset of a single corpus, process the question-and-answer dataset using placeholders, and construct training samples for prompt learning.

[0118] Training module 30 is used to iteratively fine-tune the multilingual pre-trained model based on training samples and prompting learning to obtain a generative multilingual model.

[0119] The generation module 40 is used to obtain the text to be identified and the question text, input the text to be identified and the question text into the generative multilingual model, and generate answers in different languages ​​corresponding to the question text based on beam retrieval.

[0120] The cross-language question-answering system construction apparatus based on a generative multilingual model in this application includes a pre-training module for acquiring multilingual text data and pre-training a general language model based on the multilingual text data to obtain a multilingual pre-trained model; a training sample construction module for acquiring a question-answering dataset of a single corpus, processing the question-answering dataset using placeholders, and constructing training samples for prompt learning; a training module for iteratively fine-tuning the multilingual pre-trained model based on prompt learning using the training samples to obtain a generative multilingual model; and a generation module for acquiring the text to be identified and the question text, inputting the text to be identified and the question text into the generative multilingual model, and generating answers in different languages ​​corresponding to the question text based on beam retrieval. Therefore, this invention solves the technical problems of existing question-answering systems being unable to complete zero-shot transfer learning across domains and languages, and the lack of diversity in generated answers. Based on a multilingual model with transfer learning capabilities and easily obtainable single-language corpora, this invention constructs a question-answering system capable of cross-language question answering, generating answers in a generative manner, increasing answer diversity, and achieving high-quality question-answering results.

[0121] Furthermore, in this embodiment of the application, the pre-training module is specifically used for:

[0122] Acquire multilingual text data;

[0123] For each language of the multilingual text data, multiple text segments are randomly sampled. Each sampled segment in the text sequence is replaced with a mask mark to obtain a corrupted text sequence, where each segment corresponds to a series of consecutive characters.

[0124] The corrupted text sequence is used as the first sample data, and the fragment replaced by the mask mark is used as the second sample data;

[0125] Based on the first and second sample data, the general language model is pre-trained to obtain a multilingual pre-trained model.

[0126] Furthermore, in this embodiment of the application, the pre-training module is also used for:

[0127] By changing the length and number of sampled text fragments, pre-training targets adapted to different tasks are generated to pre-train a general language model, resulting in a multilingual pre-trained model adapted to different tasks.

[0128] To implement the above embodiments, this application also proposes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the cross-language question-answering system construction method based on the generative multilingual model described in the above embodiments.

[0129] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the cross-language question-answering system construction method based on a generative multilingual model as described in the above embodiments.

[0130] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0131] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0132] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0133] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0134] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0135] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0136] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0137] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for constructing a cross-language question-answering system based on a generative multilingual model, characterized in that, Includes the following steps: Acquire multilingual text data, and pre-train a general language model based on the multilingual text data to obtain a multilingual pre-trained model. Obtain a question-and-answer dataset from a single corpus, process the dataset using placeholders, and construct training samples for prompt learning; Based on the training samples, the multilingual pre-trained model is iteratively fine-tuned and trained using prompting learning to obtain a generative multilingual model. Obtain the text to be identified and the question text, input the text to be identified and the question text into the generative multilingual model, and generate answers in different languages ​​corresponding to the question text based on bundle retrieval; The step of inputting the text to be identified and the question text into the generative multilingual model, and generating answers in different languages ​​corresponding to the question text based on beam retrieval, includes: Based on the text to be identified and the question text, candidate words and their probabilities are generated using the generative multilingual model; Set the bundle size to k, and select the k candidate words with the highest probability from the candidate words as the word examples for the first output position according to the probability of the candidate words; Based on the probability of the word example and the candidate word combination example at the first output position, select the k combination examples with the highest probability as the word examples at the second output position; Based on the probability of the word example and the combination of candidate words at the previous output position, select the k combination words with the highest probability as the word examples at the current output position, and so on until the word examples at all output positions are obtained, and use the word example with the highest probability as the generated answer.

2. The method as described in claim 1, characterized in that, The process of acquiring multilingual text data and pre-training a general language model based on the multilingual text data to obtain a multilingual pre-trained model includes: Acquire multilingual text data; For each language of the multilingual text data, multiple text segments are randomly sampled, and each sampled segment in the text sequence is replaced with a mask mark to obtain a corrupted text sequence, wherein each segment corresponds to a series of consecutive characters; The corrupted text sequence is used as the first sample data, and the segment replaced by the mask mark is used as the second sample data; Based on the first sample data and the second sample data, the general language model is pre-trained to obtain a multilingual pre-trained model.

3. The method as described in claim 2, characterized in that, The step of pre-training a general language model based on the multilingual text data to obtain a multilingual pre-trained model further includes: By changing the length and number of sampled text fragments, pre-training targets adapted to different tasks are generated to pre-train the general language model, resulting in a multilingual pre-trained model adapted to different tasks.

4. The method as described in claim 1, characterized in that, Fine-tuning the multilingual pre-trained model includes: Obtain the learning rate and batch size; Using the learning rate and batch size, the parameters of the multilingual pre-trained model are updated through backpropagation based on the loss scaling mechanism, thereby training the multilingual pre-trained model.

5. A device for constructing a cross-language question-answering system based on a generative multilingual model, characterized in that, include: The pre-training module is used to acquire multilingual text data and pre-train a general language model based on the multilingual text data to obtain a multilingual pre-trained model. The training sample construction module is used to obtain a question-and-answer dataset of a single corpus, process the question-and-answer dataset using placeholders, and construct training samples for prompt learning. The training module is used to iteratively fine-tune the multilingual pre-trained model based on the training samples and based on prompting learning to obtain a generative multilingual model. The generation module is used to acquire the text to be identified and the question text, input the text to be identified and the question text into the generative multilingual model, and generate answers in different languages ​​corresponding to the question text based on bundle retrieval. The generation module is specifically used for: Based on the text to be identified and the question text, candidate words and their probabilities are generated using the generative multilingual model; Set the bundle size to k, and select the k candidate words with the highest probability from the candidate words as the word examples for the first output position according to the probability of the candidate words; Based on the probability of the word example and the candidate word combination example at the first output position, select the k combination examples with the highest probability as the word examples at the second output position; Based on the probability of the word example and the combination of candidate words at the previous output position, select the k combination words with the highest probability as the word examples at the current output position, and so on until the word examples at all output positions are obtained, and use the word example with the highest probability as the generated answer.

6. The apparatus as claimed in claim 5, characterized in that, The pre-training module is specifically used for: Acquire multilingual text data; For each language of the multilingual text data, multiple text segments are randomly sampled, and each sampled segment in the text sequence is replaced with a mask mark to obtain a corrupted text sequence, wherein each segment corresponds to a series of consecutive characters; The corrupted text sequence is used as the first sample data, and the segment replaced by the mask mark is used as the second sample data; Based on the first sample data and the second sample data, the general language model is pre-trained to obtain a multilingual pre-trained model.

7. The apparatus as claimed in claim 5, characterized in that, The pre-training module is also used for: By changing the length and number of sampled text fragments, pre-training targets adapted to different tasks are generated to pre-train the general language model, resulting in a multilingual pre-trained model adapted to different tasks.

8. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the method as described in any one of claims 1-4.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-4.

Images