Task model training method and apparatus

Abstract

Embodiments of the present specification provide a task model training method and device, wherein the method comprises: obtaining first training data, the first training data comprising at least two training sample pairs, each training sample pair comprising a first training text and a first training sample corresponding to the first training text, the first training sample comprising a positive sample and a negative sample, and the training sample pair comprising a multilingual training sample pair; training an initial task model based on the at least two training sample pairs to obtain a pre-trained task model; constructing a to-be-trained task model according to the pre-trained task model and a preset downstream task; and training the to-be-trained task model according to second training data corresponding to the preset downstream task to obtain a target task model, the second training data comprising at least one second training sample and a sample label corresponding to each second training sample. By training the initial task model, a strong cross-language understanding capability can be learned, and the model training efficiency and the accuracy of the task execution result can be improved.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to a task model training method. Background Technology

[0002] With the development of artificial intelligence technology, pre-trained models are increasingly being applied to various fields. Due to their powerful multilingual representation capabilities, multilingual pre-trained models have made significant strides in the field of natural language processing. In particular, multilingual pre-trained models possess strong cross-language understanding capabilities, enabling them to solve a wide variety of natural language understanding tasks through fine-tuning for downstream tasks.

[0003] However, most current methods for improving the language understanding capabilities of multilingual pre-trained models involve pre-training with large amounts of multilingual data. Since there is a saturation point in the amount of data a model can receive, introducing too much pre-training data not only wastes training resources and fails to further improve cross-lingual representation capabilities, but may also have negative consequences. Therefore, there is an urgent need for a method to further improve the performance of multilingual pre-trained models on cross-lingual understanding tasks. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a task model training method. One or more embodiments of this specification also relate to a task model training apparatus, a computing device, and a computer-readable storage medium, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a task model training method is provided, comprising:

[0006] Obtain first training data, wherein the first training data includes at least two training sample pairs, each training sample pair includes a first training text and a first training sample corresponding to the first training text, the first training sample includes positive samples and negative samples, and the training sample pairs include multilingual training sample pairs.

[0007] A pre-trained task model is obtained by training an initial task model based on at least two training samples.

[0008] Based on the pre-trained task model and the preset downstream tasks, construct the task model to be trained;

[0009] The target task model is obtained by training the model of the task to be trained based on the second training data corresponding to the preset downstream task. The second training data includes at least one second training sample and the sample label corresponding to each second training sample.

[0010] According to a second aspect of the embodiments of this specification, a method for training an extractive question-answering task model is provided, comprising:

[0011] Obtain first training data, wherein the first training data includes at least two training sample pairs, each training sample pair includes a first training text and a first training sample corresponding to the first training text, the first training sample includes positive samples and negative samples, and the training sample pairs include multilingual training sample pairs.

[0012] A pre-trained task model is obtained by training an initial task model based on at least two training samples.

[0013] Based on the pre-trained task model and the extractive question answering task, construct the extractive question answering task model to be trained;

[0014] Based on the second training data corresponding to the extractive question answering task, train the extractive question answering task model to be trained to obtain the target extractive question answering task model. The second training data includes at least one question sample and the answer label corresponding to each question sample.

[0015] According to a third aspect of the embodiments of this specification, a data processing method for a task model is provided, comprising:

[0016] Receive a preset downstream task processing request sent by the user through the front end, wherein the preset downstream task processing request carries data to be processed;

[0017] The data to be processed is input into the target task model to obtain the task processing result corresponding to the preset downstream task processing request. The target task model is trained based on the second training data corresponding to the preset downstream task. The task model to be trained is constructed based on the pre-trained task model and the preset downstream task. The pre-trained task model is obtained by training the initial task model based on at least two training sample pairs. The second training data includes at least one second training sample and sample labels corresponding to each second training sample. Each training sample pair includes a first training text and a first training sample corresponding to the first training text. The first training sample includes positive samples and negative samples. The training sample pair includes multilingual training sample pairs.

[0018] Send the task processing results to the front end so that the front end can display the task processing results.

[0019] According to a fourth aspect of the embodiments of this specification, a task model training method is provided, applied to a cloud-side device, comprising:

[0020] Obtain first training data, wherein the first training data includes at least two training sample pairs, each training sample pair includes a first training text and a first training sample corresponding to the first training text, the first training sample includes positive samples and negative samples, and the training sample pairs include multilingual training sample pairs.

[0021] A pre-trained task model is obtained by training an initial task model based on at least two training samples.

[0022] Based on the pre-trained task model and the preset downstream tasks, construct the task model to be trained;

[0023] Based on the second training data corresponding to the preset downstream task, train the model of the task to be trained and obtain the model parameters of the target task model. The second training data includes at least one second training sample and the sample label corresponding to each second training sample.

[0024] Send the model parameters of the target task model to the end-side device.

[0025] According to a fifth aspect of the embodiments of this specification, a task model training apparatus is provided, comprising:

[0026] The acquisition module is configured to acquire first training data, wherein the first training data includes at least two training sample pairs, each training sample pair includes a first training text and a first training sample corresponding to the first training text, the first training sample includes positive samples and negative samples, and the training sample pairs include multilingual training sample pairs.

[0027] The first training module is configured to train the initial task model based on at least two training samples to obtain a pre-trained task model.

[0028] The building module is configured to build a model for the task to be trained based on a pre-trained task model and preset downstream tasks;

[0029] The second training module is configured to train the model of the task to be trained based on the second training data corresponding to the preset downstream task, and obtain the target task model. The second training data includes at least one second training sample and the sample label corresponding to each second training sample.

[0030] According to a sixth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0031] Memory and processor;

[0032] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the above-described task model training method.

[0033] According to a seventh aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the task model training method described above.

[0034] One embodiment of this specification implements the acquisition of first training data, wherein the first training data includes at least two training sample pairs, each training sample pair includes a first training text and a first training sample corresponding to the first training text, the first training sample includes positive samples and negative samples, and the training sample pairs include multilingual training sample pairs; an initial task model is trained based on at least two training sample pairs to obtain a pre-trained task model; a task model to be trained is constructed according to the pre-trained task model and a preset downstream task; the task model to be trained is trained according to the second training data corresponding to the preset downstream task to obtain a target task model, wherein the second training data includes at least one second training sample and sample labels corresponding to each second training sample.

[0035] In this way, by training the initial task model with the first training data, the initial task model learns a strong cross-language understanding ability from positive and negative samples, which can achieve better task performance when processing cross-language understanding tasks in the future. By setting up downstream tasks, the pre-trained task model is constructed into the task model to be trained, and the task model to be trained is trained according to the second training data. This allows the cross-language understanding ability learned by the pre-trained task model to be directly inherited in the training process of the downstream task, improving the convergence efficiency of the task model to be trained, thereby improving the training efficiency of the target task model and the accuracy of the target task model's cross-language understanding task performance results. Attached Figure Description

[0036] Figure 1 This is a flowchart illustrating a task model training method provided in one embodiment of this specification;

[0037] Figure 2 This is a schematic diagram illustrating the training process of a task model provided in one embodiment of this specification;

[0038] Figure 3 This is a flowchart illustrating the processing procedure of a task model training method provided in one embodiment of this specification.

[0039] Figure 4a This is a schematic diagram illustrating the construction of sample pairs for a task model provided in one embodiment of this specification;

[0040] Figure 4b This is a flowchart illustrating the pre-training process of a task model provided in one embodiment of this specification;

[0041] Figure 5 This is a flowchart illustrating a training method for an extractive question-answering task model, provided in one embodiment of this specification.

[0042] Figure 6 This is a flowchart illustrating a data processing method for a task model provided in one embodiment of this specification;

[0043] Figure 7 This is a flowchart illustrating a task model training method provided in one embodiment of this specification;

[0044] Figure 8 This is a schematic diagram of the structure of a task model training device provided in one embodiment of this specification;

[0045] Figure 9 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0046] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0047] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0048] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0049] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0050] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0051] Machine Reading Comprehension (MRC) is a paradigm of natural language processing models. Its input consists of a query and relevant context text, and its output is the positions of some answers within the context text, such that the answer segment satisfies the input query.

[0052] Natural Language Understanding (NLU): A collective term for the ideas, methods, and tasks that support machines in understanding textual data. Specifically, NLU refers to understanding between languages.

[0053] Pre-trained Language Model (PLM): A language model is pre-trained on a large amount of low-cost data to learn commonalities and acquire general knowledge. Unless otherwise specified, pre-trained language models specifically refer to models for English.

[0054] Multilingual Pre-trained Language Model (mPLM): A pre-trained language model based on multiple languages. This model learns common knowledge from multiple languages ​​simultaneously.

[0055] Cross-lingual language understanding (XLU): The ability to directly transfer language-related knowledge from one language (such as English) to another language (such as Chinese).

[0056] Named Entity Recognition (NER) refers to the identification of entities with specific meanings in text, mainly including names of people, places, organizations, proper nouns, etc.

[0057] Extractive Question Answering (EQA): Given a question, extract the corresponding answer from relevant text.

[0058] Attribute-based sentiment analysis (APS) is a fine-grained sentiment analysis task. It aims to detect attribute words in text that express emotion and determine their specific sentiment polarity. This task is a text segment recognition task.

[0059] Sentence-pair classification: Given two sentences, determine the relationship between them, including: implication, contradiction, and neutrality.

[0060] Entity: An entity is an objectively existing thing that can be distinguished from one another.

[0061] Fine-tuning: refers to the process of performing a small amount of training on given data after the downstream processing network loads the pre-trained model parameters.

[0062] XLM-R (XLM-RoBERTa): A large-scale unsupervised cross-lingual representation model that uses massive training data to train the model on 100 languages, resulting in good cross-lingual performance. However, once the amount of training data reaches a certain point, the overall performance of the model on both monolingual and cross-lingual benchmarks will decline.

[0063] mT5 (A massively multilingual pre-trained text-to-text transformer): A large-scale generative pre-trained model that uses an encoder-decoder model to expand the training corpus to 101 languages, improving the model's cross-language understanding capabilities.

[0064] Multilingual pre-trained language models have made significant progress in the field of natural language processing due to their powerful multilingual representation capabilities. In particular, they possess strong cross-lingual understanding capabilities; by fine-tuning them on downstream tasks in the source language, they can solve a wide variety of natural language understanding tasks in previously unseen target languages, such as named entity recognition, extractive question answering, and sentence pair classification.

[0065] To further enhance the cross-linguistic understanding capabilities of multilingual pre-trained language models, strengthen their potential interpretability, and improve their ability to understand language sentiment, existing methods for improving multilingual pre-trained language models mostly implicitly enhance their cross-linguistic representation capabilities by constructing large amounts of multilingual data. However, with a fixed model size, the model faces a situation where a large amount of multilingual training data can actually reduce its performance; that is, the amount of data the model can handle reaches a saturation point. Due to the limitation of model size, after the multilingual pre-trained model reaches data saturation, introducing additional multilingual data for pre-training will not bring further improvements in cross-linguistic representation. Not only is it difficult to achieve significant improvements in downstream task performance, but it may even have a negative impact on the model.

[0066] Exemplarily, cross-language supervision signals are currently often utilized to enhance cross-language representation capabilities. When training multilingual pre-trained models, existing solutions introduce cross-language supervision signals such as bilingual dictionaries and parallel sentence pairs to improve the alignment between two languages. For example, given the supervision signal "language" - "语言", the model can clearly learn that the Chinese corresponding to "language" is "语言". Supervision signals can provide very accurate cross-language alignment information. However, since they require manual annotation of alignment relationships, such supervision signals are extremely scarce and expensive for less frequently used minority languages, resulting in low generality. Similar methods also include directive fine-tuning. By converting annotated target language task data into a unified generation format and using this data to further fine-tune the language model, the natural understanding ability of the target language can be obtained. It is easy to understand that the cost of manually annotating training samples is very high. Moreover, for less frequently used minority languages, as well as languages with a small number of entity types, a general label system category, and unable to perform fine-grained classification and marking, it is not only likely to cause waste of sample resources, but may also have problems affecting the task execution effect, such as inaccurate model training results and slow convergence speed due to inaccurate labels.

[0067] Currently, in order to avoid the model from reaching the saturation point of the amount of data it can receive as much as possible, very large language models are often constructed to reduce the limitation of the amount of training data brought by the model scale. By introducing a large amount of multilingual data and a super-large-scale model for pre-training, strong multilingual natural language understanding capabilities can also be learned. Existing solutions show that increasing the number of languages in the pre-training corpus to a certain extent can effectively improve cross-language representation capabilities. For example, XLM-R introduced 100 languages, and mT5 introduced 101 languages. However, the problem of model scale limitation objectively exists. When the number of languages introduced reaches a certain level, introducing new languages will have very limited improvement in cross-language representation capabilities, and may even affect the alignment capabilities of existing languages, resulting in a decline in performance in downstream tasks. This solution has currently encountered a bottleneck.

[0068] When existing models perform language understanding tasks, they often resemble doing "fill in the blanks". They have a strong ability to translate words from one language to another. For example, "苹果" can be translated as "apple". However, their interpretability for sequence-level tasks, especially the ability to understand sentence sentiment, is weak. During the process of training for downstream tasks, the model has a slow convergence speed, low training efficiency, and the task execution effect of performing sequence-level tasks is not ideal enough.

[0069] To address the aforementioned issues, one or more embodiments of this specification provide a task model training method, which involves obtaining first training data, wherein the first training data includes at least two training sample pairs, each training sample pair including a first training text and a first training sample corresponding to the first training text, the first training sample including positive samples and negative samples, and the training sample pairs including multilingual training sample pairs; training an initial task model based on at least two training sample pairs to obtain a pre-trained task model; constructing a task model to be trained based on the pre-trained task model and a preset downstream task; and training the task model to be trained based on second training data corresponding to the preset downstream task to obtain a target task model, wherein the second training data includes at least one second training sample and sample labels corresponding to each second training sample.

[0070] In this way, by training the initial task model with the first training data, the initial task model learns a strong cross-language understanding ability from positive and negative samples, which can achieve better task performance when processing cross-language understanding tasks in the future. By setting up downstream tasks, the pre-trained task model is constructed into the task model to be trained, and the task model to be trained is trained according to the second training data. This allows the cross-language understanding ability learned by the pre-trained task model to be directly inherited in the training process of the downstream task, improving the convergence efficiency of the task model to be trained, thereby improving the training efficiency of the target task model and the accuracy of the target task model's cross-language understanding task performance results.

[0071] This specification provides a task model training method, and also relates to a task model training apparatus, a computing device, and a computer-readable storage medium, which will be described in detail in the following embodiments.

[0072] See Figure 1 , Figure 1 A flowchart of a task model training method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0073] Step 102: Obtain the first training data, wherein the first training data includes at least two training sample pairs, each training sample pair includes a first training text and a first training sample corresponding to the first training text, the first training sample includes positive samples and negative samples, and the training sample pairs include multilingual training sample pairs.

[0074] One or more embodiments of this specification can be applied to a client or server with text processing capabilities. In order to train a task model with better task performance, one or more embodiments of this specification can obtain first training data for training the initial task model.

[0075] Specifically, the first training data can be understood as training data used to train the initial task model. The first training data may include multiple training sample pairs. It should be noted that, in order to enable the initial task model to learn better cross-language understanding capabilities, the first training data may be multilingual training data, that is, the multiple training sample pairs may include multilingual training sample pairs.

[0076] A training sample pair can be understood as a pair of training samples consisting of a first training text and a first training sample corresponding to the first training text. By training the initial task model with multilingual training sample pairs, the initial task model can acquire strong cross-language understanding capabilities through self-learning.

[0077] The first training text can be understood as the training text describing the sample entity, and the sample entity can be understood as the target language that the initial task model needs to learn and understand. For example, the sample entity can be "apple". If the initial task model needs to learn the ability to understand the target language of "apple", then the text describing "apple" as "is a fruit" can be used as the first training text for "apple".

[0078] The first training sample can be understood as a training sample that has some kind of relationship with the sample entity. Optionally, the relationship can be related or unrelated. Based on the first training text and the first training sample, the initial task model can learn the ability to understand the sample entity.

[0079] Positive and negative samples can be understood as two different attributes corresponding to the first training sample. A first training sample with a relevant relationship to the sample entity can be considered a positive sample; a first training sample with no relevant relationship to the sample entity can be considered a negative sample. It should be noted that the first training sample with a relevant relationship to the sample entity can be uniformly assigned a sample label of 1, representing that an answer related to the sample entity can be obtained from the first training sample based on the first training text; the first training sample with no relevant relationship to the sample entity can be uniformly assigned a sample label of 0, representing that an answer related to the sample entity cannot be obtained from the first training sample based on the first training text. By pre-setting different numerical sample labels for positive and negative samples, not only can the cost of manual annotation be reduced and the efficiency of labeling samples be improved, but the initial task model can also be trained from both positive and negative perspectives to possess the ability to answer and not answer questions for the sample entity, improving the model's comprehensiveness and flexibility in multilingual understanding, and thus improving the model's performance on cross-language understanding tasks.

[0080] Multilingual training sample pairs can be understood as training sample pairs including at least two different languages. For example, the first training data includes training sample pair 1, training sample pair 2, ..., training sample pair n, where the language of training sample pair 1 is Chinese, the language of training sample pair 2 is English, and the language of training sample pair n is Japanese. Thus, the model trained based on multilingual training sample pairs possesses powerful cross-language understanding capabilities, enabling it to perform various specific downstream cross-language understanding tasks, such as named entity recognition, extractive question answering, sentence pair classification, and sentiment analysis, without requiring translation of the input text before task execution, greatly improving task execution efficiency and providing a better user experience.

[0081] Specifically, the initial task model is a machine learning model, which can be understood as a trained program that can discover patterns and make predictions in new data. These models are represented as a mathematical function that receives requests in the form of input data, makes predictions on the input data, and then provides output as a response. In one or more embodiments of this specification, the initial task model can be understood as a pre-trained reading comprehension device. The model structure of the initial task model may include an encoding module, an extraction module, and an output module. The encoding module can be used to map each keyword in the input text into a vector space and output the keyword vector corresponding to each keyword. The extraction module can be used to calculate the relevance probability between each pair of keyword vectors based on the input keyword vectors and output it to the output module. The output module can be used to calculate the relevance probability values ​​of each input and output the overall relevance parameter of the input text. For example, the stronger the calculated relevance of the input text, the closer the output value of the output module is to 1; the weaker the calculated relevance of the input text, the closer the output value of the output module is to 0.

[0082] Currently, multilingual pre-trained language models are often used to perform various downstream cross-language understanding tasks. Training a multilingual pre-trained language model involves preparing a large amount of multilingual training data, pre-training the multilingual pre-trained language model, and then fine-tuning the pre-trained model during the downstream task training process to obtain the trained multilingual pre-trained language model.

[0083] Due to limitations in model size, excessive training data cannot further improve the cross-linguistic understanding capabilities of multilingual pre-trained models. Therefore, one or more embodiments in this specification address the difficulty in improving the performance of multilingual pre-trained models by constructing a large amount of high-quality machine reading comprehension style data and based on machine reading comprehension paradigms.

[0084] Accordingly, in one or more embodiments of this specification, obtaining the first training data can be achieved through the following steps:

[0085] Retrieve the preset text database;

[0086] At least two sample entities are determined based on a pre-set text database, including multilingual sample entities;

[0087] Based on the target sample entity, obtain the target first training text and the target first training sample corresponding to the target sample entity, wherein the target sample entity is any one of at least two sample entities;

[0088] Based on the first training text and the first training sample, construct the target training sample pair corresponding to the target sample entity, and add the target training sample pair to the first training data.

[0089] In practical applications, constructing machine reading comprehension style data based on the machine reading comprehension paradigm can be achieved by constructing triples. For example, the sample entity can be used as the answer, the descriptive text describing the sample entity can be used as the query, and the mention text related to the sample entity can be used as the context text. Constructing triple training data in the machine reading comprehension style, such as (Query, Context, Answer), and inputting it into the model as training data, can enhance the model's machine reading comprehension capabilities.

[0090] However, constructing triples requires first identifying the sample entity, and then finding the appropriate query text and context text based on the sample entity. This is time-consuming, labor-intensive, and has high manual costs. Furthermore, for less commonly used languages, the cost of manually constructing triples is even more expensive.

[0091] Based on this, one or more embodiments of this specification obtain first training data for training the initial task model by acquiring a preset text database, determining at least two multilingual sample entities from the preset text database, acquiring the target first training text and target first training sample corresponding to the target sample entity based on the target sample entity, constructing a target training sample pair corresponding to the target sample entity based on the target first training text and target first training sample, and adding the target training sample pair to the first training data.

[0092] Specifically, the pre-set text database includes, but is not limited to, multilingual electronic dictionaries, multilingual electronic encyclopedias, and other databases.

[0093] The sample entities include multilingual sample entities, which enable the initial task model to be trained with cross-language understanding capabilities. Furthermore, sample entities can be marked with hyperlinks, allowing direct access to the definition text describing the sample entity and efficiently retrieving multiple reference texts mentioning the sample entity. These reference texts can then be used to construct multiple contextual texts related to the sample entity, serving as positive samples for that entity.

[0094] The target sample entity is any one of at least two sample entities. By using a pre-set text database, the sample entities in multiple languages ​​are determined. Based on the target sample entity, the first target training text and the first target training sample are obtained. Thus, a target training sample pair is constructed based on the first target training text and the first target training sample. This can greatly improve the efficiency of training data acquisition, reduce the manual cost of preparing training data, and ensure that the training data obtained based on the pre-set database is more accurate and academic in nature, thereby improving the quality of training data and ensuring that the model output results are more accurate and effective.

[0095] The target first training text is the training text corresponding to the target sample entity; the target first training sample is any one of at least one training sample corresponding to the target sample entity, and the target first training sample has positive sample attribute or negative sample attribute; the target training sample pair is the training sample pair corresponding to the target sample entity constructed based on the target first training text and the target first training sample. Adding the constructed target training sample pairs to the first training data can yield first training data including at least two training sample pairs.

[0096] By pre-setting a text database and identifying sample entities, the efficiency of obtaining multilingual sample entities can be effectively improved, and a large number of multilingual sample entities can be obtained. Based on the large number of multilingual sample entities, the first training text and first training sample corresponding to each sample entity can be obtained, thereby constructing a large amount of high-quality training data that conforms to the machine reading comprehension style, improving the efficiency and accuracy of model training.

[0097] It should be noted that multilingual and large-scale training data can indeed effectively improve the model's cross-language understanding ability. Therefore, one or more embodiments in this specification can construct multiple training sample pairs for a single sample entity. Based on this approach, a large number of multilingual machine reading comprehension training data, including both answerable and unanswerable types, can be constructed at a relatively low cost.

[0098] Accordingly, one or more embodiments of this specification may include the following steps S10202-S10210: Obtaining the target first training text and target first training sample corresponding to the target sample entity based on the target sample entity.

[0099] S10202. Based on the target sample entity, determine the definition text and at least one reference text corresponding to the target sample entity.

[0100] Specifically, the target sample entity is any one of at least two sample entities. The language of the target sample entity can be any of the existing languages. The target sample entity is an objectively existing and distinguishable entity. Specifically, in the pre-defined text database, the target sample entity can be a word or phrase with specific meaning, including but not limited to place names, personal names, organization names, and proper nouns. The pre-defined text database stores text related to the target sample entity. This can include descriptive text describing the target sample entity, which can also be understood as definition text, and reference text that mentions the target sample entity in the text content, which can also be understood as reference text.

[0101] Typically, there is one definition text for the target sample entity, and many corresponding reference texts. However, it is possible that there may be multiple definition texts for the target sample entity. The specific definition text for the target sample entity can be determined based on the actual storage situation of the preset text database. This manual does not impose any restrictions on this.

[0102] This specification describes one or more embodiments whereby, based on a target sample entity, a corresponding definition text and at least one corresponding reference text are determined from a preset text database. For example, the number of reference texts can be five. This can be achieved by first obtaining all mentions of the target sample entity, and then randomly selecting five reference texts from among the multiple mentions of the target sample entity. The method for obtaining the reference texts can be determined according to actual needs and specific circumstances, and this specification does not impose any limitations on it.

[0103] S10204. Based on the first text extraction rule, extract the target first training text from the definition text.

[0104] Specifically, the first text extraction rule is the rule for extracting the query text corresponding to the target sample entity from the definition text corresponding to that target sample entity. The query text can be understood as the question corresponding to the target sample entity. The target sample entity can be understood as the answer corresponding to the query text. The first training text can be understood as the query text corresponding to the target sample entity.

[0105] In practical applications, for a given definition text, the target first training text can be extracted using various methods. This specification provides one or more embodiments that offer at least one first text extraction rule for extracting the target first training text. Specifically, the first text extraction rule may include: using the first Q words in the definition text as the target first training text for the target sample entity, where Q is a preset constant.

[0106] By using the first text extraction rule, the target first training text corresponding to the target sample entity can be obtained, thereby improving the efficiency of training data acquisition and the quality of training data.

[0107] S10206. Based on the second text extraction rule, extract the first training positive sample of the target sample entity from each reference text.

[0108] Specifically, the second text extraction rule is a rule for extracting the context text corresponding to the target sample entity from each reference text. It should be noted that the first and second text extraction rules are different. Context text can be understood as the first training positive sample that has a relevant relationship with the target sample entity, and its corresponding sample label can be 1. Based on the target first training sample, the answer corresponding to the target first training sample, i.e., the target sample entity, can be found in the first training positive sample.

[0109] In practical applications, for a reference text, first training positive samples can be extracted using various methods. This specification provides one or more embodiments that offer at least one second text extraction rule for extracting first training positive samples. Specifically, the second text extraction rule may include: using the k words preceding and Ck words following the target sample entity as the first training positive samples of that target sample entity. Here, C is a preset constant, and k is a random constant value. By randomly setting the value of k, the target sample entity will not appear in a specific position in the context text, thereby reducing the shortcut information learned by the model and preventing inaccurate model training results.

[0110] By using the second text extraction rule, at least one first training positive sample corresponding to the target sample entity can be obtained, thereby improving the efficiency of training data acquisition and the quality of training data.

[0111] It should be noted that, according to the above steps S10202-S10206, hundreds of millions of responsive multilingual machine reading comprehension style data can be constructed. By training the initial task model with these data, a pre-trained task model with strong cross-language understanding ability can be obtained.

[0112] In one or more embodiments of this specification, in order to further improve the cross-language understanding ability of the pre-trained task model, a first training negative sample with the same number as the first training positive sample can be obtained, and training sample pairs can be constructed based on the first training text and the first training negative sample, thereby constructing unanswerable multilingual machine reading comprehension style data with the same number as the answerable multilingual machine reading comprehension style data.

[0113] Accordingly, in one or more embodiments of this specification, the first training negative sample of the target sample entity is obtained through the following steps S10208-S10210.

[0114] S10208. Determine the reference sample entity and obtain the reference training positive sample corresponding to the reference sample entity.

[0115] Specifically, a reference sample entity is any sample entity other than the target sample entity. A reference training positive sample is a positive sample corresponding to a reference sample entity. It's easy to understand that if a reference training positive sample has a relevant relationship with a reference sample entity, the likelihood of it having a relevant relationship with the target sample entity is relatively small; it may often be unrelated to the target sample entity. Therefore, obtaining the reference training positive sample corresponding to the reference sample entity, and using it as the first training negative sample for the target sample entity, can improve the efficiency and cost of obtaining negative samples, and quickly construct the same number of negative samples as the number of positive samples.

[0116] S10210. Use the reference training positive sample corresponding to the reference sample entity as the first training negative sample of the target sample entity.

[0117] In practical applications, after obtaining the first training positive sample and the first training negative sample, the target training sample pair can be added to the first training data by constructing the target training sample pair corresponding to the target sample entity based on the target first training text and the first training positive sample, or the first training negative sample. This can greatly improve the efficiency of obtaining the first training data, reduce the data acquisition cost, and improve the quality of the first training data.

[0118] It should be noted that the order in which the first training positive sample and the first training negative sample corresponding to each training sample entity are obtained can be determined according to the actual situation, and this application does not impose any restrictions on this.

[0119] After obtaining the first training data, the first training data can be input into the initial task model, and the initial task model can be trained based on the first training data and the sample labels corresponding to each first training sample in the first training data.

[0120] Step 104: Train the initial task model based on at least two training sample pairs to obtain the pre-trained task model.

[0121] In practical applications, the cross-language reading comprehension ability of the initial task model can be trained using at least two training sample pairs to obtain a pre-trained task model.

[0122] Specifically, the initial task model and the pre-trained task model can be understood as machine learning models. A machine learning model is a trained program that can discover patterns and make predictions in new data. These models are represented as a mathematical function that receives a request in the form of input data, makes a prediction on the input data, and then provides an output as a response. The pre-trained task model can be understood as the initial task model that has been trained based on at least two pairs of training samples and through multiple model iterations.

[0123] In one or more embodiments of this specification, obtaining a pre-trained task model based on at least two training samples to train an initial task model may include the following steps:

[0124] Input the target training sample pair into the initial task model to obtain the target initial probability parameters corresponding to the target training sample pair output by the initial task model, wherein the target training sample pair is any one of at least two training samples;

[0125] The first model loss value is calculated based on the first target label and the initial probability parameters of the target training sample pair, wherein the first target label is determined based on the first training sample in the target training sample pair.

[0126] Adjust the initial task model based on the loss value of the first model, and continue training the initial task model until the model training stops, thus obtaining the pre-trained task model.

[0127] Specifically, the target training sample pair is any one of at least two training samples. The initial target probability parameter can be used to characterize the correlation between the target first training text and the target first training sample in the target training sample pair, and is specifically a numerical value. For example, the closer the initial target probability parameter is to 1, the more correlated the target first training text and the target first training sample are; the closer the initial target probability parameter is to 0, the less correlated the target first training text and the target first training sample are. The probability values ​​0 and 1 mentioned above are merely illustrative examples in one or more embodiments of this specification for the purpose of facilitating understanding of the technical solution and should not be construed as limiting this specification.

[0128] In practical applications, the first model loss value of the initial task model can be calculated based on the first target label and the initial probability parameters of the target training samples.

[0129] Specifically, the target first label is the label corresponding to the target training sample pair, and the label value is determined based on the attribute of the target first training sample in the target training sample pair. For example, if the target first training sample is a positive sample, the target first label can be 1; if the target first training sample is a negative sample, the target first label can be 0.

[0130] Based on the initial probability parameters of the target output from the initial task model based on the target training samples, and the first label of the target, the cross-entropy can be calculated to obtain the first model loss value of the initial task model. The model parameters of the initial task model are then adjusted according to the first model loss value, and iterative training continues until the model training stopping condition is reached to obtain the pre-trained task model.

[0131] In one or more embodiments of this specification, the model training stopping condition for the initial task model may be that the output target initial probability parameter meets a preset parameter value, or that the loss value calculated by the target initial probability parameter and the target first label is less than a preset threshold, or that the number of iterations reaches a preset value.

[0132] By training the initial task model, the initial probability parameters of the target output by the extraction module in the initial task model are calibrated according to the first target label, so that the extractor has the ability to accurately output the correlation between any two keyword vectors, thereby improving the model's cross-language reading comprehension ability.

[0133] Optionally, the initial task model may include an encoding module and an extraction module;

[0134] Accordingly, inputting the target training sample pairs into the initial task model to obtain the target initial probability parameters corresponding to the target training sample pairs output by the initial task model can include the following steps S1042-S1046:

[0135] S1042. Concatenate the target first training text and the target first training sample in the target training sample pair to obtain the target input text.

[0136] Specifically, the target input text can be the input text obtained by concatenating the target first training text, the target first training samples, and model training special words. For example, the model training special words can include CLS and SEP, where CLS can be understood as the starting word of the input text, and SEP can be used to separate sentences in the text.

[0137] After obtaining the target input text, the target input text can be input into the encoding module, and the keyword vector set corresponding to the target input text can be obtained according to the encoding module.

[0138] S1044. Input the target input text into the encoding module to obtain the keyword vector set corresponding to the target input text.

[0139] In practical applications, since the target input text may be a single piece of text, and the encoding module typically maps individual words in the text to vectors based on word units, inputting the target input text into the encoding module to obtain the keyword vector set corresponding to the target input text can include the following steps:

[0140] The target input text is segmented into words to obtain the set of keywords corresponding to the target input text;

[0141] Input the keyword set into the encoding module;

[0142] Obtain the keyword vector set corresponding to the keyword set output by the encoding module.

[0143] In practical applications, the target input text can be segmented using word segmentation tools such as Jieba to obtain a set of keywords corresponding to the target input text. Then, each keyword in the keyword set is input into the encoding module to obtain a set of keyword vectors composed of keyword vectors output by the encoding module for each keyword.

[0144] Specifically, the encoding module can map each keyword in natural language form onto a vector space to obtain the keyword vector corresponding to each keyword.

[0145] S1046. Input each keyword vector in the keyword vector set into the extraction module to obtain the target initial probability parameters of the target training sample pairs.

[0146] In practical applications, the keyword vector set corresponding to the target input text is obtained through the encoding module. Each keyword in the keyword vector set can be input into the extraction module. The extraction module performs feature extraction on each keyword vector to obtain the target initial probability parameters of the target training sample pair.

[0147] Specifically, the extraction module can obtain the pairwise relationships between each keyword vector in the input keyword vector set. Based on the relationship between any two keywords in the input text, it can obtain the relationship between the target first training text and the target first training sample in the target training sample pair.

[0148] Optionally, the initial task model may also include an output module;

[0149] Accordingly, the initial probability parameters of the target training sample pairs are obtained by inputting each keyword vector from the keyword vector set into the extraction module, which may include the following steps:

[0150] Input the vectors of each keyword into the extraction module;

[0151] The extraction module calculates the first probability value corresponding to the first keyword vector and the second keyword vector based on the first keyword vector and the second keyword vector, wherein the first keyword vector and the second keyword vector are any two of the keyword vectors;

[0152] Input each first probability value into the output module to obtain the target initial probability parameters of the target training sample pair output by the output module.

[0153] Specifically, each keyword vector is input into the extraction module, which calculates a first probability value corresponding to the first and second keyword vectors based on the first and second keyword vectors. Here, the first and second keyword vectors are any two of the keyword vectors, and the first probability value characterizes the relevance probability between the first and second keyword vectors. For example, the more relevant the first and second keyword vectors are, the closer the first probability value is to 1; the less relevant they are, the closer the first probability value is to 0.

[0154] In practical applications, each first probability value is input into the output module, and the output module can calculate the target initial probability parameters of the target training sample pair based on each first probability value.

[0155] Specifically, the initial probability parameter can be used to characterize the correlation probability between the first training text and the first training sample in the target training sample pair. For example, the more correlated the first training text and the first training sample are, the closer the first probability value is to 1; the less correlated they are, the closer the first probability value is to 0.

[0156] The extraction module calculates the relevance probability between keyword vectors corresponding to any two keywords in the input text, thus obtaining the relevance relationship between any two keywords in the input text. The output module outputs the target initial probability parameters of the target training sample pair based on the first probability values ​​output by the extraction module. By comparing the target initial probability parameters with the target first label, the self-learning status of the initial task model in each loop can be monitored in real time, and the model parameters of the initial task model can be tuned to improve the feature extraction capability of the extraction module, thereby improving the cross-language understanding capability of the pre-trained task model.

[0157] Step 106: Construct the task model to be trained based on the pre-trained task model and the preset downstream tasks.

[0158] To avoid the limitation of training data volume on the model's own size, which makes it difficult for the model to significantly improve its performance on downstream tasks, one or more embodiments in this specification can modify the model structure based on the pre-trained task model obtained through training. This allows the reading comprehension ability acquired during the pre-training stage to be directly inherited into the downstream task training process, thereby further improving the task model's performance on cross-language understanding tasks.

[0159] Specifically, the pre-defined downstream task can be understood as the task that the trained model is actually intended to solve. Model training typically consists of a pre-training phase and a fine-tuning phase. The pre-training phase uses massive amounts of training data to learn commonalities and acquire general knowledge, such as cross-language understanding capabilities. The fine-tuning phase, on the other hand, constructs training sample data based on the specific task to be solved, and fine-tunes the model parameters based on the pre-trained data, enabling the model to perform the task for that specific task. Therefore, the pre-defined downstream task is the specific task that the trained model is intended to solve; it can be understood as any type of cross-language understanding task.

[0160] Specifically, in one or more real-time examples of this specification, the preset downstream task may include at least one of word-level tasks and sequence-level tasks.

[0161] For example, word-level tasks can include word-level extraction tasks with fixed task labels, such as named entity recognition. In a named entity recognition task, each task label can serve as query text to retrieve the corresponding entity in the context text, and the retrieved entity is identified as the answer and output.

[0162] Word-level tasks can also include word-level extraction tasks based on natural language questions, such as extractive question answering. Extractive question answering can be understood as treating the question as query text and searching for the answer in the context text.

[0163] Sequence-level tasks can include sentence pair classification tasks with fixed task labels, such as sentence pair classification. In sentence pair classification, each task label can be used as query text to determine whether it is relevant to the context text corresponding to the sentence pair.

[0164] The preset downstream tasks can include any word-level or sequence-level tasks included in cross-language understanding tasks. Therefore, by constructing a task model to be trained based on the pre-trained task model and the preset downstream tasks, the target task model obtained by training the task model to be trained can be capable of handling various specific cross-language understanding tasks, can be applied in various practical task scenarios, has better transferability, versatility, and applicability, and can effectively reduce the training and maintenance costs of the model.

[0165] Optionally, the pre-trained task model includes an encoding module, an extraction module, and an output module;

[0166] Accordingly, based on the pre-trained task model and the preset downstream tasks, a task model to be trained is constructed, including:

[0167] Obtain the downstream task module corresponding to the preset downstream task;

[0168] Replace the output module with the downstream task module to obtain the task model to be trained.

[0169] Specifically, the downstream task module can be understood as the module that obtains the answer from the input context text based on the relevance probability output by the extraction module and outputs it.

[0170] By replacing the original output module of the model to be trained with the downstream task module corresponding to the preset downstream task, the output results of the extraction module that are relevant to the answer can be directly input into the downstream task module. This allows the downstream task module to inherit the natural language understanding ability obtained in the multilingual machine reading comprehension pre-training, which can effectively improve the convergence efficiency of the model to be trained, accelerate the model convergence speed, and thus improve the training efficiency of the model to be trained and the accuracy of the cross-language reading comprehension task execution results.

[0171] It should be noted that, according to one or more embodiments of this specification, the task model training method obtains a pre-trained task model by training an initial task model with at least two multilingual training sample pairs. Furthermore, by replacing the output module of the pre-trained task model with a downstream task module, the model structure of the pre-trained task model is modified to obtain a task model to be trained. This allows the task model to be trained to directly inherit the cross-language understanding ability learned by the pre-trained task model during the training process using training data corresponding to the preset downstream task. This significantly improves the cross-language task execution performance of the task model to be trained, especially in handling sequence-level tasks where it exhibits strong interpretability. Compared to current multilingual pre-trained models, it can achieve better task execution results.

[0172] Step 108: Train the model of the task to be trained based on the second training data corresponding to the preset downstream task to obtain the target task model. The second training data includes at least one second training sample and the sample label corresponding to each second training sample.

[0173] In practical applications, the model structure of the pre-trained task model is modified to obtain the task model to be trained. The task model to be trained can be trained based on the second training data corresponding to the preset downstream task, thereby obtaining the target task model.

[0174] Specifically, the second training data can be understood as the training data corresponding to the preset downstream task. The second training data includes at least one second training sample and sample labels corresponding to each second training sample. Further, the sample labels corresponding to the second samples can be understood as the task execution result that the preset downstream task wants to obtain, or as the answer that the target task model should output. Depending on different preset downstream tasks, the second training samples and the sample labels corresponding to the second training samples will be different, and will be determined according to the actual situation. This specification does not impose any limitations on this.

[0175] The training task model and the target task model can also be understood as machine learning models. Specifically, the target task model can be understood as the training task model that has reached the model training stopping condition and completed model training.

[0176] Optionally, the task model to be trained includes an encoding module, an extraction module, and a downstream training module;

[0177] Accordingly, based on the second training data corresponding to the preset downstream task, the model for the task to be trained is trained to obtain the target task model, which may include the following steps:

[0178] Input the target second training sample and the target downstream task label into the task model to be trained to obtain the downstream task prediction result output by the task model to be trained. The target second training sample is any one of at least one second training sample, and the target downstream task label is the task label corresponding to the preset downstream task.

[0179] The loss value of the second model is calculated based on the target sample label corresponding to the second training sample and the prediction results of the downstream task.

[0180] Adjust the training task model based on the second model loss value, and continue training the training task model until the model training stops, thus obtaining the target task model.

[0181] Specifically, the target downstream task label can be understood as the task label corresponding to the preset downstream task. For example, if the preset downstream task is named entity recognition, then the target downstream task label could be "named entity". The downstream task prediction result is the prediction result output by the training task model, which is the task execution result corresponding to the preset downstream task. The target sample label is the task execution result that the training task model should output. Continuing with the previous example, if the preset downstream task is named entity recognition and the second training sample is "Tom ate an apple", then the target sample labels could be "Tom" and "apple".

[0182] The second model loss value is the model loss value of the task model to be trained, obtained by comparing the prediction results of the downstream task with the target sample labels. Specifically, it can be calculated using methods such as cross-entropy.

[0183] The training task model is adjusted based on the second model loss value, and training continues until the model training stops. This results in a target task model that has been trained and has strong cross-language understanding and task execution capabilities.

[0184] One embodiment of this specification implements the acquisition of first training data, wherein the first training data includes at least two training sample pairs, each training sample pair includes a first training text and a first training sample corresponding to the first training text, the first training sample includes positive samples and negative samples, and the training sample pairs include multilingual training sample pairs; an initial task model is trained based on at least two training sample pairs to obtain a pre-trained task model; a task model to be trained is constructed according to the pre-trained task model and a preset downstream task; the task model to be trained is trained according to the second training data corresponding to the preset downstream task to obtain a target task model, wherein the second training data includes at least one second training sample and sample labels corresponding to each second training sample.

[0185] In this way, by training the initial task model with the first training data, the initial task model learns a strong cross-language understanding ability from positive and negative samples, which can achieve better task performance when processing cross-language understanding tasks in the future. By setting up downstream tasks, the pre-trained task model is constructed into the task model to be trained, and the task model to be trained is trained according to the second training data. This allows the cross-language understanding ability learned by the pre-trained task model to be directly inherited in the training process of the downstream task, improving the convergence efficiency of the task model to be trained, thereby improving the training efficiency of the target task model and the accuracy of the target task model's cross-language understanding task performance results.

[0186] For example, see Figure 2 , Figure 2 This diagram illustrates the training process of a task model provided in one embodiment of this specification.

[0187] In one or more embodiments of this specification, a preset sample database can be obtained, and two sample entities can be determined from the preset sample database. The language of sample entity 1 can be Chinese, and the language of sample entity 2 can be English. Based on sample entity 1, definition text 1 and reference text 1 corresponding to sample entity 1 are obtained. Based on sample entity 2, definition text 2 and reference text 2 corresponding to sample entity 2 are obtained. According to a first text extraction rule, training text 1 corresponding to sample entity 1 is extracted from definition text 1, and training text 2 corresponding to sample entity 2 is extracted from definition text 2. According to a second text extraction rule, training positive sample 1 corresponding to sample entity 1 is extracted from reference text 1, and training positive sample 2 corresponding to sample entity 2 is extracted from reference text 2. Based on training text 1 and training positive sample 1, training sample pair 1 corresponding to sample entity 1 is constructed; based on training text 2 and training positive sample 2, training sample pair 2 corresponding to sample entity 2 is constructed.

[0188] Based on the training samples, the multilingual pre-trained models 1 and 2 and the initial task model are pre-trained. The cross-language understanding ability learned by the multilingual pre-trained language model and the initial task model can be directly inherited in the training process of the downstream cross-language understanding task, which accelerates the convergence speed of the model in the preset downstream training task, improves the training efficiency of the target task model, enhances the cross-language understanding ability of the target task model, and enables the target task model to achieve better task performance when performing cross-language understanding tasks.

[0189] The following is in conjunction with the appendix Figure 3 Taking the task model training method provided in this specification in the application of named entity recognition as an example, the task model training method will be further explained. Figure 3 The flowchart of a task model training method provided in one embodiment of this specification is shown, which specifically includes the following steps.

[0190] Step 302: Obtain a preset text database and determine at least two sample entities based on the preset text database.

[0191] Step 304: Based on the target sample entity, determine one definition text and five reference texts corresponding to the target sample entity, wherein the target sample entity is any one of at least two sample entities.

[0192] Step 306: Based on the first text extraction rule, extract the target first training text from the definition text. Based on the second text extraction rule, extract the five first training positive samples of the target sample entity from the five reference texts.

[0193] Step 308: Determine the reference sample entity and obtain the 5 reference training positive samples corresponding to the reference sample entity. Determine these 5 reference training positive samples as the 5 first training negative samples of the target sample entity.

[0194] Step 310: Based on the first training text, the first training positive sample, and the first training negative sample, construct 10 training sample pairs corresponding to the target sample entity, and add these 10 sample pairs to the first training data.

[0195] Specifically, the first training text can be the query text, and the first training positive sample can be the context text related to the target sample entity.

[0196] See Figure 4a , Figure 4a This specification illustrates a schematic diagram of the construction of a sample pair of a task model provided in one embodiment.

[0197] Based on the target sample entity, one definition text and five reference texts corresponding to the target sample entity are determined. Using a first text extraction method, the target first training text corresponding to the target sample entity is extracted from the definition text. Using a second text extraction method, five target first training positive samples corresponding to the target sample entity are extracted from the five reference texts. Matching the target first training text with the five target first training positive samples yields five training sample pairs corresponding to the target sample entity.

[0198] Step 312: Concatenate the target first training text and the target first training sample in the target training sample pair to obtain the input text. Input the input text into the encoding module in the initial task model to obtain the keyword set corresponding to the input text.

[0199] Step 314: Input the keyword set into the extraction module in the initial task model to obtain the first probability value between any two keywords output by the extraction module. Input each first probability value into the output module in the initial task model to obtain the initial probability parameters output by the initial task model.

[0200] Step 316: Calculate the first model loss value of the initial task model based on the initial probability parameters and the first label corresponding to each sample pair in the first training data.

[0201] Step 318: Adjust the initial task model according to the first model loss value, and continue training the initial task model until the model training stopping condition is met to obtain the pre-trained task model.

[0202] See Figure 4b See Figure 4b A flowchart illustrating the pre-training process of a task model provided in one embodiment of this specification is shown.

[0203] The input text is obtained by concatenating the Q keywords corresponding to the query text and the C keywords corresponding to the context text with the special words CLS and SEP, where Q and C are preset constants.

[0204] The input text is segmented to obtain a keyword set corresponding to the input text. This keyword set is then input into the encoding module of the initial task model to obtain M keyword vectors, where M is the sum of the number of Q, C, and special words. These M keyword vectors are then input into the extraction module of the initial task model. The extraction module outputs a first probability value between any two keywords, allowing the calculation of initial probability parameters between the query text and the context text based on these first probability values. Specifically, these initial probability parameters characterize the relevance between the query text and the context text.

[0205] Step 320: Obtain the downstream task module corresponding to named entity recognition, replace the output module included in the pre-trained task model with the downstream task module, and obtain the task model to be trained.

[0206] Step 322: Train the model for the task to be trained based on the second training data corresponding to named entity recognition, and obtain the target task model.

[0207] One embodiment of this specification implements the acquisition of first training data, wherein the first training data includes at least two training sample pairs, each training sample pair includes a first training text and a first training sample corresponding to the first training text, the first training sample includes positive samples and negative samples, and the training sample pairs include multilingual training sample pairs; an initial task model is trained based on at least two training sample pairs to obtain a pre-trained task model; a task model to be trained is constructed according to the pre-trained task model and a preset downstream task; the task model to be trained is trained according to the second training data corresponding to the preset downstream task to obtain a target task model, wherein the second training data includes at least one second training sample and sample labels corresponding to each second training sample.

[0208] In this way, by training the initial task model with the first training data, the initial task model learns a strong cross-language understanding ability from positive and negative samples, which can achieve better task performance when processing cross-language understanding tasks in the future. By setting up downstream tasks, the pre-trained task model is constructed into the task model to be trained, and the task model to be trained is trained according to the second training data. This allows the cross-language understanding ability learned by the pre-trained task model to be directly inherited in the training process of the downstream task, improving the convergence efficiency of the task model to be trained, thereby improving the training efficiency of the target task model and the accuracy of the target task model's cross-language understanding task performance results.

[0209] See Figure 5 , Figure 5 A flowchart of an extraction-based question-answering task model training method provided in one embodiment of this specification is shown.

[0210] Step 502: Obtain the first training data, wherein the first training data includes at least two training sample pairs, each training sample pair includes a first training text and a first training sample corresponding to the first training text, the first training sample includes positive samples and negative samples, and the training sample pairs include multilingual training sample pairs.

[0211] Step 504: Train the initial task model based on at least two training sample pairs to obtain the pre-trained task model.

[0212] Step 506: Construct the extractive question answering task model to be trained based on the pre-trained task model and the extractive question answering task.

[0213] Step 508: Train the extractive question answering task model to be trained based on the second training data corresponding to the extractive question answering task, and obtain the target extractive question answering task model. The second training data includes at least one question sample and the answer label corresponding to each question sample.

[0214] It should be noted that the implementation methods of steps 502-508 are the same as those of steps 102-108 above, and will not be described again in the embodiments of this specification.

[0215] The scheme of the embodiments of this specification is applied by training an initial task model with first training data, so that the initial task model learns a strong cross-language understanding ability from positive and negative samples, and can obtain better task performance when processing cross-language understanding tasks in the future. By setting a downstream task, the pre-trained task model is constructed into a task model to be trained, and the task model to be trained is trained according to the second training data, so that the cross-language understanding ability learned by the pre-trained task model can be directly inherited in the training process of the downstream task, improving the convergence efficiency of the task model to be trained, thereby improving the training efficiency of the target task model and the accuracy of the cross-language understanding task execution results of the target task model.

[0216] See Figure 6 , Figure 6 A flowchart illustrating a data processing method for a task model provided in one embodiment of this specification is shown.

[0217] Step 602: Receive a preset downstream task processing request sent by the user through the front end, wherein the preset downstream task processing request carries data to be processed.

[0218] Step 604: Input the data to be processed into the target task model to obtain the task processing result corresponding to the preset downstream task processing request. The target task model is trained based on the second training data corresponding to the preset downstream task. The task model to be trained is constructed based on the pre-trained task model and the preset downstream task. The pre-trained task model is trained based on at least two training sample pairs to obtain the initial task model. The second training data includes at least one second training sample and the sample label corresponding to each second training sample. Each training sample pair includes a first training text and the first training sample corresponding to the first training text. The first training sample includes positive samples and negative samples. The training sample pairs include multilingual training sample pairs.

[0219] Step 606: Send the task processing result to the front end so that the front end can display the task processing result.

[0220] In practical applications, receiving pre-defined downstream task processing requests sent by users through the front-end can be handled by either the client or the server. That is, the target task model can be deployed on either the client or the server. The front-end can be understood as the front-end part of the website on the client side; the client can be called the end-side device, and the server side can be called the cloud-side device. Multiple clients can establish communication connections through the server. In real-world scenarios, the server can be used to provide data processing services for the task model across multiple clients, with each client acting as either a sender or a receiver, communicating through the server.

[0221] Users can interact with the server through the client to receive data sent by other clients or send data to other clients. In practical applications, users can publish data streams to the server through the client, and the server can generate task processing results based on the data stream and push the task processing results to other clients that have established communication.

[0222] In this process, the client and server establish a connection via a network. The network provides the medium for communication between the client and server. Networks can include various connection types, such as wired or wireless communication links, or fiber optic cables. Data transmitted by the client may need to undergo encoding, transcoding, compression, or other processing before being sent to the server.

[0223] The client can be a browser, an app (application), a web application such as an H5 (HyperText Markup Language 5) application, a lightweight application (also known as a mini-program), or a cloud application. The client can be developed based on the software development kit (SDK) provided by the server, such as a real-time communication (RTC) SDK. The client can be deployed on electronic devices and depends on the device or certain apps running on the device to function. Electronic devices may have displays and support information browsing, such as personal mobile terminals like smartphones, tablets, and personal computers. Various other types of applications can also be configured on electronic devices, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, and social media platforms.

[0224] The server side can include servers providing various services, such as servers providing communication services for multiple clients, servers supporting backend training of models used on clients, and servers processing data sent by clients. It should be noted that the server side can be implemented as a distributed server cluster composed of multiple servers, or as a single server. The server can also be a server in a distributed system, or a server integrated with blockchain. The server can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0225] It is worth noting that the task model training method provided in the embodiments of this specification is generally executed by the server. However, in other embodiments of this specification, the client may also have similar functions to the server, thereby executing the task model training method provided in the embodiments of this specification. In other embodiments, the task model training method provided in the embodiments of this specification may also be executed jointly by the client and the server.

[0226] For example, suppose the downstream task processing request is an extraction question answering task processing request. The extraction question answering task carries input text and question text as data to be processed and inputs them into the target task model. After the target task model processes the data to be processed, the answer text extracted from the input text based on the question text can be obtained.

[0227] The scheme of the embodiments of this specification is applied by training an initial task model with first training data, so that the initial task model learns a strong cross-language understanding ability from positive and negative samples, and can obtain better task performance when processing cross-language understanding tasks in the future. By setting a downstream task, the pre-trained task model is constructed into a task model to be trained, and the task model to be trained is trained according to the second training data, so that the cross-language understanding ability learned by the pre-trained task model can be directly inherited in the training process of the downstream task, improving the convergence efficiency of the task model to be trained, thereby improving the training efficiency of the target task model and the accuracy of the cross-language understanding task execution results of the target task model.

[0228] See Figure 7 , Figure 7 A flowchart of a task model training method provided in one embodiment of this specification is shown, which is applied to a cloud-side device.

[0229] Step 702: Obtain the first training data, wherein the first training data includes at least two training sample pairs, each training sample pair includes a first training text and a first training sample corresponding to the first training text, the first training sample includes positive samples and negative samples, and the training sample pairs include multilingual training sample pairs.

[0230] Step 704: Train the initial task model based on at least two training sample pairs to obtain the pre-trained task model.

[0231] Step 706: Construct the task model to be trained based on the pre-trained task model and the preset downstream tasks.

[0232] Step 708: Train the model of the task to be trained according to the second training data corresponding to the preset downstream task, and obtain the model parameters of the target task model. The second training data includes at least one second training sample and the sample label corresponding to each second training sample.

[0233] Step 710: Send the model parameters of the target task model to the end-side device.

[0234] It should be noted that the implementation methods of steps 702-708 are the same as those of steps 102-108 above, and will not be described again in the embodiments of this specification.

[0235] The scheme of the embodiments of this specification is applied by training an initial task model with first training data, so that the initial task model learns a strong cross-language understanding ability from positive and negative samples, and can obtain better task performance when processing cross-language understanding tasks in the future. By setting a downstream task, the pre-trained task model is constructed into a task model to be trained, and the task model to be trained is trained according to the second training data, so that the cross-language understanding ability learned by the pre-trained task model can be directly inherited in the training process of the downstream task, improving the convergence efficiency of the task model to be trained, thereby improving the training efficiency of the target task model and the accuracy of the cross-language understanding task execution results of the target task model.

[0236] Corresponding to the above method embodiments, this specification also provides embodiments of a task model training device. Figure 8 A schematic diagram of a task model training apparatus according to one embodiment of this specification is shown. Figure 8 As shown, the device includes:

[0237] The acquisition module 802 is configured to acquire first training data, wherein the first training data includes at least two training sample pairs, each training sample pair includes a first training text and a first training sample corresponding to the first training text, the first training sample includes positive samples and negative samples, and the training sample pairs include multilingual training sample pairs.

[0238] The first training module 804 is configured to train the initial task model based on at least two training sample pairs to obtain a pre-trained task model.

[0239] Module 806 is configured to build a model for a task to be trained based on a pre-trained task model and a preset downstream task.

[0240] The second training module 808 is configured to train the model of the task to be trained based on the second training data corresponding to the preset downstream task, and obtain the target task model. The second training data includes at least one second training sample and sample labels corresponding to each second training sample.

[0241] Optionally, the acquisition module 802 is further configured as follows:

[0242] Retrieve the preset text database;

[0243] At least two sample entities are determined based on a pre-set text database, including multilingual sample entities;

[0244] Based on the target sample entity, obtain the target first training text and the target first training sample corresponding to the target sample entity, wherein the target sample entity is any one of at least two sample entities;

[0245] Based on the first training text and the first training sample, construct the target training sample pair corresponding to the target sample entity, and add the target training sample pair to the first training data.

[0246] Optionally, the acquisition module 802 is further configured as follows:

[0247] Based on the target sample entity, determine the definition text and at least one reference text corresponding to the target sample entity;

[0248] Based on the first text extraction rule, extract the target first training text from the definition text;

[0249] Based on the second text extraction rule, the first training positive sample of the target sample entity is extracted from each reference text;

[0250] Identify the reference sample entity and obtain the corresponding reference training positive sample;

[0251] The reference training positive sample corresponding to the reference sample entity is used as the first training negative sample of the target sample entity.

[0252] Optionally, the first training module 804 is further configured as follows:

[0253] Input the target training sample pair into the initial task model to obtain the target initial probability parameters corresponding to the target training sample pair output by the initial task model, wherein the target training sample pair is any one of at least two training samples;

[0254] The first model loss value is calculated based on the first target label and the initial probability parameters of the target training sample pair, wherein the first target label is determined based on the first training sample in the target training sample pair.

[0255] Adjust the initial task model based on the loss value of the first model, and continue training the initial task model until the model training stops, thus obtaining the pre-trained task model.

[0256] Optionally, the initial task model includes an encoding module and an extraction module;

[0257] Accordingly, the first training module 804 is further configured as follows:

[0258] The target input text is obtained by concatenating the target first training text and the target first training sample in the target training sample pair.

[0259] The target input text is input into the encoding module to obtain the keyword vector set corresponding to the target input text;

[0260] Input each keyword vector in the keyword vector set into the extraction module to obtain the target initial probability parameters of the target training sample pairs.

[0261] Optionally, the first training module 804 is further configured as follows:

[0262] The target input text is segmented into words to obtain the set of keywords corresponding to the target input text;

[0263] Input the keyword set into the encoding module;

[0264] Obtain the keyword vector set corresponding to the keyword set output by the encoding module.

[0265] Optionally, the first training module 804 is further configured as follows:

[0266] Accordingly, each keyword vector in the keyword vector set is input into the extraction module to obtain the target initial probability parameters of the target training sample pairs, including:

[0267] Input the vectors of each keyword into the extraction module;

[0268] The extraction module calculates the first probability value corresponding to the first keyword vector and the second keyword vector based on the first keyword vector and the second keyword vector, wherein the first keyword vector and the second keyword vector are any two of the keyword vectors;

[0269] Input each first probability value into the output module to obtain the target initial probability parameters of the target training sample pair output by the output module.

[0270] Optionally, the pre-trained task model includes an encoding module, an extraction module, and an output module;

[0271] Accordingly, module 806 is further configured as follows:

[0272] Obtain the downstream task module corresponding to the preset downstream task;

[0273] Replace the output module with the downstream task module to obtain the task model to be trained.

[0274] Optionally, the task model to be trained includes an encoding module, an extraction module, and a downstream training module;

[0275] Accordingly, the second training module 808 is further configured as follows:

[0276] Input the target second training sample and the target downstream task label into the task model to be trained to obtain the downstream task prediction result output by the task model to be trained. The target second training sample is any one of at least one second training sample, and the target downstream task label is the task label corresponding to the preset downstream task.

[0277] The loss value of the second model is calculated based on the target sample label corresponding to the second training sample and the prediction results of the downstream task.

[0278] Adjust the training task model based on the second model loss value, and continue training the training task model until the model training stops, thus obtaining the target task model.

[0279] Optionally, the preset downstream tasks include at least one of word-level tasks and sequence-level tasks.

[0280] One embodiment of this specification implements the acquisition of first training data, wherein the first training data includes at least two training sample pairs, each training sample pair includes a first training text and a first training sample corresponding to the first training text, the first training sample includes positive samples and negative samples, and the training sample pairs include multilingual training sample pairs; an initial task model is trained based on at least two training sample pairs to obtain a pre-trained task model; a task model to be trained is constructed according to the pre-trained task model and a preset downstream task; the task model to be trained is trained according to the second training data corresponding to the preset downstream task to obtain a target task model, wherein the second training data includes at least one second training sample and sample labels corresponding to each second training sample.

[0281] In this way, by training the initial task model with the first training data, the initial task model learns a strong cross-language understanding ability from positive and negative samples, which can achieve better task performance when processing cross-language understanding tasks in the future. By setting up downstream tasks, the pre-trained task model is constructed into the task model to be trained, and the task model to be trained is trained according to the second training data. This allows the cross-language understanding ability learned by the pre-trained task model to be directly inherited in the training process of the downstream task, improving the convergence efficiency of the task model to be trained, thereby improving the training efficiency of the target task model and the accuracy of the target task model's cross-language understanding task performance results.

[0282] The above is an illustrative scheme of a task model training device according to this embodiment. It should be noted that the technical solution of this task model training device and the technical solution of the task model training method described above belong to the same concept. For details not described in detail in the technical solution of the task model training device, please refer to the description of the technical solution of the task model training method described above.

[0283] Figure 9 A structural block diagram of a computing device 900 according to one embodiment of this specification is shown. The components of the computing device 900 include, but are not limited to, a memory 910 and a processor 920. The processor 920 is connected to the memory 910 via a bus 930, and a database 950 is used to store data.

[0284] The computing device 900 also includes an access device 940, which enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 940 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0285] In one embodiment of this specification, the above-described components of the computing device 900 and Figure 9 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 9 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0286] The computing device 900 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 900 can also be a mobile or stationary server.

[0287] The processor 920 is used to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described task model training method.

[0288] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the task model training method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the task model training method described above.

[0289] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the task model training method described above.

[0290] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solution of the task model training method described above. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the task model training method described above.

[0291] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the task model training method described above.

[0292] The above is an illustrative example of a computer program according to this embodiment. It should be noted that the technical solution of this computer program and the technical solution of the task model training method described above belong to the same concept. Details not described in detail in the computer program's technical solution can be found in the description of the technical solution of the task model training method described above.

[0293] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0294] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0295] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0296] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0297] It should be noted that the embodiments of this application may involve the use of user data. In practical applications, user-specific personal data may be used in the scheme described herein within the scope permitted by applicable laws and regulations, provided that it complies with the applicable laws and regulations of the country (e.g., with the user's explicit consent, with the user being properly notified, etc.).

[0298] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A method for training a task model, comprising: The method involves obtaining first training data, which includes at least two training sample pairs. Each training sample pair includes a first training text and a first training sample corresponding to the first training text. The first training sample includes positive samples and negative samples. The training sample pairs include multilingual training sample pairs. The method of obtaining the training sample pairs includes: determining a corresponding definition text and at least one corresponding reference text in a preset text database based on the target sample entity; extracting the first training text from the definition text; extracting each context text corresponding to the target sample entity from each reference text; the context text including a first training positive sample that has a related relationship with the target sample entity; and obtaining a reference training positive sample corresponding to the reference sample entity as a first training negative sample of the target sample entity. Based on the at least two training samples, train the initial task model to obtain the pre-trained task model; Based on the pre-trained task model and the preset downstream tasks, construct the task model to be trained; The target task model is obtained by training the task model based on the second training data corresponding to the preset downstream task, wherein the second training data includes at least one second training sample and sample labels corresponding to each second training sample.

2. The method according to claim 1, wherein obtaining the first training data includes: Retrieve the preset text database; At least two sample entities are determined based on the preset text database, wherein the sample entities include multilingual sample entities; Based on the target sample entity, obtain the target first training text and the target first training sample corresponding to the target sample entity, wherein the target sample entity is any one of the at least two sample entities; Based on the target first training text and the target first training sample, construct the target training sample pair corresponding to the target sample entity, and add the target training sample pair to the first training data.

3. The method according to claim 2, wherein obtaining the target first training text and the target first training sample corresponding to the target sample entity based on the target sample entity includes: Based on the first text extraction rule, extract the target first training text from the defined text; Based on the second text extraction rule, the first training positive sample of the target sample entity is extracted from each reference text.

4. The method according to claim 1, wherein obtaining a pre-trained task model based on the at least two training samples for training the initial task model comprises: Input the target training sample pair into the initial task model to obtain the target initial probability parameter corresponding to the target training sample pair output by the initial task model, wherein the target training sample pair is any one of the at least two training samples; The first model loss value is calculated based on the first target label corresponding to the target training sample pair and the target initial probability parameter, wherein the first target label is determined based on the first training sample in the target training sample pair; The initial task model is adjusted based on the first model loss value, and the initial task model is trained until the model training stops, thus obtaining a pre-trained task model.

5. The method according to claim 4, wherein the initial task model includes an encoding module and an extraction module; Accordingly, the step of inputting the target training sample pair into the initial task model to obtain the target initial probability parameters corresponding to the target training sample pair output by the initial task model includes: By concatenating the target first training text and the target first training sample in the target training sample pair, the target input text is obtained; The target input text is input into the encoding module to obtain the keyword vector set corresponding to the target input text; Each keyword vector in the keyword vector set is input into the extraction module to obtain the target initial probability parameters of the target training sample pair.

6. The method according to claim 5, wherein inputting the target input text into the encoding module to obtain the keyword vector set corresponding to the target input text includes: The target input text is segmented into words to obtain a set of keywords corresponding to the target input text; Input the keyword set into the encoding module; Obtain the keyword vector set corresponding to the keyword set output by the encoding module.

7. The method according to claim 5, wherein the initial task model further includes an output module; Accordingly, the step of inputting each keyword vector in the keyword vector set into the extraction module to obtain the target initial probability parameters of the target training sample pair includes: Input the keyword vectors into the extraction module; The extraction module calculates a first probability value corresponding to the first keyword vector and the second keyword vector based on the first keyword vector and the second keyword vector, wherein the first keyword vector and the second keyword vector are any two of the keyword vectors; Each first probability value is input into the output module to obtain the target initial probability parameters of the target training sample pair output by the output module.

8. The method according to claim 1, wherein the pre-trained task model comprises an encoding module, an extraction module, and an output module; Accordingly, the step of constructing the training task model based on the pre-trained task model and the preset downstream task includes: Obtain the downstream task module corresponding to the preset downstream task; The output module is replaced with the downstream task module to obtain the task model to be trained.

9. The method according to claim 1, wherein the task model to be trained includes an encoding module, an extraction module, and a downstream training module; Accordingly, the step of training the task model to be trained based on the second training data corresponding to the preset downstream task to obtain the target task model includes: The target second training sample and the target downstream task label are input into the task model to be trained to obtain the downstream task prediction result output by the task model to be trained. The target second training sample is any one of the at least one second training sample, and the target downstream task label is the task label corresponding to the preset downstream task. The second model loss value is calculated based on the target sample label corresponding to the second training sample and the downstream task prediction result; The training task model is adjusted according to the second model loss value, and the training task model continues until the model training stopping condition is met to obtain the target task model.

10. A method for training an extractive question-answering task model, comprising: The method involves obtaining first training data, which includes at least two training sample pairs. Each training sample pair includes a first training text and a first training sample corresponding to the first training text. The first training sample includes positive samples and negative samples. The training sample pairs include multilingual training sample pairs. The method of obtaining the training sample pairs includes: determining a corresponding definition text and at least one corresponding reference text in a preset text database based on the target sample entity; extracting the first training text from the definition text; extracting each context text corresponding to the target sample entity from each reference text; the context text including a first training positive sample that has a related relationship with the target sample entity; and obtaining a reference training positive sample corresponding to the reference sample entity as a first training negative sample of the target sample entity. Based on the at least two training samples, train the initial task model to obtain the pre-trained task model; Based on the pre-trained task model and the extractive question answering task, construct the extractive question answering task model to be trained; Based on the second training data corresponding to the extractive question answering task, the extractive question answering task model to be trained is trained to obtain the target extractive question answering task model, wherein the second training data includes at least one question sample and the answer label corresponding to each question sample.

11. A data processing method for a task model, comprising: Receive a preset downstream task processing request sent by the user through the front end, wherein the preset downstream task processing request carries data to be processed; The data to be processed is input into the target task model to obtain the task processing result corresponding to the preset downstream task processing request. The target task model is trained on the task model to be trained based on the second training data corresponding to the preset downstream task. The task model to be trained is constructed based on the pre-trained task model and the preset downstream task. The pre-trained task model is trained on the initial task model based on at least two training samples. The second training data includes at least one second training sample and sample labels corresponding to each second training sample. Each training sample pair includes a first training text and a first training sample corresponding to the first training text. The first training sample includes positive samples and negative samples. The training sample pair includes multilingual training sample pairs. The method of obtaining the training sample pair includes: determining a corresponding definition text and at least one corresponding reference text in a preset text database based on the target sample entity; extracting the first training text from the definition text; extracting each context text corresponding to the target sample entity from each reference text; the context text includes a first training positive sample that has a related relationship with the target sample entity; and obtaining a reference training positive sample corresponding to the reference sample entity as the first training negative sample of the target sample entity. The task processing result is sent to the front end so that the front end can display the task processing result.

12. A task model training method, applied to cloud-side devices, comprising: The method involves obtaining first training data, which includes at least two training sample pairs. Each training sample pair includes a first training text and a first training sample corresponding to the first training text. The first training sample includes positive samples and negative samples. The training sample pairs include multilingual training sample pairs. The method of obtaining the training sample pairs includes: determining a corresponding definition text and at least one corresponding reference text in a preset text database based on the target sample entity; extracting the first training text from the definition text; extracting each context text corresponding to the target sample entity from each reference text; the context text including a first training positive sample that has a related relationship with the target sample entity; and obtaining a reference training positive sample corresponding to the reference sample entity as a first training negative sample of the target sample entity. Based on the at least two training samples, train the initial task model to obtain the pre-trained task model; Based on the pre-trained task model and the preset downstream tasks, construct the task model to be trained; The model of the task to be trained is trained based on the second training data corresponding to the preset downstream task to obtain the model parameters of the target task model. The second training data includes at least one second training sample and sample labels corresponding to each second training sample. The model parameters of the target task model are sent to the end-side device.

13. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 12.

14. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 12.

Images