Natural language processing, model training method and device, equipment and storage medium

Abstract

The present disclosure relates to a natural language processing method and device, a model training method and device, and a storage medium. The present disclosure pre-trains a machine learning model through each triple, so that the pre-trained machine learning model can seamlessly and naturally process various natural language understanding tasks under the machine reading comprehension paradigm. In addition, because the data format used for model training in the pre-training stage is consistent with the data format used for model training in the fine-tuning stage, the pre-training target and the fine-tuning target are the same, thereby enabling seamless connection between the pre-training stage and the fine-tuning stage. After pre-training the model using a large amount of low-cost data, the pre-trained machine learning model can be calibrated using a small amount of target task data, thereby enabling the general knowledge learned in the pre-training stage to be successfully transferred to the fine-tuned model, and ensuring the accuracy of the fine-tuned model.

Description

Technical Field

[0001] This disclosure relates to the field of information technology, and in particular to a natural language processing, model training method, apparatus, device, and storage medium. Background Technology

[0002] While natural language understanding tasks can be performed by machines, they require machine learning models capable of processing natural language. Because target task data for natural language understanding is limited, machine learning models are typically pre-trained before being trained using a small amount of target task data.

[0003] However, the goals of current pre-training differ from those of natural language understanding tasks. For example, current pre-training aims to enable machine learning models to recover corrupted text, while the goal of natural language understanding tasks is to solve specific problems, such as named entity recognition, extractive question answering, sentiment analysis, and multiple-choice question answering. Therefore, using current pre-training methods to pre-train this machine learning model will result in a model that is unusable for natural language understanding tasks. Furthermore, it will be difficult to calibrate the pre-trained model using limited target task data, leading to a still inaccurate model even after fine-tuning. Summary of the Invention

[0004] To address, or at least partially address, the aforementioned technical problems, this disclosure provides a natural language processing method, model training method, apparatus, device, and storage medium to improve the accuracy of fine-tuned machine learning models.

[0005] In a first aspect, embodiments of this disclosure provide a natural language processing method, including:

[0006] Retrieve the first target information marked by the hyperlink;

[0007] Obtain the first query corresponding to the first target information from the homepage articles of the first target information, and obtain at least one first context information of the first target information from at least one cited article of the first target information;

[0008] The machine learning model is pre-trained based on the first target information, the first query, and the at least one first context information to obtain the pre-trained machine learning model.

[0009] The second target information is determined from the sample text provided by the natural language understanding task, a second query corresponding to the second target information is generated, and the pre-trained machine learning model is trained using the sample text, the second query, and the second target information.

[0010] Secondly, embodiments of this disclosure provide a model training method, including:

[0011] Get the target text;

[0012] Based on the natural language understanding task corresponding to the target text, determine the query information;

[0013] The query information and the target text are used as input to a machine learning model, which outputs the answer in the target text that corresponds to the query information. The machine learning model is trained according to the model training method described above.

[0014] Thirdly, embodiments of this disclosure provide a model training apparatus, comprising:

[0015] The first acquisition module is used to acquire the first target information marked by the hyperlink;

[0016] The second acquisition module is used to acquire a first query corresponding to the first target information from the homepage articles of the first target information, and to acquire at least one first context information of the first target information from at least one cited article of the first target information;

[0017] The pre-training module is used to pre-train the machine learning model based on the first target information, the first query, and the at least one first context information to obtain the pre-trained machine learning model.

[0018] The fine-tuning module is used to determine the second target information from the sample text provided by the natural language understanding task, generate the second query corresponding to the second target information, and train the pre-trained machine learning model using the sample text, the second query, and the second target information.

[0019] Fourthly, embodiments of this disclosure provide a natural language processing apparatus, comprising:

[0020] The acquisition module is used to acquire the target text;

[0021] The determination module is used to determine query information based on the natural language understanding task corresponding to the target text;

[0022] The input module is used to take the query information and the target text as input to the machine learning model, so that the machine learning model outputs the answer in the target text that corresponds to the query information. The machine learning model is trained according to the model training method described above.

[0023] Fifthly, embodiments of this disclosure provide an electronic device, including:

[0024] Memory;

[0025] Processor; and

[0026] Computer programs;

[0027] The computer program is stored in the memory and configured to be executed by the processor to implement the method as described in the first or second aspect.

[0028] In a sixth aspect, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the method described in the first or second aspect.

[0029] The natural language processing, model training method, apparatus, device, and storage medium provided in this disclosure obtain first target information marked by a hyperlink as the answer, obtain a first query corresponding to the first target information from the homepage article of the first target information, and obtain at least one first contextual information of the first target information from at least one cited article of the first target information, so that the first target information, the first query, and each first contextual information can constitute a triple in the machine reading comprehension style. Furthermore, a machine learning model is pre-trained based on each triple, enabling the pre-trained machine learning model to seamlessly and naturally handle various natural language understanding tasks within the machine reading comprehension paradigm. In addition, since the data format used for model training in the pre-training stage is consistent with the data format used for model training in the fine-tuning stage—both are triples including the answer, query, and contextual information—the goals of pre-training and fine-tuning are the same, thus allowing for a seamless transition between the pre-training and fine-tuning stages. Since the pre-training process and the fine-tuning process are very similar, after pre-training the machine learning model with a large amount of low-cost data, the pre-trained machine learning model can be calibrated with a small amount of target task data. This allows the general knowledge learned in the pre-training stage to be successfully transferred to the fine-tuned machine learning model, and ensures the accuracy of the fine-tuned machine learning model. Attached Figure Description

[0030] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0031] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 A schematic diagram illustrating the differences between MLM, S2S, and MRC in the pre-training and fine-tuning stages provided in embodiments of this disclosure;

[0033] Figure 2 A flowchart of the model training method provided in this embodiment of the disclosure;

[0034] Figure 3 A schematic diagram illustrating an application scenario provided by an embodiment of this disclosure;

[0035] Figure 4 A schematic diagram illustrating the homepage articles and cited articles provided in this embodiment of the disclosure;

[0036] Figure 5 A schematic diagram of a PMR provided for another embodiment of this disclosure;

[0037] Figure 6 A flowchart of a model training method provided in another embodiment of this disclosure;

[0038] Figure 7 A schematic diagram of a probability matrix provided for another embodiment of this disclosure;

[0039] Figure 8 A schematic diagram of a probability matrix provided for another embodiment of this disclosure;

[0040] Figure 9 A schematic diagram of a probability matrix provided for another embodiment of this disclosure;

[0041] Figure 10 A flowchart of a model training method provided in another embodiment of this disclosure;

[0042] Figure 11 This is a schematic diagram of the structure of the model training device provided in the embodiments of this disclosure;

[0043] Figure 12 This is a schematic diagram of the structure of a natural language processing device provided in an embodiment of the present disclosure;

[0044] Figure 13 A schematic diagram of the structure of an electronic device embodiment provided in this disclosure. Detailed Implementation

[0045] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0046] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0047] Typically, due to the limited amount of data available for the target task, directly training a machine learning model with such limited data will result in poor performance and insufficient accuracy. Therefore, pre-training is usually performed before training the model with a small amount of target task data. For example, a pre-training method can be used to pre-train the model on a large amount of low-cost data, resulting in pre-trained models. These pre-trained models learn the commonalities found in the large amount of low-cost data and acquire general knowledge. Furthermore, the pre-trained models are fine-tuned using a small amount of target task data, transferring this general knowledge to the fine-tuned model. This allows the fine-tuned model to handle the target task effectively, such as natural language understanding. Natural Language Understanding (NLU) is a collective term for the ideas, methods, and tasks that support machines in understanding text data.

[0048] However, the goals of current pre-training differ from those of natural language understanding (NLE) tasks. For example, current pre-training aims to enable machine learning models to recover corrupted text, while NLE tasks aim to solve specific problems, such as named entity recognition, extractive question answering, sentiment analysis, and multiple-choice question answering. Therefore, using current pre-training methods to pre-train the machine learning model will result in a model unsuitable for NLE tasks, and it will be difficult to calibrate the pre-trained model with limited target task data, leading to insufficient accuracy even after fine-tuning. While adjustments can be made to the pre-trained model to enable it to handle NLE tasks, these adjustments alter the training objectives and data formats, creating significant discrepancies between pre-training and fine-tuning, thus hindering the transfer of general knowledge to downstream tasks. Furthermore, limited target task data is insufficient to fine-tune the pre-trained model and eliminate the differences between pre-training and fine-tuning.

[0049] For example, pre-training based on the masked language paradigm (MLM-style pre-training) involves two steps: Step (1): Automatically replacing parts of the input text with special characters such as [MASK], and then feeding the input text containing these special characters into the encoder. Step (2): Recovering the original text based on the contextual representation of the replaced text. Masked Language Model (MLM) is a model paradigm in natural language processing. Specifically, machine learning models need to recover some contaminated words from the input. Since machine learning models can see the entire sentence, they can recover the words based on the context of the contaminated words. Figure 1 As shown, the input text is "XXX invented silicon technology," where "invented" is replaced with [MASK]. The encoder can be a Bidirectional Encoder Representation from Transformer (BERT) or a Robustly Optimized BERT (RoBERTa). The language model output layer (LM Head) recovers "invented" based on the contextual text representation of the replaced text. This pre-training scheme can be pre-trained on large-scale text corpora. Furthermore, when fine-tuning the pre-trained machine learning model, an additional randomly initialized task-related module is needed to achieve the downstream task classification objective. For example, for the Named Entity Recognition (NER) task, a multi-classifier needs to be added to each word vector output by the pre-trained machine learning model. Figure 1The named entity recognition output layer (NER Layer) shown is used to determine which entity category the word belongs to (or does not belong to an entity). This multi-classifier is randomly initialized and task-related, so it can only be fine-tuned using the data of this named entity recognition. If the data of this named entity recognition is scarce, it will be difficult to obtain a good fine-tuning effect, that is, it is difficult to calibrate the pre-trained machine learning model through a small amount of target task data (such as the data of this named entity recognition), resulting in the still inaccurate and easily over-fitting fine-tuned machine learning model. Among them, the fine-tuned machine learning model refers to the machine learning model obtained by calibrating the pre-trained machine learning model through a small amount of target task data, that is, re-training or fine-tuning. For another example, for the extractive question answering (EQA) task, an extractive question answering output layer (EQA Layer) needs to be added. Among them, named entity recognition (NER) refers to identifying entities with specific meanings in text, mainly including person names, place names, organization names, proper nouns, etc. For example, from Figure 1 the "Someone will fly to City A" shown, the person name "Someone" is recognized. Extractive question answering (EQA) refers to extracting the corresponding answer from the relevant text according to the given question. For example Figure 1 the question shown is "Who is Zhang's father?", the relevant text is "Zhang's father is Lao Zhang", the answer is "Lao Zhang" and the position information of "Lao Zhang" in the relevant text. For example Figure 1 the 14 and 15 shown indicate that "Lao Zhang" is located at the 14th and 15th characters. Among them, each character in the question and each character in the relevant text are sorted uniformly.

[0050] For another example, for the sequence-to-sequence (S2S)-style pre-training, its pre-training process is divided into two steps. Step (1): Pollute some text segments in the input text, for example, replace the text segments with some special characters, such as [X], [Y], and then send them into the encoder. Step (2): Based on the text representation output by the encoder, the text segments polluted by special characters are respectively restored through the decoder. Sequence-to-sequence (S2S) is a type of natural language processing model paradigm. Given text input, the machine learning model needs to output text related to the corresponding generation task. For example Figure 1As shown, the text segment replaced by the special character [X] is "Invented Silicon". T5 includes an encoder and a decoder. This method can convert various types of downstream natural language understanding tasks into text generation tasks, thus eliminating the need to add task-related modules. However, because the input text of this scheme is contaminated, the output text is a recovered text segment, which still differs in data form from the natural language input and output of the actual downstream task. For example, when the downstream task is a named entity recognition task, the natural language input is... Figure 1 The given text, "[spot]person[spot]location[text]So-and-so will fly to City A," outputs "Person: So-and-so, Location: City A." When the downstream task is an extractive question-answering task, the natural language input is... Figure 1 The example given, "Who is Xiao Zhang's father? Xiao Zhang's father is Lao Zhang," outputs "Lao Zhang" in natural language. This demonstrates that for the sequence-sequence paradigm, there is a significant difference between the input / output during the pre-training and fine-tuning stages. This makes it difficult to achieve good fine-tuning results in low-resource scenarios, i.e., when there is limited target task data. In other words, it is difficult to calibrate the pre-trained machine learning model with a small amount of target task data, resulting in an inaccurate fine-tuned machine learning model.

[0051] To address the aforementioned problems, this disclosure provides a model training method, which includes employing... Figure 1 The Machine Reading Comprehension (MRC) paradigm shown in the diagram uses MRC-style pre-training to pre-train the machine learning model. This method also includes fine-tuning the pre-trained model. Specifically, the MRC paradigm is a type of natural language processing model whose input consists of a query and relevant context text, and whose output is the position of some answers within the context text that satisfy the input query. For example, for... Figure 1 The MRC paradigm shown has a query that reads "It is a chemical component of semiconductors, with the chemical symbol Si," and the context text that reads "So-and-so invented silicon technology." The answer is the position of "silicon" in the context text, "24, 24," indicating that "silicon" is at the 24th character position. Each character in the query and each character in the context text are ordered uniformly. Furthermore, for the machine reading comprehension paradigm provided in this embodiment, the input and output data formats of the machine learning model during the fine-tuning stage and the pre-training stage are the same. For example, during the pre-training stage, the input of the machine learning model is the query and the context text, and the output is the answer. During the fine-tuning stage, if the target task is a NER task, the input of the machine learning model is also the query and the context text, where the query is as follows: Figure 1 The text shown is "Character?", and the context is... Figure 1 The given query, "So-and-so will fly to city A," outputs the machine learning model as the answer, which is the position of "so-and-so" in the context text: "2, 3." If the target task is an EQA task, the input to the machine learning model is also the query and the context text, where the query is as follows: Figure 1 The question "Who is Xiao Zhang's father?" is shown in the context text. Figure 1 The machine learning model outputs the answer "Xiao Zhang's father is Lao Zhang" as the position of "Lao Zhang" in the context text: "14, 15".

[0052] The method will be described below with reference to specific embodiments. Figure 2 This is a flowchart illustrating a model training method provided in an embodiment of this disclosure. The method can be executed by a model training device, which can be implemented in software and / or hardware. This device can be configured in an electronic device, such as a server or terminal, where the terminal specifically includes a mobile phone, computer, or tablet computer. Furthermore, the model training method described in this embodiment is applicable to... Figure 3 The application scenarios shown are as follows. Figure 3 As shown, this application scenario includes a terminal 31 and a server 32. The server 32 can pre-train and fine-tune the machine learning model using the model training method described in this embodiment. Furthermore, the server 32 can provide services to the terminal 31 based on the fine-tuned machine learning model. For example, the terminal 31 can send a query and context text to the server 32, which can then input the query and context text into the fine-tuned machine learning model, allowing the model to output an answer. The server 32 can then return the answer to the terminal 31. Alternatively, the terminal 31 can send context text to the server 32, which generates a query based on the specific requirements of the natural language understanding task and inputs the query and context text into the fine-tuned machine learning model, enabling the model to output an answer. Or, the server 32 can deploy the fine-tuned machine learning model on the terminal 31, allowing the terminal 31 to perform natural language understanding tasks using this model. The following section combines... Figure 3 This method will be described in detail, such as Figure 2 As shown, the specific steps of this method are as follows:

[0053] S201. Obtain the first target information marked by the hyperlink.

[0054] For example, to construct data in a machine reading comprehension style, the anchor text of a hyperlink can be used as the answer for machine reading comprehension. Furthermore, the anchor text of a hyperlink can be considered the primary target information. Specifically, this hyperlink can be a hyperlink within a webpage. This webpage can be a page from Wikipedia.

[0055] S202. Obtain the first query corresponding to the first target information from the homepage articles of the first target information, and obtain at least one first context information of the first target information from at least one cited article of the first target information.

[0056] Specifically, an anchor can link to two types of articles: homepage articles and cited articles. Homepage articles provide a detailed explanation of the anchor. For example... Figure 4 As shown, "silicon" is the anchor text marked with a hyperlink, and the homepage article provides a detailed explanation of silicon. Additionally, an anchor text may link to one or more reference articles. For example... Figure 4 The "silicon" link shown contains two cited articles: Article 41 and Article 42. Article 41 introduces semiconductors, and Article 42 introduces integrated circuits. These articles can be Wikipedia entries, and both mention "silicon," indicated by hyperlinks. In this embodiment, the query corresponding to the anchor can be obtained from the homepage articles; this query can be designated as the first query. Additionally, the context information corresponding to the anchor can be obtained from each cited article; this context information is designated as the first context information. At least one piece of context information can be obtained from each cited article.

[0057] Optionally, obtaining the first query corresponding to the first target information from the homepage article of the first target information includes: taking at least one sentence from the homepage article of the first target information as the first query corresponding to the first target information.

[0058] For example, in this embodiment, the first T sentences of the homepage article can be used as the query corresponding to the anchor, where T is greater than or equal to 1. For example Figure 4 As shown, the first two sentences of the homepage article are used as the query corresponding to this anchor.

[0059] Optionally, obtaining at least one first context information of the first target information from at least one cited article of the first target information includes: for each cited article of the first target information, determining a sentence that includes the first target information from the cited article; and taking at least one sentence before and after the sentence in the cited article and the sentence together as a first context information.

[0060] For example, targeting Figure 4 In each of the two cited articles shown, the sentence containing the anchor "silicon" is first identified. It's understandable that there may be more than one sentence containing "silicon" in that cited article; for example... Figure 4 In the cited article 41 shown, there are two sentences that contain the word "silicon". Taking one of the sentences containing "silicon" as an example, the W sentences before and after that sentence, along with the sentence itself (i.e., 2W+1 sentences), are considered as contextual information for this anchor, where W is greater than or equal to 1. In other words, for a cited article, if N sentences contain this anchor, then N pieces of contextual information can be obtained from the cited article. For example... Figure 4 As shown, assuming we extract one piece of contextual information from each cited article, specifically, we extract contextual information 1 from cited article 41, which corresponds to answer 1; and we extract contextual information 2 from cited article 41, which corresponds to answer 2. Specifically, (query, contextual information 1, answer 1) can form a triple, and (query, contextual information 2, answer 2) can form another triple. Through this data construction method, we can construct hundreds of millions of positive examples of machine reading comprehension style data. Furthermore, we can match unrelated queries and contextual information to construct negative examples of machine reading comprehension style data. For example, negative examples are also in the form of triples, which also include the query, contextual information, and answer as described above. However, the query and contextual information do not match; that is, the contextual information does not contain the answer corresponding to the query. Accordingly, the answer in this triple can be directly assigned a value such as empty or 0, etc.

[0061] S203. The machine learning model is pre-trained based on the first target information, the first query, and the at least one first context information to obtain the pre-trained machine learning model.

[0062] like Figure 5As shown, this embodiment provides a unified pre-trained machine reader (PMR), which can be a machine learning model. Further, the pre-trained reader is pre-trained using the positive and negative example data described above to obtain the pre-trained PMR, which can be referred to as the pre-trained machine learning model. For example, taking a triple from the positive example data as an example, the query and context information in the triple are used as inputs to the PMR, causing the PMR to output an answer based on the query and context information. Further, the PMR is trained based on the output answer and the standard answer in the triple, i.e., the parameters of the PMR are iteratively updated. It can be understood that the parameters of the PMR can be iteratively updated once for each triple. When the number of iterations reaches a preset number, or when the parameters of the PMR tend to stabilize, it can be determined that the pre-training of the PMR is complete.

[0063] S204. Determine the second target information from the sample text provided by the natural language understanding task, generate the second query corresponding to the second target information, and train the pre-trained machine learning model using the sample text, the second query, and the second target information.

[0064] In this embodiment, natural language understanding tasks can be categorized into several types. For each type of task, a sample text is typically provided. Further, an answer is determined from this sample text, and this answer serves as the second target information. A corresponding query can be generated based on this answer, and this query can be denoted as the second query. Thus, (sample text, second target information, second query) is treated as a triple. That is, the sample text can be considered as the context information described above, and the triple can be the target task data described above. This triple can be used to train the pre-trained PMR, i.e., for fine-tuning, correction, or calibration. Similarly, the second query and the sample text are used as inputs to the pre-trained PMR, causing the pre-trained PMR to output an answer. Further, the pre-trained PMR is fine-tuned based on the answer output by the pre-trained PMR and the second target information.

[0065] Specifically, if a natural language understanding task is a word-level extraction task with fixed task labels, such as named entity recognition (NER), then a query can be generated based on each task label. Each type of entity can correspond to a task label; entity types can be categorized as place names, people, organization names, proper nouns, etc. Assuming the sample text provided by the NER task is "So-and-so will fly to city A," various types of entities are pre-labeled from this sample text. For example, "So-and-so" is a person type entity, and "city A" is a place name type entity. Person and place name types can correspond to different task labels. For the task label corresponding to the person type, the query "Please find the entities related to the person in the following text" is generated. Further, this sample text and the query are used as input to a pre-trained PMR, allowing the pre-trained PMR to retrieve the answer corresponding to the query from the sample text. Assuming the pre-trained PMR outputs an answer, it is further fine-tuned based on the pre-trained PMR's output answer and the standard answer, "So-and-so." For another example, for the task label corresponding to the place name type, the query "Please find the entities related to the place name in the following text" is generated. Furthermore, the sample text and the query are used as input to the pre-trained PMR, so that the pre-trained PMR can retrieve the answer corresponding to the query from the sample text. Assuming that the pre-trained PMR outputs an answer, the pre-trained PMR is further fine-tuned based on the answer output by the pre-trained PMR and the standard answer, namely "City A".

[0066] If the natural language understanding task is a word-level extraction task of natural language questions, such as an extractive question answering task, then at least one question is generated based on the sample text. For example, if the sample text is "Xiaoming was born in country B," then the questions could be "Who appears in country B" or "Where was Xiaoming born?" For "Who appears in country B," this query and the sample text are used as input to the pre-trained PMR, allowing the pre-trained PMR to retrieve the answer corresponding to the query in the sample text. Furthermore, the pre-trained PMR is fine-tuned based on the answer output by the pre-trained PMR and the standard answer, which is "Xiaoming."

[0067] If a natural language understanding task is a sequence-level classification task with fixed task labels, such as sentiment analysis (SMA), where a given text is used to determine its sentiment polarity, then each sentiment can correspond to a task label. For example, "positive sentiment" and "negative sentiment" correspond to different task labels. A query is generated based on each task label. For instance, the query generated for "positive sentiment" might be "The following text represents positive sentiment." Assuming the sample text is "Xiaoming is very happy today," this query and the sample text are used as input to a pre-trained PMR (Persistent Sentiment Analysis) system. The pre-trained PMR then determines whether the query and the sample text are relevant. If they are relevant, the sentiment analysis result is "positive sentiment." Conversely, the query generated for "negative sentiment" might be "The following text represents negative sentiment." This query and the sample text are used as input to a pre-trained PMR system. The pre-trained PMR then determines whether the query and the sample text are relevant. If they are not relevant, "negative sentiment" cannot be used as the sentiment analysis result.

[0068] If the natural language understanding task is a sequence-level classification task based on multiple options for a natural language question, such as the multi-choice question answering (MCQA) task, then the task involves selecting the correct option from multiple choices given a question and relevant reference information. For example, the relevant reference information might be a given passage in reading comprehension, the question might be about that passage, and the multiple options might be the corresponding choices for that question. In this case, the question and one option can be treated as a query, and the query and the relevant reference information can be used as input to a pre-trained PMR (Problem-Based Understanding). The pre-trained PMR then determines whether the query and the relevant reference information are relevant. If they are relevant, the correct answer to the question is the option in the query; if they are irrelevant, the correct answer is not the option in the query, and further evaluation is needed.

[0069] This embodiment of the disclosure obtains the first target information marked by the hyperlink as the answer, obtains the first query corresponding to the first target information from the homepage article of the first target information, and obtains at least one first context information of the first target information from at least one cited article of the first target information, so that the first target information, the first query, and each first context information can constitute a triple in the machine reading comprehension style. Furthermore, the machine learning model is pre-trained based on each triple, enabling the pre-trained machine learning model to seamlessly and naturally handle various natural language understanding tasks within the machine reading comprehension paradigm. In addition, since the data format used for model training in the pre-training stage is the same as that used in the fine-tuning stage—both are triples including the answer, query, and context information—the goals of pre-training and fine-tuning are the same, thus allowing for a seamless transition between the pre-training and fine-tuning stages. Since the pre-training process and the fine-tuning process are very similar, after pre-training the machine learning model with a large amount of low-cost data, the pre-trained machine learning model can be calibrated with a small amount of target task data. This allows the general knowledge learned in the pre-training stage to be successfully transferred to the fine-tuned machine learning model, and ensures the accuracy of the fine-tuned machine learning model.

[0070] Furthermore, this embodiment can construct a large amount of high-quality machine reading comprehension format data, such as triples, enabling end-to-end pre-training of machine learning models. This eliminates the need for manual annotation, allowing for the construction of massive datasets and significantly reducing labor costs, thereby lowering pre-training costs and improving the accuracy of the pre-trained machine learning model. Additionally, since the pre-training provided in this embodiment is based on a machine reading comprehension paradigm, the pre-trained machine learning model can be applied to various languages, even multilingual learning. Therefore, this solution has wide applicability, strong versatility, and strong interpretability in sequence-level tasks. Moreover, this embodiment unifies the pre-training and fine-tuning stages entirely within the machine reading comprehension paradigm, eliminating differences in training objectives and data formats between pre-training and fine-tuning. This allows the general knowledge learned in the pre-training stage to be easily transferred to the fine-tuned machine learning model, improving transferability and resulting in a significant performance improvement for the machine learning model after pre-training and fine-tuning when handling natural language understanding tasks.

[0071] Figure 6 This is a flowchart illustrating a model training method according to another embodiment of the present disclosure. In this embodiment, the machine learning model includes an encoder and a decimator, where the output of the encoder is the input of the decimator. For example, the machine learning model is as follows: Figure 5The PMR shown includes an encoder and a sludge extractor, with the encoder's output serving as the sludge extractor's input. Pre-training the machine learning model based on the first target information, the first query, and at least one first contextual information to obtain a pre-trained machine learning model includes the following steps:

[0072] S601. For each of the at least one first context information, the first query and the first context information are used as inputs to the encoder, so that the encoder outputs the representation vector of each text unit in the first query and the representation vector of each text unit in the first context information.

[0073] It's understandable that a single anchor tag for a hyperlink might correspond to multiple pieces of contextual information. The anchor, the query it corresponds to, and a piece of contextual information associated with the anchor can form a triple. Let's assume the query in a triple is... Figure 5 The query shown contains context information in the triple. Figure 5 The context information shown is as follows. It is understood that since the query and the context information are each at least one sentence, and each sentence includes at least one text unit, which can be, for example, a word, character, phrase, short sentence, or character. Therefore, in this embodiment, the context information can be... Figure 5 The query shown is broken down into multiple text units, and at the same time... Figure 5 The context information shown is broken down into multiple text units. For example, the query can be broken down into Q text units, and the context information can be broken down into C text units. Each text unit can be represented as a token. Additionally, a special word such as [CLS] can be added before the query, a special word such as [SEP] can be added between the query and the context information, and a special word such as [SEP] can also be added after the context information. Furthermore, [CLS], the Q text units in the query, [SEP], the C text units in the context information, and [SEP] are used together as input to the encoder. Assume... Figure 5 The [CLS], Q text units in the query, [SEP], C text units in the context information, and [SEP] together constitute M input words. The encoder can represent these M input words in a vector space, allowing it to output the representation vector corresponding to each of the M input words. For example... Figure 5 H1, H2, H shown N-- H N H N+1 H M-1 H M It is the sequential representation vector of the M input words.

[0074] S602. The extractor calculates the probability that each text segment in the first context information is the answer to the first query, wherein each text segment is composed of at least one consecutive text unit in the first context information.

[0075] In this embodiment, the extractor can calculate a probability value given the representation vectors of any two input words. This probability value represents the probability that a text segment determined based on the two input words is the answer to the query. The text segment is at least one consecutive input word extracted from the M input words, starting with the first of the two input words and ending with the second of the two input words. For example, S 1,3 This represents the probability that the text segment consisting of the input words corresponding to H1, H2, and H3 is the answer to the query. Because... Figure 5 Given a total of M input words, we can obtain a total of M*M probabilities, thus obtaining... Figure 5 The probability matrix 51 is shown. It can be understood that since the C text units in the context information are included in the M input words, the probability matrix 51 includes the probability that a text segment consisting of any number of consecutive text units extracted from the C text units will be the answer to the query. In other words, the probability matrix 51 includes the probability that each text segment in the context information will be the answer to the query, and each text segment in the context information is composed of at least one consecutive text unit from that context information.

[0076] Optionally, calculating the probability that each text segment in the first context information is the answer to the first query through the extractor includes: calculating the probability that a text segment consisting of at least one consecutive text unit from the i-th text unit to the j-th text unit in the first context information is the answer to the first query through the extractor, where j is greater than or equal to i, and the probability is calculated based on the representation vector of the i-th text unit and the representation vector of the j-th text unit.

[0077] Assuming in Figure 5 and Figure 7In the encoder, [CLS], the Q text units in the query, [SEP], the C text units in the context information, and [SEP] are all ordered uniformly. That is, [CLS] is assumed to be the first text unit, and so on, with the last [SEP] being the Mth text unit. In other words, the index of [CLS] in the encoder's input is 1, the index of the first text unit in the query is 2, the index of the first text unit in the context information is N+1, and so on, with the index of the last [SEP] being M. For example... Figure 5 The probability matrix 51 shown can be represented in detail as follows: Figure 7 The probability matrix 71 shown includes probability matrix 72. Assuming i is greater than or equal to N+1, j is greater than or equal to i, and j is less than or equal to M-1, then any S in probability matrix 72... i,j S represents the probability that a text segment consisting of at least one consecutive text unit from the i-th text unit to the j-th text unit is the answer to the query. i,j It is calculated based on the representation vector of the i-th text unit and the representation vector of the j-th text unit.

[0078] S603. Based on the probability and the position information of the first target information in the first context information, the machine learning model is pre-trained to obtain the pre-trained machine learning model.

[0079] For example, the first target information is as follows Figure 4 Answer 2 is shown. Figure 5 The query shown is Figure 4 The query shown Figure 5 The context information shown is as follows: Figure 4 Context information 2 is shown. In constructing, as shown... Figure 4 When a triple is formed by the query, context information 2, and answer 2, the position of answer 2 within context information 2 can be predetermined. Therefore, based on the position of answer 2 within context information 2, the index corresponding to answer 2 in the encoder input can be determined. Furthermore, probability matrix 71 can be calculated using the method described above, and probability matrix 72 can be determined from probability matrix 71. It is understood that it is assumed that there exists a probability S with the largest probability value in probability matrix 72. i,j So, for the largest S i,jThis indicates that a text segment consisting of at least one consecutive text unit from the i-th text unit to the j-th text unit can be output as the answer by the PMR. However, during the pre-training phase, there may be a certain difference between the answer output by the PMR and the standard answer, such as answer 2. Therefore, it is necessary to pre-train the PMR based on this difference. In one possible implementation, a standard matrix can be generated based on the position information of answer 2 in context information 2. The size of the standard matrix is ​​the same as the size of the probability matrix 72. The standard matrix is, for example, as shown below. Figure 8 The probability matrix 81 is shown. Assuming answer 2 is the (M-1)th text unit in the encoder input, then S'M in probability matrix 81 can be... -1，M-1 Set the value to 1, and set all other probability values ​​in probability matrix 81 to 0. Further, pre-train the PMR based on the difference between probability matrices 72 and 81. For example, calculate the loss function value based on probability matrices 72 and 81, and then pre-train the PMR based on this loss function value.

[0080] Optionally, before pre-training the machine learning model based on the probability and the position information of the first target information in the first context information, the method further includes: calculating the relevance between the first query and the first context information through the extractor, wherein the relevance is calculated based on the overall representation vector of the first query and the first context information; and pre-training the machine learning model based on the probability and the position information of the first target information in the first context information, including: pre-training the machine learning model based on the relevance, the probability, and the position information of the first target information in the first context information.

[0081] For example Figure 5 As shown, although H1 corresponds to the special word [CLS], in this embodiment, H1 can be the representation vector of the query and context information as a whole in the encoder input. That is, the representation vector corresponding to the special word [CLS] can be the representation vector of the query and context information as a whole. Therefore, S calculated based on H1 1,1 This can represent the relevance of the query and contextual information. Therefore, according to S... 1,1 This can determine whether the query and the context information are relevant. For example, if S 1,1 If S = 1, then the query is related to the context information, meaning the answer to the query exists within the context information. If S 1,1 If the value is 0, it indicates that the query and the context information are irrelevant, meaning that the answer to the query does not exist in the context information. This is understandable when constructing... Figure 4When the query, context information 2, and answer 2 form a triple, since answer 2 is in context information 2, the query and context information 2 are related. In this case, S' can be used. 1，1 =1 indicates that the query is related to the context information 2. Further, by using the query and the context information 2 as input to the PMR, and obtaining probability matrix 71, the S in probability matrix 71 can be used as a basis for... 1,1 Probability matrix 71 contains probability matrix 72, S' 1，1 ,like Figure 8 The probability matrix 81 shown is used to construct the loss function, for example, the loss function includes S 1,1 and S' 1，1 The differences between the two, and the differences between probability matrices 72 and 81. Furthermore, the PMR is pre-trained based on the value of this loss function.

[0082] Alternatively, in one feasible implementation, for the triplet query, context information 2, and answer 2, a standard matrix of the same size as the probability matrix 71 can be constructed, for example... Figure 9 The probability matrix shown is 91. In this standard matrix, S' 1，1 Equal to 1, S' M-1，M-1 The value is equal to 1, and all other elements are 0. Further, a loss function is constructed based on the difference between the standard matrix and the probability matrix 71, and the PMR is pre-trained based on the loss function value.

[0083] It is understandable that for the same answer, the query can be fixed, while the context information may be multiple. Therefore, when the query and each piece of context information are used as input to the PMR, the PMR can be pre-trained once. As the context information is continuously changed, the PMR can be pre-trained multiple times. Furthermore, when the answer is changed, the corresponding query and context information will also change, allowing for multiple more pre-training sessions for the PMR. In other words, each triple can be used to pre-train the PMR once. Since this embodiment can construct a massive amount of machine reading comprehension style data, i.e., triples, the PMR can be sufficiently pre-trained. Additionally, it is understood that, as... Figure 4 The triplet consisting of the query, context information 2, and answer 2 shown is only a positive example. Similarly, the process of pre-training PMR based on negative example data is similar to the process of pre-training PMR based on this positive example data, which will not be elaborated here.

[0084] Furthermore, after pre-training the PMR, it can be fine-tuned using S204 as described above. During fine-tuning, the data used for model training is still triples, but these triples are specific to a particular natural language understanding task. Although the source of the triples used in fine-tuning differs from that used in pre-training, the data format is similar. Therefore, the principle of fine-tuning is consistent with that of pre-training; that is, fine-tuning can be achieved by using the difference between probability matrices 72 and 81 as described above to fine-tune the pre-trained PMR, or by using S204 as described above. 1,1 and S' 1，1 The differences between the two probabilities, as well as the differences between probability matrix 72 and probability matrix 81, are used to fine-tune the pre-trained PMR. Alternatively, the differences between probability matrix 71 and probability matrix 91, as described above, can be used to fine-tune the pre-trained PMR. The specific process will not be elaborated here.

[0085] This embodiment employs a unified extractor to handle various natural language understanding tasks. This extractor maintains the same training objective during both pre-training and fine-tuning, thus eliminating discrepancies between the training objectives of pre-training and fine-tuning. Since the pre-training process provided in this embodiment is based on discriminative objectives for machine reading comprehension, it significantly improves pre-training efficiency and reduces the hardware overhead required for pre-training compared to traditional generative pre-training objectives. Furthermore, after pre-training the machine learning model, this embodiment allows for accurate fine-tuning of the pre-trained machine learning model using a small amount of target task data for a specific natural language understanding task (e.g., triples used for fine-tuning) without any adjustments. This enables the fine-tuned machine learning model to accurately handle the natural language understanding task.

[0086] Furthermore, this embodiment transforms four types of natural language understanding tasks (e.g., named entity recognition, extractive question answering, sentiment analysis, and multiple-choice question answering) into machine reading comprehension paradigms, allowing for direct and seamless fine-tuning of pre-trained machine learning models, such as pre-trained PMR, on the triplets formed by these tasks. In addition, this embodiment provides a unified framework for solving downstream natural language understanding tasks, such as PMR. Therefore, only one machine learning model needs to be maintained to solve various tasks, resulting in high storage efficiency in real-world scenarios and strong transferability and versatility of the machine learning model.

[0087] The embodiments described above primarily illustrate how to pre-train and fine-tune a machine learning model, such as PMR, where both pre-training and fine-tuning belong to the training phase. After the training phase is completed, the fine-tuned PMR, i.e., the trained machine learning model, can be used to handle various natural language understanding tasks, thus entering the inference phase, i.e., the usage phase, of the machine learning model. It is understood that the training and inference phases can be executed on the same device or on separate devices. For example, both the training and inference phases can be performed on devices such as... Figure 3 The training phase is completed on server 32, as shown. Alternatively, the training phase is completed on server 32, and the trained machine learning model is then ported to other devices to implement the inference phase on those devices. The following section combines... Figure 10 The reasoning stage will be introduced.

[0088] Figure 10 A flowchart of a natural language processing method provided in another embodiment of this disclosure. In this embodiment, the specific steps of the method are as follows:

[0089] S1001, Obtain the target text.

[0090] Hypothetical reasoning stage in such Figure 3 This is executed on server 32. Specifically, when terminal 31 needs to process a natural language understanding task, such as named entity recognition, terminal 31 can send the target text to server 32.

[0091] S1002. Determine query information based on the natural language understanding task corresponding to the target text.

[0092] In one possible scenario, the user inputs target text along with query information on terminal 31. For example, the target text might be "So-and-so will fly to city A," and the query information might be "city name." In this case, when server 32 receives the target text and the query information, it can use the user-inputted query information as the query information corresponding to the natural language understanding task. This allows server 32 to retrieve the entity corresponding to "city name," i.e., "city A," from the target text based on the user-specified query information, and then return "city A" to terminal 31. In other words, "city A" is the answer to the query information.

[0093] In another possible scenario, the user inputs target text on terminal 31 but does not input query information. In this case, when server 32 receives the target text, and the natural language understanding task is named entity recognition, it can generate query information for each of all known entity types. For example, known entity types include "city name," "person's name," and "famous historical sites." For "city name," "person's name," and "famous historical sites," server 32 generates query information as follows: "Please find the entities related to the city name in the following text," "Please find the entities related to the person's name in the following text," and "Please find the entities related to the famous historical sites in the following text." Furthermore, server 32 can sequentially query the target text for the entities corresponding to "city name," "person's name," and "famous historical sites," and return the corresponding entities to terminal 31.

[0094] S1003. The query information and the target text are used as input to the machine learning model, so that the machine learning model outputs the answer in the target text that corresponds to the query information. The machine learning model is trained according to the model training method described above.

[0095] Specifically, server 32 can use the query information and the target text as input to the machine learning model that has been pre-trained and fine-tuned as described above. For example, the pre-trained and fine-tuned machine learning model is as follows: Figure 5 The PMR is shown. The PMR is as follows: Figure 5 The logic shown processes the query information and the target text to obtain a matrix similar to probability matrix 51. The maximum probability value is obtained from this matrix, and the text segment in the target text that corresponds to the maximum probability value is taken as the answer to the query information and the answer is output.

[0096] Optionally, using the query information and the target text as input to a machine learning model, so that the machine learning model outputs the answer in the target text corresponding to the query information, includes: using the query information and the target text as input to a machine learning model, so that the machine learning model determines whether the query information and the target text are related; if the query information and the target text are related, then the machine learning model outputs the answer in the target text corresponding to the query information.

[0097] For example, the query information described in this embodiment corresponds to, for example, Figure 5 The query shown in this embodiment corresponds to the target text as follows: Figure 5The context information is shown. After adding the special word [CLS] before the query information, the special word [SEP] between the query information and the target text, and the special word [SEP] after the target text, [CLS], the query information, [SEP], the target text, and [SEP] are used as input to the encoder. After processing by the encoder and extractor, a matrix similar to probability matrix 51 is obtained. The element in the first row and first column of this matrix is ​​extracted first. This element is similar to... Figure 5 S shown 1,1 Based on this element, it can be determined whether the query information and the target text are related. If they are related, it can be further determined whether the text segment should be output based on the probability of each text segment in the target text in the matrix. For example, if the probability is greater than or equal to a preset threshold, it can be determined that the text segment is the answer to the query information, and thus the answer is output.

[0098] This embodiment seamlessly connects the pre-training and fine-tuning stages. After pre-training the machine learning model with a large amount of low-cost data, the pre-trained machine learning model can be calibrated with a small amount of target task data, thereby enabling the fine-tuned machine learning model to accurately handle a variety of natural language understanding tasks.

[0099] Figure 11 This is a schematic diagram of the structure of a model training apparatus provided in an embodiment of this disclosure. The model training apparatus provided in this embodiment of the disclosure can execute the processing flow provided in the model training method embodiment, such as... Figure 11 As shown, the model training device 110 includes:

[0100] The first acquisition module 111 is used to acquire the first target information marked by the hyperlink;

[0101] The second acquisition module 112 is used to acquire a first query corresponding to the first target information from the homepage articles of the first target information, and to acquire at least one first context information of the first target information from at least one cited article of the first target information;

[0102] The pre-training module 113 is used to pre-train the machine learning model based on the first target information, the first query, and the at least one first context information to obtain the pre-trained machine learning model.

[0103] The fine-tuning module 114 is used to determine the second target information from the sample text provided by the natural language understanding task, generate the second query corresponding to the second target information, and train the pre-trained machine learning model using the sample text, the second query and the second target information.

[0104] Optionally, the machine learning model includes an encoder and an extractor, the output of which is the input of the extractor; the pre-training module 113 includes an input unit 1131, a calculation unit 1132, and a pre-training unit 1133, wherein the input unit 1131 is used to take the first query and the first context information as input to the encoder for each of the at least one first context information, so that the encoder outputs the representation vector of each text unit in the first query and the representation vector of each text unit in the first context information; the calculation unit 1132 is used to calculate the probability of each text segment in the first context information being the answer corresponding to the first query through the extractor, wherein each text segment is composed of at least one consecutive text unit in the first context information; the pre-training unit 1133 is used to pre-train the machine learning model according to the probability and the position information of the first target information in the first context information to obtain the pre-trained machine learning model.

[0105] Optionally, when the calculation unit 1132 calculates the probability that each text segment in the first context information is the answer corresponding to the first query through the extractor, it is specifically used for:

[0106] The extractor calculates the probability that a text segment consisting of at least one consecutive text unit from the i-th text unit to the j-th text unit is the answer to the first query, where j is greater than or equal to i, i is greater than or equal to N+1, and N+1 is the index of the first text unit in the first context information corresponding to the input of the encoder. The probability is calculated based on the representation vector of the i-th text unit and the representation vector of the j-th text unit.

[0107] Optionally, the calculation unit 1132 is further configured to: calculate the relevance between the first query and the first context information by the extractor before the pre-training unit 1133 pre-trains the machine learning model based on the probability and the position information of the first target information in the first context information, wherein the relevance is calculated based on the overall representation vector of the first query and the first context information; correspondingly, when the pre-training unit 1133 pre-trains the machine learning model based on the probability and the position information of the first target information in the first context information, it is specifically configured to: pre-train the machine learning model based on the relevance, the probability, and the position information of the first target information in the first context information.

[0108] Optionally, when the second acquisition module 112 acquires the first query corresponding to the first target information from the homepage article of the first target information, it is specifically used to: take the first at least one sentence in the homepage article of the first target information as the first query corresponding to the first target information.

[0109] Optionally, when the second acquisition module 112 acquires at least one first context information of the first target information from at least one cited article of the first target information, it is specifically used for:

[0110] For each of at least one cited article containing the first target information, determine the sentence that includes the first target information from the cited article;

[0111] The sentence in the cited article, along with at least one sentence before and after it, is considered as a first contextual information.

[0112] Figure 11 The model training apparatus of the illustrated embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0113] Figure 12 This is a schematic diagram of a natural language processing apparatus provided in an embodiment of this disclosure. The natural language processing apparatus provided in this embodiment can execute the processing flow provided in the natural language processing method embodiment, such as… Figure 12 As shown, the natural language processing device 120 includes:

[0114] Module 121 is used to acquire the target text;

[0115] The determining module 122 is used to determine query information based on the natural language understanding task corresponding to the target text;

[0116] The input module 123 is used to take the query information and the target text as input to the machine learning model, so that the machine learning model outputs the answer in the target text that corresponds to the query information. The machine learning model is trained according to the model training method described above.

[0117] Optionally, the input module 123 takes the query information and the target text as input to the machine learning model, so that when the machine learning model outputs the answer in the target text corresponding to the query information, it is specifically used for:

[0118] The query information and the target text are used as input to a machine learning model, enabling the machine learning model to determine whether the query information and the target text are related.

[0119] If the query information is related to the target text, then the machine learning model outputs the answer in the target text that corresponds to the query information.

[0120] Figure 12 The natural language processing apparatus of the illustrated embodiment can be used to execute the technical solutions of the above method embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0121] The above describes the internal functions and structure of a model training device or natural language processing device, which can be implemented as an electronic device. Figure 13 A schematic diagram illustrating the structure of an electronic device embodiment provided in this disclosure. For example... Figure 13 As shown, the electronic device includes a memory 131 and a processor 132.

[0122] Memory 131 is used to store programs. In addition to the programs described above, memory 131 can also be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device, contact data, phone book data, messages, pictures, videos, etc.

[0123] The memory 131 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0124] The processor 132 is coupled to the memory 131 and executes the program stored in the memory 131 for:

[0125] Retrieve the first target information marked by the hyperlink;

[0126] Obtain the first query corresponding to the first target information from the homepage articles of the first target information, and obtain at least one first context information of the first target information from at least one cited article of the first target information;

[0127] The machine learning model is pre-trained based on the first target information, the first query, and the at least one first context information to obtain the pre-trained machine learning model.

[0128] The second target information is determined from the sample text provided by the natural language understanding task, a second query corresponding to the second target information is generated, and the pre-trained machine learning model is trained using the sample text, the second query, and the second target information.

[0129] Alternatively, processor 132 can also be used for:

[0130] Get the target text;

[0131] Based on the natural language understanding task corresponding to the target text, determine the query information;

[0132] The query information and the target text are used as input to a machine learning model, which outputs the answer in the target text that corresponds to the query information. The machine learning model is trained according to the model training method described above.

[0133] Furthermore, such as Figure 13 As shown, the electronic device may also include other components such as a communication component 133, a power supply component 134, an audio component 135, and a display 136. Figure 13 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 13 The components shown.

[0134] Communication component 133 is configured to facilitate wired or wireless communication between electronic devices and other devices. The electronic devices can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 133 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 133 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0135] Power supply component 134 provides power to various components of an electronic device. Power supply component 134 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device.

[0136] Audio component 135 is configured to output and / or input audio signals. For example, audio component 135 includes a microphone (MIC) configured to receive external audio signals when the electronic device is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 131 or transmitted via communication component 133. In some embodiments, audio component 135 also includes a speaker for outputting audio signals.

[0137] Display 136 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation.

[0138] In addition, this disclosure also provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the model training method or natural language processing method described in the above embodiments.

[0139] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0140] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A model training method, wherein, The method includes: Retrieve the first target information marked by the hyperlink; Obtain the first query corresponding to the first target information from the homepage articles of the first target information, and obtain at least one first context information of the first target information from at least one cited article of the first target information; For each of the at least one first context information, the first query and the first context information are used as input to the encoder, so that the encoder outputs the representation vector of each text unit in the first query and the representation vector of each text unit in the first context information. The probability of each text segment in the first context information being the answer corresponding to the first query is calculated by the extractor. Each text segment is composed of at least one consecutive text unit in the first context information. Based on the probability and the position information of the first target information in the first context information, the machine learning model is pre-trained to obtain the pre-trained machine learning model. The machine learning model includes an encoder and an extractor, and the output of the encoder is the input of the extractor. The second target information is determined from the sample text provided by the natural language understanding task, a second query corresponding to the second target information is generated, and the pre-trained machine learning model is trained using the sample text, the second query, and the second target information.

2. The method according to claim 1, wherein, The extractor calculates the probability that each text segment in the first context information is the answer to the first query, including: The extractor calculates the probability that a text segment consisting of at least one consecutive text unit from the i-th text unit to the j-th text unit is the answer to the first query, where j is greater than or equal to i, i is greater than or equal to N+1, and N+1 is the index of the first text unit in the first context information corresponding to the input of the encoder. The probability is calculated based on the representation vector of the i-th text unit and the representation vector of the j-th text unit.

3. The method according to claim 1, wherein, Before pre-training the machine learning model based on the probability and the position information of the first target information in the first context information, the method further includes: The extractor calculates the relevance between the first query and the first context information, and the relevance is calculated based on the overall representation vector of the first query and the first context information. Based on the probability and the position information of the first target information in the first context information, the machine learning model is pre-trained, including: The machine learning model is pre-trained based on the relevance, the probability, and the position information of the first target information in the first context information.

4. The method according to claim 1, wherein, Retrieving the first query corresponding to the first target information from the homepage articles of the first target information includes: Use the first sentence of the homepage article of the first target information as the first query corresponding to the first target information.

5. The method according to claim 1, wherein, Obtain at least one first contextual information of the first target information from at least one cited article of the first target information, including: For each of at least one cited article containing the first target information, determine the sentence that includes the first target information from the cited article; The sentence in the cited article, along with at least one sentence before and after it, is considered as a first contextual information.

6. A natural language processing method, wherein, The method includes: Obtain the target text; Based on the natural language understanding task corresponding to the target text, determine the query information; The query information and the target text are used as input to a machine learning model, which outputs the answer in the target text that corresponds to the query information. The machine learning model is trained according to the method described in any one of claims 1-5.

7. The method according to claim 6, wherein, Using the query information and the target text as input to a machine learning model, the machine learning model outputs the answer in the target text that corresponds to the query information, including: The query information and the target text are used as input to a machine learning model, enabling the machine learning model to determine whether the query information and the target text are related. If the query information is related to the target text, then the machine learning model outputs the answer in the target text that corresponds to the query information.

8. A model training device, wherein, include: The first acquisition module is used to acquire the first target information marked by the hyperlink; The second acquisition module is used to acquire a first query corresponding to the first target information from the homepage articles of the first target information, and to acquire at least one first context information of the first target information from at least one cited article of the first target information; A pre-training module is used to take the first query and the first context information as input to an encoder for each of the at least one first context information, so that the encoder outputs the representation vector of each text unit in the first query and the representation vector of each text unit in the first context information. An extractor calculates the probability that each text segment in the first context information is the answer corresponding to the first query, where each text segment is composed of at least one consecutive text unit in the first context information. Based on the probability and the position information of the first target information in the first context information, a machine learning model is pre-trained to obtain a pre-trained machine learning model. The machine learning model includes an encoder and an extractor, and the output of the encoder is the input of the extractor. The fine-tuning module is used to determine the second target information from the sample text provided by the natural language understanding task, generate the second query corresponding to the second target information, and train the pre-trained machine learning model using the sample text, the second query, and the second target information.

9. A natural language processing device, wherein, include: The acquisition module is used to acquire the target text; The determination module is used to determine query information based on the natural language understanding task corresponding to the target text; An input module is used to take the query information and the target text as input to a machine learning model, so that the machine learning model outputs the answer in the target text that corresponds to the query information, wherein the machine learning model is trained according to the method of any one of claims 1-5.

10. An electronic device, wherein, include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the method as described in any one of claims 1-7.

11. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

Images