Prompt learning method and interaction system for large-scale pre-trained language models

Abstract

The present disclosure relates to a prompting learning method for a large-scale pre-trained language model and an interactive system using the same. The method comprises: constructing context examples and feeding them into the pre-trained PLM for prompting learning; and performing testing on target examples using the PLM that has learned the context examples, wherein the parameters of the PLM are fine-tuned using a pre-training task that injects entity knowledge into the PLM to obtain the pre-trained PLM. The present disclosure first proposes a knowledge pre-training algorithm for large-scale pre-trained language models in the industry, laying the foundation for building large-scale knowledge models applied in various fields. Specifically, the method proposes novel pre-training tasks based on fact knowledge injection, which can be used separately or in combination, and finds that the intelligent selection of target examples and the calibration of knowledge priors in the knowledge base can further improve the model performance.

Description

Technical Field

[0001] This disclosure relates to a prompting learning method for large-scale pre-trained language models and an interactive system utilizing this method. Background Technology

[0002] Large-scale pre-trained language models (PLMs), such as those with 1B or more parameters, achieve context learning (ICL) by adjusting some labeled training examples as text-based cues, thus achieving competitive performance without updating parameters. Existing research has explored many aspects affecting ICL performance. However, all these studies, limited to the ICL framework, have yielded limited improvements in model performance.

[0003] Therefore, an improved method is needed to further enhance the performance of large-scale pre-trained language models. Summary of the Invention

[0004] One technical problem addressed by this disclosure is to provide a prompt learning method for large-scale pre-trained language models and an interactive system utilizing this method. This method is the first in the industry to propose a knowledge pre-training algorithm for large-scale pre-trained language models, laying the foundation for building large-scale knowledge models applicable to various domains. Specifically, this method proposes a novel fact-based knowledge injection-based pre-training task that can be used alone or in combination, and finds that the intelligent selection of target examples and calibration using prior knowledge from a knowledge base can further improve model performance.

[0005] According to a first aspect of this disclosure, a method for operating a large-scale pre-trained language model (PLM) is provided, comprising: constructing context examples and feeding them into a pre-trained PLM for cue learning; and performing a test against a target example using the PLM that has learned the context examples, wherein the parameters of the PLM are fine-tuned using a pre-training task that injects entity knowledge into the PLM to obtain a pre-trained PLM.

[0006] Optionally, fine-tuning the parameters of the PLM using a pre-training task that injects entity knowledge into the PLM further includes treating the classification labels as special entities for entity knowledge injection.

[0007] Optionally, fine-tuning the parameters of the PLM using a pre-training task that injects entity knowledge into the PLM includes: fine-tuning the parameters of the PLM using a masked entity prediction task, wherein the masked entity prediction task masks entities in the input text and fine-tunes the parameters of the PLM based on the loss generated by the difference between the entity labels output by the PLM and the real entity labels corresponding to the masked entities.

[0008] Optionally, fine-tuning the parameters of the PLM using a pre-training task that injects entity knowledge into the PLM includes: fine-tuning the parameters of the PLM using an entity description generation task, wherein the entity description generation task is used to: construct a prefix text including entity templates, all entities in the input text, and text templates, and use the original text as a suffix text; mask the entities in the suffix text; and fine-tune the parameters of the PLM based on the loss generated by the difference between the text representation output by the PLM for a given entity and the original text.

[0009] Optionally, fine-tuning the parameters of the PLM using a pre-training task that injects entity knowledge into the PLM includes: fine-tuning the parameters of the PLM using a knowledge question answering task, wherein the knowledge question answering task is used to: mask entities in the input text; construct triples containing the masked entities based on a knowledge base; convert the information of the triples into a question predicting another entity in the triples; and fine-tune the parameters of the PLM based on the loss generated by the difference between the answer label output by the PLM and the corresponding label of the other entity.

[0010] Optionally, fine-tuning the parameters of the PLM using a pre-training task that injects entity knowledge into the PLM includes: randomly selecting multiple examples from the same task to form training instances until the maximum sequence length is reached; calculating the cross-entropy loss between the masked labels output by the PLM and the corresponding labels of the masked entities; and fine-tuning the parameters of the PLM based on the cross-entropy loss.

[0011] Optionally, constructing a context example and feeding it into the pre-trained PLM for cue learning includes: determining the entities contained in the training set and the target set; retrieving preferred training examples from the training set, the preferred training examples containing more entities from the target set compared to other training examples; and constructing the context example using the retrieved preferred training set.

[0012] Optionally, retrieving preferred training examples from the training set includes: calculating the semantic space difference between the training set and the target set, the semantic space difference representing the distance between the average entity representations of the training set and the target set; using the semantic space difference to calculate the knowledge relevance between training examples in the training set and target examples in the target set; determining the sampling weight of each training example in the training set based on the knowledge relevance; and selecting the preferred training examples from the training set according to the sampling weights.

[0013] Optionally, performing a test against a target example using the PLM that has learned the context example includes: calculating the prior probability of a candidate tag word or entity at the output position; and calibrating the output of the PLM test against the target example based on the prior probability.

[0014] According to a second aspect of this disclosure, an interactive system is provided, comprising: a server configured to perform the operation method as described in the first aspect based on domain-specific entity knowledge to obtain a large-scale pre-trained language model enhanced for the specific knowledge domain; and a client configured to obtain query information from a user regarding the specific knowledge domain and submit the query information to the server, wherein the server feeds the query information into the large-scale pre-trained language model enhanced for the specific knowledge domain and returns the output generated by the model to the client.

[0015] According to a third aspect of this disclosure, a computing device is provided, comprising: a processor; and a memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method described in the first aspect above.

[0016] According to a fourth aspect of this disclosure, a computer program product is provided, including executable code that, when executed by a processor of an electronic device, causes the processor to perform the method described in the first aspect above.

[0017] According to a fifth aspect of this disclosure, a non-transitory machine-readable storage medium is provided, on which executable code is stored, which, when executed by a processor of an electronic device, causes the processor to perform the method described in the first aspect above.

[0018] Therefore, this application introduces a novel Knowledge-Based Context Adjustment (KICT) framework to further improve the performance of ICL in the following ways: 1) injecting knowledge into PLM during continuous self-supervised pre-training; 2) intelligently selecting examples with high knowledge relevance; and 3) calibrating prediction results based on prior knowledge. Experimental results show that KICT significantly outperforms strong baselines, improving performance by more than 13% and 7% on text classification and question answering tasks, respectively. Attached Figure Description

[0019] The above and other objects, features and advantages of this disclosure will become more apparent from the more detailed description of exemplary embodiments thereof taken in conjunction with the accompanying drawings, wherein like reference numerals generally denote like parts.

[0020] Figure 1 An example of ICL is shown.

[0021] Figure 2 The accuracy variation of a large-scale PLM is shown when the settings are disrupted.

[0022] Figure 3 A schematic flowchart of a prompting learning method for large-scale PLMs according to an embodiment of the present invention is shown.

[0023] Figure 4 An example of masked entity prediction is shown.

[0024] Figure 5 An example of entity description generation is shown.

[0025] Figure 6 An example of a knowledge-based question-and-answer format is shown.

[0026] Figure 7 A schematic diagram of a knowledge example retrieval operation according to an embodiment of the present invention is shown.

[0027] Figure 8 The relationship between tag frequency and accuracy is shown in large-scale PLM prediction.

[0028] Figure 9 A schematic diagram of the KICT framework according to an embodiment of the present invention is shown.

[0029] Figure 10 A schematic diagram of the composition of an interactive system according to an embodiment of the present invention is shown.

[0030] Figure 11 A schematic diagram of a computing device according to an embodiment of the present invention is shown, which can be used to implement the above-described method for large-scale prompting learning. Detailed Implementation

[0031] Preferred embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0032] With the emergence of large-scale pre-trained language models (with parameter sizes of 1B, tens of B, or even more) that use deep learning to generate natural language that humans can understand, ICL (In-Context Learning) methods, which utilize the knowledge contained in large models without updating parameters, have been widely studied in the field of NLP (Natural Language Processing).

[0033] Figure 1An example of ICL is shown. The purpose of ICL is to generate predictions for target examples by conditionalizing a small number of labeled samples, and its key idea is to learn from analogies. Figure 1 The document provides an example illustrating how large-scale PLM systems can use ICL for decision-making. First, ICL requires examples to form a context serving as a demonstration. These examples are typically written using natural language templates. Figure 1 In the example, the downstream task is to classify the sentiment of text. To enable the model to learn to classify text sentiment, it is first provided with exemplary contextual examples. In the example shown, the classification labels are "Positive" and "negative," which can be labels selected from the verbbalizer corresponding to positive and negative evaluations, respectively. When the input text is "It's a symptom," the model is told to provide a "negative" classification; when the input text is "A funny work," the model is told to provide a "Positive" classification. After learning these two contextual examples, the large-scale PLM can then provide the corresponding output, i.e., a "Positive" classification, for the input "Verynice," which has the same form of target example, based on its inherent knowledge (knowledge embedded in a large number of parameters).

[0034] It is worth noting that, unlike supervised learning during the training phase which requires updating model parameters using backpropagation gradients, ICL does not require parameter updates and directly makes predictions on a pre-trained language model (this is different from hint-based, traditional demonstration learning; ICL does not require downstream P-tuning or fine-tuning). ICL aims to have the model learn patterns hidden in the demonstrations and make correct predictions accordingly. Because ICL can enable models to perform downstream tasks with considerable accuracy without updating model parameters, research on ICL has been booming in recent years. Previous work has explored several aspects affecting ICL performance, such as input-output mapping and data resource generalization.

[0035] However, these works have overlooked the impact of factual knowledge on ICL, which is an indispensable factor in the NLP field. The inventors of this invention have demonstrated through extensive experiments that factual knowledge has a significant impact on model accuracy, and this invention explores the effectiveness of ICL from the perspective of factual knowledge. This is the first time in the industry that knowledge injection into pre-training has been applied to large-scale PLMs. In this invention, "factual knowledge" can refer to knowledge about entities in an example. In the NLP field, "entity" is a very common concept used to refer to an instance of a concept. For example, "Zhang San" can be seen as an instance of a person's name, "Mid-Autumn Festival" as an instance of a time concept, "Chinese Men's Basketball Team" as an instance of a group, and so on. In some cases, such as in text classification tasks, "factual knowledge" can include not only entity knowledge but also classification label knowledge. The importance of entities and labels is combined as follows... Figure 2 This was clearly demonstrated in the destructive experiments.

[0036] To illustrate the importance of factual knowledge for the correct prediction of a model, this invention first explains the impact on model accuracy when factual knowledge is violated. Figure 2 This demonstrates the accuracy variation of a large-scale PLM system when its settings are disrupted. Specifically, Figure 2 The effects of different PLM sizes (illustrated model sizes from 0.1B to 6.7B, all of which can still be considered large-scale PLMs) on accuracy in two common downstream tasks, text classification and question answering, are presented when the settings are broken to varying degrees and in different aspects.

[0037] This invention uses component decomposition to conduct empirical experiments. Specifically, given a target example text... We randomly select K training samples Create a text-based prompt. Identify all entities in the prompt, and then design some disruptive settings as follows.

[0038] 1) Shuffle Entity refers to randomly replacing all entities in the knowledge base with other entities.

[0039] 2) Shuffle Non-Entity indicates that some non-entity words (e.g., "It", "have") are replaced with other words in the PLM vocabulary.

[0040] 3) Shuffle Label means replacing all golden labels with incorrect labels.

[0041] 4) Remove Entity and Remove Label are designed to remove all entities and labels from the prompt, respectively.

[0042] 5) No Demonstration is a typical zero-sample method that does not use any labeled data.

[0043] In the specific experiments, Model A (0.1B-1.5B) and Model B (2.7B-6.7B) at different scales were selected to evaluate eight text classification tasks and four question answering tasks. By default, K = 8 labeled samples were randomly selected for each task, and the experiments were run using five different random seeds. The results are summarized below.

[0044] The inherent knowledge of PLM itself is beneficial to the execution of downstream tasks. For example Figure 2 As shown, the model achieves significantly better few-shot performance as its size increases. This can be hypothesized to be because larger-scale models can learn more valuable semantics from the pre-training corpus. To test this hypothesis, we can refer to performing zero-shot inference (i.e., no demonstration) without using any text-based prompts. In this case, only the intrinsic knowledge learned during pre-training can provide model guidance for prediction. The figure shows a 20% performance gap between 6.7B and 0.1B for both text classification and question-answering tasks. This indicates that the intrinsic knowledge learned during pre-training is necessary.

[0045] The factual knowledge in the selected context examples is key to ICL. For example... Figure 2 As shown, the original setting outperforms all other settings across every model scale. The performance degradation is minimal with non-entity scrambling compared to the original setting; in other words, experiments show that changing non-entities does not significantly reduce performance, while entity scrambling or deletion greatly reduces the average accuracy for text classification and question answering tasks. This indicates that factual knowledge within text-based prompts is a key factor in PLM's understanding of tasks. Furthermore, labels were found to be essential for ICL, therefore the inventors believe that labels can also be considered as part of the knowledge guiding PLM to perceive semantics during model inference. This is also why, in this invention, "factual knowledge" includes entity knowledge, and in some cases, such as in text classification tasks, it can also include classification label knowledge.

[0046] As demonstrated by the experiments above, factual knowledge is also important for large-scale PLM. Therefore, this invention proposes a large-scale PLM operation method that can fully utilize factual knowledge to improve the performance of various processes in ICL. Figure 3A schematic flowchart illustrating a prompting learning method for large-scale PLMs according to an embodiment of the present invention is shown. This method is designed for large-scale PLMs with infused factual knowledge and improves the inference performance of large-scale PLMs for downstream tasks.

[0047] In step S310, a context example is constructed and fed into the pre-trained PLM for cue learning. Specifically, the parameters of the PLM are fine-tuned using a pre-training task that injects entity knowledge into the PLM to obtain a pre-trained PLM. In step S320, a test against the target example is performed using the PLM that has learned the context example. Due to the above combination... Figure 2 It has been demonstrated that even for large-scale PLMs, the injection of factual knowledge is necessary (i.e., pre-training for entity knowledge injection in large-scale PLMs is necessary, therefore, in performing ICL, step S120 can correspond to...). Figure 1 The learning steps for the example "Input: It's a symptom.\nOutput: Negative\n\nInput: A funny work.\nOutput: Positive\n\n" are as follows; step S130 corresponds to the step of inferring the output when the target example is "Input: Verynice.\n", for example, before correctly inferring the output as "Positive", the large-scale PLM is first enhanced with factual knowledge, thereby improving the performance of the PLM in ICL and subsequent inference. Therefore, the pre-training task can be regarded as a necessary operation performed before cue learning.

[0048] As mentioned earlier, due to the importance of the label words, fine-tuning the parameters of the PLM using a pre-training task that injects entity knowledge into the PLM may also include: treating the classification labels as special entities for entity knowledge injection (especially for text classification tasks).

[0049] This invention designs a novel Knowledgeable Pre-Training (KPT) task by injecting factual knowledge into the PLM during pre-training. The goal of pre-training is to enrich the language generation capabilities for important entities using an external knowledge base. Therefore, the input is the training corpus { } and knowledge base ,in It is a group of entities. It is a set of relationships. It is a set of triples that express factual knowledge.

[0050] In one embodiment, the pre-training task can be Masked Entity Prediction (MEP). This task requires the model to predict missing entities in text to learn explicit knowledge, similar to masked language modeling in BERT-type PLMs. In this case, fine-tuning the PLM parameters using the pre-training task that injects entity knowledge into the PLM includes: fine-tuning the PLM parameters using the Masked Entity Prediction task, wherein the Masked Entity Prediction task masks entities in the input text and fine-tunes the PLM parameters based on the loss generated by the difference between the entity labels output by the PLM and the true entity labels corresponding to the masked entities.

[0051] Specifically, given a text token. Identify all entities within using existing tools (e.g., entity linking toolkits). ,in It is an entity with multiple tags. For each entity This can be replaced with a special marker (e.g., "_") for subsequent model inference on the special marker placeholder. In one embodiment, for each entity... 50% of the time is replaced with special markers (e.g., "_"), while the remaining 50% is replaced with random markers. This introduction of random markers increases the task difficulty and helps the model learn the semantics of entities. A training example can be obtained from this. It can generate a label mask vector. To represent the location used for training, and ,in It is an indicator function.

[0052] Figure 4 An example of masked entity prediction is shown. In the MEP task, an example of input text is as follows:

[0053]

[0054] The corresponding Chinese translation could be: The Association for Computational Linguistics (ACL) is a scientific and professional organization dedicated to NLP work. The highlighted parts, "Association for Computational Linguistics," "ACL," and "NLP," correspond to the entities contained in the text, which are the contents that need to be masked and predicted by PLM in the MEP task.

[0055] In one embodiment, the pre-training task can be Entity Description Generation (EDG). This task aims to generate text descriptions step-by-step from given entities. In this case, fine-tuning the parameters of the PLM using the pre-training task that injects entity knowledge into the PLM includes: fine-tuning the parameters of the PLM using the EDG task, wherein the EDG task is used to: construct a prefix text including entity templates, all entities in the input text, and the text template, and use the original text as a suffix text; mask the entities in the suffix text; and fine-tune the parameters of the PLM based on the loss generated by the PLM's output text representation for a given entity compared to the original text.

[0056] Specifically, given a text and a corresponding entity set Construct a prefix text, which is formed from the template "Entities:". The text consists of all entities and the linearized string formed by the template "Text:". The suffix text is the original text. Similarly, training examples can be generated. and label mask vector ,when When located in the suffix string, .

[0057] Here, the prefix and suffix text explicitly define what the EDG task requires. Adding descriptive words before and after the entity helps the model better understand that the current task is to generate a description of that entity. Figure 5 An example of entity description generation is shown. In the EDG task, the input to the model is entity information, namely, "Association for Computational Linguistics", "ACL", and "NLP", and the model is required to generate textual descriptions of these three entities, as shown in the gray area in the figure, in complete sentences.

[0058] To fully utilize existing triples in the knowledge base, this invention also considers a knowledge-aware question-answering task. Therefore, in one embodiment, the pre-training task can be a knowledge-question-answering (KQA) task. This task aims to generate entities based on a question. In this case, fine-tuning the parameters of the PLM using the pre-training task that injects entity knowledge into the PLM includes: fine-tuning the parameters of the PLM using the knowledge-question-answering task, wherein the knowledge-question-answering task is used to: mask entities in the input text; construct triples containing the masked entities based on the knowledge base; convert the information of the triples into a question predicting another entity in the triples; and fine-tune the parameters of the PLM based on a loss arising from the difference between the answer label output by the PLM and the corresponding label of the other entity.

[0059] Specifically, given a text and a corresponding entity set Two entities can be obtained. They have a 1-hop relationship. And form a triple. ,in and These are the head entity and the tail entity, respectively. A template can be designed for each triple and transformed into a problem where the model predicts the tail entity, thus obtaining training examples. And the label mask vector. When When it is the tag of the selected tail entity, .

[0060] Figure 6 An example of knowledge-based question answering is shown. In the EQA task, the input to the model is textual information, such as a complete sentence as illustrated. Because there is a one-hop relationship between "Association for Computational Linguistics" (or "ACL") and "NLP", a triple can be formed, where "Association for Computational Linguistics" is the head entity, NLP is the tail entity, and the relationship... Therefore, it is "working for". Thus, the template can be used to generate the question "What is Association for Computational Linguistics working for?" from the triple, and ask the model to answer. When the model reasones correctly, it answers "NLP" (as shown by the gray mark in the figure), and this answer corresponds to the tail entity of the triple above.

[0061] It should be understood that, in different implementations, any one or a combination of the three self-supervised learning tasks mentioned above can be used to pre-train the model. In one embodiment, the pre-training mentioned above may include: randomly selecting multiple examples from the same task to form training instances until the maximum sequence length is reached; calculating the cross-entropy loss between the masked labels output by the PLM and the corresponding labels of the masked entities; and fine-tuning the parameters of the PLM based on the cross-entropy loss.

[0062] Specifically, during the pre-training process, multiple examples can be randomly selected from the same task to form training instances. This continues until the maximum sequence length (e.g., 2048) is reached. The output position is then calculated (where...). The cross-entropy loss at () is used. Formally, the loss function can be:

[0063]

[0064] in This is a basic fact. p (·) indicates the predicted probability. This is the number of locations where the model needs to calculate the loss.

[0065] Although a powerful and knowledge-rich PLM is obtained after pre-training, capable of performing the prompting step as described in step S310 and the prediction step as described in step S320, the performance of ICL largely depends on the selection and order of labeled examples. It has been found that the PLM itself can generate suitable text-based prompts. However, existing research has rarely focused on the tangible value of factual knowledge in the knowledge base. The inventors of this invention have discovered that wisely selecting prompts in the prompting step can also improve the reasoning accuracy of the prediction step and downstream tasks.

[0066] To this end, in one embodiment, the present invention proposes a novel Knowledge Instance Retrieval (KER) algorithm that incorporates knowledge to select contextual instances. Figure 7 A schematic diagram of a knowledge example retrieval operation according to an embodiment of the present invention is shown.

[0067] As shown in the figure, the entities included can be determined from the training set and the target set. Here, a given training set is... and a target set (i.e., the test set), where and This indicates the input text. Labels representing training examples, and This is the corresponding entity set. Recall that knowledge in text-based prompts is key to ICL. Therefore, the task of KER is to select a set of training examples that are highly knowledge-relevant to the target set. A simple approach is to retrieve examples whose entities can cover more target examples. To do this, after identifying the entities, preferred training examples can be retrieved from the training set, which contain more entities from the target set compared to other training examples, and the context examples can be constructed using the retrieved preferred training set. The similarity between two examples can be calculated using Jaccard similarity. .

[0068] However, most example pairs have a Jaccard similarity of zero, so pre-trained knowledge embeddings are further utilized to retrieve training examples that are more similar to the target set in the semantic space. At this point, retrieving preferred training examples from the training set includes: calculating the semantic space difference between the training set and the target set, where the semantic space difference characterizes the distance between the average entity representations of the training set and the target set; calculating the knowledge relevance between training examples in the training set and target examples in the target set using the semantic space difference; determining the sampling weight for each training example in the training set based on the knowledge relevance; and selecting the preferred training example from the training set according to the sampling weight.

[0069] In form, they obtained and Average representation of all entities and . and Euclidean distance between This can be used to represent differences in semantic space. Therefore, the knowledge relevance between two samples can ultimately be calculated as:

[0070]

[0071] Where 0 ≤ α ≤ 1 and These are hyperparameters. For each The sampling weights are:

[0072]

[0073] in It can be calculated as:

[0074]

[0075] Intuitively, a high-weighted training example implies that it has a high knowledge relevance to all target examples. Ultimately, we can sample K training examples based on these weights.

[0076] After model pre-training and context example selection, we can directly generate target instances using the following formula. Output:

[0077]

[0078] in It is a verbalizer that maps tags to corresponding categories (for categories, ...). This represents a set of tag words. For question-and-answer questions, This refers to the entire vocabulary set. It is a collection of context examples.

[0079] However, due to knowledge bias, PLM tends to generate common tag words. This invention selects two knowledge-intensive tasks (i.e., AGNews and TREC) to test whether prediction bias exists. First, the top 5 predictions are obtained at the output position of each training example, and the frequency statistics of each generated tag word are calculated (the training set is larger than the test set to make the statistics more obvious). Then, 4 tag examples are selected from the training set. For each category, 2 words are randomly selected from each frequency domain, and the average accuracy of all tag mapping permutations is reported. (Taking AGNews as an example, it has 4 classes, each with 2 words, for a total of 2...) 4 =16 label mappings arranged. ) Experimental results in Figure 8 It is shown in the figure. Figure 8 The figure illustrates the relationship between label frequency and accuracy in large-scale PLM prediction. As can be seen from the figure, performance is highly dependent on label frequency, indicating that the frequency of factual knowledge learned in the PLM is crucial for prediction.

[0080] To address this challenge, prior information about the tag words can be used to calibrate the prediction for each target example. Specifically, in one embodiment, the invention further includes optimizing the prediction step, where performing a test against the target example using the PLM learned from the context examples includes: calculating the prior probability of the candidate tag word or entity at the output position; and calibrating the output of the PLM's test against the target example based on the prior probability.

[0081] Specifically, training corpora can be obtained from KQA tasks. A subset, and each candidate tag or entity is calculated in the following way. Prior context regarding the output location:

[0082]

[0083] in This is a training example. Show candidates Approximate prior information. In one embodiment, tags or entities with prior probabilities less than a threshold can be deleted. Therefore, the output can be improved by calibrating the prediction:

[0084]

[0085] Much of the recent research on ICL focuses on prediction calibration. In contrast, this invention leverages prior knowledge from large-scale corpora for bias correction, rather than relying solely on in-domain data or designing task-agnostic, context-free inputs (e.g., "N / A").

[0086] Figure 9 A schematic diagram of a KICT framework according to an embodiment of the present invention is shown. This KICT framework can be considered a preferred embodiment of the large-scale PLM operation method described above. According to... Figure 2 Experiments demonstrate that factual knowledge has a significant impact on ICL. This suggests that this knowledge can be fully utilized to improve the performance of various processes in ICL, including pre-training, prompting, and prediction. To achieve this, this invention introduces KICT, a novel Knowledgeable In-Context Tuning framework that better leverages knowledge to unleash the potential of PLM for answer generation. Within this framework, Knowledgeable Pre-Training (KPT) is introduced, incorporating three carefully designed self-supervised tasks—MEP, EDG, and KQA—to extract information from a pre-training corpus and a knowledge base (knowledge pictures) to infuse factual knowledge into the PLM. Then, in the prompting stage, a Knowledgeable Example Retrieval (KER) algorithm is proposed to intelligently select knowledge-relevant contextual examples. Finally, a Knowledgeable Prediction Calibration (KPC) technique is used to calibrate the prediction distribution using prior information derived from the knowledge base. Figure 8 The left, middle, and right figures correspond to the pre-trained self-supervised learning task (KPT), the prompting step of knowledge-based example retrieval (KER), and the prediction step of obtaining prior information from a large-scale corpus to calibrate predictions (KPC), respectively. These three modules are plug-and-play knowledge-based technologies that can be used individually or in combination to improve the overall performance of large-scale PLM.

[0087] This invention proposes for the first time a knowledge pre-training algorithm for large-scale PLM (Product Lifecycle Management), enabling large-scale PLMs to fully learn various types of knowledge and laying the foundation for building large-scale knowledge GPT (Geometric Knowledge Platform) models applicable to various fields. Therefore, this invention can also be implemented as an interactive system. Figure 10A schematic diagram of an interactive system according to an embodiment of the present invention is shown. As shown, the system 1000 includes a server 1010, a database 1020, and a client 1030. The database 1020 stores a large-scale PLM and a knowledge base. The server 1010 is used to perform the operation method described above based on entity knowledge in a specific domain to obtain a large-scale pre-trained language model enhanced for that specific knowledge domain. The client 1020 is used to obtain query information from a user regarding a specific knowledge domain and submit the query information to the server 1010. The server sends the query information into the large-scale pre-trained language model enhanced for that specific knowledge domain and returns the output generated by the model to the client.

[0088] Figure 11 A schematic diagram of a computing device according to an embodiment of the present invention, which can be used to implement the above-described large-scale PLM operation method, is shown. See also Figure 11 The computing device 1100 includes a memory 1110 and a processor 1120.

[0089] Processor 1120 may be a multi-core processor or may contain multiple processors. In some embodiments, processor 920 may include a general-purpose main processor and one or more special coprocessors, such as a graphics processing unit (GPU), a digital signal processor (DSP), etc. In some embodiments, processor 920 may be implemented using custom circuitry, such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0090] Memory 1110 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by processor 1120 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation. Furthermore, memory 1110 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, memory 1110 may include a removable storage device that is readable and / or writable, such as a laser disc (CD), a read-only digital multifunction optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-high density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.

[0091] The memory 1110 stores executable code, which, when processed by the processor 1120, enables the processor 1120 to execute the large-scale PLM operation method described above.

[0092] The large-scale PLM operation method and interactive system according to the present invention have been described in detail above with reference to the accompanying drawings.

[0093] This application introduces a novel Knowledgeable In-Context Tuning (KICT) framework to further improve the performance of ICL by: 1) injecting knowledge into PLM during continuous self-supervised pre-training; 2) intelligently selecting examples with high knowledge relevance; and 3) calibrating prediction results based on prior knowledge. The inventors evaluated the performance of the proposed method on autoregressive models for multi-text classification and question answering tasks. Experimental results show that KICT significantly outperforms strong baselines, achieving improvements of over 13% and 7% on text classification and question answering tasks, respectively.

[0094] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0095] Furthermore, the method according to the present invention can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing the steps defined in the above-described method of the present invention.

[0096] Alternatively, the present invention can also be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) storing executable code (or computer program, or computer instruction code) thereon, which, when executed by a processor of an electronic device (or computing device, server, etc.), causes the processor to perform the various steps of the method described above according to the present invention.

[0097] Those skilled in the art will also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both.

[0098] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems and methods according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0099] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A cue learning method for large-scale pre-trained language models (PLMs), comprising: Constructing context examples and feeding them into the pre-trained PLM for cue learning includes: identifying entities contained in a training set and a target set; calculating the semantic space difference between the training set and the target set, the semantic space difference representing the distance between the average representations of entities in the training set and the target set; calculating the knowledge relevance between training examples in the training set and target examples in the target set using the semantic space difference; determining the sampling weight of each training example in the training set based on the knowledge relevance; selecting preferred training examples from the training set according to the sampling weights, the preferred training examples containing more entities from the target set compared to other training examples; and constructing the context examples using the retrieved preferred training set; and Perform tests against the target example using the PLM that has learned from the context example. Specifically, a pre-training task that injects entity knowledge into the PLM is used to fine-tune the parameters of the PLM to obtain a pre-trained PLM.

2. The method as described in claim 1, wherein, Fine-tuning the parameters of the PLM using a pre-training task that injects entity knowledge into the PLM also includes: The category tags are treated as special entities for entity knowledge injection.

3. The method as described in claim 1, wherein, Fine-tuning the parameters of the PLM using a pre-training task that injects entity knowledge into the PLM includes: The parameters of the PLM are fine-tuned using a masked entity prediction task, wherein the masked entity prediction task masks entities in the input text and fine-tunes the parameters of the PLM based on the loss generated by the difference between the entity labels output by the PLM and the real entity labels corresponding to the masked entities.

4. The method of claim 1, wherein, Fine-tuning the parameters of the PLM using a pre-training task that injects entity knowledge into the PLM includes: The entity description generation task is used to fine-tune the parameters of the PLM, wherein the entity description generation task is used for: Construct a prefix text that includes entity templates, all entities in the input text, and text templates, and use the original text as the suffix text; Mask entities in the suffix text; The parameters of the PLM are fine-tuned based on the loss generated by the difference between the textual representation output by the PLM for a given entity and the original text.

5. The method of claim 1, wherein, Fine-tuning the parameters of the PLM using a pre-training task that injects entity knowledge into the PLM includes: The PLM parameters are fine-tuned using a knowledge-based question-and-answer task, wherein the knowledge-based question-and-answer task is used to: Mask entities in the input text; Construct triples containing the masked entities based on the knowledge base; The information of the triple is transformed into a question of predicting another entity in the triple; The parameters of the PLM are fine-tuned based on the loss generated by the difference between the answer label output by the PLM and the corresponding label of the other entity.

6. The method according to any one of claims 3-5, wherein, Fine-tuning the parameters of the PLM using a pre-training task that injects entity knowledge into the PLM includes: Multiple examples are randomly selected from the same task to form training instances until the maximum sequence length is reached; Calculate the cross-entropy loss between the masking markers output by the PLM and the corresponding markers of the masked entity; The parameters of the PLM are fine-tuned based on the cross-entropy loss.

7. The method of claim 1, wherein, Performing tests against the target example using the PLM that has learned the context example includes: Calculate the prior probability of candidate tags or entities at the output position; and Based on the prior probability, the output of the PLM test for the target example is calibrated.

8. An interactive system, comprising: A server is configured to perform the prompting learning method as described in any one of claims 1-7 based on domain-specific entity knowledge to obtain a large-scale pre-trained language model enhanced with domain-specific knowledge. as well as The client is used to obtain query information from users regarding a specific knowledge domain and submit the query information to the server. The server feeds the query information into a large-scale pre-trained language model enhanced with the specific knowledge domain, and returns the output generated by the model to the client.

9. A computing device, comprising: processor; as well as A memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 1 to 7.

10. A computer program product comprising executable code that, when executed by a processor of an electronic device, causes the processor to perform the method as claimed in any one of claims 1 to 7.

11. A non-transitory machine-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method as claimed in any one of claims 1 to 7.

Images