Pre-trained language model processing method and intelligent question answering system based on contrastive learning

Abstract

Disclosed are a pre-trained language model processing method and an intelligent question-answering system based on contrastive learning. The method comprises: adding a prompt placeholder and a mask output placeholder to a sample to obtain a processed training sample; inputting the processed training sample into the pre-trained language model PLM to obtain an embedding for continuous characterization of the mask output; and adjusting the neural network of the PLM with a first loss function, wherein the first loss function is used to minimize the intra-group distance of the embeddings of samples of the same type and maximize the inter-group distance of the embeddings of samples of different types. The present invention is based on the idea of ​​contrastive learning, so that positive and negative examples are far away from each other under the constraint of the loss function, thereby realizing efficient small sample learning without adding any new parameters and avoiding artificially setting any templates and label words. An intelligent question-answering system equipped with a model obtained based on this method can obtain good question matching capabilities.

Description

Technical Field

[0001] The present disclosure relates to the field of deep learning, and in particular to a pre-trained language model processing method based on contrastive learning, and an intelligent question-answering system equipped with a model obtained by this method. Background Art

[0002] To achieve high-precision prediction results on specific natural language processing tasks, it is usually necessary to use a large amount of labeled data to train the pre-trained model. However, a large amount of labeled data will make the training cost too high. For this reason, the small sample learning technology developed can enable the pre-trained language model to be trained under the condition of a small number of training samples, thereby achieving higher prediction accuracy with lower training costs.

[0003] However, in the existing small sample learning technology, a lot of manual operations are still required to set the training template, label words and add new parameters. The above operations make the training process cumbersome and the training effect unstable.

[0004] To this end, an improved technique for few-shot learning is needed. Summary of the invention

[0005] A technical problem to be solved by the present disclosure is to provide a pre-trained language model processing method based on contrastive learning. Based on the idea of ​​contrastive learning, the present invention makes positive and negative examples far away from each other under the constraint of the loss function, so as to achieve efficient small sample learning without adding any new parameters and avoiding artificially setting any templates and label words. The intelligent question-answering system equipped with the model obtained by this method can obtain good question matching ability.

[0006] According to the first aspect of the present disclosure, a method for processing a pre-trained language model based on contrastive learning is provided, comprising: adding a prompt placeholder and a mask output placeholder to a sample to obtain a processed training sample; inputting the processed training sample into the pre-trained language model PLM to obtain an embedding that continuously represents the mask output; and adjusting the neural network of the PLM with a first loss function, wherein the first loss function is used to minimize the intra-group distance of the embeddings of samples of the same type and maximize the inter-group distance of the embeddings of samples of different types. Optionally, the first loss function includes paired relaxation factors to constrain the distances between different categories and the degree of similarity within the class.

[0007] Optionally, the method also includes: masking the words in the sample to obtain a processed training sample with a masked text placeholder; obtaining the corresponding prediction result of the PLM for the masked text placeholder; and adjusting the neural network of the PLM with a second loss function, wherein the second loss function calculates the loss based on whether the corresponding prediction result of the masked text placeholder is the same as the masked word.

[0008] Optionally, the method further includes: adjusting the neural network of the PLM according to a total loss function obtained by weighting the first loss function and the second loss function.

[0009] Optionally, the method further includes: after the current iteration of PLM training for the current training sample set ends, calculating the embeddings of all previous training samples, and calculating the embedding average of each category as the class center.

[0010] Optionally, the method further includes: in the inference stage, obtaining the category to which the data to be inferred belongs according to the similarity between the current embedding output by the PLM and the class center.

[0011] Optionally, the method further includes: obtaining the class center of each class in the classification task for the training sample set; and storing the class centers of the existing multiple classes in the storage network.

[0012] Optionally, the method further includes: inputting training samples in the current sample set to be trained into a text neural network model to obtain embeddings for generating prompts; sending the embeddings for generating prompts and the class centers into an attention network; and obtaining the output of the attention network as prompts for the current sample set to be trained.

[0013] Optionally, the method further includes: adding a prompt placeholder and a mask output placeholder to the sample to obtain a processed training sample includes: for the sample input as a sentence pair, adding the prompt placeholder and the mask output placeholder between two sentences to obtain the processed training sample.

[0014] According to a second aspect of the present disclosure, an intelligent question-answering system is provided, comprising: a user input receiving unit, used to obtain a user's input query; a question matching unit, comprising a pre-trained model obtained by the method described in the first aspect above, wherein the model is used to match the input query with an existing question; and an answer providing unit, used to provide an answer to the matched existing question to the user.

[0015] According to a third aspect of the present disclosure, a computing device is provided, comprising: a processor; and a memory on which executable codes are stored, and when the executable codes are executed by the processor, the processor executes the method described in the first aspect above.

[0016] According to a fourth aspect of the present disclosure, a non-transitory machine-readable storage medium is provided, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor is caused to execute the method described in the first aspect above.

[0017] Therefore, the present invention avoids the need to manually construct a mapping between label words and classification categories in traditional small sample learning technology by introducing contrastive learning, and reduces the sensitivity of text to the prompt template by setting the PCCL task. In addition, by using the output representation of the pre-trained model before the softmax layer as a feature, the semantic information of the pre-trained model is retained and utilized to the maximum extent, and the overfitting problem in small sample learning can be alleviated by using MLM as an auxiliary task. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components in the exemplary embodiments of the present disclosure.

[0019] Figure 1 An example of prompts and label selection for review sentiment analysis is shown.

[0020] Figure 2 A schematic flow chart of a method for processing a pre-trained language model based on contrastive learning according to an embodiment of the present invention is shown.

[0021] Figure 3 A diagram showing a contrastive learning adjustment method according to an embodiment of the present invention is shown.

[0022] Figure 4A -B shows an example of conventional contrastive learning and the present invention optimizing the distribution of positive and negative examples.

[0023] Figure 5 A diagram showing a contrastive learning adjustment method according to an embodiment of the present invention is shown.

[0024] Figure 6 A diagram showing a contrastive learning adjustment method according to an embodiment of the present invention is shown.

[0025] Figure 7 A schematic diagram showing the operation of the task-adaptive cue encoder.

[0026] Figure 8 An example of using the PLM trained by the present invention for intelligent question answering is shown.

[0027] Fig. 9A schematic diagram of the structure of a computing device that can be used to implement the above-mentioned pre-trained language model processing method based on contrastive learning according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0028] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0029] Small sample learning is a paradigm of machine learning, which aims to obtain a model with higher accuracy by only slightly tuning the model with very small training samples. Small sample learning has broad application prospects in today's world where the cost of training labeled data is high.

[0030] Examples of few-shot learning techniques include:

[0031] 1) Fine-tuning based on pre-trained models: refers to adding a small number of task-related neural network layers to the already trained language model and retraining for a specific task. For example, for classification problems, a classifier is added based on a pre-trained language model (such as BERT) and trained based on the task corpus. This method is more commonly used when downstream tasks have more labeled data. Since a large number of additional parameters need to be reinitialized, the results are poor in small sample cases due to insufficient data.

[0032] 2) PET (Pattern Exploting Training): In view of the special nature of small sample tasks, downstream tasks are reconstructed into "cloze problems" and prediction problems are converted into MLM (masked language model) problems. For example, for the problem of comment sentiment analysis, if you need to predict the sentiment polarity of "This phone case is very good". The PET algorithm rewrites the task as "This phone case is very good. I think [MASK] is good. The "[MASK]" characters correspond to the prediction output of the pre-trained model MLMhead (MLM head) mapped to the actual category label. For example, for the above example, if the probability of predicting "very" is high, it corresponds to the "positive" category, and if the probability of predicting "no" is high, it corresponds to the "negative" category. The advantage of the PET algorithm is that it can utilize the MLM layer of the pre-trained model without introducing new parameters, and can use manually selected templates and label words to prompt model predictions, so the model performs better in small sample situations. However, its main disadvantage is that the model performance is more sensitive to the selected templates and labels, and the selection of templates and label words is more dependent on manual experience, so it is not user-friendly from the product level.

[0033] P-Tuning: The P-tuning algorithm abandons the assumption that the template must be natural language, and believes that training continuous text representations as auxiliary inputs can also have the effect of prompting the pre-trained model. The P-tuning algorithm builds a small LSTM model to learn these continuous representations for "prompting", and finds that it has a significant improvement in small sample learning tasks for GPT-type pre-trained models, and is also applicable to BERT-type models. Although this method solves the problem of manually selecting prompt templates in the PET algorithm, the label words corresponding to the classification problem still need to be manually selected and have a greater impact on the model results; and other experiments also show that the P-tuning algorithm does not perform as well as PET when applied to BERT-type models.

[0034] Figure 1 An example of prompts and label selection for review sentiment analysis is shown. Figure 1 As shown, in order to determine the sentiment category of the sentence "Wonderful movie in every aspect (a great movie in every aspect)" (for example, positive praise or negative criticism), the prompt "It is [MASK]" can be directly added to the input text, and it is determined that the [MASK] to be predicted by the model can be "good" corresponding to the positive label, or "terrible" corresponding to the negative label. In other words, the prompt template can be constructed in the format of "It is + sentiment attribute vocabulary", and the Verbalizer is used to select two labels from the vocabulary corresponding to positive and negative emotions, in this case "good" and "terrible". Thus, the original training sample "Wonderful movie in every aspect." can be transformed into a processed training sample with a mask prompt added: "Wonderful movie in every aspect. It is [MASK]", which is then sent to the pre-trained language model (PLM) for training, for example, the loss function is obtained and adjusted based on back propagation according to whether the PLM model predicts [MASK] to be good or terrible.

[0035] exist Figure 1 In the example shown, positive and negative labels need to be selected manually. For example, the words “good” and “terrible” shown in the figure can be selected, or other words in the vocabulary (e.g., the total vocabulary) used to represent sentiment attributes, such as “awesome” and “bad” can be selected. Figure 1In the example, the prompt word "It is" is also artificially designed. The selection of the prompt words and positive and negative labels increases the uncertainty of the model, and the artificial selection has a great influence on the model results.

[0036] To this end, the present invention proposes a pre-trained language model processing method based on contrastive learning, which can be implemented as CP-Tuning (Contrastive Learning Tuning), an end-to-end contrastive prompt tuning framework for PLM, without the need to manually design task-specific prompts and language expressors. Specifically, the present invention includes two main innovations:

[0037] 1) Task-invariant continuous hint encoding: The present invention uses continuous embeddings as hints and does not use any hint encoder to avoid learning additional parameters during small-shot learning. In one embodiment, the embeddings of continuous hints can be initialized as a set of pre-trained representations of task-invariant labels, and the hint embeddings are made task-adaptive through back-propagation.

[0038] 2) Language-less class mapping: We propose a language-less mechanism to alleviate the manual labor of designing language expressors and improve the generalization ability of task-invariant models and prompts. Specifically, we introduce a pairwise cost-sensitive contrastive loss (PCCL) to train a few-shot learner, as well as an auxiliary masked language model (MLM) task as a regularizer. PCCL explicitly learns to distinguish different categories and makes the decision boundary smoother by assigning different costs to easy and hard cases. Compared with previous methods, the instance embedding before the MLM classifier is directly used for reasoning.

[0039] Figure 2 A schematic flow chart of a method for processing a pre-trained language model based on contrastive learning according to an embodiment of the present invention is shown.

[0040] Here, contrastive learning refers to a self-supervised learning method used to improve the representation of data in deep learning models. The core idea is to bring positive samples closer together and distance positive and negative samples further apart in the data representation space. Pre-trained language model is a general term for a class of natural language processing models that are used to learn low-dimensional real-valued vector representations of text. Early pre-trained language models aimed to use shallow neural networks to learn word embedding representations, which were used for various downstream natural language processing tasks; the latest pre-trained language models learn context-based word embedding representations, and the learned models are used for fine-tuning in downstream tasks.

[0041] In step S210, a prompt placeholder and a mask output placeholder are added to the sample to obtain a processed training sample. In step S220, the processed training sample is input into the pre-trained language model PLM to obtain an embedding that continuously represents the mask output. In step S230, a first loss function (L PCCL ) The neural network of the PLM is adjusted, and the first loss function is used to minimize the intra-group distance of the embedding of samples of the same type and maximize the inter-group distance of the embedding of samples of different types.

[0042] Figure 3 A diagram of a contrastive learning adjustment method according to an embodiment of the present invention is shown. Here, [CLS] refers to a text input header identifier, [TXT] refers to the original input text, [PRO] refers to a learnable prompt, and [OMSK] refers to a mask for label prediction. Further, [EMB] refers to a representation for label prediction, i.e., an embedding of a continuous representation.

[0043] Here, taking the original text "Wonderful movie in every aspect." as an example, different from Figure 1 As shown, "Wonderful movie in every aspect. It is [MASK]." is fed into the PLM as a training sample. In the present invention, the training sample fed into the PLM is a training sample in which a prompt placeholder and a mask output placeholder are directly added to the original text, which can be "Wonderful movie in every aspect. [PRO] [PRO] [OMSK]." Since the prompt has not been learned at this time, [PRO] corresponds to the prompt placeholder; and since the mask has not been predicted by the model at this time, [OMSK] corresponds to the mask output placeholder.

[0044] It should be understood that Figure 3 An example of single sentence input is shown in . In the case where sentence pairs need to be input, that is, for sentence pair tasks, the input format can be [CLS][TXT][TXT][PRO][PRO][OMSK][TXT][TXT][TXT]. The "[PRO]" and "[OMSK]" tags are placed between text pairs to better capture the relationship between them. For example, in a text matching task, for a sample input as a sentence pair, the prompt placeholder and the mask output placeholder can be added between two sentences to obtain a processed training sample.

[0045] Furthermore, unlike Figure 1The MLM head (i.e., the classification result sent to the softmax layer) is directly used to calculate the loss function. The first loss function L in the present invention is PCCL The computation object is [EMB], which is the representation used for label prediction, that is, the embedding of the continuous representation. Using the continuous representation of label prediction instead of MLM head to calculate the loss value based on contrastive learning can bring many benefits, which are explained as follows.

[0046] As a parameter, "[PRO]" needs to be learned from a given task, but this is difficult to achieve in small sample learning due to the lack of training data. To this end, the present invention initializes the prompt embedding to a pre-trained representation of a general task invariant prompt. At this time, the general task invariant prompt for single sentence classification is "it is"; the invariant prompt for the sentence pair classification task is "?". This setting can be regarded as a knowledge prior for the prompt embedding. During model training, the representation of the prompt can be automatically adapted to a specific task. It can be further verified that there is no need to design task-specific prompts for the method of the present invention. Therefore, there is no need to change the number and position of the "[PRO]" prompts for model tuning.

[0047] A common property of existing prompt-based methods is that they require hand-crafted language expressors to establish the mapping between tokens and class labels. Figure 1 In the review analysis example shown, the Verbalizer-based approach generates a distribution over the entire vocabulary (which may contain more than 10,000 words) and focuses on only a very small number of words (such as "good" and "terrible" in the example). The semantic associations between words are also largely ignored. For example, the probabilities of words such as "nice", "fantastic", "bad", and "horrible" are also strong indicators of class labels. In contrast, the present invention uses low-dimensional, dense representations (i.e., embeddings) instead of high-dimensional, sparse distributions, which can greatly improve the generalization ability and flexibility of the underlying model.

[0048] Therefore, the present invention proposes a new language-free method to generate PCCL-based model output. Represented as a set of instances in a batch Each instance can be considered as an anchor point, with y i You can also set the positive set P(i) and negative set N(i) for instance i and batch and That is, the same labels are positive sets, and different labels are negative sets.

[0049] Make 1 for the last layer of the bottom layer PLM (before the sofimax function) marked with "[OMSK]" 2 Normalized embeddings. In the context of contrastive learning, the training goal is to maximize the intra-class similarity Where p∈P(i), while minimizing the similarity between classes Where n∈N(i). Following the supervised contrastive learning model, the sample contrast loss can be directly derived:

[0050]

[0051] where τ is the temperature value. When considering multiple instances in P(i) and N(i), we will CL (i) Rewrite as follows:

[0052]

[0053] The collection This provides the model with more generalization ability as multiple intra-class and inter-class similarity values ​​are averaged, making the learned decision boundaries smoother.

[0054] Just minimize L CL (i) may be insufficient because it does not take sample difficulty into account. For example, if s i，p = 0.2 and s i，p′ = 0.95 where p, P'∈P(i). The model should pay more attention to s i，p To achieve the most optimized, and pay less attention to s i，p′ To avoid overfitting of the model. Therefore, in one embodiment, a pairwise relaxation factor can be introduced and a new loss function named pairwise cost-sensitive contrast loss (PCCL) is proposed, as shown below:

[0055]

[0056] in is the normalization factor:

[0057]

[0058] α i，p and α i，n is the pairwise relaxation factor, which is defined as follows:

[0059]

[0060] Compared with the original L CL Compared with (i), PCCL adds two new features. i，p and α i，n Internally, the marginal factor m is used to estimate s i，p>1-m and s i，n <m. Therefore, in s i，p and i，n There is a loose margin between α i，p and α i，n The use of also makes the model focus on learning difficult cases and avoid overfitting on easy cases. Another empirical setting is to use a separate temperature τ p and τ n To represent the similarity within and between classes, instead of a uniform temperature τ. We further set τ p =ξ·τ n (ξ>1) gives more relaxation to the positive samples so that the similarity within the class will not be too large, from which it is easy to conclude:

[0061]

[0062] in In this way, the small sample learner of the present invention will be less prone to overfitting the training examples.

[0063] Figure 4A -B shows an example of conventional contrastive learning and the present invention optimizing the distribution of positive and negative examples. As mentioned above, the present invention constrains the distance between groups by a marginal factor, so that Figure 4B As shown, the positive and negative examples are further apart from each other. p and τ n , so that the positive examples are not too close and the negative examples are not too far away.

[0064] Preferably, since the learning objective of PCCL is significantly different from that of the MLM task, only L PCCL (i) It may lead to catastrophic forgetting of pre-training knowledge. To this end, MLM can be regarded as an auxiliary task during few-shot learning to improve the performance of the model on previously unseen instances. Figure 5 A diagram showing a contrastive learning adjustment method according to an embodiment of the present invention is shown.

[0065] like Figure 5 As shown, the training of PLM can also include auxiliary training tasks of MLM. To this end, the processing method of the present invention can also include: masking the words in the sample to obtain a processed training sample with a masked text placeholder; obtaining the corresponding prediction result of the PLM for the masked text placeholder; and using a second loss function (corresponding to L MLM ) adjusts the neural network of the PLM, and the second loss function calculates the loss according to whether the corresponding prediction result of the masked text placeholder is the same as the masked word.

[0066] Denote the sample-level MLM loss as L MLM (i). Then the total loss function L(i) for the sample can be the weighted sum of the first loss function and the second loss function. In one example, the total loss function L(i) can be written as follows: L(i) = λ·L PCCL (i)+(1-λ)·L MIM (i), where λ is a predefined equilibrium overrange. Figure 5 In the example, the auxiliary MLM task is applied to the “[TMSK]” instruction ([TMSK] refers to the random mask in the text), and PCCL is applied to the “[OMSK]” instruction. This approach can be viewed as performing continuous pre-training and few-shot learning for a specific task at the same time.

[0067] During model inference, the embedding of test instance i can be directly used By using the K nearest neighbors in the small sample training set and Comparison to generate class labels When CP-Tuning is applied to a larger training set, for better scalability, the labels It is predicted by the following formula:

[0068]

[0069] Where C is the set of class labels, is the prototype embedding of class c∈C (i.e., the average embedding of all training instances with class label c).

[0070] To this end, the processing method of the present invention may also include: after the current iteration of PLM training for the current training sample set is completed, the embedding of all previous training samples is calculated, and the embedding average of each category is calculated as the class center. To this end, in the inference stage, the category of the data to be inferred is obtained according to the similarity between the current embedding output by PLM and the class center.

[0071] Furthermore, the class center of each class in the classification task for the training sample set can be obtained, and the class centers of the existing multiple classes can be stored in the storage network. The stored class centers can be updated after each iteration and can be used for determining the reasoning and prompt embedding. To this end, the processing method of the present invention can also include: inputting the training samples in the current sample set to be trained into the text neural network model to obtain the embedding for generating the prompt; sending the embedding for generating the prompt and the class center into the attention network; and obtaining the output of the attention network as the prompt of the current sample set to be trained. Figure 6 FIG. 2 shows a diagram of a contrastive learning adjustment method according to an embodiment of the present invention. Figure 5As shown, a task-adaptive cue encoder and a storage network are added. Figure 7 A schematic diagram showing the operation of the task-adaptive cue encoder.

[0072] Specific as Figure 6 and 7 The input of the task-adaptive prompt encoder consists of two parts: text input and the set of all centroids. The task-adaptive prompt encoder accepts both inputs and obtains the final prompt representation by calculating the attention of the text and the centroid. Specifically, TextCNN can be used to encode the text, and the result of text encoding (embedding) is used together with these centroids to calculate the attention result as the final input to PLM [PRO].

[0073] When performing data inference, after each epoch of model training, the model can automatically calculate the [EMB] representation of the training data and calculate the [EMB] average of each category, which is the center point, and store it in Figure 6 In the storage network shown in Figure 1, when the model is used for inference, the [EMB] of the data can be directly used to calculate the similarity with these center points, and the classification label can be output.

[0074] Thus, the model can support iterative incremental learning. After each iteration of model training, the center points of the corresponding classes can be recalculated. These center points can then be input into the task adaptive prompt encoder, so that the prompts for the next round of model training are more updated than the prompts for the previous round. To this end, the model can automatically learn the calculation process of the prompts during the learning process until the model converges.

[0075] The model trained in the above method is particularly suitable for text matching tasks. Here, text matching refers to the task of matching the similarity of two input texts, which is often used in information retrieval. For example, in a retrieval robot, the similarity between a given user question and a question in the knowledge base is evaluated. For this reason, the PLM trained in the above method can be implemented in an intelligent question-answering system. Figure 8 An example of using the PLM trained by the present invention for intelligent question answering is shown.

[0076] The intelligent question-answering system may include: a user input receiving unit, used to obtain the user's input query (for example, it may be a text model obtained by voice recognition, or a text model directly input by the user); a question matching unit, including a pre-trained model obtained by the method described above, wherein the model is used to match the input query with an existing question; and an answer providing unit, used to provide the answer to the matched existing question to the user.

[0077] Fig. 9A schematic diagram of the structure of a computing device that can be used to implement the above-mentioned pre-trained language model processing method based on contrastive learning according to an embodiment of the present invention is shown.

[0078] See also Fig. 9 , the computing device 900 includes a memory 910 and a processor 920 .

[0079] The processor 920 may be a multi-core processor or may include multiple processors. In some embodiments, the processor 920 may include a general-purpose main processor and one or more special coprocessors, such as a graphics processing unit (GPU), a digital signal processor (DSP), etc. In some embodiments, the processor 920 may be implemented using a customized circuit, such as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0080] The memory 910 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, ROM can store static data or instructions required by the processor 920 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at run time. In addition, the memory 910 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 910 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, minSD card, Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0081] The memory 910 stores executable codes. When the executable codes are processed by the processor 920, the processor 920 can execute the pre-trained language model processing method based on contrastive learning mentioned above.

[0082] The pre-trained language model processing method based on contrastive learning according to the present invention has been described in detail above with reference to the accompanying drawings. The processing method of the present invention avoids the need to manually construct the mapping between label words and classification categories in traditional PET and P-tuning algorithms by introducing contrastive learning. The implemented PCCL task reduces the sensitivity of the text to the prompt template, and selecting the same template for different matching tasks can achieve better results, thus solving the problem that the previous algorithm needs to manually construct the prompt template. In addition, by using the output representation of the pre-trained model MLM head before the softmax layer as a feature, the semantic information of the pre-trained model is retained and utilized to the maximum extent, and no new parameters are introduced compared to methods such as tuning or p-tuning. By using MLM as an auxiliary task, the overfitting problem in small sample learning is alleviated to a certain extent.

[0083] In addition, the method according to the present invention may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing the above steps defined in the above method of the present invention.

[0084] Alternatively, the present invention may also be implemented as a non-temporary machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or computing device, server, etc.), the processor executes the various steps of the above-mentioned method according to the present invention.

[0085] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or a combination of both.

[0086] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system and method according to multiple embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0087] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A pre-trained language model processing method based on contrastive learning, comprising: adding a hint placeholder and a mask output placeholder to the sample to obtain a processed training sample; The sample is text; Inputting the processed training sample into the pre-trained language model PLM to obtain an embedding that continuously represents the mask output; as well as Adjusting the neural network of the PLM with a first loss function, wherein the first loss function is used to minimize the intra-group distance of embeddings of samples of the same type and maximize the inter-group distance of embeddings of samples of different types; Among them, the first loss function is: ； is the normalization factor: ； and is the pairwise relaxation factor, defined as: ； Among them, P(i) is the positive set with the same label, and N(i) is the negative set with different labels; and is the temperature value; m is the marginal factor; is the intra-class similarity; is the similarity between classes.

2. The method of claim 1, further comprising: Masking the words in the sample to obtain processed training samples with masked text placeholders; Obtaining a corresponding prediction result of the PLM for the mask text placeholder; as well as The neural network of the PLM is adjusted using a second loss function, and the second loss function calculates the loss according to whether the corresponding prediction result of the masked text placeholder is the same as the masked word.

3. The method of claim 2, further comprising: The neural network of the PLM is adjusted according to a total loss function obtained by weighting the first loss function and the second loss function.

4. The method of claim 1, further comprising: After the current iteration of PLM training for the current training sample set is completed, the embeddings of all previous training samples are calculated, and the average embedding value of each category is calculated as the class center.

5. The method of claim 4, further comprising: In the inference stage, the category of the data to be inferred is obtained based on the similarity between the current embedding output by PLM and the class center.

6. The method of claim 4, further comprising: Find the class center of each class in the classification task for the training sample set; as well as The class centers of the existing multiple classes are stored in the storage network.

7. The method of claim 6, further comprising: Input the training samples in the current sample set to be trained into the text neural network model to obtain embeddings for generating prompts; Feeding the embedding used to generate the prompt and the class center into an attention network; The output of the attention network is obtained as a hint for the current sample set to be trained.

8. The method of claim 1, wherein: Adding hint placeholders and mask output placeholders to the sample to get the processed training sample includes: For a sample input as a sentence pair, the prompt placeholder and the mask output placeholder are added between two sentences to obtain a processed training sample.

9. An intelligent question-answering system, comprising: A user input receiving unit, used to obtain a user's input query; a question matching unit, comprising a pre-trained model obtained by the method of any one of claims 1 to 8, wherein the model is used to match the input query with an existing question; and The answer providing unit is used to provide the answer of the matching existing question to the user.

10. A computing device comprising: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 8.

11. A non-transitory machine-readable storage medium having executable codes stored thereon, which, when executed by a processor of an electronic device, causes the processor to execute the method according to any one of claims 1 to 8.

Images