A medical conversation positive-negative discrimination method based on prompt learning
By using a prompt-based learning approach, templates and vocabulary are designed to train a pre-trained language model, solving the problem that existing technologies cannot fully utilize the knowledge of pre-trained models and achieving efficient gender discrimination in medical dialogues.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SOUTHWEST JIAOTONG UNIV
- Filing Date
- 2023-08-31
- Publication Date
- 2026-07-03
AI Technical Summary
Existing methods for determining the masculinity of medical dialogues cannot fully utilize the prior knowledge of pre-trained language models, ignore the independent relationship between context and entities, and consume a large amount of computational resources.
We employ a prompt-based learning approach, designing hard-coded and soft-coded templates, and combining them with hand-crafted and soft vocabularies to train a pre-trained language model. Through template and vocabulary reconstruction tasks, we fully utilize the prior knowledge of the pre-trained model.
It improves the accuracy of positive/negative identification in medical dialogues, reduces the consumption of computing resources, and achieves effective classification of context and entities.
Smart Images
Figure CN117056830B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical dialogue, and in particular to a method for determining the positive or negative status of medical dialogue based on cue learning. Background Technology
[0002] Traditional methods for determining gender in medical dialogues typically employ a fine-tuning approach based on pre-trained models. This is also the mainstream paradigm for solving natural language tasks. Large-scale pre-trained models, due to their complex pre-training objectives and massive model parameters, can effectively acquire knowledge from large amounts of labeled and unlabeled data. By storing this knowledge in a large number of parameters and fine-tuning it for specific tasks, the rich hidden knowledge and massive parameter encoding can benefit downstream tasks. In traditional fine-tuning methods, additional objective functions are usually introduced into the pre-trained language model to adapt it to downstream tasks. However, this approach creates a certain disconnect between the pre-trained task and the downstream task, thus failing to fully utilize the prior knowledge contained in the pre-trained model. Furthermore, the need to introduce additional parameters during the fine-tuning stage to adapt to different task requirements can easily lead to overfitting with limited samples, significantly reducing generalization ability. The essence of fine-tuning is to change the weights of the pre-trained model. Since the pre-trained model already performs very well on the original pre-training task, transferring it to downstream tasks will inevitably be influenced by the original pre-training task. At the same time, to achieve better performance in downstream tasks, larger pre-trained models and more training data are required, and the pre-trained models need to be retrained multiple times, which leads to a large consumption of computing resources.
[0003] Existing discrimination methods cannot fully utilize the prior knowledge of pre-trained language models, ignore the independent relationship between context and entities, and consume a lot of computer resources. Summary of the Invention
[0004] To address the aforementioned shortcomings in existing technologies, this invention provides a medical dialogue positive / negative discrimination method based on cue learning. This method solves the problems that existing discrimination methods cannot fully utilize the prior knowledge of pre-trained language models, ignore the independent relationship between context and entities, and consume a large amount of computer resources.
[0005] To achieve the aforementioned objectives, the technical solution adopted by this invention is as follows: a method for determining the sex of a patient in medical dialogue based on prompting learning, comprising the following steps:
[0006] S1: Design a prompt template based on the characteristics of prompt learning and positive / negative judgment in medical dialogue. The prompt template includes the original input text x and a special token. The prompt text output after being packaged by the prompt template is T(x);
[0007] S2: Copy the entity [Ent] mentioned in the original input text x that needs to be predicted into the prompt text, and add a conjunction designed according to the characteristics of the task before and after the entity [Ent]. At the same time, add the identifier [MASK] as a special token that needs to be predicted, and complete the design of the prompt template.
[0008] S3: Design a vocabulary and use the vocabulary and prompt templates to train the pre-trained language model to obtain a prompt-based discrimination model, and complete the discrimination of positive and negative medical dialogues.
[0009] The beneficial effects of the above scheme are as follows: This invention designs hard-coded templates and soft-coded templates, and also designs manual vocabularies and soft vocabularies for the vocabulary. By introducing prompt learning and designing templates and vocabularies, the original task is reconstructed into a pre-training task, thereby making full use of the prior knowledge of the pre-trained language model. This solves the problems of existing discrimination methods being unable to make full use of the prior knowledge of the pre-trained language model, ignoring the independent relationship between context and entities, and consuming a lot of computer resources.
[0010] Furthermore, the prompt template in S2 includes a hard-coded template and a soft-coded template, wherein the hard-coded template is:
[0011] T1(x) = x. Patient [Ent] is [MASK]
[0012] T2(x) = x. Does the patient have [Ent]? [MASK]
[0013] T3(x) = x. Does the patient have symptoms? [Ent] [MASK]
[0014] T1(x), T2(x), and T3(x) are the outputs of three different hard-coded templates.
[0015] The beneficial effect of the above-mentioned further solutions is that, through the above technical solutions, three hard-coded templates are provided for the task of identifying symptoms in medical dialogues, so as to complete the classification of symptom entities in the context.
[0016] Furthermore, the soft-coding template is as follows:
[0017] T4(x)=x.[soft1][soft2]…[Ent][soft n [MASK]
[0018] Where T4(x) is the output of the soft-coded template, [soft1], [soft2], ..., [soft... n ] is a special identifier introduced, and n is a predefined hyperparameter.
[0019] The beneficial effect of the above-mentioned further solution is that, through the above technical solution, a soft coding strategy is adopted, and a soft coding template is designed by introducing some additional special identifiers.
[0020] Furthermore, the vocabulary in S3 includes both a hand-written vocabulary and a soft vocabulary.
[0021] The beneficial effects of the above-mentioned further scheme are: the vocabulary design mainly includes a handmade vocabulary and a soft vocabulary that participates in the model learning, which is mainly used for model training.
[0022] Furthermore, the hand-crafted vocabulary is obtained using a thesaurus search website. After constructing the hand-crafted vocabulary, when training the pre-trained language model, the confidence scores of all words in the vocabulary are used to construct the final scores of the labels. For the original input text x and its label category y, the conditional probability p(y|x) of the original input text x is:
[0023]
[0024] Where, λ j For the current word w j The importance parameter, j is the index of a different word in the vocabulary, m is the number of words in the vocabulary, p([MASK]=w j |T(x)) represents the word predicted at the [MASK] position. j The probability of.
[0025] The beneficial effect of the above-mentioned further solutions is that a hand-crafted vocabulary can be obtained through the above technical solutions, and the model can be trained using the hand-crafted vocabulary.
[0026] Furthermore, the soft vocabulary incorporates the embeddings of each label as part of the pre-trained language model learning process, resulting in continuous soft labels. During model training, the similarity between the embeddings output at the mask position and the embeddings of each label word is calculated, and the softmax function is used to obtain the probability p1(y|x) for each category:
[0027]
[0028] Where exp is an exponential function with base e. Let f(T(x)) be the embedding vector for the label category y, and let f(T(x)) be the output of the pre-trained language model. Let be the embedding vector of label i, and C be all categories.
[0029] The beneficial effect of the above further scheme is that when the soft vocabulary is input into the model for training, the probability of each category in softmax can be obtained through the above formula.
[0030] Furthermore, S3 employs the cross-entropy loss function L when training the pre-trained language model:
[0031] L=-log∑p(y|x;θ,φ,Θ)
[0032] Where p(y|x;θ,φ,Θ) represents the predicted probability of being classified as the label category y given the original input text x, the first model parameter θ, the second model parameter φ, and the third model parameter Θ.
[0033] The beneficial effect of the above-mentioned further scheme is that the training model is evaluated using the above loss function, the model is optimized, and the accuracy of the model output is improved. Attached Figure Description
[0034] Figure 1 This is a flowchart of a medical dialogue gender discrimination method based on prompt learning.
[0035] Figure 2 This is an example diagram of a medical dialogue for determining positive or negative results.
[0036] Figure 3 This is a framework diagram of a medical dialogue gender discrimination method based on prompt learning. Detailed Implementation
[0037] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.
[0038] like Figure 1 As shown, a method for determining the sex of a patient in medical conversation based on prompting learning includes the following steps:
[0039] S1: Design a prompt template based on the characteristics of prompt learning and positive / negative judgment in medical dialogue. The prompt template includes the original input text x and a special token. The prompt text output after being packaged by the prompt template is T(x);
[0040] S2: Copy the entity [Ent] mentioned in the original input text x that needs to be predicted into the prompt text, and add a conjunction designed according to the characteristics of the task before and after the entity [Ent]. At the same time, add the identifier [MASK] as a special token that needs to be predicted, and complete the design of the prompt template.
[0041] S3: Design a vocabulary and use the vocabulary and prompt templates to train the pre-trained language model to obtain a prompt-based discrimination model, and complete the discrimination of positive and negative medical dialogues.
[0042] The prompt template in S2 includes a hard-coded template and a soft-coded template. The hard-coded template is as follows:
[0043] T1(x) = x. Patient [Ent] is [MASK]
[0044] T2(x) = x. Does the patient have [Ent]? [MASK]
[0045] T3(x) = x. Does the patient have symptoms? [Ent] [MASK]
[0046] T1(x), T2(x), and T3(x) are the outputs of three different hard-coded templates.
[0047] The software coding template is:
[0048] T4(x)=x.[soft1][soft2]…[Ent][soft n [MASK]
[0049] Where T4(x) is the output of the soft-coded template, [soft1], [soft2], ..., [soft... n ] is a special identifier introduced, and n is a predefined hyperparameter.
[0050] The vocabulary in S3 includes both a handwritten vocabulary and a soft vocabulary.
[0051] The hand-crafted vocabulary was obtained using a thesaurus search website. After constructing the hand-crafted vocabulary, when training the pre-trained language model, the confidence scores of all words in the vocabulary were used to construct the final scores of the labels. For the original input text x and its label category y, the conditional probability p(y|x) of the original input text x is:
[0052]
[0053] Where, λ j For the current word w j The importance parameter, j is the index of a different word in the vocabulary, m is the number of words in the vocabulary, p([MASK]=w j |T(x)) represents the word predicted at the [MASK] position. j The probability of.
[0054] The soft vocabulary incorporates the embeddings of each label as part of the pre-trained language model learning process, resulting in continuous soft labels. During model training, the similarity between the embeddings output at the mask position and the embeddings of each label word is calculated, and the softmax function is used to obtain the probability p1(y|x) for each category:
[0055]
[0056] Where exp is an exponential function with base e. Let f(T(x)) be the embedding vector for the label category y, and let f(T(x)) be the output of the pre-trained language model. Let be the embedding vector of label i, and C be all categories.
[0057] Hard-coded and soft-coded template designs offer different template initializations, both of which can be parameterized by φ and optimized along with the pre-trained language model during training. The soft vocabulary searches for the optimal label representation in a continuous space through embeddings, and this part also participates in optimization along with the pre-trained language model.
[0058] In S3, the cross-entropy loss function L is used when training the pre-trained language model:
[0059] L=-log∑p(y|x;θ,φ,Θ)
[0060] Where p(y|x;θ,φ,Θ) represents the predicted probability of being classified as the label category y given the original input text x, the first model parameter θ, the second model parameter φ, and the third model parameter Θ.
[0061] In one embodiment of the present invention, such as Figure 2 The diagram illustrates an example of positive / negative identification in a medical dialogue. It shows an example of symptom identification in a medical dialogue. The patient initially states their symptoms, and the doctor then asks further questions. The patient's statement of "constant abdominal pain" is positive, so the symptom category is determined to be positive. When the doctor asks if the patient has "upper abdominal pain," the patient replies negatively, so the symptom category is negative based on the context. When the doctor asks if the patient has "stabbing pain," the patient is unsure, so the entity category is determined to be other.
[0062] like Figure 3 The diagram illustrates a framework for a medical dialogue positive / negative gender discrimination method based on cue learning. The example output, "The patient's stomach has been hurting because [MASK]", is a cue text output obtained from a hard-coded template. Here, [MASK] is a special token that needs to be predicted, thus transforming the original classification task into a [MASK] prediction task, predicting the probability distribution of [MASK] on the constructed vocabulary. The vocabulary is then designed. Taking a manually designed vocabulary as an example, a label vocabulary is first constructed for each original label. The sub-vocabulary is a subset of the pre-trained language model vocabulary. Then, by taking the union of the sub-vocabularies corresponding to each label, a complete vocabulary designed specifically for the task is obtained. The label vocabulary represents words associated with the label; for example, the vocabulary for the "other" label contains "fuzzy, unclear," etc., and the vocabulary for the "positive" label contains related words such as "yes, have," etc.
[0063] The discriminative model based on cue learning proposed in this invention does not design a complex network model or introduce too many additional training parameters. Instead, it leverages the simple and effective mechanism of cue learning, formalizing the target task into a pre-training task solely through template and vocabulary design, thereby fully utilizing the true capabilities of the pre-trained language model. This invention does not adapt the pre-trained language model to downstream tasks by introducing additional objective functions; instead, it redefines the form of downstream tasks to make them more similar to the tasks solved using textual cues during the training of the original pre-trained model.
[0064] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of the invention.
Claims
1. A method for determining the positive or negative of a medical dialogue based on prompt learning, characterized in that, Includes the following steps: S1: Design a prompt template based on the characteristics of prompt learning and positive / negative gender discrimination in medical dialogues. The prompt template includes the original input text. And a special token, the prompt text output after being wrapped in a prompt template is as follows ; S2: Transfer the original input text The document mentions copying the entity [Ent] that needs to be predicted into the prompt text, adding conjunctions designed according to the characteristics of the task before and after the entity [Ent], and adding the identifier [MASK] as a special token that needs to be predicted, thus completing the design of the prompt template; The prompt template in S2 includes a hard-coded template and a soft-coded template, wherein the hard-coded template is: in, , and Outputs for three different hard-coded templates; The soft-coding template is: in, For the output of the soft-coded template, For the introduction of special identifiers, These are predefined hyperparameters; S3: Design a vocabulary and use the vocabulary and prompt templates to train the pre-trained language model to obtain a prompt-based discrimination model to complete the discrimination of positive and negative medical dialogues; The vocabulary in S3 includes a manual vocabulary and a soft vocabulary; The hand-crafted vocabulary was obtained using a synonym lookup website. After constructing the vocabulary, when training the pre-trained language model, the confidence scores of all words in the vocabulary were used to construct the final scores of the labels for the original input text. and Tag categories Then the original input text conditional probability for: in, For the current word Importance parameters For indexing different words in the vocabulary, The number of words in the vocabulary. The word predicted at the [MASK] position is The probability of; The soft vocabulary incorporates the embeddings of each tag as part of the pre-trained language model learning process, resulting in continuous soft tags. During model training, the embeddings output at the mask position are used to calculate the similarity with the embeddings of each tag word, and a softmax function is calculated to obtain the probability of each category. for: in, For An exponential function with base 0. For tag categories Embedding vector, The output of the pre-trained language model, For tags Embedding vector, For all categories.
2. The method for determining the sex of a medical dialogue based on prompting learning according to claim 1, characterized in that, In step S3, the cross-entropy loss function is used when training the pre-trained language model. : in, Indicates that given the original input text First model parameters Second model parameters and the third model parameters In the case of, it is determined to be a tag category. The predicted probability.