Model generation and intelligent question answering method and system based on common sense graph

Abstract

The present invention provides a model generation and intelligent question-answering method and system based on a common sense graph, comprising the following steps: Step S1: Based on the common sense graph, question data is sequentially queried in the common sense graph to obtain relevant common sense of the question, which is used to construct an input sequence for the question-answering generation model; Step S2: The input sequence obtained based on the common sense graph is input into a teacher network to obtain classification probabilities, which are transferred as task knowledge and input into a student network in the form of soft labels for learning; Step S3: The same input sequence is input into the student network, and the obtained hard labels and the soft labels transferred by the teacher network are used as weights to calculate the classification loss and optimize, thereby obtaining a question-answering generation model based on the common sense graph; Step S4: Based on the question-answering generation model, an input sequence is constructed for user questions to generate answers. This invention improves the semantic understanding ability and reasoning accuracy of the question-answering model, while also meeting the real-time requirements of the question-answering system.

Description

Technical Field

[0001] The present invention relates to the technical field of model generation and intelligent question answering, and in particular to a method and system for model generation and intelligent question answering based on common sense graphs. Background Art

[0002] The present invention is applied to the COVID-19 medical knowledge generation and intelligent question-answering system, which provides intelligent answers to common-sense popular science and professional diagnostic knowledge on COVID-19 and epidemic prevention issues of concern to students, the elderly, ordinary adults, close contacts of COVID-19 and confirmed patients, thereby reducing the resource consumption of professional clinics and government citizen education.

[0003] There are currently relevant research results and technical solutions, which mainly use the following two technologies.

[0004] 1. Training of question-answering generation model

[0005] Traditional question-answering methods don't rely on predefined responses. Instead, the system automatically generates a response after receiving natural language input from the user. While this approach offers the advantage of flexible contextual integration, covering any topic and sentence structure, it often results in less-than-satisfactory conversational performance, with low-level errors such as incoherent sentences and grammatical errors. Therefore, training requires a large-scale training corpus and increased training times to mitigate or avoid these low-level errors, resulting in a significant learning cost.

[0006] In order to improve the performance of the question-answering system, researchers proposed the idea of ​​a question-answering generation model. Based on the pre-trained language model, they fine-tuned the model on the answering task to obtain a generation model to guide the generation of answers that meet a certain confidence level.

[0007] Both of the above methods use deep learning technology to perform feature representation on question and answer text or entity attributes, and then perform feature storage or model inference. However, deep learning models require a large amount of training data. The collection of new crown common sense corpus is still in its infancy, and the cost of fine-tuning the language model commonly used in the medical field is high, resulting in certain obstacles to training a high-credibility new crown medical common sense question and answer model. Therefore, the combination of the Attention mechanism and the Seq2Seq model architecture is used to train the question and answer model. Compared with the pre-trained large language model, the demand for corpus data can be effectively reduced, and the training process is more flexible and simple, and different feature modules can be added to enhance the information content of the output text. Representative related results are as follows:

[0008] Xu Tongyu et al. proposed a Seq2Seq question-answering model based on word2vec and attention mechanism optimization ("Research on Intelligent Question-Answering Method for Rice Pests and Diseases Based on Word2vec and Attention-Seq2Seq", Xu Tongyu et al., Journal of Shenyang Agricultural University, 2019, 50(3): 378-384). The model segmented the input text to obtain a refined text sequence, then used the word2vec model to express the features of the input text sequence, and then used LSTM as the encoder-decoder to implement the encoder-decoder. The encoder representation of the input word vector sequence was extracted and the hidden unit was injected into the initial decoder output, and finally the question-answer matching was performed.

[0009] In her paper "Context-Assisted SEQ2SEQ Medical Guide Automatic Question-Answering System" (Master's Thesis, 2020), Liu Qingli implemented an encoder-decoder using a variant RNN-based model. First, the question-and-answer corpus (Q, A) between the patient and the medical guide was fed into a Seq2Seq model for training to generate a pre-trained model. However, the responses in the prediction phase did not sufficiently correspond to the actual human expression A of the medical guide. This is because the question content in Q itself makes it difficult for the model to generate an informative response. Therefore, by using keyword extraction and other techniques on the input question corpus to match the relevant background information in Q, this information was fed into the encoder for retraining with the question-and-answer corpus. This allowed the model to generate responses not only by referencing the content of the labeled answer A but also by incorporating relevant background knowledge. The result was a more informative, human-like automatic question-answering model with a medical background. During the model retraining process, the background information was extracted through the network and then weighted with the question content Q to obtain encoded features. These features were then summed with the question feature vector as context for optimizing the generation of A.

[0010] The BART (Bidirectional and Auto-Regressive Transformers) pre-training model proposed by Facebook AI Research in (Mike Lewis, Yinhan Liu, Naman Goyal et al. "BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension" in ACL 2020, pp. 7871–7880) is a Transformer with a built-in Attention module. It also follows a Seq2Seq architecture, consisting of an encoder and a decoder. The BART encoder is also a Transformer Encoder, responsible for encoding, calculating, and extracting bidirectional features from the input text. The BART decoder is a unidirectional feature autoregressive model composed of a Transformer Decoder. It uses a cross-multi-head attention aggregation operation and the hidden state results of the encoder's last layer to perform attention aggregation calculations, thereby generating text using an autoregressive approach based on unidirectional feature representation.

[0011] 2. Model Distillation

[0012] Model distillation transfers the representational capabilities of a large model to a small model, helping the small model achieve the inference accuracy of the large model, thereby overcoming the shortcomings of the large model such as slow inference speed during deployment. The following achievements are groundbreaking research and applications in model distillation.

[0013] Scholars Geoffrey Hinton et al. ("Distilling the Knowledge in a Neural Network" in the NIPS workshop, 2014) pioneered the use of a large model as a teacher network, using its prediction output to define and quantify "knowledge." In classification tasks, this is expressed as the predicted probabilities of each label category, thereby guiding the effective update of gradients for each category. This paper also demonstrated the application of distillation techniques to ensemble learning paradigms for large models targeting large datasets.

[0014] Patent document CN111126573A (application number: CN201911387562.5) discloses an improved model distillation method based on individual learning. The method first inputs a preselected training set into a teacher network, and generates representative samples and non-representative samples after screening by the teacher network; secondly, the first loss function for processing representative samples and the second loss function for processing non-representative samples of the student network are determined; finally, the preselected training set is input into the student network, and the trained student network is obtained by training with the first loss function and the second loss function.

[0015] However, traditional question-answering models are targeted at specific fields, and the scale of training corpus is relatively limited, which affects the model's reasoning ability; it cannot effectively handle multiple rounds of answers; the question-answering model parameters are huge, making it difficult to meet the requirements of real-time answers; finally, the traditional question-answering model training method has a relatively simple ability to extract question text representations, making it impossible for the model to reach the upper limit of its reasoning ability with the current number of parameters.

[0016] In order to overcome the above-mentioned shortcomings of traditional methods, the present invention trains the question-answering model based on the BART architecture, and incorporates structured data from common sense graphs and common sense libraries. This injects common sense knowledge related to the question into the question-answering model training, enabling accurate answers to be returned. Secondly, a prompt-based representation method and a feature representation of multiple rounds of answers are designed at the data input end of the encoder Enocder, optimizing the encoder's ability to represent contextual question-answering text. In addition, the distilled representation of the pre-trained large question-answering model is added to the model training process to guide the small question-answering model, achieving rapid convergence. Summary of the Invention

[0017] In response to the deficiencies in the prior art, the present invention aims to provide a method and system for model generation and intelligent question answering based on common sense graphs.

[0018] The model generation and intelligent question-answering method based on common sense graph provided by the present invention includes:

[0019] Step S1: Based on the common sense graph, the question data is sequentially queried in the common sense graph to obtain the relevant common sense of the question, which is used to construct the input sequence of the question-answering generation model;

[0020] Step S2: The input sequence obtained based on the common sense graph is input into the teacher network to obtain the classification probability, which is transferred as task knowledge and input into the student network in the form of soft labels for learning;

[0021] Step S3: Input the same input sequence into the student network, use the obtained hard labels and the soft labels delivered by the teacher network as weights, calculate the classification loss and optimize, and obtain a question-answering generation model based on the common sense graph;

[0022] Step S4: Based on the question-answer generation model, construct an input sequence for the user's question and generate the answer to the question;

[0023] The teacher network is the BART-QA fine-tuning model, and the student network is the BART-tiny pre-training model.

[0024] Preferably, the step S1 includes:

[0025] Step S1.1: For the input question, common sense extraction is performed based on the common sense entity extraction model and intent recognition model;

[0026] Step S1.2: Based on the common sense extraction results, the entity linking algorithm is further used to construct a query statement and perform query matching in the common sense graph;

[0027] The step S1.1 includes:

[0028] Step S1.1.1: Using the entity extraction model and the part-of-speech named entity recognition model, extract nouns from the input question text as common sense entities and nodes in the common sense graph query;

[0029] Step S1.1.2: Using the intent recognition model, identify the intent type of the input question text as an edge in the common sense graph query;

[0030] Step S1.1.3: Integrate the extraction results by extraction type as the final common sense extraction result;

[0031] In step S1.2, the entity linking algorithm shown in the following formula is used to make it easier for the extracted common sense results to find relevant knowledge information in the common sense graph;

[0032]

[0033] Where n is the shorter length of the query word and the candidate word, k is the total length of the query word and the candidate word, and p is the length of the query word and the candidate word. n Indicates the ratio of the number of matching query and candidate n-grams to the total number of n-grams. The index term will have a higher matching score for candidate words with longer matches.

[0034] Preferably, the step S2 includes:

[0035] Step S2.1: Based on the common sense graph query results, obtain the input sequence;

[0036] The input text of the input sequence will be represented as: Input = [CLS] Dialogue history: <question>Knowledge: <knowledge>[SEP], where the existing conversation history of multiple rounds of dialogue is used as <question>The knowledge obtained by querying the results of the current round of dialogue in the common sense graph is used as <knowledge>, thus forming a complete input sequence, where <question>Represents the question text in the question data; <knowledge>The information is composed of several answers obtained by matching common sense graph queries. The remaining sequence encoding and position encoding information will be learned by the model's learnable Embedding layer. The input sequence is obtained by adding the corresponding positions of the three types of encodings.

[0037] Step S2.2: Calculate the output classification probability of the teacher network and obtain the soft labels in the model distillation;

[0038]

[0039] in, The id of the special word to start prediction; E j represents the dictionary id value corresponding to the jth word in the input vector of the BART Encoder; i represents the number of words currently predicted, with the maximum being the length of the sentence of the true answer; concat(·) is the vector concatenation function; T(·) represents the score vector composed of the predicted scores of each category word obtained by the autoregressive output of the BART Decoder in the teacher network; W T is the projection function of the teacher network, which projects the hidden layer output of the current model into the category space; softmax(·) is the normalized activation function with temperature τ, Where q corresponds to the vocabulary size; h i+1 Represents the probability of each predicted word; argmax(·) will get the subscript with the largest value in the vector, corresponding to Indicates the id of the predicted word in the vocabulary;

[0040] Step S2.3: Select The predicted probability corresponding to the true category id in each row is used as the soft label soft_label=(h i+1 ) i , as the knowledge of the teacher network.

[0041] Preferably, step S3 includes:

[0042] Step S3.1: Input the input sequence feature E into the Encoder of the student network and calculate the hard label loss hard_loss of the student network;

[0043]

[0044] Where S(·) represents the score vector composed of the predicted scores of each category word obtained by the BART Decoder autoregressive method in the student network; W s is the projection function of the student network, which projects the hidden layer output of the current model into the category space; same formula as above It is composed of the id of each predicted word.

[0045] Then use the cross entropy loss function Used to calculate hard_loss and soft_loss, where j is the predicted sequence id; q is the maximum length of the predicted sequence; y is the true category id of each sequence, y j is the real word category id; is the probability that the predicted word is predicted as the true category, so,

[0046] Step S3.2: Combine the soft label loss soft_label of the teacher network to calculate the soft label loss of the student network.

[0047]

[0048] Step S3.3: Calculate the total loss of the student network;

[0049] loss=λ·soft_loss+(1-λ)·hard_loss

[0050] Step S3.4: Train the common sense graph-based question-answering generation model using the above total loss function until convergence;

[0051] Step S3.5: Use the student model as the final question-answering generation model.

[0052] Preferably, the step S4 includes: the question-answer generation model is a trained student network S; when Stop prediction when the prediction is the id corresponding to the end-of-sentence word, and get a sequence of candidate prediction answers. The output confidence of several answers is sorted and the one with the highest confidence is selected as the answer result.

[0053] The common sense graph-based model generation and intelligent question-answering system provided by the present invention includes:

[0054] Module M1: Based on the common sense graph, the question data is sequentially queried in the common sense graph to obtain the relevant common sense of the question, which is used to construct the input sequence of the question-answering generation model;

[0055] Module M2: The input sequence obtained based on the common sense graph is input into the teacher network to obtain the classification probability, which is transferred as task knowledge and input into the student network in the form of soft labels for learning;

[0056] Module M3: Inputs the same input sequence into the student network, uses the obtained hard labels and the soft labels delivered by the teacher network as weights, calculates the classification loss and optimizes it, and obtains a question-answering generation model based on the common sense graph;

[0057] Module M4: Based on the question-answer generation model, it constructs an input sequence for user questions and generates answers to them.

[0058] The teacher network is the BART-QA fine-tuning model, and the student network is the BART-tiny pre-training model.

[0059] Preferably, the module M1 includes:

[0060] Module M1.1: For input problems, common sense extraction is performed based on the common sense entity extraction model and intent recognition model;

[0061] Module M1.2: Based on the common sense extraction results, we further use the entity linking algorithm to construct query statements and perform query matching in the common sense graph;

[0062] The module M1.1 includes:

[0063] Module M1.1.1: Using the entity extraction model and the part-of-speech named entity recognition model, the extracted nouns from the input question text are used as common sense entities and nodes in the common sense graph query;

[0064] Module M1.1.2: Using the intent recognition model, the identified intent type for the input question text is used as an edge in the common sense graph query;

[0065] Module M1.1.3: Integrate the extraction results by extraction type as the final common sense extraction result;

[0066] The module M1.2 adopts the entity linking algorithm shown in the following formula, which makes it easier to find relevant knowledge information in the common sense graph after extracting common sense results;

[0067]

[0068] Where n is the shorter length of the query word and the candidate word, k is the total length of the query word and the candidate word, and p is the length of the query word and the candidate word. n Indicates the ratio of the number of matching query and candidate n-grams to the total number of n-grams. The index term will have a higher matching score for candidate words with longer matches.

[0069] Preferably, the module M2 includes:

[0070] Module M2.1: Obtain input sequence based on common sense graph query results;

[0071] The input text of the input sequence will be represented as: Input = [CLS] Dialogue history: <question>Knowledge: <knowledge>[SEP], where the existing conversation history of multiple rounds of dialogue is used as <question>The knowledge obtained by querying the results of the current round of dialogue in the common sense graph is used as <knowledge>, thus forming a complete input sequence, where <question>Represents the question text in the question data; <knowledge>The information is composed of several answers obtained by matching common sense graph queries. The remaining sequence encoding and position encoding information will be learned by the model's learnable Embedding layer. The input sequence is obtained by adding the corresponding positions of the three types of encodings.

[0072] Module M2.2: Calculate the output classification probability of the teacher network and obtain the soft labels in the model distillation;

[0073]

[0074]

[0075] in, The id of the special word to start prediction; E j represents the dictionary id value corresponding to the jth word in the input vector of the BART Encoder; i represents the number of words currently predicted, with the maximum being the length of the sentence of the true answer; concat(·) is the vector concatenation function; T(·) represents the score vector composed of the predicted scores of each category word obtained by the autoregressive output of the BART Decoder in the teacher network; W T is the projection function of the teacher network, which projects the hidden layer output of the current model into the category space; softmax(·) is the normalized activation function with temperature τ, Where q corresponds to the vocabulary size; h i+1 Represents the probability of each predicted word; argmax(·) will get the subscript with the largest value in the vector, corresponding to Indicates the id of the predicted word in the vocabulary;

[0076] Module M2.3: Selection The predicted probability corresponding to the true category id in each row is used as the soft label soft_label=(h i+1 ) i , as the knowledge of the teacher network.

[0077] Preferably, the module M3 includes:

[0078] Module M3.1: Input the input sequence feature E into the Encoder of the student network and calculate the hard label loss hard_loss of the student network;

[0079]

[0080] Where S(·) represents the score vector composed of the predicted scores of each category word obtained by the BART Decoder autoregressive method in the student network; W S is the projection function of the student network, which projects the hidden layer output of the current model into the category space; same formula as above It is composed of the id of each predicted word.

[0081] Then use the cross entropy loss function Used to calculate hard_loss and soft_loss, where j is the predicted sequence id; q is the maximum length of the predicted sequence; y is the true category id of each sequence, y j is the real word category id; is the probability that the predicted word is predicted as the true category, so,

[0082] Module M3.2: Combine the soft label loss soft_label of the teacher network to calculate the soft label loss of the student network.

[0083]

[0084] Module M3.3: Calculate the total loss of the student network;

[0085] loss=λ·soft_loss+(1-λ)·hard_loss

[0086] Module M3.4: Train the common sense graph-based question-answering generation model using the above total loss function until convergence;

[0087] Module M3.5: Use the student model as the final question-answering generation model.

[0088] Preferably, the module M4 includes: the question-answer generation model is a trained student network S; Stop prediction when the prediction is the id corresponding to the end-of-sentence word, and get a sequence of candidate prediction answers. The output confidence of several answers is sorted and the one with the highest confidence is selected as the answer result.

[0089] Compared with the prior art, the present invention has the following beneficial effects:

[0090] (1) The present invention uses the BART question-answering model as a teacher network to transfer knowledge to a question-answering model with much smaller parameters, thereby overcoming the problem of the traditional pre-training + fine-tuning paradigm causing the model to be large and the inference speed to be unable to meet the requirements of real-time question answering;

[0091] (2) This paper utilizes the knowledge injection of entity extraction model and common sense graph to construct an input sequence based on the prompt paradigm, thereby enhancing the semantic understanding ability of the question-answering model and improving the reasoning accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0093] Figure 1 This is an architecture diagram of a generative model and intelligent question-answering method based on common sense graphs in an embodiment of the present invention;

[0094] Figure 2 is the input sequence of the common sense graph-based question-answering generation model in an embodiment of the present invention;

[0095] Figure 3 Schematic diagram of knowledge transfer for a knowledge distillation task in an embodiment of the present invention;

[0096] Figure 4 Schematic diagram of the model generation and intelligent question-answering system based on common sense graph in an embodiment of the present invention. DETAILED DESCRIPTION

[0097] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0098] Example 1:

[0099] According to the present invention, a generative model and intelligent question-answering method based on common sense graphs are provided, including:

[0100] Step S1: Based on the common sense graph, the question data is sequentially queried in the common sense graph to obtain the relevant common sense of the question, which is used to construct the input sequence of the question-answering generation model;

[0101] Step S2: Based on the common sense graph, the input sequence is obtained and input into the teacher network to obtain the classification probability of temperature τ, which is transferred as task knowledge and input into the student network in the form of soft labels for learning;

[0102] Step S3: The same input sequence is input to the student network, and the obtained hard labels are weighted with the soft labels transferred by the teacher network. The classification loss is calculated and optimized to obtain a question-answering generation model based on the common sense graph.

[0103] Step S4: Based on the question-answer generation model, construct an input sequence for the user's question and generate the answer to the question.

[0104] The teacher network is based on the BART-QA fine-tuned model, and the student network is based on the BART-tiny pre-trained model.

[0105] Specifically, the step S1 adopts:

[0106] Step S1.1: For the input question, common sense extraction is performed based on the common sense entity extraction model and intent recognition model;

[0107] Step S1.2: Based on the common sense extraction results, the entity linking algorithm is further used to construct a query statement and perform query matching in the common sense graph;

[0108] Specifically, the step S1.1 adopts:

[0109] Step S1.1.1: Common sense entity extraction based on the entity extraction model;

[0110] Through entity extraction models, such as the part-of-speech named entity recognition model, the nouns extracted from the input question text are used as common sense entities and as "nodes" in the common sense graph query;

[0111] Step S1.1.2: Extract common sense relations based on the intent recognition model;

[0112] Through the intent recognition model, the identified intent type of the input question text is used as the "edge" in the common sense graph query;

[0113] Step S1.1.3: Integrate the extraction results by extraction type as the final common sense extraction result;

[0114] Specifically, step S1.2 uses the entity linking algorithm shown in the following formula, which makes it easier for the extracted common sense results to find relevant "knowledge" information in the common sense graph;

[0115]

[0116] Where n represents the shorter length of the query word and the candidate word, and p n Indicates the ratio of the number of matching query and candidate n-grams to the total number of n-grams (no repeated matches). The index term will give a higher matching score to candidate words with longer matches.

[0117] Specifically, the step S2 adopts:

[0118] Step S2.1: Based on the common sense graph query results, obtain the input sequence;

[0119] like Figure 2 As shown, the input text of the input sequence will be represented as "Input = [CLS] Dialogue History: <question>Knowledge: <knowledge>[SEP]", which mainly consists of the existing "dialogue history" of multiple rounds of dialogue as " <question>" and the question of the current round of dialogue, the "knowledge" obtained by querying the common sense graph is used as " <knowledge>", thus forming a complete input sequence, where " <question>" represents the question text in the question data; <knowledge>The information is composed of several answers obtained by matching common sense graph queries. The remaining sequence encoding and position encoding information will be learned by the model's learnable Embedding layer. The input sequence is obtained by adding the corresponding positions of the three types of encoding.

[0120] Step S2.2: Calculate the output classification probability of the teacher network and obtain the soft labels in the model distillation;

[0121]

[0122] in, The id of the special word to start prediction; E j express Figure 2 The dictionary id value corresponding to the jth word in the input vector of the BART Encoder shown in the figure; where i represents the number of words currently predicted, and the maximum is the length of the sentence of the true answer; concat(·) is the vector concatenation function; T(·) represents the score vector composed of the predicted scores of each category word obtained by the autoregressive output of the BART Decoder in the teacher network; W T is the projection function of the teacher network, which projects the hidden layer output of the current model into the category space; softmax(·) is the normalized activation function with temperature τ, as shown in the following formula, where q corresponds to the vocabulary size; h i+1 Represents the probability of each predicted word; argmax(·) will get the subscript with the largest value in the vector, corresponding to Indicates the id of the predicted word in the vocabulary;

[0123]

[0124] Specifically, the step S2.3 adopts: selecting The predicted probability corresponding to the true category id in each row is used as the soft label soft_label=(h i+1 ) i , as the knowledge of the teacher network.

[0125] Specifically, the step S3 adopts:

[0126] Step S3.1: Input the input sequence feature E into the Encoder of the student network and calculate the hard label loss hard_loss of the student network;

[0127]

[0128] S(·) represents the score vector composed of the predicted scores of each category word obtained by the BART Decoder autoregressive method in the student network; W S is the projection function of the student network, which projects the hidden layer output of the current model into the category space; The id of each predicted word; y is the true category id of each sequence; cross_entropy is the cross entropy loss function, and hard_loss is calculated as shown in the following formula, where j is the predicted sequence id; q is the maximum length of the predicted sequence; y j is the real word category id; The probability that the predicted word is the true category.

[0129]

[0130] Step S3.2: Figure 3 As shown, the soft label loss of the student network is calculated by combining the soft label loss soft_label of the teacher network;

[0131]

[0132] Step S3.3: Figure 3 As shown, calculate the total loss of the student network;

[0133] loss=λ·soft_loss+(1-λ)·hard_loss

[0134] Step S3.4: Train the common sense graph-based question-answering generation model using the above total loss function until convergence;

[0135] Step S3.5: Use the student model as the final question-answer generation model;

[0136] Specifically, the step S4 adopts:

[0137] Step S4.1: For the input question, common sense extraction is performed based on the common sense entity extraction model and the intent recognition model;

[0138] Step S4.2: Based on the common sense extraction results, an entity linking algorithm is used to construct a query statement, and the query is matched in the common sense graph to obtain common sense information related to the question;

[0139] Step S4.3: Construct an input sequence and generate answers based on the question-answering generation model;

[0140] Specifically, the step S4.1 adopts:

[0141] Step S4.1.1: Common sense entity extraction based on the entity extraction model;

[0142] Through entity extraction models, such as the part-of-speech named entity recognition model, the nouns extracted from the input question text are used as common sense entities and as "nodes" in the common sense graph query;

[0143] Step S4.1.2: Extract common sense relations based on the intent recognition model;

[0144] Through the intent recognition model, the identified intent type of the input question text is used as the "edge" in the common sense graph query;

[0145] Step S4.1.3: Integrate the extraction results by extraction type as the final common sense extraction result;

[0146] Specifically, the step S4.2 adopts the entity linking algorithm shown above Where n represents the shorter length of the query word and the candidate word, and p n Indicates the ratio of the number of matching query and candidate n-grams to the total number of n-grams (no repeated matches). The index term will give a higher matching score to candidate words with longer matches. This makes it easier for the extracted common sense results to find relevant "knowledge" information in the common sense graph.

[0147] Specifically, the step S4.3 adopts:

[0148] Step S4.3.1: Obtain an input sequence based on the common sense graph query results;

[0149] like Figure 1 As shown, the input sequence will be represented as "Input = [CLS] Dialogue History: <question>Knowledge: <knowledge>[SEP]", which mainly consists of the existing "dialogue history" of multiple rounds of dialogue as " <question>" and the question of the current round of dialogue, the "knowledge" obtained by querying the common sense graph is used as " <knowledge>", thus forming a complete input sequence, where " <question>" represents the question text in the question data; <knowledge>The information consists of several answers obtained through common sense graph query matching.

[0150] Step S4.3.2: Input sequence features and autoregressive results into the question-answer generation model to obtain the predicted answer result sequence;

[0151]

[0152] Among them, the question-answer generation model is S; when Stop prediction when the prediction is the id corresponding to the end-of-sentence word. The final predicted answer sequence is The output confidence of several answers is sorted and the one with the highest confidence is selected as the answer result.

[0153] According to the present invention, a model generation and intelligent question-answering system based on common sense graph is provided, comprising:

[0154] Module M1: User question input interface;

[0155] Module M2: Intent-based user question recognition, which extracts common sense information of interest to users contained in the model input questions;

[0156] Module M3: Question-answer generation model, which constructs query statements based on common sense information to obtain the relevant "knowledge" of the input question in the common sense graph; constructs input sequences based on the relevant "knowledge" of the question; and trains to obtain a generation model;

[0157] Module M4: Answer generation, which generates predicted answers based on the user question recognition model and the question-answer generation model;

[0158] Module M5: Common sense maps, as the basis for acquiring medical common sense;

[0159] Module M6: Answer display interface.

[0160] The three core modules are detailed below:

[0161] In detail, the module M2 adopts:

[0162] Module M2.1: Using an entity extraction model, such as a part-of-speech named entity recognition model, extracts nouns from the input question text as common sense entities and uses them as "nodes" in the common sense graph query.

[0163] Module M2.2: Using the intent recognition model, the identified intent type is used as an edge in the common sense graph query for the input question text.

[0164] Module M2.3: Finally, the extraction results are integrated according to the extraction type as the final common sense extraction results;

[0165] In detail, the module M3 adopts:

[0166] Module M3.1: Based on the common sense extraction results, a simple entity linking algorithm is used to construct query statements, and query matching is performed on the common sense graph to obtain "knowledge" information related to the question;

[0167]

[0168] The entity linking algorithm shown in the above formula is used to make it easier for the extracted common sense results to find relevant "knowledge" information in the common sense graph. n represents the shorter length of the query word and the candidate word, p n Indicates the ratio of the number of matching query and candidate n-grams to the total number of n-grams (no repeated matches). The index term will give a higher matching score to candidate words with longer matches.

[0169] In detail, the module M4 adopts:

[0170] Module S4.1: Construct input sequences and complete answer prediction based on the question-answer generation model;

[0171] like Figure 2 As shown, the input text of the input sequence will be represented as "Input = [CLS] Dialogue History: <question>Knowledge: <knowledge>[SEP]", which mainly consists of the existing "dialogue history" of multiple rounds of dialogue as " <question>" and the question of the current round of dialogue, the "knowledge" obtained by querying the common sense graph is used as " <knowledge>", thus forming a complete input sequence, where " <question>" represents the question text in the question data; <knowledge>The information is composed of several answers obtained by matching common sense graph queries. The remaining sequence encoding and position encoding information will be learned by the model's learnable Embedding layer. The input sequence is obtained by adding the corresponding positions of the three types of encoding.

[0172] Module M4.2: Input sequence features and autoregressive results into the question-answer generation model to obtain the predicted answer result sequence

[0173] Among them, the question-answering generation model is the trained student network S; when Stop prediction when the prediction is the id corresponding to the end-of-sentence word, and get a sequence of candidate prediction answers. Sort by the output confidence of several answers and select the one with the highest confidence as the answer result

[0174] Example 2:

[0175] The present invention also provides a model generation and intelligent question answering system based on common sense graph, which is mainly implemented based on the method provided in the above embodiment, such as Figure 4 As shown, the core modules of the system include:

[0176] The model building unit is used to build a multi-granularity extraction model for common sense information extraction, including: a named entity recognition model and an intent recognition model; a question-answering generation model for answer prediction;

[0177] The training unit is applied to the training phase; the training phase includes: performing multi-granular common sense extraction on the input question based on the common sense entity extraction model and the intent recognition model, taking the input question text, and using the extracted nouns as common sense entities and intent types, as "nodes" and "edges" in the common sense graph query; for the common sense extraction results, further adopting a simple entity linking algorithm to construct a query statement, and performing query matching in the common sense graph, so that the extracted common sense results are easier to find relevant "knowledge" information in the common sense graph; the algorithm makes it easier for the extracted common sense results to find relevant "knowledge" information in the common sense graph; and then obtaining the input sequence "Input = [CLS] Dialogue history:" based on the common sense graph query results. <question>Knowledge: <knowledge>[SEP]"; calculate the output classification probability of the teacher network and obtain the soft label in the model distillation as the knowledge of the teacher network; input the input sequence feature E into the encoder of the student network and calculate the hard label loss hard_loss of the student network; combine the soft label loss soft_label of the teacher network to calculate the soft label loss soft_loss of the student network, and weight it with the hard label loss to calculate the total loss of the student network; train the question-answering generation model based on the common sense graph using the above total loss function until convergence, and use the student model as the final question-answering generation model;

[0178] The intelligent question-answering unit is applied in the testing phase; the testing phase includes: performing multi-granular common sense extraction on the input question based on the common sense entity extraction model and the intent recognition model, taking the input question text, and using the extracted nouns as common sense entities and intent types, as "nodes" and "edges" in the common sense graph query; further adopting a simple entity linking algorithm to construct a query statement for the common sense extraction result, and performing query matching in the common sense graph, so that the extracted common sense results are easier to find relevant "knowledge" information in the common sense graph; the algorithm makes it easier for the extracted common sense results to find relevant "knowledge" information in the common sense graph; and then obtaining the input sequence "Input = [CLS] Dialogue history:" based on the common sense graph query result. <question>Knowledge: <knowledge>[SEP]"; input sequence features and autoregressive results into the question-answer generation model to obtain several candidate predicted answer result sequences The output confidence of several answers is sorted and the one with the highest confidence is selected as the answer result.

[0179] Those skilled in the art will appreciate that, in addition to implementing the system, device, and various modules provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, and the like by logically programming the method steps. Therefore, the system, device, and various modules provided by the present invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; the modules for implementing various functions can also be considered both software programs for implementing the method and structures within the hardware component.

[0180] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.< / knowledge> < / question> < / knowledge> < / question> < / knowledge> < / question> < / knowledge> < / question> < / knowledge> < / question> < / knowledge> < / question> < / knowledge> < / question> < / knowledge> < / question> < / knowledge> < / question> < / knowledge> < / question> < / knowledge> < / question> < / knowledge> < / question> < / knowledge> < / question> < / knowledge> < / question> < / knowledge> < / question> < / knowledge> < / question> < / knowledge> < / question>

Claims

1. A model generation and intelligent question answering method based on common sense graph, characterized in that: include: Step S1: Based on the common sense graph, the question data is sequentially queried in the common sense graph to obtain the relevant common sense of the question, which is used to construct the input sequence of the question-answering generation model; Step S2: The input sequence obtained based on the common sense graph is input into the teacher network to obtain the classification probability, which is transferred as task knowledge and input into the student network in the form of soft labels for learning; Step S3: Input the same input sequence into the student network, use the obtained hard labels and the soft labels delivered by the teacher network as weights, calculate the classification loss and optimize, and obtain a question-answering generation model based on the common sense graph; Step S4: Based on the question-answer generation model, construct an input sequence for the user's question and generate the answer to the question; The teacher network is a BART-QA fine-tuned model, and the student network is a BART-tiny pre-trained model; The step S2 comprises: Step S2.1: Based on the common sense graph query results, obtain the input sequence; The input text of the input sequence will be represented as: Input = [CLS] Dialogue history: <question>Knowledge: <knowledge>[SEP], where the existing conversation history of multiple rounds of dialogue is used as <question>The knowledge obtained by querying the results of the current round of dialogue in the common sense graph is used as <knowledge>, thus forming a complete input sequence, where <question>Represents the question text in the question data; <knowledge> The information is composed of several answers obtained by matching common sense graph queries. The remaining sequence encoding and position encoding information will be learned by the model's learnable Embedding layer. The input sequence is obtained by adding the corresponding positions of the three types of encodings.< / knowledge> < / question> < / knowledge> < / question> < / knowledge> < / question> Step S2.2: Calculate the output classification probability of the teacher network and obtain the soft labels in the model distillation; in, The id of the special word to start prediction; E j represents the dictionary id value corresponding to the jth word in the input vector of the BART Encoder; i represents the number of words currently predicted, with the maximum being the length of the sentence of the true answer; concat(·) is the vector concatenation function; T(·) represents the score vector composed of the predicted scores of each category word obtained by the autoregressive output of the BART Decoder in the teacher network; W T is the projection function of the teacher network, which projects the hidden layer output of the current model into the category space; softmax(·) is the normalized activation function with temperature τ, Where q corresponds to the vocabulary size; h i+1 Represents the probability of each predicted word; argmax(·) will get the subscript with the largest value in the vector, corresponding to Indicates the id of the predicted word in the vocabulary; Step S2.3: Select The predicted probability corresponding to the true category id in each row is used as the soft label soft_label=(h i+1 ) i , as the knowledge of the teacher network.

2. The common sense graph-based model generation and intelligent question answering method according to claim 1, characterized in that: The step S1 comprises: Step S1.1: For the input question, common sense extraction is performed based on the common sense entity extraction model and intent recognition model; Step S1.2: Based on the common sense extraction results, the entity linking algorithm is further used to construct a query statement and perform query matching in the common sense graph; The step S1.1 includes: Step S1.1.1: Using the entity extraction model and the part-of-speech named entity recognition model, extract nouns from the input question text as common sense entities and nodes in the common sense graph query; Step S1.1.2: Using the intent recognition model, identify the intent type of the input question text as an edge in the common sense graph query; Step S1.1.3: Integrate the extraction results by extraction type as the final common sense extraction result; In step S1.2, the entity linking algorithm shown in the following formula is used to make it easier for the extracted common sense results to find relevant knowledge information in the common sense graph; Where n is the shorter length of the query word and the candidate word, k is the total length of the query word and the candidate word, and p is the length of the query word and the candidate word. n Indicates the ratio of the number of matching query and candidate n-grams to the total number of n-grams. The index term will have a higher matching score for candidate words with longer matches.

3. The common sense graph-based model generation and intelligent question answering method according to claim 1, characterized in that: The step S3 comprises: Step S3.1: Input the input sequence feature E into the Encoder of the student network and calculate the hard label loss hard_loss of the student network; Where S(·) represents the score vector composed of the predicted scores of each category word obtained by the BART Decoder autoregressive method in the student network; W S is the projection function of the student network, which projects the hidden layer output of the current model into the category space; same formula as above It is composed of the id of each predicted word. Then use the cross entropy loss function Used to calculate hard_loss and soft_loss, where j is the predicted sequence id; q is the maximum length of the predicted sequence; y is the true category id of each sequence, y j is the real word category id; is the probability that the predicted word is predicted as the true category, so, Step S3.2: Combine the soft label loss soft_label of the teacher network to calculate the soft label loss of the student network. Step S3.3: Calculate the total loss of the student network; loss=λ·soft_loss+(1-λ)·hard_loss Step S3.4: Train the common sense graph-based question-answering generation model using the above total loss function until convergence; Step S3.5: Use the student model as the final question-answering generation model.

4. The common sense graph-based model generation and intelligent question answering method according to claim 1, characterized in that: The step S4 includes: the question-answer generation model is a trained student network S; Stop prediction when the prediction is the id corresponding to the end-of-sentence word, and get a sequence of candidate prediction answers. The output confidence of several answers is sorted and the one with the highest confidence is selected as the answer result.

5. A model generation and intelligent question answering system based on common sense graph, characterized in that: include: Module M1: Based on the common sense graph, the question data is sequentially queried in the common sense graph to obtain the relevant common sense of the question, which is used to construct the input sequence of the question-answering generation model; Module M2: The input sequence obtained based on the common sense graph is input into the teacher network to obtain the classification probability, which is transferred as task knowledge and input into the student network in the form of soft labels for learning; Module M3: Inputs the same input sequence into the student network, uses the obtained hard labels and the soft labels delivered by the teacher network as weights, calculates the classification loss and optimizes it, and obtains a question-answering generation model based on the common sense graph; Module M4: Based on the question-answer generation model, it constructs an input sequence for user questions and generates answers to them. The teacher network is a BART-QA fine-tuned model, and the student network is a BART-tiny pre-trained model; The module M2 includes: Module M2.1: Obtain input sequence based on common sense graph query results; The input text of the input sequence will be represented as: Input = [CLS] Dialogue history: <question>Knowledge: <knowledge>[SEP], where the existing conversation history of multiple rounds of dialogue is used as <question>The knowledge obtained by querying the results of the current round of dialogue in the common sense graph is used as <knowledge>, thus forming a complete input sequence, where <question>Represents the question text in the question data; <knowledge> The information is composed of several answers obtained by matching common sense graph queries. The remaining sequence encoding and position encoding information will be learned by the model's learnable Embedding layer. The input sequence is obtained by adding the corresponding positions of the three types of encodings.< / knowledge> < / question> < / knowledge> < / question> < / knowledge> < / question> Module M2.2: Calculate the output classification probability of the teacher network and obtain the soft labels in the model distillation; in, The id of the special word to start prediction; E j represents the dictionary id value corresponding to the jth word in the input vector of the BART Encoder; i represents the number of words currently predicted, with the maximum being the length of the sentence of the true answer; concat(·) is the vector concatenation function; T(·) represents the score vector composed of the predicted scores of each category word obtained by the autoregressive output of the BART Decoder in the teacher network; W T is the projection function of the teacher network, which projects the hidden layer output of the current model into the category space; softmax(·) is the normalized activation function with temperature τ, Where q corresponds to the vocabulary size; h i+1 Represents the probability of each predicted word; argmax(·) will get the subscript with the largest value in the vector, corresponding to Indicates the id of the predicted word in the vocabulary; Module M2.3: Selection The predicted probability corresponding to the true category id in each row is used as the soft label soft_label=(h i+1 ) i , as the knowledge of the teacher network.

6. The common sense graph-based model generation and intelligent question-answering system according to claim 5, characterized in that: The module M1 includes: Module M1.1: For input problems, common sense extraction is performed based on the common sense entity extraction model and intent recognition model; Module M1.2: Based on the common sense extraction results, we further use the entity linking algorithm to construct query statements and perform query matching in the common sense graph; The module M1.1 includes: Module M1.1.1: Using the entity extraction model and the part-of-speech named entity recognition model, the extracted nouns from the input question text are used as common sense entities and nodes in the common sense graph query; Module M1.1.2: Using the intent recognition model, the identified intent type for the input question text is used as an edge in the common sense graph query; Module M1.1.3: Integrate the extraction results by extraction type as the final common sense extraction result; The module M1.2 adopts the entity linking algorithm shown in the following formula, which makes it easier to find relevant knowledge information in the common sense graph after extracting common sense results; Where n is the shorter length of the query word and the candidate word, k is the total length of the query word and the candidate word, and p is the length of the query word and the candidate word. n Indicates the ratio of the number of matching query and candidate n-grams to the total number of n-grams. The index term will have a higher matching score for candidate words with longer matches.

7. The common sense graph-based model generation and intelligent question-answering system according to claim 5, characterized in that: The module M3 includes: Module M3.1: Input the input sequence feature E into the Encoder of the student network and calculate the hard label loss hard_loss of the student network; Where S(·) represents the score vector composed of the predicted scores of each category word obtained by the BART Decoder autoregressive method in the student network; W S is the projection function of the student network, which projects the hidden layer output of the current model into the category space; same formula as above It is composed of the id of each predicted word. Then use the cross entropy loss function Used to calculate hard_loss and soft_loss, where j is the predicted sequence id; q is the maximum length of the predicted sequence; y is the true category id of each sequence, y j is the real word category id; is the probability that the predicted word is predicted as the true category, so, Module M3.2: Combine the soft label loss soft_label of the teacher network to calculate the soft label loss of the student network. Module M3.3: Calculate the total loss of the student network; loss=λ·soft_loss+(1-λ)·hard_loss Module M3.4: Train the common sense graph-based question-answering generation model using the above total loss function until convergence; Module M3.5: Use the student model as the final question-answering generation model.

8. The common sense graph-based model generation and intelligent question-answering system according to claim 5, characterized in that: The module M4 includes: the question-answer generation model is a trained student network S; Stop prediction when the prediction is the id corresponding to the end-of-sentence word, and get a sequence of candidate prediction answers. The output confidence of several answers is sorted and the one with the highest confidence is selected as the answer result.

Images