Intelligent question-answering method and storage medium

Abstract

The application discloses an intelligent question-answering method, which comprises the following steps: constructing an intention recognition dataset and a semantic matching dataset based on question-answering data of problems in a selected professional field, constructing an intention recognition model based on the intention recognition dataset, and constructing a semantic matching model based on the semantic matching dataset. The application adopts a contrast learning technology to shorten the distance between positive sample pairs and lengthen the distance between negative sample pairs, thereby improving the accuracy of intention recognition. The application adopts a graph convolutional neural network technology to fuse syntactic features and text features obtained by BERT, thereby improving the semantic representation capability of the model for question texts and improving the semantic matching effect.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to an intelligent question-answering method and storage medium. Background Technology

[0002] With the continuous advancement of deep learning technology, deep learning-based question-answering systems have already found applications in daily life, such as Tmall Genie and Microsoft Xiaoice. Most existing question-answering systems are casual or fact-based; systems designed and applied to specialized fields such as law and healthcare are still relatively rare. Because these questions are complex and the knowledge within these fields is highly specialized, users cannot effectively filter answers when searching through historical question-and-answer materials. Therefore, an intelligent question-answering system is needed to provide services to users.

[0003] The current mainstream question answering method uses a large pre-trained language model (such as BERT) to fine-tune the tasks at each stage of the question answering system. With the help of the powerful text representation capabilities of the pre-trained model, this method has achieved good results, but there are still the following problems: (1) In terms of question intent recognition, the BERT model is affected by high-frequency words and maps questions with different intents to similar positions in the vector space, which increases the difficulty of distinguishing different intents and reduces the accuracy of intent recognition; (2) In terms of semantic matching, the BERT model fails to make full use of syntactic features, which is an important factor in representing semantics, affecting semantic representation and reducing the accuracy of semantic matching. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention provides an intelligent question-answering method for professional domain questions. It mainly employs techniques such as BERT model, contrastive learning, and graph convolutional neural networks, which significantly improve the accuracy of intent recognition and semantic matching.

[0005] The technical solution of the present invention mainly comprises the following steps:

[0006] S1: Based on multiple professional consulting websites, obtain question and answer data for questions according to the selected field type.

[0007] S2: Construct an intent recognition dataset using the acquired question-and-answer data, where the intent recognition dataset includes the description of the question and the category to which the question belongs.

[0008] S3: Based on the intent recognition dataset, combine the BERT model and contrastive learning method to train and build an intent recognition model.

[0009] In step S3, the specific steps for constructing the intent recognition model using the BERT model and contrastive learning method are as follows:

[0010] S3-1: Perform text vectorization and data augmentation on the input training data. Specifically, for a batch of training data Q = (Q1, Q2, ... Q... N The vectorized result E is obtained by passing Q through a BERT model with Dropout operation in two separate steps. (1) =(E1) (1) E2 (1) ,...E N (1) ) and E (2) =(E1) (2) E2 (2) ,...E N (2) N represents the number of sample data in a batch.

[0011] S3-2: Use KL divergence to constrain the distribution of data augmentation and calculate the corresponding KL loss. For Q i The two vectorization results E i (1) and E i (2) The formula for calculating KL loss is:

[0012] Where D KL (E (1) ||E (2) ) is E (1) For E (2) The KL divergence.

[0013] S3-3: Calculate the contrastive learning loss based on the contrastive learning method. In E (1) and E (2) In this model, two samples with the same legal category are considered positive sample pairs, and all other samples are considered negative sample pairs. The formula for calculating the contrastive learning loss is as follows:

[0014] in They represent E respectively i The positive and negative sample pairs are formed, τ is the weight hyperparameter, and sim(·) is the cosine similarity calculation function, which is used to measure the degree of similarity between two vectors.

[0015] S3-4: E (1) and E (2) Category prediction is performed through fully connected classification layers, and cross-entropy loss is calculated. The KL loss, contrastive learning loss, and cross-entropy loss are combined as the loss function for the entire intent recognition model, as shown in the formula:

[0016] Loss = CrossEntropy loss +α×Contrastloss +β×KL loss CrossEntropy loss Let α be the cross-entropy loss, and β be the weight hyperparameters of the contrastive loss and the KL loss, respectively.

[0017] S3-5: Train the model until it converges or reaches the set number of iterations, and select the best intent recognition model.

[0018] S4: Construct a semantic matching dataset using the obtained question-and-answer data. The data format of the semantic matching dataset is: {Question 1, Question 2, Similarity}, which includes two question texts and labels representing whether the two are similar.

[0019] S5: Based on the semantic matching dataset, the semantic matching model is trained and constructed by combining the BERT model and graph convolutional neural network, fusing syntactic features with BERT text features.

[0020] In step S5, the specific implementation steps for constructing the semantic matching model using the BERT model and graph convolutional neural network are as follows:

[0021] S5-1: Use the BERT model to vectorize the question text and obtain the word and sentence features of the question.

[0022] S5-2: Use the Natural Language Processing Toolkit LTP to perform dependency parsing on the sentence to obtain the dependency parsing tree of the sentence. Then, construct the adjacency matrix and dependency type matrix based on the dependency parsing tree as the syntactic features of the sentence.

[0023] S5-3: Construct a graph G representing the sentence, using the words in the sentence as nodes of the graph, the vector representation of the words as the values ​​of the graph nodes, and the incidence matrix and dependency type matrix as the edges and weights between nodes, respectively.

[0024] S5-4: Use the attention mechanism to calculate the importance of the neighboring nodes of a node in graph G when it is being updated. The calculation formula is:

[0025] Where a i,j These are the values ​​of points i and j in the adjacency matrix, s i s j The intermediate values ​​of points i and j are formed by concatenating the corresponding word features and dependency type matrix values; n represents the number of graph nodes.

[0026] S5-5: Input the graph G, word features, and attention matrix into the graph convolutional neural network for feature fusion to obtain the intermediate feature vector h. The calculation formula of the graph convolutional neural network is:

[0027] Where σ represents the ReLU activation function, and W and b represent the trainable matrix and bias, respectively. It is the value of the neighbor node j in the upper layer of the graph convolutional neural network.

[0028] S5-6: The intermediate feature vector h is fused with the sentence features to obtain the semantic representation vector u of the fused syntactic features of the text in question.

[0029] S5-7: Repeat steps S5-1 to S5-6 for another problem text to obtain the corresponding semantic representation vector v that integrates syntactic features. Concatenate vectors u, v, and their difference vector |uv|. Input the concatenated vector into a fully connected classification layer for semantic matching prediction, using cross-entropy as the loss function.

[0030] S5-8: Train the model until it converges or reaches the set number of iterations, and select the best semantic matching model.

[0031] S6: Enter the user's question / comment.

[0032] S7: Use an intent recognition model to identify user inquiries and obtain a corresponding set of historical question candidates.

[0033] S8: Use a semantic matching model to calculate the semantic similarity between the user's inquiry and each question in the historical question candidate set.

[0034] S9: Based on semantic similarity, the answer to the most similar historical question is used as the answer to the user's question, and the candidate set of historical questions is sorted from high to low semantic similarity and returned to the user.

[0035] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the intelligent question-answering method described above.

[0036] Technical features and effects of the present invention:

[0037] This invention employs contrastive learning technology to narrow the distance between positive sample pairs and widen the distance between negative sample pairs, thereby improving the accuracy of intent recognition. It also uses graph convolutional neural network technology to fuse syntactic features with text features obtained by BERT, thereby enhancing the model's ability to semantically represent problem texts and improving semantic matching performance. Detailed Implementation

[0038] The invention will now be described in further detail with reference to an embodiment concerning legal issues.

[0039] Example 1:

[0040] Step 1: Obtain legal Q&A data from multiple professional legal consultation websites.

[0041] We collected Q&A data from multiple legal consultation websites, such as FindLaw and China Legal Service Network, using web crawling technology. The collected data fields include question description, question time, question category, and question answer. "Question description" refers to the user's description of the legal issue they are consulting; "question time" refers to the time the user asked the question; "question category" refers to the legal category to which the question belongs, such as labor disputes, marriage and family law, or debt collection; and "question answer" refers to the answer provided by a professional lawyer.

[0042] Step 2: Construct an intent recognition dataset using the acquired legal question-and-answer data. The intent recognition dataset includes the description of the question and the legal category to which the question belongs.

[0043] Step 3: Based on the intent recognition dataset, train and build an intent recognition model by combining the BERT model and contrastive learning methods.

[0044] First, the input training data is vectorized into text and augmented. Specifically, for a batch of training data Q = (Q1, Q2, ... Q... N The vectorized result E is obtained by passing Q through a BERT model with Dropout operation in two separate steps. (1) =(E1) (1) E2 (1) ,...E N (1) ) and E (2) =(E1) (2) E2 (2) ,...E N (2) Then, KL divergence is used to constrain the distribution of data augmentation, and the corresponding KL loss is calculated. For Q i The two vectorization results E i (1) and E i (2) The formula for calculating KL loss is: Where D KL (E (1) ||E (2) ) is E (1) For E (2) The KL divergence. Next, based on the contrastive learning method, the contrastive learning loss is calculated, in E... (1) and E (2) In this model, two samples with the same legal category are considered positive sample pairs, and all other samples are considered negative sample pairs. The formula for calculating the contrastive learning loss is as follows:

[0045] in They represent E respectively i The positive and negative sample pairs are formed, where τ is the weight hyperparameter, and sim(·) is the cosine similarity calculation function used to measure the degree of similarity between two vectors. Finally, E (1) and E (2) Category prediction is performed through fully connected classification layers, and cross-entropy loss is calculated. The KL loss, contrastive learning loss, and cross-entropy loss are combined as the loss function for the entire intent recognition model, as shown in the formula:

[0046] Loss = CrossEntropy loss +α×Contrast loss +β×KL loss CrossEntropy loss The loss function is cross-entropy, and α and β are the weight hyperparameters of the contrastive loss and KL loss, respectively. The model is trained using the above loss function until it converges or reaches the set number of iterations, at which point training stops, and the best intent recognition model is selected.

[0047] Step 4: Construct a semantic matching dataset using the obtained legal question and answer data. The data format of the semantic matching dataset is: {Question 1, Question 2, Similarity}, which includes two question texts and labels representing whether the two are similar.

[0048] Step 5: Based on the semantic matching dataset, combine the BERT model and graph convolutional neural network to fuse syntactic features with BERT text features, train and build a semantic matching model.

[0049] First, the BERT model is used to vectorize the question text, obtaining word and sentence features. Then, the LTP (Language Processing Toolkit) is used to perform dependency parsing on the sentence, obtaining the dependency parsing tree. Based on the dependency parsing tree, an adjacency matrix and a dependency type matrix are constructed, serving as the sentence's syntactic features. Next, using the words in the sentence as nodes, the word vector representations as node values, and the adjacency matrix and dependency type matrix as edges and weights between nodes, a graph G representing the sentence is constructed. An attention mechanism is then used to calculate the importance of a node's neighbors during updates in graph G, using the following formula: Where a i,j These are the values ​​of points i and j in the adjacency matrix, s i s j The intermediate values ​​of points i and j are concatenated from the corresponding word features and dependency type matrix values. Next, the graph G, word features, and attention matrix are input into a graph convolutional neural network for feature fusion to obtain the intermediate feature vector h. The calculation formula for the graph convolutional neural network is: Where σ represents the ReLU activation function, and W and b represent the trainable matrix and bias, respectively. This refers to the value of the neighbor node j in the previous layer of the graph convolutional neural network. Finally, the intermediate feature vector h is fused with the sentence features to obtain the semantic representation vector u, which is the fused syntactic feature of the text in question.

[0050] Repeat the above operation for another question text in the data sample to obtain the corresponding semantic representation vector v that integrates syntactic features. Concatenate vectors u, v, and their difference vector |uv|, and input the concatenated vector into a fully connected classification layer for semantic matching prediction. Use cross-entropy as the loss function for training. Stop training when the model converges or reaches the set number of iterations, and select the best semantic matching model.

[0051] Step 6: Enter the user's question / complaint.

[0052] Step 7: Use the intent recognition model to identify user inquiries and obtain a corresponding set of historical question candidates.

[0053] Step 8: Use a semantic matching model to calculate the semantic similarity between the user's inquiry and each question in the historical question candidate set.

[0054] Preferably, to accelerate semantic similarity retrieval and calculation, this method can pre-use a semantic matching model to obtain corresponding semantic representation vectors for all historical questions, and store the resulting vectors in the Faiss vector retrieval library. When a user inputs a question, the semantic matching model only needs to be used once to obtain the corresponding semantic vector, and then this vector is calculated against the vectors stored in Faiss corresponding to the historical question candidate set. This avoids repeatedly generating semantic vectors for historical questions and speeds up the retrieval response.

[0055] Step 9: Based on semantic similarity, select the answer to the most similar historical question as the answer to the user's question, and sort the candidate set of historical questions from high to low semantic similarity, and return them to the user together.

Claims

1. An intelligent question-answering method, comprising: An intent recognition dataset and a semantic matching dataset are constructed based on question-and-answer data from selected professional fields. An intent recognition model is built based on the intent recognition dataset, and a semantic matching model is built based on the semantic matching dataset. A user's inquiry is input, and the intent recognition model is applied to the inquiry to obtain a corresponding set of historical question candidates. The semantic matching model is then used to calculate the semantic similarity between the user's inquiry and each question in the historical question candidate set. Based on the semantic similarity, the answer to the most similar historical question is used as the answer to the user's inquiry. The historical question candidate set is then sorted from highest to lowest semantic similarity and returned to the user. The construction of the intent recognition model includes the following steps: (1) Perform text vectorization and data augmentation on the input training data: For a batch of training data ,Will The vectorized results are obtained twice, sequentially through a BERT model with Dropout operation. and ; Indicates the number of sample data in a batch; (2) Use KL divergence to constrain the distribution of data augmentation and calculate the corresponding KL loss: For Two vectorization results and The formula for calculating KL loss is: ,in for right KL divergence; (3) Calculate the contrastive learning loss based on the contrastive learning method: and In this model, two samples with the same legal category are considered positive sample pairs, and all other samples are considered negative sample pairs. The formula for calculating the contrastive learning loss is as follows: ,in , Respectively represent and The positive and negative sample pairs formed For weight hyperparameters, It is a cosine similarity calculation function used to measure the degree of similarity between two vectors; (4) and Category prediction is performed through fully connected classification layers, and cross-entropy loss is calculated. The KL loss, contrastive learning loss, and cross-entropy loss are combined as the loss function for the entire intent recognition model, as shown in the formula: ,in For cross-entropy loss, , These are the weight hyperparameters for the contrastive loss and the KL loss, respectively. The construction of the semantic matching model includes the following steps: (1) Use the BERT model to vectorize the question text to obtain the word features and sentence features of the question; (2) Use the Natural Language Processing Toolkit LTP to perform dependency parsing on the sentence to obtain the dependency parsing tree of the sentence. Then, construct the adjacency matrix and dependency type matrix based on the dependency parsing tree as the syntactic features of the sentence. (3) Construct a graph representing the sentence by using the characters in the sentence as nodes of the graph, the vector representation of the characters as the values ​​of the graph nodes, and the incidence matrix and dependency type matrix as the edges and weights between nodes, respectively. ; (4) Computing graphs using attention mechanisms The importance of a node's neighboring nodes during an update is calculated using the following formula: ,in Points on the adjacency matrix and The corresponding value, , They are points and The intermediate value is formed by concatenating the corresponding word features and dependency type matrix values. Indicates the number of nodes in the graph; (5) The diagram The word features and attention matrix are input into a convolutional neural network, where they are fused to obtain an intermediate feature vector. The calculation formula for a graph convolutional neural network is: ,in Represents the ReLU activation function. and Let these represent the trainable matrix and the bias, respectively. It is a neighboring node in the upper layer of the graph convolutional neural network. The value; (6) Transfer the intermediate feature vector By fusing features with sentence features, a semantic representation vector of fused syntactic features for the text in question is finally obtained. ; (7) Repeat steps (1) to (6) for another problem text to obtain the corresponding semantic representation vector that integrates syntactic features. , will vector ,vector and the difference vector between the two Vector concatenation is performed, and the concatenated vector is input into a fully connected classification layer for semantic matching prediction, using cross-entropy as the loss function; (8) Train the model until it converges or reaches the set number of iterations, and select the best semantic matching model.

2. A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the intelligent question-answering method of claim 1.

Images