A few-shot question answering method with prior knowledge

Abstract

The present invention discloses a method for question-answering with few samples that introduces prior knowledge, comprising the following steps: S101 prefabricates a knowledge graph of a few sample business domain; S102 obtains a user-side query; S103 copies the query in two copies, respectively inputs a character feature module based on query semantics and a word feature module integrating the business domain knowledge for feature extraction, and outputs the extracted character feature vector and word feature vector; S104 concatenates the character feature vector and the word feature vector, inputs a correlation calculation model enhanced by prior knowledge, and obtains a correlation score for each answer; S105 returns the answer with the highest correlation score to the user. The present invention provides a method for question-answering with few samples that introduces prior knowledge, so that the accuracy of the question-answering model is improved in the case of few samples.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a small sample question answering method that introduces prior knowledge. Background Art

[0002] The question-and-answer system is an intelligent product that meets the needs of B-side enterprises to reduce labor costs and improve question-and-answer efficiency. Compared with manual customer service, the question-and-answer system can accumulate data through user feedback in actual scenarios to improve itself. The accumulated data can also play a role in reducing costs and increasing efficiency in mining customer service value information.

[0003] A common question-and-answer method is to select the answer that best matches the current user query from the candidate answers. A high-performance question-and-answer model can accurately locate the answer that best matches the user query from the candidate answers, thereby improving the user experience. However, high question-and-answer performance often relies on a large number of high-quality question-and-answer samples, which means a lot of manpower and time costs.

[0004] However, for some more professional or niche business areas, there is often only a small amount of sample question and answer data, which results in the question and answer model being unable to fully learn the characteristics of business problems. As a result, there is a problem of poor question and answer results due to the small sample size. Summary of the invention

[0005] Purpose of the invention: The purpose of the present invention is to provide a few-sample question-answering method that introduces prior knowledge, so as to improve the accuracy of the question-answering model in the case of few samples.

[0006] Technical solution: A few-sample question answering method that introduces prior knowledge, including the following steps:

[0007] S101 prefabricates a few-sample business domain knowledge graph;

[0008] S102 obtains a user-side query;

[0009] S103: copy the query into two copies, input them into a character feature module based on query semantics and a word feature module integrating the business domain knowledge respectively for feature extraction, and output the extracted character feature vector and word feature vector;

[0010] S104 concatenates the character feature vector and the word feature vector, inputs the concatenation into a correlation calculation model enhanced by prior knowledge, and obtains a correlation score for each answer;

[0011] S105 returns the answer with the highest relevance score to the user.

[0012] The step S101 includes:

[0013] S201 collects the business points of the business field where the small sample is located;

[0014] S202 summarizes the categories of the business points and the relationships between the business points based on the expert experience in the business field;

[0015] S203 constructs a business domain knowledge graph by taking the business points as entities, taking the relationships between the business points as relationships between entities, and taking the categories of the business points and the relationships as ontology.

[0016] The word feature module extraction based on query semantics in step S103 includes:

[0017] S301: Segment the query into words;

[0018] S302: pre-process the word segmentation sequence and input it into the RoBERTa pre-training model to extract features, and obtain a feature matrix;

[0019] S303 outputs the first vector of the feature matrix as the character feature vector.

[0020] The word feature extraction integrating the business domain knowledge in step S103 includes:

[0021] S401 uses a word segmentation tool to segment the query, wherein the word segmentation tool introduces a custom dictionary of all entity names in the business domain knowledge graph;

[0022] S402: linking each word in the word segmentation sequence with an entity in the business domain knowledge graph, injecting triple information associated with the linked entity into the word segmentation sequence to obtain a word segmentation graph;

[0023] S403: input the word segmentation graph into the K-BERT pre-training model to extract features, and obtain a feature matrix;

[0024] S404 outputs the first vector of the feature matrix as the word feature vector.

[0025] The step S402 links each word in the word segmentation sequence with an entity in the business domain knowledge graph, including:

[0026] Count the number of occurrences of each word in the word to obtain the word bag;

[0027] Count the number of occurrences of each word in each entity name in the business domain knowledge graph to obtain a word bag for each entity;

[0028] The similarity between the word and the name of each entity in the business domain knowledge graph is calculated, and the similarity score is calculated by the following formula:

[0029]

[0030] Among them, simscore k is the similarity score between the kth entity and the word, n is the length of the union of the word bag and the entity word bag, x i is the number of occurrences of the i-th character in the word in the union, y i is the number of occurrences of the i-th word in the union in the name of the entity;

[0031] If the entity score with the highest similarity score exceeds the link threshold, it is determined that the word is linked to the entity.

[0032] The correlation calculation model in step S104 is:

[0033] S501: Input each of the answers into the character feature module for feature extraction, and output the character feature vector of the answer;

[0034] S502 inputs the concatenated vector into a first coding layer based on a self-attention mechanism for encoding, and outputs a first coding vector;

[0035] S503 concatenates the first encoding vector with the word feature vector of the answer, inputs the concatenation into a second encoding layer based on a self-attention mechanism for encoding, and outputs a second encoding vector;

[0036] S504: input the second encoding vector into a BatchNormalization normalization layer based on batch normalization for normalization to accelerate convergence, and output a normalized vector;

[0037] S505: Input the normalized vector into a 2-dimensional Softmax fully connected layer for classification, and the 2-dimensional classification probability value is the relevance score of the answer.

[0038] The process of enhancing the correlation calculation model with prior knowledge in step S104 includes:

[0039] Segmenting a small number of original samples with answer annotation information, wherein the answer annotation information is determined by business personnel based on the questions of the samples;

[0040] Linking each word in the word segmentation sequence with an entity in the business domain knowledge graph;

[0041] According to the ontology of the business domain knowledge graph, the successfully linked words are replaced with the names of other entities of the same entity type to obtain a large number of candidate enhancement samples;

[0042] The text of the original sample is subjected to the character feature extraction by the character feature module to extract the character feature vector of the original sample;

[0043] The candidate enhanced sample text is passed through the character feature module to extract the character features, and the feature vector extracted from the candidate sample is calculated with the character feature vector of the original sample. The vector distance is calculated by the following formula:

[0044]

[0045] Among them, x i and i are the values ​​of the original sample word feature vector and the candidate sample word feature vector in the i-th dimension respectively;

[0046] The candidate samples whose vector distance exceeds the vector distance threshold are discarded, and the remaining screened candidate samples constitute the prior knowledge enhanced sample data;

[0047] The correlation calculation model is trained by taking the question of the sample data enhanced with prior knowledge as input and taking the answer annotation information of the sample data enhanced with prior knowledge as target output.

[0048] Beneficial effect: Different from the prior art, the present invention provides a small-sample question-answering method that introduces prior knowledge, and uses a feature extraction module based on semantics and fusion knowledge to realize feature extraction of user queries by introducing prior knowledge, thereby solving the problem of insufficient feature learning of the question-answering model caused by the small sample size; using a correlation calculation model enhanced by prior knowledge, prior knowledge is introduced into sample data for data enhancement, so that the correlation calculation model can learn the feature information of the enhanced samples to adapt to the user query features in more situations and improve the accuracy of the question-answering model. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a flow chart of the present invention.

[0050] Figure 2 A schematic diagram of the process of prefabricating a knowledge graph of a small number of samples in a business field for the present invention.

[0051] Figure 3 It is a schematic diagram of the word feature extraction process based on query semantics of the present invention.

[0052] Figure 4 The figure is a flow chart of word feature extraction integrating the business domain knowledge in the present invention.

[0053] Figure 5 Schematic diagram of the flow of the correlation calculation model enhanced by prior knowledge of the present invention. DETAILED DESCRIPTION

[0054] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0055] See also Figure 1 , Figure 1 A schematic diagram of a few-sample question answering method with prior knowledge provided by the present invention. The method comprises:

[0056] S101 prefabricates a few-sample business domain knowledge graph;

[0057] S102 obtains a user-side query;

[0058] S103: copy the query into two copies, input them into a character feature module based on query semantics and a word feature module integrating the business domain knowledge respectively for feature extraction, and output the extracted character feature vector and word feature vector;

[0059] S104 concatenates the character feature vector and the word feature vector, inputs the concatenation into a correlation calculation model enhanced by prior knowledge, and obtains a correlation score for each answer;

[0060] S105 returns the answer with the highest relevance score to the user.

[0061] For further information, see Figure 2 In this embodiment, in step S101: pre-fabricate a few-sample business domain knowledge graph, the process includes:

[0062] S201 collects the business points of the business field where the small sample is located;

[0063] S202 summarizes the categories of the business points and the relationships between the business points based on the expert experience in the business field;

[0064] S203 constructs a business domain knowledge graph by taking the business points as entities, taking the relationships between the business points as relationships between entities, and taking the categories of the business points and the relationships as ontology.

[0065] The primary problem caused by a small sample size is that the business information in the sample is also scarce. If the question-answering model relies only on the text data of a small number of samples, it cannot fully learn the business information of the sample business field. The reason why a small sample business field knowledge graph needs to be prefabricated in this embodiment is to integrate business point information and the experience information of business field experts into the question-answering model to supplement the learned business information.

[0066] For further information, see Figure 3 In this embodiment, in step S103: extracting word features based on query semantics, the process includes:

[0067] S301: Segment the query into words;

[0068] S302: pre-process the word segmentation sequence and input it into the RoBERTa pre-training model to extract features, and obtain a feature matrix;

[0069] S303 outputs the first vector of the feature matrix as the character feature vector.

[0070] For example, in the power industry service assistant scenario:

[0071] Input user query "What time does the shuttle bus to the power company leave?"

[0072] Split the words to get the sequence ['go', 'to', 'electricity', 'power', 'company', 'of', 'shift', 'car', 'is', 'how many', 'point'];

[0073] Preprocess the word sequence by first adding '[CLS]' and '[SEP]' to the beginning and end of the sequence, and then convert_tokens_to_ids to get the id sequence [101,1343,2518,4510,1213,1062,1385,4638,4408,6756,3221,1126,4157,102];

[0074] Input the id sequence into the RoBERTa pre-trained model for feature extraction and obtain the matrix [[-5.9931e-01,6.4724e-01,…,3.9449e-01],…];

[0075] The first vector is output as the word feature vector.

[0076] The small sample size also leads to the problem of insufficient sample semantic information learned by the question-answering model, specifically, insufficient information about common Chinese semantics such as syntactic structure and daily words in the sample. In this embodiment, the character feature module based on query semantics essentially utilizes the learning of large-scale Chinese sentences by the RoBERTa pre-training model, and uses the prior knowledge of Chinese syntactic structure, daily words, etc. learned by the RoBERTa pre-training model to extract the semantic features of user query sentences, thereby supplementing the common Chinese semantic information in the question-answering model.

[0077] For further information, see Figure 4 In this embodiment, in step S103: extracting word features integrating the business domain knowledge, the process includes:

[0078] S401 segment the query using a segmentation tool;

[0079] S402 links each word in the word segmentation sequence with an entity in the business domain knowledge graph, and injects triple information (entity, relationship, entity) associated with the linked entity into the word segmentation sequence to obtain a word segmentation graph.

[0080] Specifically, the process of linking each word in the word segmentation sequence with an entity in the business domain knowledge graph includes:

[0081] Count the number of occurrences of each word in the word to obtain the word bag;

[0082] Count the number of occurrences of each word in each entity name in the business domain knowledge graph to obtain a word bag for each entity;

[0083] The similarity between the word and the name of each entity in the business domain knowledge graph is calculated, and the similarity score is calculated by the following formula:

[0084]

[0085] Among them, simscore k is the similarity score between the kth entity and the word, n is the length of the union of the word bag and the entity word bag, x i is the number of occurrences of the i-th character in the word in the union, y i is the number of occurrences of the i-th word in the union in the name of the entity;

[0086] If the entity score with the highest similarity score exceeds the link threshold, it is determined that the word is linked to the entity.

[0087] S403: input the word segmentation graph into the K-BERT pre-training model to extract features, and obtain a feature matrix;

[0088] S404 outputs the first vector of the feature matrix as the word feature vector.

[0089] For example, in the power industry service assistant scenario:

[0090] Input user query "What time does the shuttle bus to the power company leave?"

[0091] After adding the custom dictionary and using the pkuseg word segmentation tool, we get the sequence ['go to', 'power company', 'of', 'shuttle bus', 'is', 'what time'];

[0092] After entity linking, the sequence becomes ['go to', 'power company', 'of', 'shuttle service', 'is', 'what time'];

[0093] After the associated triples are injected, the sequence is in the form of a word segmentation graph of ['go to' (1), 'power company' (2), 'of' (3), 'shuttle service' (4), 'type' (5), 'service' (6), 'is' (5), 'what time' (6)], where the index position in the sequence is in brackets. Since BERT-like models use index position vectors to represent sequence positions to learn sequence positions, the word segmentation graph uses the "soft index" method of using the same index position vector to convert the graph structure into a sequence form.

[0094] The word segmentation graph is input into the K-BERT pre-training model for feature extraction, and the matrix [[-3.9386e-01,-7.8880e-01,…,5.3451e-02],…] is obtained;

[0095] The first vector is output as the word feature vector.

[0096] In this embodiment, a knowledge graph has been prefabricated in S101, and the word feature extraction that integrates the business domain knowledge in S103 is to inject the business knowledge of the graph structure into the question-answering model. Since the graph structure is essentially a graph structure, and the question-answering model is oriented to text-type sequence structure data, this embodiment uses word segmentation, linking and association triple injection operations to convert the business knowledge of the graph structure into a sequence index, thereby being able to integrate business domain knowledge information into user queries and expand the knowledge features of user queries.

[0097] For further information, see Figure 5 In this embodiment, in step S104: prior knowledge enhanced correlation calculation, the process includes:

[0098] S501: Input each of the answers into the character feature module for feature extraction, and output the character feature vector of the answer;

[0099] S502 inputs the concatenated vector into a first coding layer based on a self-attention mechanism for encoding, and outputs a first coding vector;

[0100] S503 concatenates the first encoding vector with the word feature vector of the answer, inputs the concatenation into a second encoding layer based on a self-attention mechanism for encoding, and outputs a second encoding vector;

[0101] S504: input the second encoding vector into a BatchNormalization normalization layer based on batch normalization for normalization to accelerate convergence, and output a normalized vector;

[0102] S505: Input the normalized vector into a 2-dimensional Softmax fully connected layer for classification, and the 2-dimensional classification probability value is the relevance score of the answer.

[0103] For example, in the power industry service assistant scenario:

[0104] The candidate answers "Shuttle service schedule: ...", "Car maintenance time: ...", ... are input into the word feature module for feature extraction to obtain word feature vectors wv respectively. 1 ,wv 2 ,……,wv n ;

[0105] Input the character feature and word feature concatenation vector of the user query;

[0106] After encoding in the first encoding layer, the encoding vector v is obtained 1 ;

[0107] v 1 Copy n copies, respectively with wv 1 ,wv 2 ,……,wv n Concatenate and encode in the second encoding layer to obtain the second encoding vector v 21 ,v 22 ,……,v 2n ;

[0108] The second encoding vector is normalized and input into the fully connected layer for classification, and the correlation scores are 0.93, 0.5, ...;

[0109] The answer with the highest relevance score, “Schedule for shuttle service”, is the answer returned by the question-answering model.

[0110] Specifically, the process of using prior knowledge to enhance the correlation calculation model includes:

[0111] Segmenting a small number of original samples with answer annotation information, wherein the answer annotation information is determined by business personnel based on the questions of the samples;

[0112] Linking each word in the word segmentation sequence with an entity in the business domain knowledge graph;

[0113] According to the ontology of the business domain knowledge graph, the successfully linked words are replaced with the names of other entities of the same entity type to obtain a large number of candidate enhancement samples;

[0114] The text of the original sample is subjected to the character feature extraction by the character feature module to extract the character feature vector of the original sample;

[0115] The candidate enhanced sample text is passed through the character feature module to extract the character features, and the feature vector extracted from the candidate sample is calculated with the character feature vector of the original sample. The vector distance is calculated by the following formula:

[0116]

[0117] Among them, x i and i are the values ​​of the original sample word feature vector and the candidate sample word feature vector in the i-th dimension respectively;

[0118] The candidate samples whose vector distance exceeds the vector distance threshold are discarded, and the remaining screened candidate samples constitute the prior knowledge enhanced sample data;

[0119] The correlation calculation model is trained by taking the question of the sample data enhanced with prior knowledge as input and taking the answer annotation information of the sample data enhanced with prior knowledge as target output.

[0120] S104 enhances data by introducing business knowledge into sample data, and uses the semantics pre-learned by RoBERTa to constrain the enhanced sample data by enhancing the semantic distance between the sample data and the original sample, thereby increasing the rationality of the enhanced sample data. The correlation calculation model is introduced by introducing two prior knowledge, semantics and business knowledge, to adapt to user query features in more situations.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, ordinary technicians in this business field should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A few-sample question answering method that introduces prior knowledge, characterized in that: The following steps are involved: S101 prefabricates a few-sample business domain knowledge graph; S102 obtains a user-side query; S103: copy the query into two copies, input them into a character feature module based on query semantics and a word feature module integrating the business domain knowledge respectively for feature extraction, and output the extracted character feature vector and word feature vector; S104 concatenates the character feature vector and the word feature vector, inputs the concatenation into a correlation calculation model enhanced by prior knowledge, and obtains a correlation score for each answer; S105 returns the answer with the highest relevance score to the user; The word feature module integrating the business domain knowledge in step S103 performs feature extraction, including: S401 uses a word segmentation tool to segment the query, wherein the word segmentation tool introduces a custom dictionary of all entity names in the business domain knowledge graph; S402: linking each word in the word segmentation sequence with an entity in the business domain knowledge graph, injecting triple information associated with the linked entity into the word segmentation sequence to obtain a word segmentation graph; S403: input the word segmentation graph into the K-BERT pre-training model to extract features, and obtain a feature matrix; S404 outputs the first vector of the feature matrix as the word feature vector; The step S402 links each word in the word segmentation sequence with an entity in the business domain knowledge graph, including: Count the number of occurrences of each word in the word to obtain a word bag; Count the number of occurrences of each word in each entity name in the business domain knowledge graph to obtain a word bag for each entity; The word is similar to each entity name in the business domain knowledge graph, and the similarity score is calculated using the following formula: in, It is The similarity score between the entity and the word, is the length of the union of the bag-of-words and bag-of-entity-words, is the first The number of times a word appears in the word. is the first The number of times a word appears in the name of the entity; If the score of the entity with the highest similarity score exceeds the link threshold, it is determined that the word is linked to the entity; The process of enhancing the correlation calculation model with prior knowledge in step S104 includes: Segmenting a small number of original samples with answer annotation information, where the answer annotation information is determined by business personnel based on the questions of the samples; Linking each word in the word segmentation sequence with an entity in the business domain knowledge graph; According to the ontology of the business domain knowledge graph, the successfully linked words are replaced with the names of other entities of the same entity type to obtain a large number of candidate enhancement samples; Extracting character features from the text of the original sample through the character feature module to obtain a character feature vector of the original sample; The candidate enhanced sample text is subjected to the character feature module to extract the character features, and the vector distance between the extracted candidate sample character feature vector and the original sample character feature vector is calculated; The candidate enhancement samples whose vector distance exceeds the vector distance threshold are discarded, and the remaining screened candidate enhancement samples constitute the prior knowledge enhancement sample data; The correlation calculation model is trained by taking the question of the sample data enhanced with prior knowledge as input and taking the answer annotation information of the sample data enhanced with prior knowledge as target output.

2. The method for answering a few samples using prior knowledge according to claim 1, characterized in that: The step S101 includes: S201 collects the business points of the business field where the small sample is located; S202 summarizes the categories of the business points and the relationships between the business points based on the expert experience in the business field; S203 constructs a business domain knowledge graph by taking the business points as entities, taking the relationships between the business points as relationships between entities, and taking the categories of the business points and the relationships as ontology.

3. The method for answering a few samples using prior knowledge according to claim 1, characterized in that: The word feature module extraction based on query semantics in step S103 includes: S301: Segment the query into words; S302: pre-process the word segmentation sequence and input it into the RoBERTa pre-training model to extract features, and obtain a feature matrix; S303 outputs the first vector of the feature matrix as the character feature vector.

4. The method for answering a few-sample questions with prior knowledge according to claim 1, characterized in that: The correlation calculation model in step S104 is: S501: Input each of the answers into the character feature module for feature extraction, and output the character feature vector of the answer; S502 inputs the concatenated vector into a first encoding layer based on a self-attention mechanism for encoding, and outputs a first encoding vector; S503 concatenates the first encoding vector with the word feature vector of the answer, inputs the concatenation into a second encoding layer based on a self-attention mechanism for encoding, and outputs a second encoding vector; S504: input the second encoding vector into a BatchNormalization normalization layer based on batch normalization for normalization to accelerate convergence, and output a normalized vector; S505 inputs the normalized vector into a 2-dimensional Softmax fully connected layer for classification, and the second-dimensional classification probability value is the relevance score of the answer.

Images