A similar question recognition method for epidemic question-answering system integrating category information

Abstract

The present invention proposes a method for identifying similar questions in an epidemic question-and-answer system that integrates category information. It is a study on the representation and processing of questions in a specific field and belongs to the field of data recognition. Compared with the prior art, the method for calculating the similarity of questions in the epidemic field question-and-answer system described in the present invention embeds the category information of the question as an important factor in the generation process of the sentence vector, and adopts a multi-head attention mechanism. Based on the above reasons, the present invention can not only better solve the problem identification related to the epidemic, but also can be widely promoted in other limited fields.

Description

Technical Field

[0001] The present invention proposes a similar question recognition method for an epidemic question-answering system that integrates category information, which is a study on the representation and processing of questions in a specific field and belongs to the field of data recognition. Background Art

[0002] In early 2020, the COVID-19 outbreak occurred. Information about the prevalence and development of COVID-19 spread rapidly on social media, attracting widespread public attention and discussion, and triggering a wave of information searches on medical and health Q&A platforms and social media. Due to the particularity of the epidemic, accurate and scientific Q&A information can answer netizens' questions in a timely manner. The construction of such an epidemic information Q&A system can provide a platform for the collection and aggregation of epidemic-related information.

[0003] Text similarity calculation technology is an important technology in the field of natural language processing. It is widely used in natural language processing tasks such as text summary generation, limited domain question-answering system, plagiarism detection, and similar content recommendation. However, for Chinese, synonyms, negative meanings, and complex and diverse sentence structures increase the difficulty of calculation. In the question-answering system, due to the short length of questions, the lack of context, and the diversity and complexity of user questions, certain requirements are placed on the accuracy and reasoning ability of the Chinese question-answering system.

[0004] Traditional semantic text similarity calculation methods based on statistics and vector space cannot effectively capture the semantic information of sentences. Methods based on semantic trees are not accurate enough and require continuous manual maintenance. Deep learning methods can not only effectively extract sentence features, but also capture sentence context information to achieve a higher accuracy. Most existing deep learning methods seek to improve model performance by improving model structure or studying model integration. In fact, the subject and focus of a sentence can also reflect the main background of the question and indicate the main areas of concern and direction of the questioner.

[0005] The innovation of this invention is to integrate the classification information of epidemic-related questions into the generation of question vectors, so that the vector generation part carries more semantic information, which can better learn the deep semantic features of sentences, and has important practical significance in the recognition of similar questions in the question-answering system in the field of epidemics. It can effectively solve various related problems of the people during the epidemic, help the people track the epidemic situation and take timely self-protection measures, enrich the existing health information classification system and support relevant departments to grasp the dynamics of the people's health information needs and formulate more practical and effective public health policies. Summary of the invention

[0006] The purpose of the present invention is to solve the technical problem that the above-mentioned similar question recognition in the epidemic field question-answering system is not accurate enough, and to provide a similar question recognition method for the epidemic question-answering system that integrates category information. In the process of generating sentence vectors, the present invention uses the category information of the epidemic question as one of the embedding layers, and uses the attention mechanism to better obtain the semantic information of the sentence, thereby improving the accuracy and speed of similar semantic recognition.

[0007] The purpose of the present invention is achieved through the following technical solutions.

[0008] A method for identifying similar questions in an epidemic question-answering system by integrating category information comprises the following steps:

[0009] Step 1: Collect real questions in the field of COVID-19 on the Internet to form a question set D = {d1, d2, ..., d j ,...d n}, where d j In order to collect specific questions, the question dj is encoded according to the vector generation method of the Bert model, and the keywords of the question are used as the corresponding categories of the question, and finally a classification information set of questions in the field of the new crown pneumonia epidemic is formed, which contains category information and corresponding questions;

[0010] Step 2: Encode the category information of dj in step 1 using a single-bit effective encoding method c j ={0,0,0,1,0……,0};

[0011] Step 3: Select any question in D as q, and calculate the correlation between q and all other questions in the question set D; decompose the input question q into a set of morphemes, represented by q = {q1, q2, q3, q4, ...q i ...}, (1≤i≤m), according to the morpheme q i In question d j and the frequency information of the question set D to calculate q i With d j , and for all morphemes relative to d j The correlation between question q and a question d in question set D is weighted. j The correlation score Score(q,d j ); the calculation formula is as follows:

[0012]

[0013] q i represents the morpheme after q is decomposed, W i Represents q i The weight, R(q i ,d j ) represents the morpheme qi With sentence d j relevance;

[0014] W i The calculation formula is as follows:

[0015]

[0016] N is the number of questions in set D, df(q i ) means it contains the morpheme q i The number of sentences; R(q i ,d j ) is calculated as follows:

[0017]

[0018]

[0019]

[0020] Among them, 1.2≤k1≤2.0, which indicates the relationship between the frequency of morphemes in sentences and scores; 0≤b≤1, which indicates the influence of sentence length on the calculation of average length; tf(q i ,d j ) is d j q i The frequency of occurrence, len(d j ) is d j , avdl is the average length of all questions in question set D;

[0021] Step 4: Score(q,d j ) are sorted in descending order, and the top 20 corresponding questions constitute the question set D';

[0022] Step 5: According to the vector generation method of the Bert model, the category information of all questions in the question set D' is embedded into the Bert model using a one-bit efficient encoding method; first, each question in the set D' in step 4 is converted into a linear sequence of words X = (x1, x2, ... x t ,...x n ); Then, the linear sequence of words X = (x1, x2, ... x t ,...x n ) is input to the Bert embedding layer;

[0023] The category information c j Three different vector representations of the Bert model j Average, the synthetic vector E = (e1, e2, ..., e t,...e n ), as the input of the Transformer encoder part;

[0024] Step 6: Replace X in step 5 with (x1, x2, … x t ,...x n ) Input the attention mechanism model to learn the global features of the question; first, record the input sentence information as Source, decompose Source into a series of (Key, Value) pairs, (Key, Value) pairs refer to the vectors and values ​​of the corresponding position characters of each sentence in D', and then use the dot multiplication method to calculate the correlation Sim between any character Query in the input question and each Key i (Key i ,Query)=Key i ·Query; then use the Softmax function to normalize to get the attention weight of the characters in the question in the entire question, and use a i Indicates that a i = L x is the length of the sentence; finally, the character attention weight a i The actual value of the character i Perform weighted summation and finally get the attention score

[0025] Calculation process: First, create three matrices Q, K, and V converted from Query, Key, and Value respectively; And by introducing the scaling factor The inner product is adjusted in such a way that it does not become too large and affect the calculation, dim k Indicates the dimension of Key, dim k =64; Here, the attention mechanism is operated using the result of multi-dimensional learning, that is, the multi-head attention mechanism, which can obtain richer semantic feature expressions:

[0026] Multihead(Q,K,V)=Concat(head1,...,head h )W O

[0027] Among them head i =Attention(QW i Q ,QW i K ,QW i V )

[0028] represents the Q, K, V weight matrix of the i-th head (i=1,2,……,h,h=8), h represents the number of attentions, d model is the dimension of the concatenated vector, dim k =dim v =d model / h, each attention head captures the information of a subspace in the question, and the h attention heads are used as the matrix The multi-head attention value is obtained by splicing. The function is to splice the multi-head attention, which is obtained through training;

[0029] Step 7: After the attention mechanism layer, a fully connected layer is added to integrate all semantic and category information for similarity recognition, and output the similarity value Similarity between question q and the questions in the candidate question set D'; the fully connected layer consists of a RELU activation function layer and a linear function layer, and the overall formula is expressed as Similarity = max(0,xW1+bias1)W2+bias2, W1, W2 are two trainable weight matrices, bias1 and bias2 represent two offset vectors, and x is the input of the activation function.

[0030] Beneficial Effects

[0031] Compared with the prior art, the question similarity calculation method of the epidemic field question-answering system described in the present invention embeds the category information of the question as an important factor in the generation process of the sentence vector, and adopts a multi-head attention mechanism. Based on the above reasons, the present invention can not only better solve the problem identification related to the epidemic, but also can be widely promoted in other limited fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A flowchart of a method for identifying similar questions in a question-answering system integrating information on the categories of COVID-19 epidemics in an embodiment;

[0033] Figure 2 Flowchart of the similar question recognition method based on attention mechanism. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solution of the present invention will be described in detail below in conjunction with the accompanying drawings and specific implementation methods in the embodiments of the present invention.

[0035] like Figure 1 As shown, the similar question recognition method of the present invention comprises the following steps:

[0036] Step 1: Collect 4506 real questions in the field of COVID-19 on the Internet to form a question set D = {d1, d2..., dj ,...d n}, where d j (1≤j≤n) is the specific question collected, and the question d is generated according to the vector generation method of the Bert model. j Encode and use the keywords of the questions as the corresponding categories of the questions, and finally form a classification information set of questions in the field of COVID-19 containing category information and corresponding questions;

[0037] Step 2: For d in step 1 j The category information is encoded using a single bit efficient encoding method. j ={0,0,0,1,0……,0};

[0038] Step 3: Select any question in D and denote it as q, calculate the correlation between q = "What should I do if I get COVID-19?" and all other questions in the question set D; decompose the input question q into a set of morphemes, represented as q = {q1, q2, q3, q4, ...q i ...}, (1≤i≤m), according to the morpheme q i In question d j and the frequency information of the question set D to calculate q i With d j and for all morphemes relative to d j The correlation between question q and a question d in question set D is weighted. j The correlation score Score(q,d j ); the calculation formula is as follows:

[0039]

[0040] q i represents the morpheme after q is decomposed, W i Represents q i The weight, R(q i ,d j ) represents the morpheme q i With sentence d j relevance;

[0041] W i The calculation formula is as follows:

[0042]

[0043] N is the number of questions in set D, df(q i ) indicates that it contains the morpheme q i The number of sentences; R(q i ,d j ) is calculated as follows:

[0044]

[0045]

[0046]

[0047] Among them, 1.2≤k1≤2.0, which indicates the relationship between the frequency of morphemes in sentences and scores; 0≤b≤1, which indicates the influence of sentence length on the calculation of average length; tf(q i ,d j ) is d j q i The frequency of occurrence, len(d j ) is d j , avdl is the average length of all questions in question set D;

[0048] Step 4: Score(q,d j ) Sort them in descending order, and the top 20 corresponding questions constitute the question set

[0049] D'={

[0050] “What to do if you get COVID-19: 13.82169669”;

[0051] “What to do after being diagnosed with COVID-19: 12.36328026”;

[0052] "What medicine is the most effective for COVID-19: 12.35062094";

[0053] "What is the prognosis of COVID-19: 11.89526051";

[0054] "How to treat COVID-19: 11.89526051";

[0055] “What are the treatments for the novel coronavirus: 11.45995442”;

[0056] “What are the common symptoms of infection with the new coronavirus: 11.05538355”;

[0057] “What it feels like to have COVID: 11.05538355”;

[0058] “What are the quarantine measures during the epidemic: 11.05538355”;

[0059] “How to prevent the new coronavirus: 10.67840377”;

[0060] "COVID-19 infection performance: 10.67840377";

[0061] "How to judge the new coronavirus pneumonia: 10.67840377";

[0062] "How to cure COVID-19, what to eat if you have COVID-19: 10.46877347";

[0063] "What to do if you get COVID-19 while breastfeeding: 10.32628565";

[0064] "How to treat and prevent COVID-19: 9.996648272":

[0065] "How to treat COVID-19 combined with hypertension: 9.687405337";

[0066] "How to take care of yourself after contracting COVID-19: 9.687405337";

[0067] “What are the preventive measures for COVID-19: 9.506486953”;

[0068] “What is the difference between COVID-19 and influenza: 8.636723046”;

[0069] “How to control COVID-19: 8.611672945”;

[0070] }.

[0071] Step 5: According to the vector generation method of the Bert model, the category information of all questions in the question set D' is embedded into the Bert model using a one-bit efficient encoding method; first, each question in the set D' in step 4 is converted into a linear sequence of words X = (x1, x2, ... x t ,...x n ); Then, the linear sequence of words X = (x1, x2, ... x t ,...x n ) is input to the Bert embedding layer;

[0072] The category information c j Three different vector representations of the Bert model j Average, the synthetic vector E = (e1, e2, ..., e t ,...e n ), as the input of the Transformer encoder part;

[0073] Step 6: Replace X in step 5 with (x1, x2, … x t ,...x n) Input the attention mechanism model to learn the global features of the question; first, record the input sentence information as Source, decompose Source into a series of (Key, Value) pairs, (Key, Value) pairs refer to the vectors and values ​​of the corresponding position characters of each sentence in D', and then use the dot multiplication method to calculate the correlation Sim between any character Query in the input question and each Key i (Key i ,Query)=Key i ·Query; then use the Softmax function to normalize to get the attention weight of the characters in the question in the entire question, and use a i express, L x is the length of the sentence; finally, the character attention weight a i The actual value of the character i Perform weighted summation and finally get the attention score like Figure 2 As shown;

[0074] Calculation process: First, create three matrices Q, K, and V converted from Query, Key, and Value respectively; Attention(Query, Source) = Softmax()V, and adjust it by introducing a scaling factor so that the inner product is not too large to affect the calculation, indicating the dimension of Key, dim k =64; Here, the attention mechanism is operated using the result of multi-dimensional learning, that is, the multi-head attention mechanism, which can obtain richer semantic feature expressions:

[0075] Multihead(Q,K,V)=Concat(head1,...,head h )W O

[0076] Among them head i =Attention(QW i Q ,QW i K ,QW i V )

[0077] represents the Q, K, V weight matrix of the i-th head (i=1,2,……,h,h=8), h represents the number of attentions, d model is the dimension of the concatenated vector, dim k =dimv =d model / h, each attention head captures the information of a subspace in the question, and the h attention heads are used as the matrix The multi-head attention value is obtained by splicing. The function is to splice the multi-head attention, which is obtained through training;

[0078] Step 7: After the attention mechanism layer, a fully connected layer is added to integrate all semantic and category information for similarity recognition, and the similarity value Similarity between the question q and the question in the candidate question set D' is output. The result is:

[0079] {

[0080] “How to treat COVID-19 Similarity = 0.894480407”;

[0081] “How to treat and prevent COVID-19 Similarity = 0.13762705”;

[0082] “What should I do after being diagnosed with COVID-19? Similarity = 0.033919051”;

[0083] "What medicine is the most effective for COVID-19? Similarity = 0.033315331";

[0084] “What are the treatments for the new coronavirus? Similarity = 0.032382693”

[0085] };

[0086] The fully connected layer consists of a RELU activation function layer and a linear function layer. The overall formula is expressed as Similarity = max(0,xW1+bias1)W2+bias2, where W1 and W2 are two trainable weight matrices, bias1 and bias2 represent two offset vectors, and x is the input of the activation function.

[0087] The current calculation method can improve the accuracy of text similarity calculation, has good recognition and matching functions for different categories of questions, and can be applied to the question similarity calculation process of question answering systems in limited fields. Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention, rather than to limit it. Therefore, improvements and modifications made to the present invention by those skilled in the art based on the disclosure of the present invention should be within the scope of protection of the present invention.

Claims

1. A similar question recognition method for an epidemic question-answering system integrating category information, characterized in that The following steps are involved: Step 1: Collect real questions in the field of COVID-19 on the Internet to form a question set D = {d1, d2..., d j ,...d n }, where d j In order to collect specific questions, the question d is generated according to the vector generation method of the Bert model. j Encoding is performed, and the keywords of the questions are used as the corresponding categories of the questions, and finally a classification information set of questions in the field of COVID-19 epidemic is formed, which includes category information and corresponding questions; Step 2: For d in step 1 j The category information is encoded using a single bit efficient encoding method. j ={0,0,0,1,0……,0}; Step 3: Select any question in the question set D as q, and calculate the correlation between q and all other questions in the question set D; decompose the input question q into a set of morphemes, represented by q = {q1, q2, q3, q4, ...q i ...}, (1≤i≤m), according to the morpheme q i In question d j and the frequency information of the question set D to calculate q i With d j , and for all morphemes relative to d j The correlation between question q and a question d in question set D is weighted. j The correlation score Score(q,d j ); the calculation formula is as follows: q i represents the morpheme after q is decomposed, W i Represents q i The weight, R(q i ,d j ) represents the morpheme q i With sentence d j relevance; W i The calculation formula is as follows: N is the number of questions in set D, df(q i ) means it contains the morpheme q i The number of sentences; R(q i ,d j ) is calculated as follows: Among them, 1.2≤k1≤2.0, which indicates the relationship between the frequency of morphemes in sentences and scores; 0≤b≤1, which indicates the influence of sentence length on the calculation of average length; tf(q i ,d j ) is d j q i The frequency of occurrence, len(d j ) is d j , avdl is the average length of all questions in question set D; Step 4: Score(q,d j ) are sorted in descending order, and the corresponding questions before sorting constitute the question set D'; Step 5: According to the vector generation method of the Bert model, the category information of all questions in the question set D' is embedded into the Bert model using a one-bit efficient encoding method; first, each question in the set D' in step 4 is converted into a linear sequence of words X = (x1, x2, ... x t ,...x n ); Then, the linear sequence of words X = (x1, x2, ... x t ,...x n ) is input to the Bert embedding layer; The category information c j Three different vector representations of the Bert model j Average, the synthetic vector E = (e1, e2, ..., e t ,...e n ), as the input of the Transformer encoder part; Step 6: Replace X in step 5 with (x1, x2, … x t ,...x n ) Input the attention mechanism model to learn the global features of the question; Step 7: After the attention mechanism layer, a fully connected layer is added to integrate all semantic and category information for similarity recognition, and the similarity value Similarity between the question q and the questions in the candidate question set D' is output.

2. According to claim 1, a similar question recognition method for an epidemic question-answering system integrating category information is characterized in that: In step 6, the method of inputting X into the attention mechanism model to learn the global features of the question is: first, the input sentence information is recorded as Source, and Source is decomposed into a series of (Key, Value) pairs, where the (Key, Value) pair refers to the vector of the corresponding position character of each sentence in D' and its value, and then the correlation Sim between any character Query in the input question and each Key is calculated by dot multiplication. i (Key i ,Query)=Key i ·Query; then use the Softmax function to normalize to get the attention weight of the characters in the question in the entire question, and use a i express, L x is the length of the sentence; finally, the character attention weight a i The actual value of the character i Perform weighted summation and finally get the attention score 3. According to claim 2, a similar question recognition method for an epidemic question-answering system integrating category information is characterized in that: The calculation method is as follows: firstly, three matrices Q, K, and V are created which are converted from Query, Key, and Value respectively; dim k Represents the dimension of Key, and the attention mechanism is: Multihead(Q,K,V)=Concat(head1,......,head h )W O Among them head i =Attention(QW i Q ,QW i K ,QW i V ) represents the Q, K, V weight matrix of the i-th head (i=1,2,……,h,h=8), h represents the number of attentions, d model is the dimension of the concatenated vector, dim k =dim v =d model / h, each attention head captures the information of a subspace in the question, and the h attention heads are used as the matrix The multi-head attention value is obtained by splicing. The function is to splice the multi-head attention, which is obtained through training.

4. According to claim 3, a similar question recognition method for an epidemic question-answering system integrating category information is characterized in that: dim k =64。 5. According to claim 4, a similar question recognition method for an epidemic question-answering system integrating category information is characterized in that: In step 7, the fully connected layer is composed of a RELU activation function layer and a linear function layer, and the overall formula is expressed as Similarity = max(0,xW1+bias1)W2+bias2, W1, W2 are two trainable weight matrices, bias1 and bias2 represent two offset vectors, and x is the input of the activation function.

Images