A query intention recognition method based on BERT+BILSTM+CRF and integrated sentence analysis

Abstract

The present invention discloses a query intention recognition method based on BERT+BILSTM+CRF and integrated with sentence pattern analysis, including: marking and constructing a data set, defining domain intention, clarifying intention classification and intention word slot, constructing a domain term set corresponding to the word slot, collecting a question set, marking the question intention classification and word slot information; generating a sentence pattern according to the marked questions and the corresponding intention information for matching verification in the intention recognition model reasoning stage; and constructing a query intention recognition model based on BERT+BILSTM+CRF. The present invention integrates slot dictionary matching, sentence pattern analysis, slot entity recognition based on BERT+BILSTM+CRF, and intention classification based on BERT, which better solves the problem of model error recognition when the distribution of model training data and model prediction data is inconsistent, and improves the accuracy of the model in recognizing the intention of user input colloquial content.

Description

Technical Field

[0001] The present invention relates to the fields of natural language processing and artificial intelligence technology, and in particular to a query intention recognition method based on BERT+BILSTM+CRF and integrated with sentence pattern analysis. Background Art

[0002] The so-called intent recognition is actually natural language understanding (NLU), which is one of the important tasks of natural language processing. Natural language understanding is to make machines like humans, have normal language understanding ability, accurately identify intent, clarify user needs, and provide users with accurate services.

[0003] With the development of deep learning, neural network models based on deep learning algorithms such as CNN, RNN, LSTM, and attention mechanism have begun to be widely used in the fields of text classification and intent recognition, and have continuously achieved SOTA results. The intent recognition rate of the intent recognition model based on deep learning is high, but the recognition results are unexplainable, and the model recognition rate is heavily dependent on the quality of the training data. Due to the complexity and diversity of natural language, the training data used to train the model is often biased and inconsistent with the actual input data distribution, which puts a great test on the generalization ability of the model. In the actual use of the model, due to the sparse training data, there must be a large number of inputs that the model has never seen. At this time, it is very important to evaluate the model recognition results, whether the model's recognition errors can be found, and even make appropriate corrections to the recognition results. When the entity or sentence structure in the input question has never appeared in the training data, the model may give an incorrect recognition result.

[0004] In summary, in practical engineering applications, there is an urgent need for an enhanced intent recognition method in which intent recognition is reliable and recognition errors can be corrected. Summary of the invention

[0005] To solve the above technical problems, the purpose of the present invention is to provide a query intention recognition method based on BERT+BILSTM+CRF and integrated with sentence analysis. The method integrates multiple strategies to realize the correctness judgment of the intention recognition results of a single neural network, moderate correction of erroneous results, and enhancement of the neural network recognition results, so as to achieve accurate semantic recognition and meet the recognition of diversified query intentions.

[0006] The purpose of the present invention is achieved through the following technical solutions:

[0007] A query intention recognition method based on BERT+BILSTM+CRF and integrated with sentence analysis includes the following steps:

[0008] A. Annotate and construct a data set, define domain intent, clarify intent classification and intent word slots, construct a domain term set corresponding to the word slots, collect question sets, and annotate question intent classification and word slot information;

[0009] B. Generate sentence patterns based on the annotated questions and corresponding intent information for matching verification in the inference phase of the intent recognition model;

[0010] C builds a query intent recognition model based on BERT+BILSTM+CRF.

[0011] Compared with the prior art, one or more embodiments of the present invention may have the following advantages:

[0012] The integration of slot dictionary matching, sentence pattern analysis, slot entity recognition based on BERT+BILSTM+CRF, and intent classification based on BERT has effectively solved the problem of model misrecognition when the distribution of model training data and model prediction data is inconsistent, and improved the accuracy of the model's intent recognition of user input colloquial content. The intent recognition method that integrates traditional methods and deep learning algorithms, and the results of the two verify each other, can make the recognition rate of accurate questions higher and the recognition of open questions closer to user needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a flow chart of building a query intention recognition model based on BERT+BILSTM+CRF provided by an embodiment of the present invention;

[0014] Figure 2 It is a full flow chart of a query intention recognition method based on BERT+BILSTM+CRF and integrated with sentence pattern analysis provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below in conjunction with embodiments and drawings.

[0016] The query intent recognition task is to input a question Q, output the intent category Intent to which the question belongs, as well as the word slot name slot of the intent and the corresponding slot value information.

[0017] like Figure 1 and Figure 2 As shown in the figure, the intention recognition method based on BERT+BILSTM+CRF and integrated sentence analysis includes the following steps:

[0018] Step 1: Annotate and construct a data set, define domain intent, clarify intent classification and intent word slots, construct a domain term set corresponding to the word slot, that is, a word slot value set, collect question sets, and annotate question intent classification and word slot information;

[0019] The construction of a word slot value set helps to further accurately determine the intent classification and identify the intent content in intent recognition in a specific field. It is one of the effective means to improve the accuracy and recall rate of intent recognition.

[0020] For training data, the present invention uses two annotation forms:

[0021] The first annotation form directly annotates the data with its intent category and word slot. For example, for the book domain question “Find a Dream of Red Mansions published in 2007”, the annotation result is “Find a Dream of Red Mansions published in

[2007] (slot_time) (slot_book): intent_book”.

[0022] The second form of annotation is to use the word slot label as a universal representation of the slot value, and to replace the slot value to form the annotation. For example, the question "Find a Dream of Red Mansions published in 2007" is annotated as "Find a [#book#](slot_book):intent_book published by [#time#](slot_time)". This replaces the word slot value with a universal slot label and inputs it into the neural network to reduce the probability of recognition errors caused by not having seen this slot value during network training. In addition, the construction of the word slot value set can be used for slot value recognition judgment in the prediction of the consciousness recognition model to further verify the correctness of the model recognition results.

[0023] Step 2 generates sentence patterns based on the annotated questions and corresponding intent information for matching verification in the model reasoning phase;

[0024] The matching verification in the model reasoning stage uses two sentence patterns. The first is the slot intent pattern, which is to retain only the rectangular pattern formed by the slot label and the intent classification label of the annotated data, such as the question "New books published by Posts and Telecommunications Press in 2019". The annotated data of this question is "New books published by [Posts and Telecommunications Press] (slot_press) in

[2019] (slot_time): intent_book". The slot intent pattern generated based on its annotated data is "slot_press, slot_time: intent_book". This pattern is used to verify the recognition results of the neural network model. When the recognition result is consistent with the pattern, the recognition result is trusted and returned. The second pattern relies on dependency syntactic analysis. Based on the syntactic analysis results, sentence components are selectively retained to form a syntactic pattern. The rules for retention are to retain nouns that are not personal pronouns; retain the "core relationship", "subject-predicate relationship", "verb-object relationship", and "preposition-object relationship"; if the "core relationship" is a verb and there is a "verb-object relationship" in the form of a verb, then the "core relationship" is not retained, and only the "verb-object relationship" is retained; if the sentence component retained is a slot word, then the word is replaced with a slot label. For example, for the question "Which hall is Dream of the Red Chamber in?", according to the rules, its syntactic analysis result is: "slot_book in the hall: intent_book". The "slot label + question core word + intent classification" mode is used to verify the recognition results of the neural network model and enhance the recognition accuracy.

[0025] Step 3: Build a query intent recognition model based on BERT+BILSTM+CRF;

[0026] The intent recognition model usually consists of two tasks: "intent classification" and "slot filling". "Intent classification" can be considered as a classification task of semantic expression, which is implemented through the BERT pre-trained model. The encoding end is the pre-trained BERT, and the encoding feature output is connected to the softmax layer to predict the input category probability. BERT uses the Transformer encoder as the main model structure. The core idea of ​​the attention mechanism used by Transformer is to calculate the relationship between each word in a sentence and all the words in the sentence. The relationship between words reflects the correlation and importance between different words in the sentence to a certain extent. Then use these relationships to adjust the importance (weight) of each word to obtain a new representation of each word. This new representation not only contains the word itself, but also contains the relationship between other words and this word. Therefore, it is a more global expression compared to a simple word vector. Transformer obtains the final text expression by continuously overlapping such an attention mechanism layer and a normal nonlinear layer on the input text. The characteristics of Transformer enable BERT to extract the deep semantic features of the sentence.

[0027] Slot filling can be considered as a sequence labeling task. The BERT model is used to embed the corpus samples, and then the BILSTM+CRF network training model is input. The conditional random field CRF mechanism is used to constrain the output sequence. The transfer score is added based on the feature score output by the decoder. The output of the current label no longer depends only on the decoded features, but also on the label output of the previous time step. The label transfer score will give positive score rewards for legal transfers and negative score penalties for illegal transfers. The decoded output after adding the transfer score is not directly used to predict labels, because the probability of the label sequence obtained by the maximum probability label at each output position may not be the largest. We use the Vertebi algorithm to find the label sequence with the maximum probability as the final output.

[0028] Step 4: Intention recognition model reasoning ①

[0029] According to the first type of training data, the intent is predicted. The questions do not need any processing and are directly input into the model to predict the starting and ending positions of the slot value, the word slot name, and the question intent classification. Assume that the input question in the prediction stage contains slot values ​​that the model has never seen before, such as the question "Which hall is Harry Potter and the Goblet of Fire in?". Since the model has generalization ability, the model may give the correct recognition result "Which hall is [Harry Potter and the Goblet of Fire] (slot_book) in intent_book." The result given by the model contains the slot value "Harry Potter and the Goblet of Fire", and determines that the type of this slot value is "slot_book". At this time, the original question is matched in the slot value dictionary to see if it can be matched to "Harry Potter and the Goblet of Fire" through the longest prefix in the dictionary of "slot_book". If it cannot be matched, it means that the slot recognition is wrong or the slot value is not included in the dictionary, and the recognition result of the model cannot be trusted; if the slot value is matched, but the result of the dictionary matching shows multiple different explanations as shown in step 6, it is transferred to the intention recognition model reasoning ③ for processing. If the dictionary matching only gives a single explanation, the recognition result of the model is converted to the first-class sentence pattern 'slot_book:intent_book', and the sentence pattern is matched in the first-class sentence pattern obtained from the training data. If the pattern exists, choose to believe the recognition result of the model and return the recognition result. If it cannot be matched, it is assumed that the question intent category recognized by the model is 'welcome' 'intent_greet'. Since the first-class sentence pattern 'slot:book:intent_greet' does not exist in the training data, the recognition result of the model cannot be trusted. The model recognition error may be affected by the slot value or sentence pattern that has never been seen before. At this time, it is transferred to the intention recognition model reasoning ② to try the matching of the second-class sentence pattern again.

[0030] Step 5: Intent Recognition Model Reasoning ②

[0031] In the intention recognition model reasoning ①, the recognition result of the model did not pass the sentence pattern verification. This may be because the model has little training data and encountered unseen features in the reasoning stage. At this time, replacing the slot value in the question with the word slot name in the way of the second type of training data may reduce the impact of unseen features. For example, the question "Which hall is Dream of the Red Chamber in" is converted to '[#book#] in which hall', and the adjusted question is input into the model. Assuming that the model does not correctly recognize the slot value '#book#' as its true slot value type 'slot_book', or does not correctly recognize the intention of the adjusted question as 'intent_book', the only choice is to discard the model recognition result. If all aspects of the adjusted question are correctly recognized, the recognition result cannot be directly trusted, because the question has been adjusted to reduce the difficulty of recognition. The model may overfit on this feature that reduces the difficulty. The model recognition result must be verified by a more rigorous sentence pattern before it can be accepted. According to the processing method of the second type of sentence pattern, the original input is syntactically analyzed to form the second type of sentence pattern 'slot_book in the hall: intent_book'. Since this pattern exists in the training data, the model recognition result can be supported by the slot dictionary and sentence pattern, so we choose to trust the model recognition result. If the recognition result fails in the second type of sentence pattern, the recognition result is discarded.

[0032] Step 6: Intention recognition model reasoning ③

[0033] Intent recognition model reasoning ③ handles the situation where there are multiple different interpretations of the dictionary matching slot value, such as the example question in step 6, "Looking for a guide to Zhu Ziqing's prose collection published by Tsinghua University Press." This sentence will match three slot values, '[Tsinghua University Press](slot_press)', '[Zhu Ziqing](slot_author)', and '[Guide to Zhu Ziqing's prose collection](slot_book)'. Slot values ​​2-3 overlap and are excluded, and the dictionary gives 1-2 and 1-3 combinations. If the model recognition result matches the longest combination in the dictionary result, that is, 1-3 combinations are recognized, and the sentence pattern formed by the slot type and the recognized intent type can pass the verification of the first type of sentence pattern, then the model recognition result is trusted. If the slot combination recognized by the model matches the shorter result recognized by the dictionary or does not match the dictionary result, the optional combinations given by the dictionary are sorted from high to low by length, and this sorted combination is traversed to match the second type of sentence pattern according to the steps in intent recognition model reasoning ②. First, the slot value in the combination is replaced with the slot category given by the dictionary in the original question to form an adjusted question. The rewritten question is input into the model. If the model still recognizes the replaced slot type text as its true slot type and the recognized intent is correct and can be verified by the second type of sentence pattern, then the slot combination and recognition intent category are returned as the recognition result. Take the example question "Find a guide to Zhu Ziqing's prose collection published by Tsinghua University Press", the slot value of the combination of '1-3' given by the dictionary is replaced by the slot type in the question to form an adjusted question "Find a book of [#author#] published by [#press#]", and input the adjusted question into the model for recognition. If the model still recognizes '[#press#]' as 'slot_press' and '[#author#]' as 'slot_author', and correctly recognizes the intent 'intent_book', the question is syntactically analyzed and combined with the recognized intent type and slot type to form the second type of sentence pattern 'Find slot_press published by slot_author', which can pass the verification of the second type of sentence pattern, and then this recognition result is returned.

[0034] Although the embodiments disclosed in the present invention are as above, the above contents are only embodiments adopted for facilitating the understanding of the present invention and are not intended to limit the present invention. Any technician in the technical field to which the present invention belongs can make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed in the present invention, but the patent protection scope of the present invention shall still be subject to the scope defined in the attached claims.

Claims

1. A query intention recognition method based on BERT+BILSTM+CRF and integrated sentence analysis, characterized in that: The method comprises: Step A: Annotate and construct a data set, define domain intent, clarify intent classification and intent word slots, construct a domain term set corresponding to the word slots, collect question sets, and annotate question intent classification and word slot information; Step B generates sentence patterns based on the marked questions and corresponding intent information for matching verification in the inference phase of the intent recognition model; Step C builds a query intent recognition model based on BERT+BILSTM+CRF; The matching verification in the model reasoning phase in step B uses two sentence patterns. The first is the slot intention pattern, which is to retain only the rectangular pattern formed by the slot label and the intention classification label of the annotated data. This pattern is applied to the verification of the recognition result of the neural network model. When the recognition result is consistent with the pattern, the recognition result is selected to be trusted and returned. The second pattern relies on dependency syntactic analysis, and selectively retains sentence components based on the syntactic analysis results to form a syntactic pattern. The method also includes reasoning of the intent recognition model and prediction of the model; the reasoning of the intent recognition model includes: predicting intent in the form of class training data, predicting intent in the form of first-class training data, directly inputting the model without any processing of the question, predicting the start and end positions of the slot value, the word slot name and the question intent classification; Assume that the question input in the prediction stage contains a slot value that has never been seen in the model, the model gives a correct recognition result, and the result given by the model contains the slot value, and determines the type of this slot value, and then matches the original question in the slot value type dictionary; if it cannot be matched, the slot recognition is wrong or the slot value is not included in the dictionary, and the recognition result of the model cannot be trusted; if the slot value is matched, but the result of the dictionary matching shows multiple different explanations, then the second type of training data is used; if the dictionary matching only gives a single explanation, the recognition result of the model is converted into the first type of sentence pattern, and the sentence pattern is matched in the first type of sentence pattern obtained from the training data. If the pattern exists, choose to believe the recognition result of the model and return the recognition result. If it cannot be matched, the recognition result of the model cannot be trusted; then match through the second type of sentence pattern, that is, the second type of training data method.

2. The query intention recognition method based on BERT+BILSTM+CRF and integrated sentence analysis as claimed in claim 1 is characterized in that: The annotations for data construction in step A include two types: In the first form of annotation, the intent category and word slot are directly labeled on the data; in the second form of annotation, the word slot label is used as a general representation of the slot value, and the slot value is replaced to form the annotation.

3. The query intention recognition method based on BERT+BILSTM+CRF and integrated sentence analysis as claimed in claim 1 is characterized in that: The intent recognition model in step C consists of intent classification and slot filling; wherein, intent classification is a classification task of semantic expression, which is implemented through the BERT pre-training model, the encoding end is a pre-trained BERT, the encoding feature output is connected to the softmax layer, and the input category probability is predicted; the slot filling is a sequence labeling task, the BERT model is used to embed the corpus samples, and then the BILSTM+CRF network training model is input, and the conditional random field CRF mechanism is used to constrain the output sequence.

4. The query intention recognition method based on BERT+BILSTM+CRF and fusion sentence analysis as claimed in claim 1 is characterized in that: The second type of training data is to replace the slot value in the question with the word slot name, and input the adjusted question into the model. If the model does not correctly identify the slot value as its true slot value type or does not correctly identify the intention of the adjusted question, the model recognition result is discarded; if all aspects of the adjusted question are correctly recognized, the original input is subjected to syntactic analysis through the second type of sentence pattern processing method to form the second type of sentence pattern, and the model recognition result is selected.

5. The query intention recognition method based on BERT+BILSTM+CRF and integrated sentence analysis as claimed in claim 1 is characterized in that: The intention recognition model reasoning also includes processing the situation where multiple different interpretations appear when the dictionary matches the slot value. If the model recognition result matches the longest combination in the dictionary result, that is, 1 group or 3 groups are recognized, and the sentence pattern formed by the slot type and the recognized intention type can pass the verification of the first type of sentence pattern, then the model recognition result is trusted; if the slot combination recognized by the model matches the shortest result recognized by the dictionary, the combinations given by the dictionary are sorted from high to low by length, and this sorted combination is traversed and matched according to the second type of sentence pattern.

6. The query intention recognition method based on BERT+BILSTM+CRF and integrated sentence analysis as claimed in claim 5 is characterized in that: The second type of sentence pattern matching includes: first, replacing the slot value in the combination with the slot category given by the dictionary in the original question to form an adjusted question, and inputting the rewritten question into the model. If the model recognizes the replaced slot type text as the real slot type, and the recognized intention is correct and can be verified by the second type of sentence pattern, the current slot combination and recognition intention category are returned as the recognition result.

Images