A short text similarity calculation method incorporating prior knowledge

Abstract

The present invention discloses a short text similarity calculation method incorporating prior knowledge, comprising the following steps: S10, constructing a keyword dictionary; S20, adding keyword annotations to training data; S30, training the training data with added keyword annotations through a BERT model, and saving the optimal result; S40, performing text error correction on the query input by the user; S50, performing keyword extraction and dependency syntactic analysis on the query after error correction and the question to be matched; S60, constructing a model input for the analysis result in step S50, and inputting it into the trained BERT model to obtain text similarity. The present invention adds keyword and dependency syntactic analysis annotations to the training data, adds more prior knowledge to the model, and accelerates the convergence speed.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a short text similarity calculation method integrating prior knowledge. Background Art

[0002] Short text similarity calculation is a difficult and hot topic in the field of natural language processing (NLP) and even machine learning. It is an important task in NLP. It can be regarded as a separate task and can also serve as the basis for other NLP applications. It can be defined as: for a given set of short texts, based on the study of the short text structure, a variety of short text features (such as word frequency, grammar, semantic features) are extracted and quantified, so as to use data to reflect the similarities and differences between short texts. The more similarities there are, the higher the similarity, and vice versa.

[0003] As a classic problem in the field of natural language processing, the question-answering system has attracted more and more attention with the development of application scenarios such as intelligent customer service and human-computer dialogue systems. As a key technology in the question-answering system, short text similarity calculation is very difficult because it contains little information, has diverse natural language expressions, and is sensitive to key information. It has always been a research hotspot and difficulty in the industry and academia. In order to solve this problem, researchers mainly start from two perspectives: data dimension and model dimension. Data determines the upper limit. Positive samples generally rely on manual annotation, while negative samples are generated by random negative sampling, WMD+multi-person annotation, entity word replacement and other methods. However, these methods either have low confusion of negative samples, high labor costs, or too much noise, which greatly disturbs the model. These methods have their own shortcomings, and there is currently no optimal strategy to obtain high-quality data. The model dimension is a more well-known improvement angle. From the perspective of model structure, it can be divided into two categories: representation-based and interaction-based. The representation-based model first represents the query-question at the bottom level, and then interacts and classifies at the high level. The interaction-based model allows query-question to interact at the bottom level, which can better grasp the semantic focus.

[0004] For example, the invention with publication number CN112329430A discloses a model training method, a text similarity determination method and device, and the invention with publication number CN111737438A discloses a data processing method, device and electronic device based on text similarity, which have explored and optimized the difficulties of text similarity, but still have not solved the problems of difficulty in capturing keyword information and difficulty in processing OOV (Out Of Vocabulary). Therefore, there is an urgent need for a short text similarity calculation method to solve these problems to a certain extent. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the technical problem to be solved by the present invention is: how to provide a short text similarity calculation method incorporating prior knowledge to effectively improve the short text similarity calculation accuracy.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A short text similarity calculation method incorporating prior knowledge includes the following steps:

[0008] S10, constructing a keyword dictionary;

[0009] S20, adding keyword annotations to the training data;

[0010] S30, training the training data with added keyword annotations through the BERT model, and saving the optimal result;

[0011] S40, performing text error correction on the query input by the user;

[0012] S50, extracting keywords and performing dependency syntax analysis on the query after error correction and the question to be matched;

[0013] S60. Construct a model input for the analysis result in step S50, and input it into the trained BERT model to obtain text similarity.

[0014] Furthermore, in step S10, the step of constructing a keyword dictionary includes:

[0015] S11. Extract candidate keywords in the open domain and / or vertical domain by using a word clustering method, and select the top-ranked candidate keywords by using a scoring formula. The scoring formula is:

[0016]

[0017] In the formula: w represents the candidate keyword, domain represents a certain field, ^domain represents a field other than the field, and tf represents the word frequency of a certain w;

[0018] S12, after the scored candidate keywords are sorted and truncated by score, they are processed by noise removal, entity normalization, etc., and then integrated with the entries in the knowledge base to construct a keyword dictionary.

[0019] Further, in step S11, the word clustering method includes one or more of TF-IDF, TextRank, Word2Vec and the like.

[0020] Furthermore, in step S30, the BERT model includes a keyword attention layer and uses an attention mechanism to obtain keyword information.

[0021] Furthermore, the BERT model also includes a dependency syntactic self-attention layer to assist transformer_layer in context encoding through dependency syntactic analysis.

[0022] Furthermore, in step S40, text correction is performed using a text correction tool.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] 1. The present invention constructs a keyword dictionary and a dependency syntax analysis tool to annotate the training data, and trains the training data with keyword annotations through the BERT model to save the optimal training results, thereby introducing more prior information for the data and model, accelerating the convergence speed, and fundamentally solving the pain points in the data and model, and can effectively solve the problems of difficulty in capturing keyword information and difficulty in handling OOV (Out Of Vocabulary).

[0025] 2. By correcting the query input by the user, performing keyword extraction and dependency syntactic analysis on the corrected query and the question to be matched, and building a model input based on the analysis results, and then inputting it into the trained BERT model to obtain text similarity, the accuracy of knowledge text calculation can be effectively improved.

[0026] 3. By improving the BERT model, the model can more effectively extract key information from the text and have stronger context encoding capabilities.

[0027] 4. The added keywords and dependency parsing information during prediction combined with the improved BERT model can have a negative impact on fewer OOV issues. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to make the purpose, technical solution and advantages of the invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0029] Figure 1 It is the overall structure flow chart of the present invention;

[0030] Figure 2 This is a schematic diagram of the BERT model structure;

[0031] Figure 3 Schematic diagram of keyword attention mechanism;

[0032] Figure 4This is a schematic diagram of the mask vector corresponding to [self] in Masked_keyword_self-attention;

[0033] Figure 5 This is a schematic diagram of the syntactic analysis results of sentence A;

[0034] Figure 6 Schematic diagram of the mask vector corresponding to [self] in Masked_DSP_self-attention. DETAILED DESCRIPTION

[0035] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0036] like Figure 1 As shown, a short text similarity calculation method incorporating prior knowledge in this specific implementation includes the following steps:

[0037] S10, constructing a keyword dictionary;

[0038] S20, adding keyword annotations to the training data;

[0039] S30, training the training data with keyword annotations through the BERT model, and saving the optimal model result;

[0040] S40, performing text error correction on the query input by the user;

[0041] S50, extracting keywords and performing dependency syntax analysis on the query after error correction and the question to be matched;

[0042] S60. Construct a model input for the analysis result in step S50, and input it into the trained BERT model to obtain text similarity.

[0043] In the present invention, a keyword dictionary is constructed, and a dependency syntactic analysis tool is used to annotate the training data, and the training data with keyword annotations is trained through a BERT model to save the optimal training results, thereby introducing more prior knowledge into the data and model, accelerating the convergence speed, fundamentally solving the pain points in the data and model, and effectively solving the problems of difficulty in capturing keyword information and difficulty in handling OOV (Out Of Vocabulary). By performing text correction on the query input by the user, performing keyword extraction on the query after correction and the question to be matched, and performing dependency syntactic analysis, and constructing a model input for the analysis results, and then inputting it into the trained BERT model to obtain text similarity, the accuracy of the known text calculation is effectively improved.

[0044] In specific implementation, the steps of constructing a keyword dictionary include:

[0045] S11. Use word clustering methods such as TF-IDF (term frequency-inverse document frequency), TextRank, and Word2Vec to extract candidate keywords in the open domain and / or vertical domain, and select the top-ranked candidate keywords through a scoring formula. The scoring formula is:

[0046]

[0047] Where w represents the candidate keyword, domain represents a certain field, ^domain represents a field other than that field, and tf represents the word frequency of a certain w.

[0048] S12, after the scored candidate keywords are sorted and truncated by score, they are processed by noise removal, entity normalization, etc., and then integrated with the entries in the knowledge base to construct a keyword dictionary.

[0049] In this way, through a variety of word clustering methods, candidate keywords are extracted in the open domain and / or vertical domain (such as automobile and medical fields), and the top-ranked candidate keywords are selected through a scoring formula, and then they are processed by noise removal and entity normalization, and then integrated with the entries in the knowledge base to build a keyword dictionary. The keyword dictionary can reduce the negative impact of the OOV problem.

[0050] In the specific implementation, keywords in the training data are extracted based on the keyword dictionary and annotated; the training data is then analyzed using a dependency syntax analysis tool, and the training data is annotated based on the analysis results.

[0051] BERT is a method of pre-training language representation. It trains a general "language understanding" model on a large amount of text corpus (Wikipedia), and then uses this model to perform the desired NLP task. BERT is better than previous methods because it is an unsupervised, deep bidirectional system used in pre-training NLP.

[0052] BERT inputs random masks into the sequence of words, and its goal is to predict the masked words using contextual information (compared to the traditional left-to-right language model, the mask model can use both the left and right context of the masked words for prediction). The use of the left and right context information of the masked words is achieved through the bidirectional encoder of the Transformer.

[0053] like Figure 2As shown, in order to enable it to capture the added keyword information and better utilize the context for encoding, the present invention improves on the basis of the BERT model structure. In order to obtain additional input keyword information, a keyword attention layer is added to the last layer of BERT: Keyword_attention_layer, and the attention mechanism is used to obtain keyword information. In order to use the dependency syntactic analysis results to assist context encoding, the transformer_layer is improved in the last three layers of BERT, and a dependency syntactic self-attention layer based on dependency syntactic analysis is added: Transformer+DSP (Dependency Syntactic Parsing)_layer, and each is explained separately.

[0054] 1. Keyword_attention_layer

[0055] In order to force the BERT model to pay more attention to the difference between the keywords of the two sentences and learn the essential difference between the two sentences, the keyword information capture layer will pay more attention to the phrases containing keywords.

[0056] like Figure 3 As shown, the attention mechanism is used in this scheme to make each key word in sentence A only focus on each key word in sentence B. In this example, sentence A is "how to start the engine in automatic transmission" and sentence B is "how to start the engine in manual transmission". The specific implementation method is the same as Transformer (existing technology).

[0057] like Figure 4 As shown in the figure, Mask needs to be changed to the corresponding keyword mask, and then the average pooling layer is used to generate the representation vectors of the two sentences. At the same time, the difference vector of the two sentences is also introduced to represent the difference between the two sentences. The formula is as follows:

[0058]

[0059] 2. Transformer+DSP_layer

[0060] Since dependency parsing can analyze the dependency relationship between words in a sentence, it can be used to assist transformer_layer in context encoding. To achieve this goal, when encoding the words of each phrase, only the words of the phrase to which it belongs and other phrases that are dependent or dependent on it are considered.

[0061] The syntactic analysis result of sentence A "How to start the engine in automatic mode" is as follows Figure 5 shown. Figure 6It is the mask corresponding to "[self]" in this sentence. Subsequently, the results of masked_DSP self-attention and multi_head self-attention will be concatenated and input into the next layer.

[0062] Concatenate the hidden vector corresponding to CLS and the h obtained in the keyword_attention_layer kw (A), h kw (B) and the kw_diff vector directly, then input them into the fully connected layer and output the result. Finally, the training data with keyword annotation added is trained through the improved BERT model, and the optimal model result is saved.

[0063] In specific implementation, in step S40, text error correction uses a text error correction tool (prior art).

[0064] During model prediction, first extract keywords and perform dependency syntactic analysis on the query input by the user, then perform the same keyword extraction and dependency syntactic analysis on the question to be matched (this part can be calculated and saved in advance), then form sentence pairs by pairing the query and all questions to be matched one by one and construct the model input in combination with the keyword extraction and dependency syntactic analysis results, and finally input it into the model to obtain the result.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described by referring to the preferred embodiments of the present invention, those of ordinary skill in the art should understand that various changes can be made in form and details. Any obvious changes derived from the technical solutions of the present invention still fall within the protection scope of the present invention.

Claims

1. A short text similarity calculation method incorporating prior knowledge, characterized in that: The steps include: S10, constructing a keyword dictionary; S20, adding keyword annotations to the training data; S30, training the training data with added keyword annotations through the BERT model, and saving the optimal result; S40, performing text error correction on the query input by the user; S50, extracting keywords and performing dependency syntax analysis on the query after error correction and the question to be matched; S60, constructing a model input for the analysis result in step S50, and inputting it into the trained BERT model to obtain text similarity; In step S10, the step of constructing a keyword dictionary includes: S11. Extract candidate keywords in the open domain and / or vertical domain by using a word clustering method, and select the top-ranked candidate keywords by using a scoring formula. The scoring formula is: In the formula: w represents the candidate keyword, domain represents a certain field, ^domain represents a field other than the field, and tf represents the word frequency of a certain w; S12, after the scored candidate keywords are sorted and truncated by score, they are subjected to noise removal and entity normalization processing, and then integrated with the entries in the knowledge base to construct a keyword dictionary.

2. According to the method for calculating short text similarity by incorporating prior knowledge in claim 1, it is characterized in that: In step S11, the word clustering method includes one or more of TF-IDF, TextRank and Word2Vec.

3. The short text similarity calculation method incorporating prior knowledge according to claim 1 is characterized in that: In step S30, the BERT model includes a keyword attention layer and uses an attention mechanism to obtain keyword information.

4. The method for calculating short text similarity by incorporating prior knowledge according to claim 3, characterized in that: The BERT model also includes a dependency syntactic self-attention layer to assist the transformer layer in context encoding through dependency syntactic analysis.

5. The method for calculating short text similarity by incorporating prior knowledge according to claim 1, characterized in that: In step S40, text correction is performed using a text correction tool.

Images