A short text matching method and system

Abstract

The application provides a short text matching method and system. The method can be divided into two aspects: on the one hand, in the aspect of knowledge representation, the method of knowledge representation learning enhancement is adopted, the feature extraction capability is increased through knowledge representation learning in the aspects of word embedding and sentence feature of the text; on the other hand, in the aspect of feature fusion, the recognition rate of the short text matching task is improved by means of an external knowledge base, and the semantic origin knowledge information, the syntax dependency information and the semantic similarity information between sentences of a single sentence are used for multi-feature fusion enhancement. Through knowledge learning representation enhancement and multi-feature fusion, the recognition accuracy of the text matching task is improved, and the development of the sub-tasks such as paraphrasing, reading comprehension and text entailment is promoted.

Description

Technical Field

[0001] This invention relates to the field of text matching technology, and in particular to a short text matching method and system. Background Technology

[0002] Sentence matching is a task that studies the relationships between two natural language processing texts, including calculating their similarity, semantic relationships, and logical reasoning relationships. Based on text length, it can be divided into long text matching and short text matching; based on task type, it can be divided into subtasks such as textual entailment, text paraphrasing, and reading comprehension. Currently, deep learning-based text matching methods can be mainly divided into two types. The first type uses encoders to independently obtain the features of two sentences, and then uses an aggregation mechanism to construct the corresponding feature vectors, such as SWEM and InferSent. The second type uses attention mechanisms to obtain different granularities of sentence encoding for matching, obtaining feature vectors of sentence relationships. The latter appears to be better at obtaining more information about the encoding interactions; typical models include Decomp-Att, ESIM, and DRCN.

[0003] The first approach focuses on encoding the two sentences to be matched independently. For example, Bowman et al. first used deep neural networks to handle text matching tasks. This work used LSTM and RNN to represent the two sentences in SNLI, and finally concatenated the two vectors and input them into an MLP, achieving a result of 77.6% on SNLI. Wang et al. first used LSTM to encode and compare the two sentences, then used an attention mechanism to construct weight matrices for the premise and hypothesis respectively, and then used LSTM for matching. This was an early work that combined attention mechanisms and neural networks for text matching. The SWEM (Simple Word Embedding Model) model proposed by Shen et al. is based on max pooling and average pooling of word vectors. It was evaluated on 17 datasets including SNLI and MultiNLI, and achieved state-of-the-art results in most of them. The InferSent model proposed by Conneau et al., based on the bidirectional recurrent neural network Bi-LSTM, achieved an accuracy of 84.5% on SNLI by using supervised training of sentence representations.

[0004] The second approach, building upon the independent encoding model, significantly enhances the information interaction component, making attention mechanisms a crucial element of text matching models. A representative example is ESIM (Enhanced LSTM) proposed by Chen et al., a two-part model: one uses a sequence model to collect contextual information from words, and the other uses a tree-structured model to collect clause information. Both parts employ attention mechanisms for word matching within sentences, achieving an SNLI accuracy exceeding 88% for the first time. Shen et al. used the bidirectional self-attention mechanism-based classification model DiSAN (Directional Self-Attention Network), achieving 85.6% accuracy on SNLI. This model, based on self-attention, incorporates only a multilayer perceptron in its neural network component. The Decomposable Attention Model (Decomp-Att) proposed by Parikh et al., primarily based on matching mechanisms, combines sentence pair matching with attention mechanisms, calculating word pairs within two sentences while maintaining structural simplicity, achieving the then-current best SNLI result of 86.8%.

[0005] The two deep learning-based text matching algorithms mentioned above can be broadly divided into three stages: text encoding, information interaction, and classifier classification. The corresponding algorithm models can be divided into an encoding layer, an information interaction layer, and an aggregation classification layer, such as... Figure 6 As shown.

[0006] The encoding layer consists of two parts: input encoding and context encoding. Input encoding typically uses word embedding generation models such as Word2vec and GloVe to generate vector representations, while context encoding further extracts text representations from the input word vectors through methods such as convolutional neural networks (CNNs).

[0007] The information interaction layer interacts with the vectors extracted by the encoding layer. The interaction layer typically uses attention mechanisms or recurrent neural networks (RNNs) for information exchange, and employs methods such as vector dot product, vector concatenation, and cosine similarity for similarity calculation, aiming to obtain deep semantic relationship information between two texts.

[0008] The aggregation classification layer is the part that classifies and outputs the comparative information of two sentences. It typically uses pooling, dot product, concatenation, etc. to aggregate text features at the sentence level, and finally uses a softmax-like function for final classification.

[0009] In text matching tasks, although traditional deep learning methods have significantly improved accuracy compared to machine learning methods, their performance in the short text domain remains unsatisfactory, with considerable room for improvement on datasets such as Chinese paraphrasing and NLI. On one hand, while existing deep learning matching algorithms are relatively intelligent, they suffer from poor interpretability and unstable feature extraction processes. The knowledge representations they typically use, such as Word2vec and GloVe, are insufficient to capture global textual knowledge solely from word vectors. On the other hand, commonly used matching methods suffer from grammatical ambiguity, meaning that short text recognition is prone to ambiguity or unclear feature values. In summary, existing text matching algorithms still suffer from insufficient feature learning in knowledge representation and weak information recognition capabilities in short text matching. Summary of the Invention

[0010] To address the problem of low recognition accuracy in current deep learning-based short text matching methods, this invention provides a short text matching method and system.

[0011] On one hand, the present invention provides a short text matching method, comprising:

[0012] Step 1: Obtain the sentence text pairs to be matched, which include premise sentences and hypothesis sentences;

[0013] Step 2: Perform data preprocessing on the sentence text pairs to obtain the word sequence P of the premise sentence and the word sequence H of the hypothesis sentence;

[0014] Step 3: Perform word encoding and sentence encoding on both word sequence P and word sequence H respectively, and then concatenate the word encoding results and sentence encoding results to obtain the enhanced vector sequence of the premise sentence. and the enhanced vector sequence of the hypothetical sentence ;

[0015] Step 4: Extract the semantic primitive feature vector, syntactic dependency feature vector, and semantic similarity vector between the premise sentence and the hypothesis sentence for word sequence P and word sequence H respectively;

[0016] Step 5: Enhance the vector sequence of the premise sentence The semantic primitive feature vector, syntactic dependency feature vector, and semantic similarity vector are fused to obtain the fused feature vector. ; Enhanced vector sequence of the hypothetical sentence The semantic primitive feature vector, syntactic dependency feature vector, and semantic similarity vector are fused to obtain the fused feature vector. ;

[0017] Step 6: Based on the fused feature vector and the fused feature vector The mutual attention mechanism is used to compute the attention vector of the premise sentence after the interaction. Attention vector of the hypothetical sentence ;

[0018] Step 7: Process the fused feature vectors clause by clause. and attention vector Aggregation is performed to obtain the vector weights of each clause in the premise sentence, and the fused feature vector is then analyzed. and attention vector Aggregate the vectors to obtain the weights of each clause in the hypothetical sentence;

[0019] Step 8: Obtain the text matching result based on the vector weights of each clause in the premise sentence and the vector weights of each clause in the hypothesis sentence.

[0020] Furthermore, the data preprocessing includes:

[0021] First, the sentence text to be processed is segmented into Chinese words. Then, PAD padding is used to fill in the positions where the text length is insufficient. Next, CLS tags are constructed at the beginning of the text and SEP tags are constructed at the end of the text. Finally, word order encoding is performed on all segmented words to obtain the word sequence of the sentence text to be processed.

[0022] Furthermore, in step 3, a pre-trained language model is used to perform word encoding processing on word sequence P and word sequence H respectively.

[0023] Furthermore, an LSTM network is used to encode the word sequence P and the word sequence H into complete sentences.

[0024] Furthermore, in step 4, the semantic primitive feature vector is the HowNet semantic primitive feature vector, the semantic similarity vector is the WordNet semantic similarity vector, and the syntactic dependency feature vector is obtained using the DDParser software tool.

[0025] Furthermore, step 5 specifically includes:

[0026] A multilayer perceptron network is used to fuse the semantic primitive feature vector, syntactic dependency feature vector, and semantic similarity vector of the premise sentence or the hypothesis sentence to obtain the multi-feature fusion vector of the premise sentence or the hypothesis sentence.

[0027] A feedforward neural network is used to fuse the multi-feature fusion vector and the enhanced vector sequence of the premise sentence or the hypothesis sentence to obtain the fused feature vector of the premise sentence or the hypothesis sentence.

[0028] Furthermore, step 6 specifically includes:

[0029] The fused feature vector and the fused feature vector Perform alignment to generate a size of Alignment matrix;

[0030] Calculate the fused feature vector using the following formula. Clauses and fused feature vectors Aligned attention weights and the fused feature vector Clauses and fused feature vectors Aligned attention weights ;

[0031]

[0032]

[0033] in, To align the elements in the i-th row and j-th column of the matrix, where m represents the dimension of the word sequence P and n represents the dimension of the word sequence H, express The j-th clause in , express The i-th clause, .

[0034] Furthermore, step 7 specifically includes:

[0035] A feedforward neural network is used to aggregate the fused feature vector and attention vector of the premise sentence or the hypothesis sentence according to the following formula;

[0036]

[0037]

[0038] Where G represents a feedforward neural network, and [;] represents a vector concatenation operation. This represents the vector difference operation. This represents the vector product operation. Indicating clauses Vector weights, Indicating clauses Vector weights.

[0039] Furthermore, step 8 specifically includes:

[0040] The premise sentence or the hypothesis sentence is represented as a whole based on the vector weights of each clause in the premise sentence or the hypothesis sentence, to obtain the overall representation vector of the premise sentence or the hypothesis sentence;

[0041] Maximization pooling layers are used to obtain the maximum weight features of the premise sentence or the hypothesis sentence from the overall representation vector of the premise sentence or the hypothesis sentence, respectively.

[0042] A classifier is used to classify the text based on the maximum weight features of the premise sentence and the hypothesis sentence to obtain the text matching result.

[0043] On the other hand, the present invention provides a short text matching system, comprising:

[0044] A data preprocessing layer is used to obtain sentence text pairs to be matched, wherein the sentence text pairs include a premise sentence and a hypothesis sentence; the sentence text pairs are preprocessed to obtain the word sequence P of the premise sentence and the word sequence H of the hypothesis sentence;

[0045] The knowledge representation encoding layer is used to perform word encoding and whole-sentence encoding on both word sequence P and word sequence H, and then concatenates the word encoding results and whole-sentence encoding results to obtain the enhanced vector sequence of the premise sentence. and the enhanced vector sequence of the hypothetical sentence ;

[0046] A multi-feature fusion layer is used to extract the semantic primitive feature vectors, syntactic dependency feature vectors, and semantic similarity vectors between the premise sentence and the hypothesis sentence, respectively; and to enhance the vector sequence of the premise sentence. The semantic primitive feature vector, syntactic dependency feature vector, and semantic similarity vector are fused to obtain the fused feature vector. ; Enhanced vector sequence of the hypothetical sentence The semantic primitive feature vector, syntactic dependency feature vector, and semantic similarity vector are fused to obtain the fused feature vector. ;

[0047] Attention interaction layer, used to adjust the fused feature vector and the fused feature vector The mutual attention mechanism is used to compute the attention vector of the premise sentence after the interaction. Attention vector of the hypothetical sentence ;

[0048] Aggregation layer, used to process the fused feature vectors on a clause-by-clause basis. and attention vector Aggregation is performed to obtain the vector weights of each clause in the premise sentence, and the fused feature vector is then processed. and attention vector Aggregate the vectors to obtain the weights of each clause in the hypothetical sentence;

[0049] The output layer is used to obtain text matching results based on the vector weights of each clause in the premise sentence and the vector weights of each clause in the hypothesis sentence.

[0050] The beneficial effects of this invention are:

[0051] This invention can be broadly divided into two aspects: Firstly, in terms of knowledge representation, it employs a knowledge representation learning enhancement method, increasing feature extraction capabilities by learning knowledge representations of both word embeddings and sentence features in the text. Secondly, in terms of feature fusion, it utilizes an external knowledge base to improve the recognition rate of short text matching tasks, using multi-feature fusion enhancement based on the semantic primitives of individual sentences, syntactic dependency information, and semantic similarity information between sentences. Through knowledge learning representation enhancement and multi-feature fusion, the recognition accuracy of text matching tasks is improved, thereby promoting the development of sub-tasks such as paraphrasing, reading comprehension, and textual entailment. Attached Figure Description

[0052] Figure 1 This is one of the flowcharts illustrating a short text matching method provided in an embodiment of the present invention;

[0053] Figure 2 This is a second flowchart illustrating a short text matching method provided in an embodiment of the present invention.

[0054] Figure 3 The data preprocessing process provided in the embodiments of the present invention;

[0055] Figure 4 This is the application scenario of the present invention;

[0056] Figure 5 A structural block diagram of a short text matching system provided in an embodiment of the present invention;

[0057] Figure 6 This is the traditional structure of deep learning text matching models in existing technologies. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0059] To better understand the technical solution of this invention, the relevant technical terms involved will be explained here first.

[0060] Premise: also known as the antecedent of text matching, which is the text that forms the basis of reasoning.

[0061] Hypothesis: also known as the consequent of text matching, that is, the result of reasoning.

[0062] Textual relations refer to the inferential relationships between premise sentences and hypothetical sentences. These are generally divided into three types: entailment, contradiction, and neutrality. An entailment indicates that the hypothetical sentence can be inferred from the premise sentence; a contradiction indicates that the premise sentence contradicts the hypothetical sentence; and a neutrality, also known as irrelevance, indicates that the hypothetical sentence cannot be inferred from the premise sentence, nor can a contradiction be deduced.

[0063] Textual Entailment: Textual Entailment Recognition (RTE), also known as Natural Language Inference, is a fundamental task in the field of Natural Language Processing (NLP). The goal of this task is to determine the unidirectional inference relationship between two sentences.

[0064] Text paraphrasing: The main object of text paraphrasing research is language units "above words and below sentences," and does not involve paragraph-level rewriting. In addition to textual similarity, semantic similarity also needs to be considered.

[0065] Attention Mechanism: The attention mechanism, applied to the field of natural language processing, is used to extract the most noteworthy paragraphs or words from a text, which often contains deep semantic information. Mathematically, the attention mechanism can be described as matching a query Q with a series of key-value pairs (K, V) and then outputting the result.

[0066] HowNet: Also known as CNKI, HowNet is a semantic primitive dictionary constructed by Dong Zhendong and others. It includes a fine-grained semantic description framework of more than 2,000 semantic primitives and annotates semantic primitive concepts for hundreds of thousands of Chinese and English words. HowNet adheres to the reductionist approach, that is, the meaning of all words can be composed of smaller semantic units, and these semantic units are called "semantic primitives" (Sememe), which are the most basic and indivisible smallest semantic units.

[0067] WordNet: WordNet is an English lexical semantic network built by a team at Princeton University. This semantic network has a wide coverage, with nouns, verbs, adjectives, and adverbs each organized into a network of synonyms. Each set of synonyms represents a basic semantic concept, and these sets are connected by various relationships. After the English version of WordNet was proposed, several Chinese versions of WordNet were released in China, realizing a semantic network for Chinese.

[0068] The text matching method provided by this invention is for matching sentences of medium length, that is, it can cover most meaningful Chinese texts of 5 to 100 words in a conversation.

[0069] Example 1

[0070] Combination Figure 1 and Figure 2 As shown, this embodiment of the invention provides a short text matching method, including the following steps:

[0071] S101: Obtain the sentence text pairs to be matched, wherein the sentence text pairs include premise sentences and hypothesis sentences;

[0072] S102: Perform data preprocessing on the sentence text pairs to obtain the word sequence P of the premise sentence and the word sequence H of the hypothesis sentence;

[0073] Specifically, the purpose of data preprocessing is to perform preliminary encoding on the two sentence texts to be processed, and output a normalized word sequence. As one possible implementation, data preprocessing specifically includes: first, performing Chinese word segmentation on the sentence texts to be processed; then, padding the text with padding pads at positions where the text length is insufficient; next, constructing a CLS header tag and a SEP footer tag; then, performing word order encoding on all segmented words, finally obtaining the word sequence of the sentence text to be processed, such as... Figure 3 As shown.

[0074] In practical applications, for Chinese text processing, a stop word list is first established. Then, Chinese word segmentation is performed using the third-party library Jieba. Next, text completion is performed using the Unicodedata module. Finally, word order encoding is applied to the text sequence to achieve preliminary data clarification, transformation, and standardization. Stop words refer to common words that are typically automatically filtered or ignored in natural language processing.

[0075] To facilitate understanding of the technical solution of this invention, in this embodiment, the word sequences P and H of the premise sentence and the hypothesis sentence are represented as follows:

[0076] (1)

[0077] (2)

[0078] in, This refers to the i-th participle or clause in the premise sentence. This indicates the j-th participle or clause in a hypothetical sentence.

[0079] S103: Perform word encoding and whole-sentence encoding on both word sequence P and word sequence H, and then concatenate the word encoding results and whole-sentence encoding results to obtain the enhanced vector sequence of the premise sentence. and the enhanced vector sequence of the hypothetical sentence ;

[0080] Specifically, in order to improve the problem that traditional text matching methods do not learn word vector encoding and sentence encoding sufficiently at the knowledge representation level, this invention uses a combination of word embedding encoding and whole sentence encoding compared to traditional vector encoding. By adding attention to the contextual information of the whole sentence on the basis of extracting word vector features, the accuracy of knowledge vector representation is enhanced.

[0081] As one possible implementation, a pre-trained language model (such as BERT) is used to encode word sequences P and H separately, increasing the input of prior knowledge; an LSTM network is used to encode whole sentences for word sequences P and H separately, and the ability of LSTM to acquire temporal text information is combined to enhance its overall sentence-level knowledge representation capability.

[0082] Furthermore, the training dataset for the pre-trained language model uses the BooksCorpus corpus, which covers multiple topics and includes knowledge from over 7,000 books. The goal of pre-training is to train samples against input sentence sequences. Find a set of model parameters that make the predicted probability Maximize this to construct the relative position of a word in the entire word vector space, as shown in Equation (3), where k is the size of the moving window. Generally, the word vector space in the BERT model has a dimension of 768.

[0083] (3)

[0084] Furthermore, the BERT model consists of different self-attention layers, which initialize different Q values ​​through linear projection. The word encoding process of the BERT model for word sequences P and H can be represented as follows:

[0085] (4)

[0086] in, and These are word vector sequences encoded by the BERT model.

[0087] Each control unit in an LSTM network mainly includes an input gate, a forget gate, a cell state, and an output gate. The input gate captures the state from the previous time step, the forget gate uses a sigmoid function to retain or discard some cell state information, the cell state records the information from the current time step, and finally, the output gate outputs the hidden state of the next cell. Similarly, the process of encoding the entire sentence of word sequence P and word sequence H by an LSTM network can be represented as follows:

[0088] (5)

[0089] in, and These are the sentence vector sequences after LSTM encoding.

[0090] Furthermore, the enhanced vector sequence of the premise sentence Enhanced vector sequence of hypothetical sentences It can be represented as follows:

[0091] (6)

[0092] in, This indicates a splicing operation.

[0093] S104: Extract the semantic primitive feature vector, syntactic dependency feature vector, and semantic similarity vector between the premise sentence and the hypothesis sentence, respectively;

[0094] S105: Enhanced vector sequence of the aforementioned premise sentence The semantic primitive feature vector, syntactic dependency feature vector, and semantic similarity vector are fused to obtain the fused feature vector. ; Enhanced vector sequence of the hypothetical sentence The semantic primitive feature vector, syntactic dependency feature vector, and semantic similarity vector are fused to obtain the fused feature vector. ;

[0095] Specifically, in order to improve the problem that existing deep learning-based text matching methods are insufficient in recognizing shorter texts, this invention adopts an external knowledge feature fusion approach to enhance the text matching ability after knowledge representation enhancement encoding. Specifically, it includes using semantic primitive features, semantic similarity features, and syntactic dependency features for multi-feature enhancement to improve the accuracy of text matching.

[0096] Many words in Chinese vocabulary have multiple meanings, which leads to ambiguity. In the field of text matching, ambiguity is even more likely to occur when calculating between two short texts. Therefore, this embodiment of the invention uses semantic primitives to fuse and improve these features.

[0097] S106: Based on the fused feature vector and the fused feature vector The mutual attention mechanism is used to compute the attention vector of the premise sentence after the interaction. Attention vector of the hypothetical sentence ;

[0098] S107: Analyze the fused feature vectors on a clause-by-clause basis. and attention vector Aggregation is performed to obtain the vector weights of each clause in the premise sentence, and the fused feature vector is then analyzed. and attention vector Aggregate the vectors to obtain the weights of each clause in the hypothetical sentence;

[0099] S108: Obtain the text matching result based on the vector weights of each clause in the premise sentence and the vector weights of each clause in the hypothesis sentence.

[0100] The short text matching method provided in this invention is a knowledge representation enhancement and multi-feature fusion-based short text matching method (KPFEM). It can be broadly divided into two aspects: First, in terms of knowledge representation, a knowledge representation learning enhancement method is adopted. This involves learning word embeddings and sentence features from the pre-trained language model BERT and the Long Short-Term Neural Network (LSTM) to increase feature extraction capabilities. Second, in terms of feature fusion, an external knowledge base is used to improve the recognition rate of the short text matching task. This is achieved by using semantic primitives, syntactic dependencies, and semantic similarity information between sentences for multi-feature fusion enhancement. Through knowledge representation enhancement and multi-feature fusion, the recognition accuracy of the text matching task is improved, thereby promoting the development of sub-tasks such as paraphrasing, reading comprehension, and textual entailment.

[0101] Example 2

[0102] Based on the above embodiments, the present invention mainly provides an implementation method for further knowledge enhancement processing of sentence pairs to be matched by combining three parts of external knowledge: semantic primitive features, syntactic dependency features, and semantic similarity features between sentences.

[0103] In this embodiment, the semantic primitive feature vector adopts HowNet semantic primitive feature vector, the semantic similarity vector adopts WordNet semantic similarity vector, and the syntactic dependency feature vector is obtained using the DDParser software tool.

[0104] This embodiment first embeds target word semantic primitives into the premise and hypothesis sentences using fixed HowNet semantic primitives, constructs a semantic primitive matrix, and then calculates a weighted average to obtain the semantic primitive feature information. Second, it uses the Chinese WordNet semantic network to semantically represent entity words and calculates the semantic similarity between the premise and hypothesis sentences, using this as a vector for semantic feature fusion. Finally, it uses syntactic dependency parsing to decompose the sentence into a dependency relation tree, obtaining a syntactic dependency feature vector. In the output of the multi-feature fusion part, the HowNet semantic primitive features, WordNet semantic similarity features, and syntactic dependency feature vectors are fused using a multilayer perceptron neural network to obtain a multi-feature fusion vector used for final text matching information interaction.

[0105] (1) HowNet semantic primitive feature fusion

[0106] HowNet uses a fixed concept to represent the different semantics of words, which can effectively solve the problem of ambiguity. In the semantic primitive feature fusion part, vector concatenation is used to integrate feature information into word vectors, mainly in two steps:

[0107] Step 1: Randomly initialize a primitive vector matrix. The vector matrix is ​​then regularized to obtain an orthogonal matrix. .

[0108] (7)

[0109] in, For one The semantic primitive matrix, where Let n be the orthonormal bases for the corresponding primitive vectors, and let m be the dimension of the primitive vectors.

[0110] Step 2: Treat the target word as a projection of the semantic primitive vectors into the word embedding space. Since each word embedding consists of multiple HowNet semantic primitives, the weighted average of these multiple semantic primitive vectors is used to represent the target word embedding, as shown in the following formula:

[0111] (8)

[0112] in, Let be the semantic primitive vector representation of target word i, and m be the number of semantic primitive vectors of the current target word.

[0113] (2) WordNet semantic feature fusion

[0114] WordNet is a comprehensive lexical semantic network. Nouns, verbs, adjectives, and adverbs are each organized into a network of synonyms, each set representing a basic semantic concept, and these sets are connected by various relations. WordNet itself typically presents information in a categorical manner, without considering modality; all entity relations are represented as true relations, without probabilistic or selective representation. This invention uses a terminology-translated version of Chinese WordNet for further sentence feature enhancement. Furthermore, this embodiment uses the Chinese-English entity mapping of WordNet 3.0 to construct a semantic intermediate layer to calculate the similarity between premise and hypothesis sentences in the text matching task of this invention. Let c represent the features related to the semantic feature concept. Then, the expression for the semantic similarity algorithm model is:

[0115] (9)

[0116] in, Representing concepts Unique characteristics Representing concepts Unique characteristics , , These are the parameters of semantic common features and conceptual features, and F is the significance test function for the feature parameters.

[0117] Describes the concept and The Minimum Specific Common Abstraction (MSCA) and the metric for semantic feature similarity, assuming a form for quantifying semantic features can be found, can be used to derive semantic feature similarity from the above expression:

[0118] (10)

[0119] Wherein, IC represents the amount of semantic feature information. and Quantization function.

[0120] (3) Fusion of dependency syntax features

[0121] Syntactic dependency relations include subject-verb (SBV), verb-object (VOB), prepositional phrase object (POB), and coordinate (COO) tags, which are automatically annotated using the DDParser software tool to obtain 14 types of relations. Dependency parsing is the process of analyzing statements... Parse into a dependency syntax tree, then add a root vector and calculate the transition probability sequence:

[0122] (11)

[0123] (12)

[0124] in, It is a line from the central word To modifiers The dependency arc, the arc is labeled as Let L be the set of tags. The input premise and hypothesis sentences are mapped into n vectors respectively, resulting in a vector sequence of dependency syntax tags.

[0125] Furthermore, feature fusion is performed on HowNet semantic primitives, WordNet semantic similarity vectors, and syntactic dependency feature vectors using a multilayer perceptron network (MLP).

[0126] (13)

[0127] (14)

[0128] in, This is a text sequence after multi-feature fusion. and These are the multi-feature fusion sequence results for the premise sentence and the hypothesis sentence, respectively; both the premise sentence and the hypothesis sentence are input as parallel text sequences, and are processed by a feedforward network G with the representation-enhanced sequence. , To integrate.

[0129] Based on the vector representation of the previous knowledge representation encoding layer, this invention enriches and enhances the semantic and syntactic information of sentences by combining HowNet semantic primitives, WordNet semantic features, and dependency syntax features. This allows the text representation to incorporate more external knowledge, resulting in more significant feature extraction.

[0130] This invention employs external knowledge encoding as a representation enhancement method for text matching to address the issues of limited sentence information and insufficient contextual semantics in short texts. In the knowledge representation enhancement process, firstly, a pre-trained language model BERT is used for word-level vector encoding to enhance the knowledge features of word embeddings. Secondly, a Long Short-Term Neural Network (LSTM) is used to perform contextual comprehensive representation of the entire text sentence to enhance the knowledge features at the whole sentence level. Finally, the word embedding enhancement vector and the overall sentence enhancement vector are combined through vector concatenation to obtain external knowledge enhancement representations at both the word and sentence levels. This effectively improves the insufficient pre-existing information in short sentences during text matching tasks and enhances the accuracy of posterior probabilities during text encoding.

[0131] Example 3

[0132] Based on the above embodiments, the present invention mainly provides a method for calculating the attention vector of the premise sentence after interaction using a mutual attention mechanism. Attention vector of the hypothetical sentence The feasible implementation method includes the following steps:

[0133] The fused feature vector and the fused feature vector Perform alignment to generate a size of The alignment matrix; that is, using the sequence , Each row and column is represented by a row and column of a matrix, respectively, and the words or clauses in the sequence are used to construct a... matrix ;

[0134] Calculate the fused feature vector using the following formula. Clauses and fused feature vectors Aligned attention weights and the fused feature vector Clauses and fused feature vectors Aligned attention weights ;

[0135] (15)

[0136] (16)

[0137] in, To align the elements in the i-th row and j-th column of the matrix, where m represents the dimension of the word sequence P and n represents the dimension of the word sequence H, express The j-th clause in , express The i-th clause, .

[0138] The embodiments of the present invention are as follows: Calculate respectively Compared to attention weights ,as well as Compared to attention rights This allows us to obtain more closely related parts of the relationships between words or clauses. Because of the use of mutual attention mechanisms, more interaction information is obtained compared to self-attention computation of the sentence itself, which helps in determining the relationships between sentences.

[0139] Furthermore, in this embodiment of the invention, the fused feature vector and attention vector of the premise sentence or hypothesis sentence are aggregated using the following aggregation method:

[0140] A feedforward neural network is used to aggregate the fused feature vector and attention vector of the premise sentence or the hypothesis sentence according to the following formula;

[0141] (17)

[0142] (18)

[0143] Where G represents a feedforward neural network, and [;] represents a vector concatenation operation. This represents the vector difference operation. This represents the vector product operation. Indicating clauses Vector weights, Indicating clauses Vector weights.

[0144] In this embodiment of the invention, to highlight the vector nature of text matching relationships, concatenation, dot product, and vector difference operations are used as the aggregation mechanism of KPFEM during the aggregation process. First, the vectors in the two sequences of the external knowledge enhancement layer are... , vectors in , Sentence weights in the multi-feature fusion layer , Aggregation calculations are performed to obtain information aggregation for knowledge representation enhancement and multi-feature fusion. The attention weight vector and text vector are compared to further extract interaction information between the two texts. Then, a feedforward neural network G is used to... and , and The two pairs of sequences are aggregated to obtain the clause vector weights in the premise and hypothesis sentences. and .

[0145] Furthermore, in this embodiment of the invention, based on the vector weights of each clause in the premise sentence and the vector weights of each clause in the hypothesis sentence, the text matching result is obtained in the following manner:

[0146] The premise sentence or the hypothesis sentence is represented as a whole based on the vector weights of each clause in the premise sentence or the hypothesis sentence, resulting in the overall representation vector of the premise sentence or the hypothesis sentence; as shown in formula (19):

[0147] (19)

[0148] Maximization pooling layers are used to obtain the maximum weight features of the premise sentence or the hypothesis sentence from the overall representation vector of the premise sentence or the hypothesis sentence, respectively.

[0149] A classifier is used to classify the text based on the maximum weight features of the premise sentence and the hypothesis sentence, thus obtaining the text matching result. This is shown in formulas (20) and (21).

[0150] (20)

[0151] (twenty one)

[0152] Where H represents a classifier composed of fully connected neural networks, designed for feature compression; MaxPooling represents the max-pooling function. These represent the aggregate vectors of the premise and hypothesis sentences after max pooling, respectively. This indicates that the last sentence matches the category label.

[0153] Example 4

[0154] Corresponding to the above methods, embodiments of the present invention provide a short text matching system based on knowledge representation enhancement and multi-feature fusion, such as... Figure 4 As shown, it includes a data preprocessing layer, a knowledge representation and encoding layer, a multi-feature fusion layer, an attention interaction layer, an aggregation layer, and an output layer.

[0155] A data preprocessing layer is used to obtain sentence text pairs to be matched, wherein the sentence text pairs include a premise sentence and a hypothesis sentence; the sentence text pairs are preprocessed to obtain the word sequence P of the premise sentence and the word sequence H of the hypothesis sentence;

[0156] The knowledge representation encoding layer is used to perform word encoding and whole-sentence encoding on both word sequence P and word sequence H, and then concatenates the word encoding results and whole-sentence encoding results to obtain the enhanced vector sequence of the premise sentence. and the enhanced vector sequence of the hypothetical sentence ;

[0157] A multi-feature fusion layer is used to extract the semantic primitive feature vectors, syntactic dependency feature vectors, and semantic similarity vectors between the premise sentence and the hypothesis sentence, respectively; and to enhance the vector sequence of the premise sentence. The semantic primitive feature vector, syntactic dependency feature vector, and semantic similarity vector are fused to obtain the fused feature vector. ; Enhanced vector sequence of the hypothetical sentence The semantic primitive feature vector, syntactic dependency feature vector, and semantic similarity vector are fused to obtain the fused feature vector. ;

[0158] Attention interaction layer, used to adjust the fused feature vector and the fused feature vector The mutual attention mechanism is used to compute the attention vector of the premise sentence after the interaction. Attention vector of the hypothetical sentence ;

[0159] Aggregation layer, used to process the fused feature vectors on a clause-by-clause basis. and attention vector Aggregation is performed to obtain the vector weights of each clause in the premise sentence, and the fused feature vector is then analyzed. and attention vector Aggregate the vectors to obtain the weights of each clause in the hypothetical sentence;

[0160] The output layer is used to obtain text matching results based on the vector weights of each clause in the premise sentence and the vector weights of each clause in the hypothesis sentence.

[0161] It should be noted that the system provided in this embodiment of the invention is for implementing the above methods, and its specific functions can be referred to the above method embodiments, which will not be repeated here.

[0162] This invention can be applied to document similarity recognition, logical implication reasoning, and security classification. For similarity calculation, two texts of approximately the same length can be input, and the KPFEM text matching model can be used to calculate their similarity. This model is more accurate and efficient than commonly used cosine similarity and Euclidean distance algorithms, with applications including expert question answering and medical diagnosis matching. For logical implication reasoning, the input consists of a preceding text and a following text. The model can be used to infer whether the preceding text can lead to the following text, i.e., the logical implication relationship between the two. Applications typically include reading comprehension, text paraphrasing, and automatic condensation. In the field of security classification, the model compares the text to be identified with existing texts to obtain the security classification of the text to be identified, which can be used for security checks and for confidential evidence collection. The overall application background of this invention is as follows: Figure 5 As shown.

[0163] This invention proposes a text matching method and system based on knowledge representation enhancement and multi-feature fusion. Building upon existing text matching methods, it enhances knowledge at two levels: word embedding and sentence representation using BERT and LSTM, respectively. By incorporating HowNet semantic primitives, WordNet semantic similarity information, and syntactic dependency label sequences, it effectively improves the text matching recognition success rate. Simultaneously, it uses a pre-trained model, recurrent neural networks, and external knowledge to fuse "world knowledge," enhancing the system's robustness and accuracy. Finally, in text matching and approximation tasks in certain domains, support from natural language rules such as grammatical and semantic analysis is required. Therefore, using dependency parsing and semantic primitives for feature fusion at the grammatical rule level can alleviate the gradient information decay problem associated with using deep neural networks alone.

[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A short text matching method, characterized in that, include: Step 1: Obtain the sentence text pairs to be matched, which include premise sentences and hypothesis sentences; Step 2: Perform data preprocessing on the sentence text pairs to obtain the word sequence P of the premise sentence and the word sequence H of the hypothesis sentence; Step 3: Perform word encoding and sentence encoding on both word sequence P and word sequence H respectively, and then concatenate the word encoding results and sentence encoding results to obtain the enhanced vector sequence of the premise sentence. and the enhanced vector sequence of the hypothetical sentence Specifically, a pre-trained language model is used to encode word sequences P and H respectively; an LSTM network is used to encode entire sentences for word sequences P and H respectively. Step 4: Extract the semantic primitive feature vector, syntactic dependency feature vector, and semantic similarity vector between the premise sentence and the hypothesis sentence for word sequence P and word sequence H respectively; the semantic primitive feature vector is HowNet semantic primitive feature vector, the semantic similarity vector is WordNet semantic similarity vector, and the syntactic dependency feature vector is obtained using the DDParser software tool; Step 5: Enhance the vector sequence of the premise sentence The semantic primitive feature vector, syntactic dependency feature vector, and semantic similarity vector are fused to obtain the fused feature vector. ; The enhanced vector sequence of the hypothetical sentence The semantic primitive feature vector, syntactic dependency feature vector, and semantic similarity vector are fused to obtain the fused feature vector. ; Step 6: Based on the fused feature vector and the fused feature vector The mutual attention mechanism is used to compute the attention vector of the premise sentence after the interaction. Attention vector of the hypothetical sentence ; Step 7: Process the fused feature vectors clause by clause. and attention vector Aggregation is performed to obtain the vector weights of each clause in the premise sentence, and the fused feature vector is then analyzed. and attention vector Aggregate the vectors to obtain the weights of each clause in the hypothetical sentence; Step 8: Obtain the text matching result based on the vector weights of each clause in the premise sentence and the vector weights of each clause in the hypothesis sentence.

2. The short text matching method according to claim 1, characterized in that, The data preprocessing includes: First, the sentence text to be processed is segmented into Chinese words. Then, PAD padding is used to fill in the positions where the text length is insufficient. Next, CLS tags are constructed at the beginning of the text and SEP tags are constructed at the end of the text. Finally, word order encoding is performed on all segmented words to obtain the word sequence of the sentence text to be processed.

3. The short text matching method according to claim 1, characterized in that, Step 5 specifically includes: A multilayer perceptron network is used to fuse the semantic primitive feature vector, syntactic dependency feature vector, and semantic similarity vector of the premise sentence or the hypothesis sentence to obtain the multi-feature fusion vector of the premise sentence or the hypothesis sentence. A feedforward neural network is used to fuse the multi-feature fusion vector and the enhanced vector sequence of the premise sentence or the hypothesis sentence to obtain the fused feature vector of the premise sentence or the hypothesis sentence.

4. The short text matching method according to claim 1, characterized in that, Step 6 specifically includes: The fused feature vector and the fused feature vector Perform alignment to generate a size of Alignment matrix; Calculate the fused feature vector using the following formula. Clauses and fused feature vectors Aligned attention weights and the fused feature vector Clauses and fused feature vectors Aligned attention weights ; in, To align the elements in the i-th row and j-th column of the matrix, where m represents the dimension of the word sequence P and n represents the dimension of the word sequence H, express The j-th clause in , express The i-th clause, .

5. The short text matching method according to claim 4, characterized in that, Step 7 specifically includes: A feedforward neural network is used to aggregate the fused feature vector and attention vector of the premise sentence or the hypothesis sentence according to the following formula; Where G represents a feedforward neural network, and [;] represents a vector concatenation operation. This represents the vector difference operation. This represents the vector product operation. Indicating clauses Vector weights, Indicating clauses Vector weights.

6. The short text matching method according to claim 1, characterized in that, Step 8 specifically includes: The premise sentence or the hypothesis sentence is represented as a whole based on the vector weights of each clause in the premise sentence or the hypothesis sentence, to obtain the overall representation vector of the premise sentence or the hypothesis sentence; Maximization pooling layers are used to obtain the maximum weight features of the premise sentence or the hypothesis sentence from the overall representation vector of the premise sentence or the hypothesis sentence, respectively. A classifier is used to classify the text based on the maximum weight features of the premise sentence and the hypothesis sentence to obtain the text matching result.

7. A short text matching system, characterized in that, include: A data preprocessing layer is used to obtain sentence-text pairs to be matched, wherein the sentence-text pairs include premise sentences and hypothesis sentences; The sentence text pairs are preprocessed to obtain the word sequence P of the premise sentence and the word sequence H of the hypothesis sentence; The knowledge representation encoding layer is used to perform word encoding and whole-sentence encoding on both word sequence P and word sequence H, and then concatenates the word encoding results and whole-sentence encoding results to obtain the enhanced vector sequence of the premise sentence. and the enhanced vector sequence of the hypothetical sentence Specifically, a pre-trained language model is used to encode word sequences P and H respectively; an LSTM network is used to encode entire sentences for word sequences P and H respectively. A multi-feature fusion layer is used to extract the semantic primitive feature vectors, syntactic dependency feature vectors, and semantic similarity vectors between the premise sentence and the hypothesis sentence, respectively; and to enhance the vector sequence of the premise sentence. The semantic primitive feature vector, syntactic dependency feature vector, and semantic similarity vector are fused to obtain the fused feature vector. ; Enhanced vector sequence of the hypothetical sentence The semantic primitive feature vector, syntactic dependency feature vector, and semantic similarity vector are fused to obtain the fused feature vector. The semantic primitive feature vector is the HowNet semantic primitive feature vector, the semantic similarity vector is the WordNet semantic similarity vector, and the syntactic dependency feature vector is obtained using the DDParser software tool. Attention interaction layer, used to adjust the fused feature vector and the fused feature vector The mutual attention mechanism is used to compute the attention vector of the premise sentence after the interaction. Attention vector of the hypothetical sentence ; Aggregation layer, used to process the fused feature vectors on a clause-by-clause basis. and attention vector Aggregation is performed to obtain the vector weights of each clause in the premise sentence, and the fused feature vector is then analyzed. and attention vector Aggregate the vectors to obtain the weights of each clause in the hypothetical sentence; The output layer is used to obtain text matching results based on the vector weights of each clause in the premise sentence and the vector weights of each clause in the hypothesis sentence.

Images