A method for calculating text semantic similarity combined with grammar

Abstract

The present invention discloses a method for calculating text semantic similarity in combination with grammar, comprising obtaining two sentences whose semantic similarity is to be calculated; extracting semantic feature vectors of the two sentences through a deep semantic interaction model; constructing grammatical dependency trees of the two sentences respectively, and performing structured embedding to obtain grammatical tree feature vectors of the two sentences respectively; concatenating the semantic feature vectors of the two sentences with the grammatical tree feature vectors to obtain final semantic feature vectors of the two sentences respectively; and calculating the semantic similarity of the two sentences based on the final semantic feature vectors of the two sentences. The sentence features extracted by the solution of the present invention combine the grammatical information in the sentences, and the extracted features are more comprehensive and in-depth, fully considering the contextual relationship in the sentences, and the calculated similarity is more accurate, while taking into account both accuracy and computational efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a method for calculating text semantic similarity combined with grammar. Background Art

[0002] Research on semantic similarity calculations can help computers better understand the meaning of sentences. Semantic understanding plays a crucial role in various research areas. In information retrieval, semantic similarity calculations can identify search results that best match a question. In community Q&A, semantic similarity calculations can be used to group similar questions together, concentrating responses to a single question. In translation software, semantic similarity can be used as an evaluation metric between source sentences and translations. Therefore, semantic similarity calculations hold significant research significance and value across a wide range of fields.

[0003] Computing semantic similarity is a challenging problem in natural language processing and is also the most widely used technique in text processing. Due to polysemy, synonyms, and complex grammatical structures in text, sentences expressing the same meaning can have a wide variety of forms. Currently, approaches for calculating semantic similarity are generally categorized into two types: semantic representation and semantic interaction. The semantic representation approach calculates a semantic vector for each sentence in a sentence pair and uses these two vectors to calculate similarity. The semantic interaction approach, on the other hand, requires semantic modeling of both sentences simultaneously, taking into account the interaction characteristics between the two sentences during the modeling process to calculate a similarity score. Both approaches have their advantages and disadvantages. The semantic representation approach generally offers lower computational complexity and higher efficiency, but relatively lower accuracy. The semantic interaction approach, on the other hand, is generally more complex and less efficient, but offers relatively higher accuracy. Summary of the Invention

[0004] In view of the shortcomings of the existing technology, the present invention provides a text semantic similarity calculation method combined with grammar to solve the problem that the existing semantic similarity calculation method is difficult to strike a balance between accuracy and computational efficiency.

[0005] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions.

[0006] A method for calculating text semantic similarity combined with grammar, comprising:

[0007] Get two sentences whose semantic similarity is to be calculated;

[0008] Extract semantic feature vectors of two sentences through deep semantic interaction model;

[0009] Construct the grammatical dependency trees of the two sentences respectively, and perform structured embedding to obtain the grammatical tree feature vectors of the two sentences respectively;

[0010] Concatenate the semantic feature vectors of the two sentences with the grammar tree feature vectors to obtain the final semantic feature vectors of the two sentences respectively;

[0011] The semantic similarity of the two sentences is calculated based on the final semantic feature vectors of the two sentences.

[0012] Furthermore, the deep semantic interaction model is a BERT-derived model trained using the word MASK.

[0013] Furthermore, the process of constructing the grammatical dependency tree of the two sentences includes:

[0014] Perform grammatical analysis on the two sentences separately;

[0015] Based on grammatical analysis and using the dependency tree structure defined by Stanford Dependencies, the grammatical dependency trees of the two sentences are obtained.

[0016] Furthermore, the grammatical dependency trees of the two sentences are structured and embedded to obtain the grammatical tree feature vectors of the two sentences respectively. The process includes:

[0017] For each sentence’s grammatical dependency tree, define the grammatical sequence C p are all child nodes of the dependency tree node, and the order is the original order of the words in the sentence;

[0018] Define a maximum sequence length l;

[0019] The grammar sequence C p All elements in the word embedding model are input to calculate and obtain their word embedding; if the grammatical sequence C p The length of the elements in is less than l, the word embedding is padded with zero matrices to reach the maximum sequence length; if the grammatical sequence C p If the length of the elements in the string exceeds l, the elements exceeding the length are truncated and only the first l elements are retained. The newly obtained grammatical sequence is defined as

[0020] Calculate the value of each word in each of the two sentences. Get the grammatical sequence of two sentences and

[0021]

[0022]

[0023] The grammatical sequences of the two sentences are input into a bidirectional LSTM neural network respectively to obtain the grammatical tree embedding of each word in each sentence, and the grammatical tree feature vector of each sentence is obtained by splicing.

[0024] Furthermore, the grammatical sequences of the two sentences are input into a bidirectional LSTM neural network to obtain the grammatical tree embedding of each word in each sentence. The grammatical tree feature vector of each sentence is obtained by splicing, which specifically includes:

[0025] The grammatical sequence of the two sentences is input into a bidirectional LSTM neural network, and the output of the bidirectional LSTM neural network at time t is:

[0026]

[0027] in, represents the forward output of the bidirectional LSTM neural network at time t, represents the reverse output of the bidirectional LSTM neural network at time t, w f and w b represents the hidden layer state of the forward LSTM and the reverse LSTM, b t Represents the offset;

[0028] For the word p, obtain the last layer state of its grammatical sequence in the forward network and the reverse network of the bidirectional LSTM neural network, and construct the grammatical tree embedding V = [E w ,f m ,b n ],in, represents the word embedding of vocabulary p, f m Indicates the calculation result of the last layer of the forward network in the bidirectional LSTM network, b n Represents the calculation result of the last layer of the reverse network in the bidirectional LSTM network;

[0029] For each word in each sentence, its syntax tree embedding is obtained, and the syntax tree feature vector of each sentence is obtained by splicing.

[0030] Furthermore, the semantic similarity of the two sentences is calculated based on the final semantic feature vectors of the two sentences, specifically including:

[0031] The final semantic feature vectors of the two sentences are input into the final prediction layer to calculate the final semantic similarity;

[0032] The calculation process of the prediction layer includes: fusing the final semantic feature vectors of the two sentences and inputting them into a multi-layer perceptron. Each hidden layer of the multi-layer perceptron uses the hyperbolic tangent function tanh as the activation function. The calculation formula of the multi-layer perceptron is:

[0033]

[0034] Among them, s represents the output of the multilayer perceptron, W1 and W2 are parameters, b1 and b2 are offsets, and M A and M B Represent the final semantic feature vectors of the two sentences respectively, and σ represents the tanh activation function;

[0035] The output of the multi-layer perceptron is further fed into a fully connected layer, and the sigmoid function is used as the activation function to obtain a final similarity in the range of [0, 1], which is the semantic similarity between the two sentences.

[0036] Beneficial effects

[0037] The present invention proposes a method for calculating text semantic similarity in combination with grammar. For sentence pairs that need to be calculated, feature extraction is performed on the sentence pairs respectively through a deep semantic interaction model to obtain the semantic feature vectors of the sentence pairs; the grammatical structure of the sentences is analyzed respectively to obtain the grammatical dependency tree of the sentence pairs; the grammatical tree is calculated through a neural network to convert it into a grammatical tree feature vector; the semantic feature vectors of the sentences are spliced ​​with the grammatical tree feature vectors respectively to obtain the final semantic feature vector combined with grammar; the vector distance between the final semantic feature vectors of the sentences is calculated to obtain the semantic similarity between the sentence pairs. The sentence features extracted by the solution of the present invention not only include vocabulary and sentence encoding, but also combine the grammatical features of the sentences. The extracted features are more comprehensive and in-depth, the calculated similarity is more accurate, and the calculation efficiency is high, taking into account the balance between accuracy and calculation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 This is a flow chart of a method for calculating text semantic similarity in combination with grammar provided by an embodiment of the present invention;

[0040] Figure 2 This is a diagram of a semantic feature vector extraction structure provided by an embodiment of the present invention;

[0041] Figure 3 (a) and (b) are two ways of representing the grammatical dependency tree example provided by the embodiment of the present invention;

[0042] Figure 4 Schematic diagram of the syntax tree feature vector extraction structure provided by an embodiment of the present invention;

[0043] Figure 5 It is a schematic diagram of the prediction layer structure provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] To make the objectives, technical solutions, and advantages of the present invention more apparent, the technical solutions of the present invention will be described in detail below. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other implementations obtained by those of ordinary skill in the art without inventive effort are within the scope of protection of the present invention.

[0045] like Figure 1 As shown, the embodiment of the present invention provides a method for calculating text semantic similarity combined with grammar, including:

[0046] S1: Get two sentences whose semantic similarity is to be calculated.

[0047] S2: Extract semantic feature vectors of two sentences through deep semantic interaction model.

[0048] Specifically, because the grammatical dependency tree nodes are words instead of BERT base The word granularity used by the model, the deep semantic interaction model should apply the BERT derivative model trained with full-word mask, such as Chinese-BERT-wwm.

[0049] like Figure 2 As shown in Figure 2, for the sentence pairs A = {A1, A2, ..., An} and B = {B1, B2, ..., Bm} input into the BERT derivative model, they are first concatenated into a sequence x and then input into the embedding layer to obtain their structured embedding representation:

[0050]

[0051] H (0) =Embedding(x)

[0052] H (i) =Transformer(H (i-1) )

[0053] Where L represents the number of layers in the BERT derivative model, N is the maximum sequence length, and d is the dimension of the hidden layer. H represents the hidden layer calculation result, H (0)This is the result of layer 0, which is the initialized matrix vector. Each subsequent layer uses the output of the previous layer as input. Therefore, the initial input for the first layer is a sentence, and the input for each subsequent layer is the output of the previous layer. In the semantic feature vector, the algorithm uses the output of the last layer as the final semantic feature representation. The Embedding calculation method is consistent with that of the word embedding model, and the Transformer calculation method is consistent with that of the Transformer framework proposed by Google. Both are existing technologies and will not be discussed in detail here.

[0054] S3: Construct the grammatical dependency trees of the two sentences respectively, and perform structured embedding to obtain the grammatical tree feature vectors of the two sentences respectively.

[0055] Specifically, the process of constructing the grammatical dependency tree of two sentences includes:

[0056] Perform grammatical analysis on the two sentences separately;

[0057] Based on grammatical analysis and using the dependency tree structure defined by Stanford Dependencies, the grammatical dependency trees of the two sentences are obtained.

[0058] For each sentence, the grammatical dependency tree defines the way in which the words in the sentence express dependency relationships. The way the tree is connected represents its grammatical definition, and the value of the tree node is a word in the sentence. In this way, the grammatical structure in the sentence can be converted into a tree representation, and the grammatical relationship between different components in the sentence can be determined by the connection between the nodes, such as Figure 3 In the example shown, two ways of presentation are used to illustrate "The red car quickly turned around the corner." It can be seen that the relationship between the root node "Turn" and its child node "Direction" is dobj, which represents the direct object.

[0059] like Figure 4 As shown in the figure, the grammatical dependency trees of the two sentences are structured and embedded to obtain the grammatical tree feature vectors of the two sentences respectively. The process includes:

[0060] For each sentence’s grammatical dependency tree, define the grammatical sequence C p are all child nodes of the dependency tree node, and the order is the original order of the words in the sentence;

[0061] In order to prevent the final result dimension from exploding, a maximum sequence length l is defined;

[0062] The grammar sequence C p All elements in are input into the word embedding model to calculate and obtain their word embedding; considering the dimension alignment problem, if the grammatical sequence C pThe length of the elements in C is less than l, and the word embeddings are padded with zero matrices to reach the longest sequence length; if the length of the elements in the syntactic sequence C p is greater than l, the elements exceeding the length are truncated, and only the first l elements are retained. At this time, the newly obtained syntactic sequence is defined as The syntactic sequence here is still the word sequence of all child nodes of a single word. Taking the Figure 3 syntactic dependency tree in as an example, for the word "turn around", its word sequence C p is [sedan, corner, quickly, already, direction]. If l is specified as 3, then the last two words need to be discarded, and only the first 3 are retained. The word sequence is updated to [sedan, corner, quickly]. Similarly, if the sequence length is less than 3, zero matrices are used for padding during the calculation.

[0063] Calculate the for all words in each of the two sentences respectively to obtain the syntactic sequences of the two sentences and

[0064]

[0065]

[0066] Input the syntactic sequences of the two sentences into a bidirectional LSTM neural network respectively. The output of the bidirectional LSTM neural network at time t is:

[0067]

[0068] Among them, represents the forward output of the bidirectional LSTM neural network at time t, represents the backward output of the bidirectional LSTM neural network at time t, w f and w b represent the hidden layer states of the forward LSTM and the backward LSTM, b t represents the offset; the hidden layer state of the neural network is updated according to the following formula: where i is the i-th node in the syntactic sequence; is the hidden layer state of the i-th layer in the sequence of the vocabulary p. The calculation methods of the hidden layer states are the same, but there are forward and backward due to different calculation orders. is the state of the vocabulary p at the i-th layer (i.e., the input at this moment).

[0069] For the vocabulary p, obtain the last layer states of its syntactic sequence in the forward network and the backward network of the bidirectional LSTM neural network, and construct the syntactic tree embedding V of the vocabulary p = [E w , f m , b n],in, represents the word embedding of vocabulary p, f m Indicates the calculation result of the last layer of the forward network in the bidirectional LSTM network, b n Represents the calculation result of the last layer of the reverse network in the bidirectional LSTM network;

[0070] For each word in each sentence, its syntax tree embedding is obtained, and the syntax tree feature vector of each sentence is obtained by splicing.

[0071] S4: Concatenate the semantic feature vectors of the two sentences with the syntax tree feature vectors to obtain the final semantic feature vectors of the two sentences.

[0072] S5: Calculate the semantic similarity of the two sentences based on the final semantic feature vectors of the two sentences. Specifically including:

[0073] The final semantic feature vectors of the two sentences are input into the final prediction layer to calculate the final semantic similarity;

[0074] like Figure 5 As shown in the figure, the calculation process of the prediction layer includes: fusing the final semantic feature vectors of the two sentences and inputting them into a multi-layer perceptron with three hidden layers. The hidden units of the hidden layers are set to 256, 64, and 16 respectively (the number of hidden units can be determined according to the vector calculated in the previous step). Each hidden layer of the multi-layer perceptron uses the hyperbolic tangent function tanh as the activation function. The calculation formula of the multi-layer perceptron is:

[0075]

[0076] Among them, s represents the output of the multilayer perceptron, W1 and W2 are parameters, b1 and b2 are offsets, and M A and M B Represent the final semantic feature vectors of the two sentences respectively, and σ represents the tanh activation function;

[0077] The output of the multi-layer perceptron is further fed into a fully connected layer, and the sigmoid function is used as the activation function to obtain a final similarity in the range of [0, 1], which is the semantic similarity between the two sentences.

[0078] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0079] The present invention can calculate the semantic similarity in sentence pairs. Compared with the traditional literal similarity detection method, the present invention mines the deep semantic information contained in the sentence text through a deep neural network, and can detect more accurate results than the literal similarity method when the literal similarity is very high but the semantics are different. At the same time, the algorithm takes the grammatical structure of the sentence into consideration as semantic information in the calculation process by introducing the grammatical dependency tree, and uses the grammatical information as a feature to enhance the semantic information of the sentence. The present invention can be used as an important auxiliary function of the duplicate checking system, and can also be used as the main calculation method for some classification and question-answering tasks. By combining the semantic calculation model of grammar, the algorithm can extract the semantic features of the text from multiple angles and levels. It mainly overcomes the problem of insufficient Chinese support of the native BERT model by using derivative models trained with full-word masks such as Chinese-BERT-wwm, and uses a multi-head attention mechanism to extract interactive information between texts. At the same time, a mature grammatical dependency tree representation method is introduced to conduct an in-depth analysis of the grammar of the sentence and perform feature extraction from a grammatical perspective. The extracted syntax tree is then fed into a bidirectional LSTM network after word embedding and similarity matrix calculations. The LSTM captures information from the syntax tree and integrates it into final semantic features. These measures enable more effective semantic analysis of sentences, helping computers better understand similar sentences under certain grammatical transformations. These semantic features are then fed into a prediction layer comprised of a multi-layer perceptron and a fully connected layer for dimensionality reduction and interaction. Finally, a sigmoid activation function is used to convert similarity into a specific value in the range [0, 1]. This value is used to measure the semantic similarity between any two sentences. This approach overcomes the semantic gap and improves the accuracy of calculating text semantic similarity.

[0080] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for calculating text semantic similarity combined with grammar, characterized in that: include: Get two sentences whose semantic similarity is to be calculated; Extract semantic feature vectors of two sentences through deep semantic interaction model; Construct the grammatical dependency trees of the two sentences respectively, and perform structured embedding to obtain the grammatical tree feature vectors of the two sentences respectively; Concatenate the semantic feature vectors of the two sentences with the grammar tree feature vectors to obtain the final semantic feature vectors of the two sentences respectively; The semantic similarity of the two sentences is calculated based on the final semantic feature vectors of the two sentences; The grammatical dependency trees of the two sentences are structurally embedded to obtain the grammatical tree feature vectors of the two sentences respectively. The process includes: For each sentence’s grammatical dependency tree, define the grammatical sequence C p are all child nodes of the dependency tree node, and the order is the original order of the words in the sentence; Define a maximum sequence length l; The grammar sequence C p All elements in the word embedding model are input to calculate and obtain their word embedding; if the grammatical sequence C p The length of the elements in is less than l, the word embedding is padded with zero matrices to reach the maximum sequence length; if the grammatical sequence C p If the length of the elements in the string exceeds l, the elements exceeding the length are truncated and only the first l elements are retained. The newly obtained grammatical sequence is defined as Calculate the value of each word in each of the two sentences. Get the grammatical sequence of two sentences and The grammatical sequences of the two sentences are input into a bidirectional LSTM neural network respectively to obtain the grammatical tree embedding of each word in each sentence, and the grammatical tree feature vector of each sentence is obtained by splicing.

2. The method for calculating text semantic similarity in combination with grammar according to claim 1, characterized in that: The deep semantic interaction model is a BERT-derived model trained using the word MASK.

3. The method for calculating text semantic similarity in combination with grammar according to claim 1, characterized in that: The process of constructing the grammatical dependency tree of two sentences includes: Perform grammatical analysis on the two sentences separately; Based on grammatical analysis and using the dependency tree structure defined by Stanford Dependencies, the grammatical dependency trees of the two sentences are obtained.

4. The method for calculating text semantic similarity in combination with grammar according to claim 1, characterized in that: The grammatical sequences of the two sentences are input into a bidirectional LSTM neural network to obtain the grammatical tree embedding of each word in each sentence. The grammatical tree feature vector of each sentence is obtained by splicing, which includes: The grammatical sequence of the two sentences is input into a bidirectional LSTM neural network, and the output of the bidirectional LSTM neural network at time t is: in, represents the forward output of the bidirectional LSTM neural network at time t, represents the reverse output of the bidirectional LSTM neural network at time t, w f and w b represents the hidden layer state of the forward LSTM and the reverse LSTM, b t Represents the offset; For the word p, obtain the last layer state of its grammatical sequence in the forward network and the reverse network of the bidirectional LSTM neural network, and construct the grammatical tree embedding V = [E w ,f m ,b n ],in, represents the word embedding of vocabulary p, f m Indicates the calculation result of the last layer of the forward network in the bidirectional LSTM network, b n Represents the calculation result of the last layer of the reverse network in the bidirectional LSTM network; For each word in each sentence, its syntax tree embedding is obtained, and the syntax tree feature vector of each sentence is obtained by splicing.

5. The method for calculating text semantic similarity in combination with grammar according to claim 1, characterized in that: The semantic similarity of the two sentences is calculated based on the final semantic feature vectors of the two sentences, specifically including: The final semantic feature vectors of the two sentences are input into the final prediction layer to calculate the final semantic similarity; The calculation process of the prediction layer includes: fusing the final semantic feature vectors of the two sentences and inputting them into a multi-layer perceptron. Each hidden layer of the multi-layer perceptron uses the hyperbolic tangent function tanh as the activation function. The calculation formula of the multi-layer perceptron is: Among them, s represents the output of the multilayer perceptron, W1 and W2 are parameters, b1 and b2 are offsets, and M A and M B Represent the final semantic feature vectors of the two sentences respectively, and σ represents the tanh activation function; The output of the multi-layer perceptron is further fed into a fully connected layer, and the sigmoid function is used as the activation function to obtain a final similarity in the range of [0, 1], which is the semantic similarity between the two sentences.

Images