Short text similarity calculation method based on deep learning and topic model
By combining deep learning and thematic model, the context and emotional characteristics of short text are extracted, and the problems of "one word multiple meanings" and emotional color neglect in the prior art are solved, and a more accurate calculation of short text similarity is achieved.
Patent Information
- Application Number
- CN202210544327.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-19
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-05-19
AI Technical Summary
The existing short text semantic similarity calculation method has deviations in dealing with the problem of "one word's multiple meanings" and ignoring the emotional color of text, resulting in a large difference between the calculation results and the actual situation.
Using a method based on deep learning and thematic model, word segmentation and vector transformation is performed through BERT, context information is extracted in combination with bidirectional LSTM and CNN networks, and emotional features are extracted using LDA sentiment analysis model, and similarity calculation is performed by fusing semantics and emotional information.
The accuracy of short text similarity calculation is improved, the impact of "one word multiple meanings" on the results is reduced, and the consideration of text emotional color is enhanced, and the calculation results are obtained that are closer to reality.
Smart Images

Figure CN114897078B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of text similarity matching, and in particular relates to a short text similarity calculation method based on deep learning and a topic model. Background Art
[0002] With the rapid development of the Internet, the field of natural language processing has developed rapidly, and short text information processing is particularly important. It is not only the basis of various functions, but also the focus of research in recent years. The problem of short text similarity calculation is an important branch of short text information processing. It has a large number of applications in Internet search engines, intelligent question answering, machine translation, information filtering and retrieval. At home and abroad, most of the cutting-edge technologies for studying the similarity between two entities are at the semantic level.
[0003] At present, most of the short text semantic similarity calculation methods are based on deep learning, such as RNN recurrent neural network, CNN convolutional neural network and other different networks or their combinations. The basic process of traditional deep learning text semantic similarity calculation is as follows: Figure 2 shown.
[0004] In deep learning networks, one-hot encoding is often used to segment and vectorize the input content for deep learning model calculation. The advantage of this encoding method is that some discrete data can be calculated in a centralized manner, and on the other hand, the relevant features of the text can be deeply mined to expand the semantic relevance of the text. However, this encoding method does not consider the connection between the context of the text content, which makes some words affect the final calculation results when calculating the text similarity. For example, the word "proud" has both the meaning of being conceited and despising others, and the meaning of pride. "Xiao Ming got the first place in the exam, he is very proud." and "Xiao Ming got the first place in the exam, he is the pride of his mother." Both sentences use pride, but the meanings behind them are very different. If the context is not considered, using only one word to segment and encode the sentence will cause a huge deviation from the actual result. Therefore, this method uses BERT for word segmentation and vector conversion, and solves the problem of polysemy by connecting the context.
[0005] RNN is a recurrent computing network. Although it can obtain the connection between vector contexts, it can only maintain a one-way order, which limits the computing power of the model to a certain extent and does not solve the problem of long-term dependence on text. LSTM is a special RNN recurrent neural network. It adds three interactive layers to the ordinary RNN network, which solves the long-term dependency problem to a certain extent, but cannot focus on extracting features of some areas. CNN network is often used in the field of graphics processing. Its characteristic is to highlight the features within the receptive field, in short, it highlights the features of some areas. However, when the network is used in the text field, it cannot consider the connection between the text content context. Therefore, in this model, the LSTM recurrent neural network and the CNN convolutional neural network are integrated to connect the context and highlight the key parts of the text content.
[0006] When using a single deep learning model to calculate text similarity, facing the problem of "polysemy", the emotional color of the text cannot be considered, which will cause the calculation results to deviate from the results of human judgment. Therefore, introducing the LDA topic model into the similarity calculation of this article can play its role in sentiment classification of texts, thereby further enhancing the accuracy of text similarity calculation. Summary of the invention
[0007] The purpose of the present invention is to provide a short text similarity calculation method based on deep learning and topic model, which integrates the traditional semantic text similarity calculation method based on deep learning with the sentiment analysis topic model to improve the technical problem of text similarity calculation accuracy.
[0008] To achieve the above object, the present invention adopts the following technical solution:
[0009] The short text similarity calculation method based on deep learning and topic model includes the following steps:
[0010] 1) Use BERT to perform word segmentation and vector conversion on the two short texts input into the system, and save the word segmentation results and vector conversion results separately;
[0011] The specific method of word segmentation and vector conversion is: segment the two short texts through the BERT model respectively, save the results of word segmentation, and then vectorize the results of BERT word segmentation into word vector representations that can be processed by the deep learning network, and use the word vector x as input.
[0012] 2) Extract fused feature information from the word vector after vector conversion: First, use a bidirectional LSTM network to extract context information, then use a 1D convolutional neural network to fuse the extracted context information with the word embedding information, and then use global maximum pooling to extract key information to obtain multi-dimensional, multi-feature fused feature information;
[0013] The specific method for extracting the fusion feature information of word vectors is:
[0014] Input the word vector x into the deep learning model and use hash encoding to get l 1 layer;
[0015] l 1 =W 1 *x
[0016] Among them l 1 is the output after word vector conversion, W 1 is the hash mapping matrix, and x is the word vector input;
[0017] Then, the 128-dimensional low-dimensional vector y is obtained by sequentially passing through the three-layer network of LSTM and CNN fusion, and then the eigenvalue output is optimized through global maximum pooling. The calculation process is as follows:
[0018] l i =f(W i *l i-1 +b i ),i=2,…,N-1
[0019]
[0020] in represents the jth pooling operation, and the final output vector dimension is M, l i represents the hidden layer nodes, W i It is the first i The weight matrix of the layer, b i For the first i The bias of , f is the activation function of the hidden layer and the output layer, and the activation function used is tanh:
[0021]
[0022] 3) Extract sentiment features from the two texts after word segmentation: input them into the LDA sentiment analysis topic model, use Gibbs sampling to extract the sentiment color of the text, and extract the sentiment features of the two texts;
[0023] The specific method for extracting the sentiment features of two paragraphs of text is:
[0024] Input the results of BERT word segmentation into the model and calculate the probability distribution of sentiment attributes for each paragraph of text:
[0025] P(w,z|α,β)=p(w|z,w)*p(z|α)
[0026] Gibbs sampling is used to calculate the relationship between data and topics. w represents a word in the text, α and β are preset hyperparameters, and z represents the number of the topic.
[0027] According to the Bayesian formula and the Dirichlet prior distribution, the Dirichlet distribution expectation is calculated:
[0028]
[0029]
[0030] k represents the number of topics, θ m,k represents the distribution of topic k in data m, φ k,t represents the word distribution of feature word t in topic k, n m,(k) Represents the characteristic vocabulary of topic k in data m, n k,(t) Indicates the number of occurrences of feature word t in topic k;
[0031] Probability distribution obtained by Gibbs sampling:
[0032]
[0033] represents the number of feature words in data topic k that do not belong to data m, Indicates the number of times a feature word is not assigned to topic word k.
[0034] 4) performing similarity calculation on the fusion features obtained in step 2), performing sentiment classification judgment on the sentiment features obtained in step 3), and performing hybrid text semantic similarity calculation on the above two results to obtain the text semantic similarity between the two texts;
[0035] The similarity calculation of the fusion features uses the following formula:
[0036]
[0037] where y Q Represents the output result of the first short text after the deep learning network, y D Represents the output result of the second sentence after passing through the deep learning network.
[0038] The emotional features are classified and judged using the following formula:
[0039] Li =max(softmax(P)),i=1,2
[0040]
[0041] Where P represents the sentiment features output by the LDA model, i represents the different texts input into the LDA model, when B is 0, it indicates that the sentiment classification of the two texts is different, and when B is 1, it indicates that the sentiment classification of the two texts is the same.
[0042] The formula for calculating the text semantic similarity between two texts is:
[0043] S=εR+(1-ε)B
[0044] Where R represents the similarity extracted by the deep learning network, B represents the similarity of the sentiment categories to which the two texts belong, and ε represents the weight of the similarity extracted by the deep learning network in the final result output.
[0045] Compared with the prior art, the present invention has the following beneficial effects: after the two text vectors are converted, they are simultaneously input into the deep learning network and the LDA sentiment topic model, the output results of the deep learning network and the LDA sentiment topic model are integrated to obtain multi-dimensional and multi-granular semantic similarity feature information, and then the text similarity is calculated by the fusion semantic similarity calculation method. Under the premise of considering the text semantics, the emotional color of the two texts is fully analyzed to avoid the influence of "polysemy" on the final judgment result. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 It is a step diagram of the short text similarity calculation method based on deep learning and topic model in the present invention.
[0047] Figure 2 It is a flow chart of the text similarity calculation system under the existing technology.
[0048] Figure 3 It is a flowchart of the short text similarity calculation method based on deep learning and topic model in the present invention.
[0049] Figure 4 This is an architecture diagram of the short text similarity calculation model based on deep learning and topic model in the present invention. DETAILED DESCRIPTION
[0050] The present invention will be further described below with reference to the accompanying drawings and examples.
[0051] The embodiment of the present invention provides a short text similarity calculation method based on deep learning and topic model, which is used to solve the problem of calculating the semantic similarity of short texts through a deep learning network and a topic model.
[0052] The short text similarity calculation method based on deep learning and topic model includes the following steps:
[0053] 1) Use BERT to perform word segmentation and vector conversion on the two short texts input into the system, and save the word segmentation results and vector conversion results separately;
[0054] The specific method of word segmentation and vector conversion is: segment the two short texts through the BERT model respectively, save the results of word segmentation, and then vectorize the results of BERT word segmentation into word vector representations that can be processed by the deep learning network, and use the word vector x as input.
[0055] 2) Extract fused feature information from the word vector after vector conversion: First, use a bidirectional LSTM network to extract context information, then use a 1D convolutional neural network to fuse the extracted context information with the word embedding information, and then use global maximum pooling to extract key information to obtain multi-dimensional, multi-feature fused feature information;
[0056] The specific method for extracting the fusion feature information of word vectors is:
[0057] Input the word vector x into the deep learning model and use hash encoding to get l 1 layer;
[0058] l 1 =W 1 *x
[0059] Among them l 1 is the output after word vector conversion, W 1 is the hash mapping matrix, and x is the word vector input;
[0060] Then, the 128-dimensional low-dimensional vector y is obtained by sequentially passing through the three-layer network of LSTM and CNN fusion, and then the eigenvalue output is optimized through global maximum pooling. The calculation process is as follows:
[0061] l i =f(W i *l i-1 +b i ),i=2,…,N-1
[0062]
[0063] in represents the jth pooling operation, and the final output vector dimension is M, l i represents the hidden layer nodes, W i It is the first i The weight matrix of the layer, b i For the firsti The bias of , f is the activation function of the hidden layer and the output layer, and the activation function used is tanh:
[0064]
[0065] 3) Extract sentiment features from the two texts after word segmentation: input them into the LDA sentiment analysis topic model, use Gibbs sampling to extract the sentiment color of the text, and extract the sentiment features of the two texts;
[0066] The specific method for extracting the sentiment features of two paragraphs of text is:
[0067] Input the results of BERT word segmentation into the model and calculate the probability distribution of sentiment attributes for each paragraph of text:
[0068] P(w,z|α,β)=p(w|z,w)*p(z|α)
[0069] Gibbs sampling is used to calculate the relationship between data and topics. w represents a word in the text, α and β are preset hyperparameters, and z represents the number of the topic.
[0070] According to the Bayesian formula and the Dirichlet prior distribution, the Dirichlet distribution expectation is calculated:
[0071]
[0072]
[0073] k represents the number of topics, θ m,k represents the distribution of topic k in data m, φ k,t represents the word distribution of feature word t in topic k, n m,(k) Represents the characteristic vocabulary of topic k in data m, n k,(t) Indicates the number of occurrences of feature word t in topic k;
[0074] Probability distribution obtained by Gibbs sampling:
[0075]
[0076] represents the number of feature words in data topic k that do not belong to data m, Indicates the number of times a feature word is not assigned to topic word k.
[0077] 4) performing similarity calculation on the fusion features obtained in step 2), performing sentiment classification judgment on the sentiment features obtained in step 3), and performing hybrid text semantic similarity calculation on the above two results to obtain the text semantic similarity between the two texts;
[0078] The similarity calculation of the fusion features uses the following formula:
[0079]
[0080] where y Q Represents the output result of the first short text after the deep learning network, y D Represents the output result of the second sentence after passing through the deep learning network.
[0081] The emotional features are classified and judged using the following formula:
[0082] L i =max(softmax(P)),i=1,2
[0083]
[0084] Where P represents the sentiment features output by the LDA model, i represents the different texts input into the LDA model, when B is 0, it indicates that the sentiment classification of the two texts is different, and when B is 1, it indicates that the sentiment classification of the two texts is the same.
[0085] The formula for calculating the text semantic similarity between two texts is:
[0086] S=εR+(1-ε)B
[0087] Where R represents the similarity extracted by the deep learning network, B represents the similarity of the sentiment categories to which the two texts belong, and ε represents the weight of the similarity extracted by the deep learning network in the final result output.
[0088] Example 1
[0089] S100. For two short texts input into the system, the short text Q is: "Xiao Ming did well in the final exam this time, he is the pride of his mother", and the short text D is: "Xiao Ming did well in the final exam this time, he is a little proud". BERT is used to perform word segmentation and vector conversion respectively, and the word segmentation results and vector conversion results are saved separately.
[0090] S200, for the converted word vector, first use a bidirectional LSTM network to extract context information, then use a 1D convolutional neural network to fuse the extracted context information with the word embedding information, and then use global maximum pooling to extract key information to obtain multi-dimensional, multi-feature fused feature information.
[0091] S300, the two text segments after word segmentation are input into the LDA sentiment analysis topic model, and Gibbs sampling is used to extract the sentiment color of the text, thereby extracting the sentiment features of the two text segments.
[0092] S400, performing similarity calculation on the fusion features, and performing sentiment classification judgment on the sentiment features, and performing hybrid text semantic similarity calculation on the results of the two to obtain the text semantic similarity between the two texts.
[0093] Wherein, S100 includes the following steps:
[0094] (1) The two short texts are segmented using the BERT model. The short text Q is: "Xiao Ming did well in the final exam this time. He is the pride of his mother", and the short text D is: "Xiao Ming did well in the final exam this time. He is a little proud". The segmented results are saved.
[0095] (2) The result of BERT word segmentation is transformed into a vector representation that can be processed by a deep learning network, and the word vector x is used as input.
[0096] Wherein, S200 includes the following steps:
[0097] (1) Input the word vector x into the deep learning model and use hash encoding to obtain l 1 layer.
[0098] l 1 =W 1 *x
[0099] Among them l 1 is the output after word vector conversion, W 1 is the hash mapping matrix and x is the word vector input.
[0100] (2) Then, the 128-dimensional low-dimensional vector y is obtained by sequentially passing through the three-layer network of LSTM and CNN. Then, the feature value output is optimized by global maximum pooling. The calculation process is as follows:
[0101] l i =f(W i *l i-1 +b i ), i = 2, ..., N-1
[0102]
[0103] in represents the jth pooling operation, and the final output vector dimension is M, l i represents the hidden layer nodes, W i It is the first i The weight matrix of the layer, b i For the first i The bias of , f is the activation function of the hidden layer and the output layer. The activation function used in this paper is tanh:
[0104]
[0105] Among them, S300 includes the following steps: input the result of BERT word segmentation into the model, and calculate the probability distribution of the sentiment attribute of each text segment:
[0106] P(w,zα,β)=p(wz,w)*p(zα)
[0107] k represents the number of topics, θ m,k represents the distribution of topic k in data m, φ k,t represents the word distribution of feature word t in topic k, n m,(k) Represents the characteristic vocabulary of topic k in data m, n k,(t) Represents the number of occurrences of feature word t in topic k.
[0108] 1) According to the Bayesian formula and the Dirichlet prior distribution, calculate the Dirichlet distribution expectation:
[0109]
[0110]
[0111] k represents the number of topics, θ m,k represents the distribution of topic k in data m, φ k,t represents the word distribution of feature word t in topic k, n m,(k) Represents the characteristic vocabulary of topic k in data m, n k,(t) Represents the number of occurrences of feature word t in topic k.
[0112] 2) Obtain probability distribution through Gibbs sampling:
[0113]
[0114] In the above formula, represents the number of feature words in data topic k that do not belong to data m, Indicates the number of times a feature word is not assigned to topic word k.
[0115] Wherein, S400 includes the following steps:
[0116] (1) Use the following formula to calculate the similarity between the features extracted and fused by the deep learning network:
[0117]
[0118] where y Q represents the output result of the first short text after step S200, y DIt represents the output result of the second sentence after step S200.
[0119] (2) Use the following formula to determine the sentiment similarity of the short text:
[0120] L i =max(softmax(P)),i=1,2
[0121]
[0122] Where P represents the sentiment feature output by the LDA model. i represents different texts input to the LDA model. When B is 0, it indicates that the sentiment classifications of the two texts are different, and when B is 1, it indicates that the sentiment classifications of the two texts are the same. In this embodiment, B is 0.
[0123] (3) Use the hybrid text semantic similarity calculation method to calculate the semantic relevance between two texts:
[0124] S=εR+(1-ε)B
[0125] Wherein R represents the similarity extracted by the deep learning network, B represents the similarity of the sentiment categories to which the two texts belong, and in this embodiment, its value is 0, ε represents the weight of the similarity extracted by the deep learning network in the final output, and the output result is that the similarity between the two short texts is 42.37%, and the conclusion is that the short text Q and the short text D express different meanings.
[0126] When using only the short text similarity calculation method based on deep learning, the similarity of the two short texts is calculated to be 67.85%, and the judgment result is that the two short texts have similar semantics. According to our sense of language, the two texts want to express different meanings. Short text Q wants to express praise for Xiao Ming, and short text D wants to express criticism of Xiao Ming. Therefore, the meanings expressed by the two short texts are different. Therefore, the short text similarity calculation method based on deep learning and topic model proposed in this paper is closer to real life and has higher calculation accuracy.
Claims
1. Short text similarity calculation method based on deep learning and topic model, It is characterized in that The steps are as follows: 1) Use BERT to perform word segmentation and vector conversion on the two short texts input into the system, and save the word segmentation results and vector conversion results separately; 2) Extract fusion feature information from the word vector after vector conversion: Firstly, a bidirectional LSTM network is used to extract context information. Then, a 1D convolutional neural network is used to fuse the extracted context information with word embedding information. Finally, global maximum pooling is used to extract key information to obtain multi-dimensional and multi-feature fused feature information. The specific method for extracting the fusion feature information of word vectors is: Input the word vector x into the deep learning model and use hash encoding to get l 1 layer; l 1 =W 1 *x Among them l 1 is the output after word vector conversion, W 1 is the hash mapping matrix, and x is the word vector input; Then, the 128-dimensional low-dimensional vector y is obtained by sequentially passing through the three-layer network of LSTM and CNN fusion, and then the eigenvalue output is optimized through global maximum pooling. The calculation process is as follows: l i =f(W i *l i-1 +b i ),i=2,…,N-1 in represents the jth pooling operation, and the final output vector dimension is M, l i represents the hidden layer nodes, W i It is the first i The weight matrix of the layer, b i For the first i The bias of , f is the activation function of the hidden layer and the output layer, and the activation function used is tanh: 3) Extract sentiment features from the two texts after word segmentation: input them into the LDA sentiment analysis topic model, use Gibbs sampling to extract the sentiment color of the text, and extract the sentiment features of the two texts; The specific method for extracting the sentiment features of two paragraphs of text is: Input the results of BERT word segmentation into the model and calculate the probability distribution of sentiment attributes for each paragraph of text: P(w,z|α,β)=p(w|z,w)*p(z|α) Gibbs sampling is used to calculate the relationship between data and topics. w represents a word in the text, α and β are preset hyperparameters, and z represents the number of the topic. According to the Bayesian formula and the Dirichlet prior distribution, the Dirichlet distribution expectation is calculated: k represents the number of topics, θ m,k represents the distribution of topic k in data m, φ k,t represents the word distribution of feature word t in topic k, n m,(k) Represents the characteristic vocabulary of topic k in data m, n k,(t) Indicates the number of occurrences of feature word t in topic k; Probability distribution obtained by Gibbs sampling: represents the number of feature words in data topic k that do not belong to data m, Indicates the number of times the feature word is not assigned to the topic word k; 4) performing similarity calculation on the fusion features obtained in step 2), performing sentiment classification judgment on the sentiment features obtained in step 3), and performing hybrid text semantic similarity calculation on the above two results to obtain the text semantic similarity between the two texts; The similarity calculation of the fusion features uses the following formula: where y Q Represents the output result of the first short text after the deep learning network, y D Represents the output result of the second sentence after passing through the deep learning network.
2. The short text similarity calculation method based on deep learning and topic model according to claim 1, Features: In the step 1), the specific method of word segmentation and vector conversion is: segment the two short texts respectively through the BERT model, save the results after word segmentation, and then vectorize the results after BERT word segmentation into a word vector representation that can be processed by the deep learning network, and use the word vector x as input.
3. The short text similarity calculation method based on deep learning and topic model according to claim 1, Features: In the step 4), the emotional features are classified and determined using the following formula: L i =max(softmax(P)),i=1,2 Where P represents the sentiment features output by the LDA model, i represents the different texts input into the LDA model, when B is 0, it indicates that the sentiment classification of the two texts is different, and when B is 1, it indicates that the sentiment classification of the two texts is the same.
4. The short text similarity calculation method based on deep learning and topic model according to claim 1, Features: In the step 4), the text semantic similarity calculation formula between two texts is: S=εR+(1-ε)B Where R represents the similarity extracted by the deep learning network, B represents the similarity of the sentiment categories to which the two texts belong, and ε represents the weight of the similarity extracted by the deep learning network in the final result output.
Citation Information
Patent Citations
Deep learning multi-classification emotion analysis model combined with attention mechanism
CN110287320A
Semantic sentiment analysis method fusing in-depth features and time sequence models
US11194972B1
