A Chinese short text similarity calculation method based on CNN and pinyin features

Abstract

The application relates to a Chinese short text similarity calculation method based on a CNN and pinyin features, which comprises the following steps: generating a sentence list of two Chinese short texts, then generating a sentence granularity pinyin embedding matrix and a sentence granularity word embedding matrix, and splicing and fusing the vectors of the same characters; calculating the attention scores of two characters in the two texts, generating an attention matrix of each character between the two texts, multiplying the attention matrix with a weight matrix, and generating a high-granularity sentence feature mapping; calculating the attention weights of single characters and performing average pooling to generate a sentence vector of the two Chinese short texts; and inputting the sentence vector into a logistic regression layer to calculate the similarity of the two Chinese short texts. According to the method, the pinyin information of Chinese characters is combined into word analysis to learn more accurate representations, so that the pronunciation of characters is comprehensively considered in the Chinese short text similarity matching task, and the similarity is more accurately judged in combination with the semantics represented by the pronunciation.

Description

Technical Field

[0001] The present invention belongs to the field of text similarity calculation, and particularly relates to a method for calculating the similarity of Chinese short texts based on CNN and Chinese character features. Background Art

[0002] With the popularity of intelligent customer service robots, retrieval-based question-and-answer systems are more and more widely used. The text similarity method is the basis of the retrieval-based question-and-answer system. However, most of the existing Chinese short text similarity matching methods are modified based on the English short text similarity matching methods. Since there are great differences between Chinese and English, for example, Chinese has the unique feature of pinyin.

[0003] The pinyin sequence is used to represent the pronunciation of a character, which is crucial for modeling semantic and syntactic information, and this information cannot be obtained through glyph and context embedding. Also, due to the widespread existence of homographs in Chinese, the pinyin feature is even more important. For example, at the semantic level, the character "乐" has two completely different pronunciations. When pronounced as "lè", it means "happy, glad", and when pronounced as "yuè", it means "music". At the syntactic level, pronunciation can help identify the word class. For example, the character "还". When pronounced as "hái", it is an adverb meaning "also"; when pronounced as "huán", it is a verb meaning "return". Word embedding and character id cannot distinguish the different pronunciations of the same character. Therefore, the conventional text similarity calculation methods based on word embedding and character id will be affected by the same character with different pronunciations and lead to misjudgment when judging Chinese texts. Summary of the Invention

[0004] Based on the above-mentioned disadvantages and deficiencies existing in the prior art, one of the purposes of the present invention is to at least solve one or more of the above problems existing in the prior art. In other words, one of the purposes of the present invention is to provide a method for calculating the similarity of Chinese short texts based on CNN and pinyin features that meets one or more of the foregoing requirements.

[0005] In order to achieve the above-mentioned invention purpose, the present invention adopts the following technical solutions:

[0006] A method for calculating the similarity of Chinese short texts based on CNN and pinyin features specifically includes the following steps:

[0007] S1. Preprocess two preselected Chinese short texts to generate a sentence list of the two Chinese short texts;

[0008] S2. Load the pre-trained pinyin vector representation to obtain the pinyin embedding matrix, and use the pinyin embedding matrix to convert the characters in the sentence list into pinyin vector representations in turn to generate the sentence-level pinyin embedding matrix of each Chinese short text;

[0009] S3, load the pre-trained word embedding representation to obtain a word embedding matrix, and use the word embedding matrix to convert the characters in the sentence list into word vector representations in sequence to generate sentence granularity word embedding matrices of the two Chinese short texts;

[0010] S4, concatenate the vectors of the same characters in the sentence granularity pinyin embedding matrix and the sentence granularity word embedding matrix, and pass through a fusion layer to obtain word pinyin fusion embeddings of the two Chinese short texts, and generate a word pinyin fusion embedding matrix;

[0011] S5, calculate the attention scores of any two characters in the two Chinese short texts according to the word pinyin fusion embedding matrix of the two characters to generate an attention matrix of each character in one Chinese short text to each character in the other Chinese short text;

[0012] S6, multiply the attention matrix by the weight matrix, and superimpose it with the word pinyin fusion embedding matrix to generate a high-granularity sentence feature mapping;

[0013] S7, add all the attention scores of one character in the attention matrix to each character in the other Chinese short text to obtain the attention weight of the single character, weight the attention scores of the character with the attention weight, and use a window of several characters wide to average pooling to generate sentence vectors of the two Chinese short texts;

[0014] S8, input the sentence vectors of the two Chinese short texts into a logistic regression layer to calculate the similarity probability of the two Chinese short texts.

[0015] As a preferred solution, step S2 specifically comprises:

[0016] S21, use an open source tool to convert each character in the corpus into a pinyin sequence;

[0017] S22, use “-” to supplement the pinyin sequence whose length is not 8 to 8 bits;

[0018] S23, segment the pinyin sequence into a Roman character sequence, and use a CNN with a width of 2 on the Roman character sequence for training, and generate a sentence granularity pinyin embedding matrix through a max-pooling operation.

[0019] As a preferred solution, step S3 specifically comprises:

[0020] S31, load the pre-trained word embedding representation to generate a word embedding matrix;

[0021] S32, convert the characters in the sentence list into word vector representations in sequence according to the word embedding matrix to generate a sentence granularity pinyin embedding matrix of each Chinese short text.

[0022] As a preferred solution, the fusion layer in step S4 uses a full connection to map the concatenated vector into the word-pinyin fusion embedding.

[0023] As a preferred solution, step S5 specifically includes:

[0024] S51, obtaining the word-pinyin fusion embedding of any two characters in the two Chinese short texts according to the word-pinyin fusion embedding matrix;

[0025] S52, calculating the attention score of the two characters according to the distance of the word-pinyin fusion embedding;

[0026] S53, repeating the combination of any two characters in the two Chinese short texts to perform steps S51-S52 until the attention scores of all combinations are generated;

[0027] S54, generating an attention matrix according to all the attention scores.

[0028] As a preferred solution, in step S7, the average pooling uses a multi-layer stacked convolutional layer and a pooling layer.

[0029] Compared with the prior art, the present application has the following beneficial effects:

[0030] The method of the present application combines the pinyin information of Chinese characters into the attention mechanism to learn more accurate representations, so as to consider the pronunciation of the characters in the Chinese short text similarity matching task, and more accurately determine the similarity by combining the semantics represented by the pronunciation. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 is a flowchart of a Chinese short text similarity calculation method based on CNN and pinyin features according to an embodiment of the present application;

[0032] Figure 2 is a calculation structure diagram of a Chinese short text similarity calculation method based on CNN and pinyin features according to an embodiment. DETAILED DESCRIPTION

[0033] In order to more clearly illustrate the embodiments of the present application, the specific embodiments of the present application will be described below with reference to the accompanying drawings. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings and other embodiments from these drawings without creative labor.

[0034] Embodiment: The present embodiment provides a Chinese short text similarity calculation method based on CNN and pinyin features, the flowchart is as shown in Figure 1 , the calculation structure diagram is as shown in Figure 2 , the method comprises the following steps:

[0035] S1. Preprocess the dataset containing multiple short Chinese texts to be compared to obtain a sentence list WList. i {i = 0, 1}, the dataset is represented as {S} j |S j = (s0, s1, label), 1≤j≤N}, where S j Let S represent the j-th Chinese short text to be compared in the j-th dataset. i It consists of three parts: the first sentence s0, the second sentence s1, and the label; N is the size of the dataset.

[0036] The above preprocessing operations include Chinese word segmentation and punctuation removal. Each short Chinese text sentence is segmented and punctuation removed to generate a list of sentences.

[0037] Here is a specific implementation process of step S1 above. Let s0 = "Xiaoxin, do you still have money today?", s1 = "How much money do you have left today, Xiaoxin?", and label = 1.

[0038] The dataset S was preprocessed using a Chinese word segmentation tool to obtain the character lists s0 and s1.

[0039] WList0=[s0word1,s0word2,s0word3,…,s0word s ];

[0040] and

[0041] WList1=[s1word1,s1word2,s1word3,…,s1word s ].

[0042] In this embodiment, WList0 = ['small', 'new', 'you', 'today', 'still', 'have', 'money', 'do'], WList1 = ['you', 'today', 'still', 'remaining', 'more', 'less', 'money', 'small', 'new'], LabelList = ['1'].

[0043] S2. Load the pre-trained pinyin vector representation to obtain the pinyin embedding matrix, and convert the characters in the sentence list into pinyin vector representations in turn to generate the sentence-level pinyin embedding matrix for each Chinese short text.

[0044] Specifically, step S2 includes the following sub-steps:

[0045] S2.1, Generate a Chinese Pinyin dictionary using the pinyin package: pinyinDict = {<w1:p1> ,<w2:p2> , ..., <wm : p m}, wherein w m represents a Chinese character, p i represents a pinyin sequence of w i , and m is the size of a pinyin dictionary pinyinDict; in this embodiment, the size of the pinyin dictionary is 3000, and part of the entries are as follows:

[0046] Chinese character Pinyin Chinese character Pinyin Chinese character Pinyin Chinese character Pinyin Small xiao3 Now jin1 Have you3 Left sheng4 New xin1 Day tian1 Money qian2 Much duo1 You ni3 Also hai2 Ma ma1 Few shao3

[0047] S2.2, segment the pinyin into a sequence of Roman characters;

[0048] S2.3, train using 50 convolution kernels with a width of 2 and maximum pooling to obtain a pinyin embedding matrix PE t×p .

[0049]

[0050] Load the above pinyin embedding matrix to convert the characters in WList0 and WList1 into pinyin vector representations in turn to obtain a sentence-level pinyin embedding matrix F 0，p = [pv1, …, pv n ] 50×9 , F 1，p = [pv1, …, pv n ] 50×10 . Wherein pv m = PE(word m ) 1×50 .

[0051] Based on the actual example in step S1 above, here, taking “huan” and “sheng” as examples, in the sentence s0, pv6 = PE(“huan”) = [0.0173, 0.0546, …, -0.0326, 0.0145] 1×50 , and in the sentence s1, pv7 = PE(“sheng”) = [0.0273, -0.0506, …, 0.0116, 0.0875] 1×50 .

[0052] S3, load the pre-trained word embedding to obtain a word embedding matrix, and convert the characters in the sentence list into word vector representations in turn to generate two sentence-level word embedding matrices of Chinese short texts;

[0053] Specifically, step S3 specifically includes the following steps:

[0054] S31, load the pre-trained word embedding vector to generate a word embedding matrix WE l×d , wherein l represents the number of characters in the word embedding vector, and d represents the dimension of the word vector.

[0055] S32, sequentially convert the list of sentences WList i into word vector representations, generating a sentence-level word embedding matrix F i，w = [wv1, …, wvn] n ] d×n . Where wvm= WE(wordm) m m 1×d , wordmrepresents the mth word in s m , and n represents the number of words in sentence s i .

[0056] In the above example, a pre-trained Word2Vector model is used to obtain the Chinese character dictionary and word embedding matrix WE i . The number of words is 87668, and the dimension of the word vector d is 300. The WE l×d matrix is as follows:

[0057]

[0058] Load the matrix WE to convert the words in WList0 and WList1 into word vector representations, obtaining the sentence-level word embedding matrix F l×d = [wv1, …, wvn] 0，w n 300×9 , F 1，w = [wv1, …, wvn] n 300×10 .

[0059] Take the two words “still” and “remaining” as an example, in s0:

[0060] wv6= WE(“still”) = [0.7105446, -0.2524085, …, 0.07105446, -0.25240803] 1×300 ;

[0061] In sentence s1:

[0062] wv6= WE(“remaining”) = [0.3102461, 0.4508503, …, -0.0205347, -0.1554082] 1×300 .

[0063] S4, sequentially concatenate the vectors of the same words in the sentence-level pinyin embedding matrix F i，p and the sentence-level word embedding matrix F i，w , and then input the concatenated vector into the fusion layer, which uses a fully connected layer to map the concatenated vector to the final input vector ​​​​​A word-pinyin fusion embedding matrix is composed of the input vectors of each character.

[0064] Among them word m represents the m-th character in s i , n represents the number of characters in sentence s i , and the symbol represents vector concatenation, d c represents the dimension of the vector after mapping, and FC represents fully connected.

[0065] Based on the above example, a specific implementation process of step S4 is also given here:

[0066] Taking the character "hai" in s0 and the character "sheng" in s1 as examples. In s0, concatenate the pinyin embedding and word embedding of the character "hai":

[0067]

[0068] In s1, concatenate the pinyin embedding and word embedding of the character "sheng":

[0069]

[0070] S5. Calculate the attention scores of any two characters in the two Chinese short texts based on the word-pinyin fusion embedding matrix of the two characters, and generate an attention matrix of each character in one Chinese short text for each character in the other Chinese short text;

[0071] Specifically, step S5 includes the following steps:

[0072] S51. Obtain the word-pinyin fusion embedding vectors of any two characters in the two Chinese short texts according to the word-pinyin fusion embedding matrix;

[0073] S52. Calculate the attention A 0，r between F 1，r and F i，j (i and j represent the attention score of the i-th character in s0 relative to the j-th character in s1, and at the same time, it is also the attention score of the j-th character in s1 relative to the i-th character in s0);

[0074] Among them, A i，j = match-score(F 0，r [:, i], F 1，r [:, j]), and the match-score function is defined as |·| represents calculating the Euclidean distance between the word-pinyin fusion embedding vectors of two characters.

[0075] S53, repeat the selection of any two-character combination in the two Chinese short texts to perform steps S51-S52 until all combinations of attention scores are generated;

[0076] S54, combine all the attention scores obtained in the above steps to obtain an attention matrix A.

[0077] In the above example, the length of sentence s0 in this step is 9, and the length of sentence s1 is 10. The calculated attention matrix A is:

[0078]

[0079] S6, multiply the attention matrix with the weight matrix, i.e., calculate the attention feature mapping F i，a of s0 and s1 respectively. T , the attention feature mapping F 1，a of s1. The weight matrix and are model parameters learned in training.

[0080] In the above example, the attention feature mapping F

[0081] of sentence s0 is

[0082] The feature mapping F i，r and the attention feature mapping F i，a are each padded with w-1 columns on the left and right to extract more features of edge words. Concatenate F i，r and F i，a to obtain a 3-order tensor Use d c convolution kernels with a size of [300, 3, 2] on F i to extract features, and generate higher-level representation feature mappings F i (i=0, 1) for sentences S Specifically, it is expressed as

[0083] In this embodiment, w takes 3, and F 0，r and F 0，a are each padded with 2 columns on the left and right to obtain matrices:

[0084]

[0085] and

[0086]

[0087] The two matrices are spliced to obtain a 3-dimensional tensor:

[0088]

[0089] Use 50 convolution kernels of size [300, 3, 2] on F0 and F1 to extract features, and obtain the convolution output of s0

[0090] F 1，r and F 1，a also perform the same operation as described above to obtain the convolution output of s1

[0091] S7, add all the attention scores of one word in the attention matrix to each word in the other Chinese short text, that is, add each column or row in the attention matrix A between to obtain the single attention weight a i，j of the word corresponding to the row in s0 (s1)

[0092] Multiply the corresponding columns in a i，j and , and weight all the attention scores of the word with the attention weight. Then the window of size w=3 is averaged and pooled to obtain the high-granularity sentence vector representation

[0093] Based on the above example, the specific implementation process of step S7 is as follows:

[0094] The attention weight of the jth word in s0 is a 0，j =∑A[j,:],and the attention weight of the jth word in s1 is a 1，j =∑A[:,j].

[0095] Attention matrix after s0 and s1 convolution:

[0096]

[0097] The attention weight of the first unit in s0 is the sum of the first row in the matrix A 13×14 : a 0，1 =[0.78478246], and the attention weight of the first unit in s1 is the sum of the first column in the matrix A 13×14 : a 1，1 =[0.94378066].

[0098] Multiply each column in a i，j and the corresponding a i , then add every w columns, and perform average pooling to obtain new feature map Then the new feature map of the jth column in the sentence i is The calculation method is as follows:

[0099]

[0100] in, Since pooling and convolution have the same output dimension, we can stack multiple convolutional-pooling blocks to extract more abstract features. When we reach the last pooling layer, we perform average pooling with a window size equal to the sentence length, obtaining the final sentence s. i sentence vector

[0101] Specifically, the feature mapping of the first column in s0

[0102]

[0103] Feature mapping of column 1 in sentence s1

[0104]

[0105] When the output is the last convolutional-pooling layer, w is the length of the sentence, then s0 s1

[0106] S8. Extract sentence vectors from two short Chinese texts. and Input the logistic regression layer to calculate the similarity probability P between two short Chinese texts;

[0107]

[0108] If P is greater than a threshold, the sentences are considered similar; otherwise, they are considered dissimilar. In this embodiment, the threshold is set to 0.75. The model calculates a result of 0.83365618, indicating that the two sentences are similar.

[0109] It should be noted that the above embodiments are merely detailed descriptions of preferred embodiments and principles of the present invention. For those skilled in the art, there may be changes in specific implementation methods based on the ideas provided by the present invention, and these changes should also be considered within the scope of protection of the present invention.

Claims

1. A Chinese short text similarity calculation method based on CNN and pinyin features, characterized in that, Specifically comprising the following steps: S1, preprocessing two preselected Chinese short texts to generate a sentence list of the two Chinese short texts; S2, loading a pre-trained pinyin vector representation to obtain a pinyin embedding matrix, and using the pinyin embedding matrix to convert the characters in the sentence list into pinyin vector representations in sequence to generate a sentence granularity pinyin embedding matrix of each Chinese short text; S3, loading a pre-trained word embedding representation to obtain a word embedding matrix, and using the word embedding matrix to convert the characters in the sentence list into word vector representations in sequence to generate a sentence granularity word embedding matrix of the two Chinese short texts; S4, concatenating the vectors of the same characters in the sentence granularity pinyin embedding matrix and the sentence granularity word embedding matrix and passing through a fusion layer to obtain word-pinyin fusion embeddings of the two Chinese short texts and generate a word-pinyin fusion embedding matrix; S5, calculating attention scores of any two characters in the two Chinese short texts according to the word-pinyin fusion embedding matrix of the two characters to generate an attention matrix of each character in one Chinese short text to each character in the other Chinese short text; S6, multiplying the attention matrix with a weight matrix and superimposing the word-pinyin fusion embedding matrix to generate a high-granularity sentence feature mapping; S7, adding all the attention scores of one character in the attention matrix to each character in the other Chinese short text to obtain an attention weight of the single character, weighting the attention weight to all the attention scores of the character, and using a window of several characters wide for average pooling to generate a sentence vector of the two Chinese short texts; S8, inputting the sentence vector of the two Chinese short texts into a logistic regression layer to calculate a similarity probability of the two Chinese short texts. 2.The Chinese short text similarity calculation method based on CNN and pinyin features of claim 1, wherein, The step S2 specifically comprises: S21, converting each character in the corpus into a pinyin sequence using an open source tool; S22, using "-" to supplement the pinyin sequence with a length not equal to 8 to 8 bits; S23, tokenizing the pinyin sequence into a Roman character sequence, and using a CNN with a width of 2 on the Roman character sequence to train and generate a sentence granularity pinyin embedding matrix through a max-pooling operation.

3. The Chinese short text similarity calculation method based on CNN and pinyin features according to claim 1, characterized in that, The step S3 specifically comprises: S31, loading a pre-trained word embedding representation to generate a word embedding matrix; S32, converting the characters in the sentence list into word vector representations in sequence according to the word embedding matrix to generate a sentence granularity pinyin embedding matrix of each Chinese short text.

4. The Chinese short text similarity calculation method based on CNN and pinyin features according to claim 1, characterized in that, The fusion layer in the step S4 uses a full connection to map the concatenated vectors into word-pinyin fusion embeddings.

5. The Chinese short text similarity calculation method based on CNN and pinyin features according to claim 1, characterized in that, The step S5 specifically comprises: S51, obtaining word-pinyin fusion embeddings of any two characters in the two Chinese short texts according to the word-pinyin fusion embedding matrix; S52, calculating attention scores of the two characters according to the distance of the word-pinyin fusion embeddings; S53, repeating steps S51-S52 to select combinations of any two characters in the two Chinese short texts until attention scores of all combinations are generated; S54, generating an attention matrix according to all the attention scores.

6. The Chinese short text similarity calculation method based on CNN and pinyin features according to claim 1, characterized in that, In the step S7, the average pooling uses a convolution layer and a pooling layer stacked in multiple layers.

Images