A method for calculating chapter-level document similarity and a readable storage medium

Abstract

The present invention relates to a method and a readable storage medium for calculating the similarity of a chapter-level document. The method comprises: obtaining m first sentences of an original document and n second sentences of a comparison document; performing Chinese word segmentation and stop word removal processing on the first sentence and the second sentence to obtain a general dictionary; obtaining a common annotation based on the word frequency and part of speech of the remaining words after the stop words are removed; obtaining the word frequency and inverse document frequency value of the word based on the common annotation of the word to obtain the word frequency and inverse document frequency vector of the first sentence and the second sentence; finding the second sentence most similar to each first sentence of the original document from the comparison document to obtain a first similarity score; obtaining a final document vector output by the final Doc2Vec; obtaining a second similarity score based on the final document vector; and determining a similarity result based on the average of the first and second similarity scores. The method takes into account word frequency, part of speech and semantics, and can effectively process long texts.

Description

Technical Field

[0001] The invention belongs to the technical field of text similarity calculation, and relates to a method for calculating chapter-level document similarity and a readable storage medium. Background Art

[0002] With the development of science and technology and the advent of the big data era, a large amount of messy and repetitive information has appeared on the Internet. The huge amount of information easily leads to information overload, which greatly reduces the efficiency of users in obtaining effective information. Therefore, it is of great application value and social benefit to compare and remove duplicate information from massive data. For text data, by calculating its similarity, it can be judged whether it is similar and then deduplicated. For example, most text data on the Internet are long texts, such as video copywriting, news articles and forum comments. Therefore, it is necessary to implement the technical problem of text similarity calculation for long texts at the chapter level.

[0003] The text similarity calculation method generally consists of two parts: text representation model and similarity measurement. The text representation model extracts features from the text and represents the text as a feature vector. The similarity measurement algorithm is used to calculate the similarity between feature vectors.

[0004] However, the current methods cannot effectively process long texts. Summary of the invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a method for calculating chapter-level document similarity and a readable storage medium. The technical problem to be solved by the present invention is achieved by the following technical solutions:

[0006] An embodiment of the present invention provides a method for calculating chapter-level document similarity, the method for calculating chapter-level document similarity comprising:

[0007] Step 1, obtaining m first sentences of the original document and n second sentences of the comparison document;

[0008] Step 2: performing Chinese word segmentation and stop word removal processing on all the first sentences and the second sentences to obtain a general dictionary consisting of all words after the stop words are removed, wherein each word in the general dictionary is configured with a mapping identifier;

[0009] Step 3: obtaining a common labeling of the word frequency and part of speech of the word according to the word frequency and part of speech of the remaining words after removing the stop words in the first sentence and the second sentence;

[0010] Step 4: obtaining the word frequency and inverse document frequency value of the word based on the common annotation of the word, and obtaining the word frequency and inverse document frequency vectors of the first sentence and the second sentence based on the word frequency and inverse document frequency value of the word;

[0011] Step 5: based on the word frequency and the inverse document frequency vector, find the second sentence most similar to each first sentence of the original document from the comparison document to obtain a first similarity score between the original document and the comparison document;

[0012] Step 6: Input the original document and the comparison document after Chinese word segmentation and stop word removal into Doc2Vec respectively, and iteratively update the Doc2Vec to obtain the final document vector output by the final Doc2Vec;

[0013] Step 7: Obtain a second similarity score according to the final document vector of the original document and the final document vector of the compared document;

[0014] Step 8: Determine a similarity result according to a relationship between an average value of the first similarity score and the second similarity score and a score threshold.

[0015] In one embodiment of the present invention, step 1 comprises:

[0016] Step 1.1, obtaining the original document and the comparison document;

[0017] Step 1.2: performing regularized sentence segmentation on the original document and the comparison document to obtain m first sentences of the original document and n second sentences of the comparison document.

[0018] In one embodiment of the present invention, step 2 comprises:

[0019] Step 2.1, performing Chinese word segmentation on all the first sentences and the second sentences to obtain a plurality of words;

[0020] Step 2.2, removing stop words from all the words obtained in step 2.1, to obtain the words after removing the stop words;

[0021] Step 2.3: construct the general dictionary based on all the words obtained in step 2.2 after removing the stop words, and configure a mapping identifier for each word in the general dictionary.

[0022] In one embodiment of the present invention, step 3 comprises:

[0023] Step 3.1, counting the number of occurrences of each word remaining after removing stop words in the first sentence and the second sentence, recording the number as the word frequency, and converting the first sentence and the second sentence into a bag-of-words format to obtain a first label according to the word frequency of each word;

[0024] Step 3.2, marking the part of speech for each word remaining after removing the stop words in the first sentence and the second sentence, and assigning a corresponding second tag to each word according to the priority order of the part of speech;

[0025] Step 3.3: Obtain a common annotation of the word according to the first annotation and the second annotation of the word.

[0026] In one embodiment of the present invention, the calculation formula of the word frequency and the inverse document frequency value is:

[0027]

[0028] Among them, tfidf(w) is the word frequency and inverse document frequency of word w, tf(d,w) is the common annotation of word w in the sentence, is the total number of all sentences in the document where word w is located, and N(w) is the total number of sentences in which word w appears in the document.

[0029] In one embodiment of the present invention, step 5 comprises:

[0030] Step 5.1, calculating the cosine similarity between the word frequency and inverse document frequency vector of each first sentence in the original document and the word frequency and inverse document frequency vector of all second sentences in the comparison document, and taking the second sentence corresponding to the largest cosine similarity as the sentence most similar to the first sentence;

[0031] Step 5.2: Count the cosine similarities greater than a preset score among the maximum cosine similarities corresponding to the first sentence, and take the ratio of the number of sentences corresponding to the cosine similarities greater than the preset score to the total number of sentences in the original document as the first similarity score.

[0032] In one embodiment of the present invention, step 6 comprises:

[0033] Step 6.1, inputting the original document and the comparison document after Chinese word segmentation and stop word removal into Doc2Vec respectively;

[0034] Step 6.2, the Doc2Vec samples a number of words from the first sentence of the original document and the second sentence of the comparison document respectively according to a preset number;

[0035] Step 6.3, select a word from the sampled words as the prediction word, and use the remaining words as input words;

[0036] Step 6.4, calculating the loss value of the loss function according to the first word vector and the second word vector of the predicted word, wherein the second word vector is obtained according to the current document vector and the word vector corresponding to the input word;

[0037] Step 6.5: Optimize the loss function using stochastic gradient descent.

[0038] Step 6.6, slide the window according to a preset number to sample new words from the first sentence of the original document and the second sentence of the comparison document respectively, and loop through steps 6.3 to 6.5 until the final document vector output by Doc2Vec is obtained.

[0039] In one embodiment of the present invention, step 7 comprises:

[0040] The cosine similarity between the final document vector of the original document and the final document vector of the comparison document is calculated, and the obtained cosine similarity is used as the second similarity score.

[0041] In one embodiment of the present invention, step 8 comprises:

[0042] Step 8.1, obtaining a final similarity score between the original document and the comparison document according to an average value of the first similarity score and the second similarity score;

[0043] Step 8.2: Determine the relationship between the final similarity score and the score threshold. If the final similarity score is greater than or equal to the score threshold, the similarity result is similar; otherwise, it is dissimilar.

[0044] The present invention also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] The present invention is aimed at chapter-level long text documents. The present invention proposes a method for calculating document similarity by combining sentence-by-sentence comparison based on TF-IDF and part-of-speech tagging to generate sentence vectors and direct comparison based on Doc2vec to generate document vectors. The method comprehensively considers word frequency, part of speech and semantics, and can effectively process long texts. Both methods are unsupervised methods and do not require training.

[0047] Other aspects and features of the present invention will become apparent from the following detailed description with reference to the accompanying drawings. It should be understood, however, that the drawings are designed for illustrative purposes only and are not intended to limit the scope of the present invention, as reference should be made to the appended claims. It should also be understood that, unless otherwise indicated, the drawings are not necessarily drawn to scale and are intended merely to conceptually illustrate the structures and processes described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1A flowchart of a method for calculating chapter-level document similarity provided by an embodiment of the present invention;

[0049] Figure 2 A flowchart of another method for calculating chapter-level document similarity provided by an embodiment of the present invention;

[0050] Figure 3 A schematic diagram of a computer device module provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0052] Embodiment 1

[0053] At present, the statistical methods based on TF (term frequency) or TF-IDF (IDF stands for inverse document frequency in Chinese) simply measure the importance of words by frequency, ignoring semantic information, which is not comprehensive enough. If the same word appears multiple times in two documents, the similarity is high, and the ability to distinguish documents with many shared words is not high. The language model method based on word vector or sentence vector generation requires large-scale corpus for training, the model is not general enough, and it is difficult to process long texts at the chapter level.

[0054] Based on this, the embodiment of the present invention provides a method for calculating chapter-level document similarity, see Figure 1 and Figure 2 , Figure 1 A flowchart of a method for calculating chapter-level document similarity provided by an embodiment of the present invention is provided. Figure 2 A flow chart of another method for calculating chapter-level document similarity provided by an embodiment of the present invention. The present invention provides a method for calculating chapter-level document similarity. The method for calculating chapter-level document similarity includes:

[0055] Step 1: Obtain m first sentences of the original document and n second sentences of the comparison document.

[0056] Specifically, the original document is the document obtained in this embodiment that needs to be compared for similarity, and the comparison document is the document used for comparison with the original document. The original document and the comparison document both consist of multiple sentences, so the original document can be divided into m first sentences and the comparison document can be divided into n second sentences.

[0057] In a specific embodiment, step 1 may specifically include steps 1.1 to 1.2, wherein:

[0058] Step 1.1: Obtain the original document and the comparison document.

[0059] Step 1.2: Regularize the original document and the comparison document to obtain m first sentences of the original document and n second sentences of the comparison document.

[0060] Specifically, the original document and the comparison document to be compared therewith are regularized and segmented. Chinese text sentences are generally segmented by sentence-end punctuation marks such as "?!", and a total of m sentences in the original document and n sentences in the comparison document are obtained after the segmentation.

[0061] Step 2: Perform Chinese word segmentation and stop word removal on all the first and second sentences to obtain a general dictionary consisting of all words after the stop words are removed, and each word in the general dictionary is configured with a mapping identifier.

[0062] Specifically, in order to quickly and accurately compare the original document and the comparison document, it is necessary to perform Chinese word segmentation on the first sentence of the original document and the second sentence of the comparison document, and remove stop words from the words obtained after word segmentation. In this way, a general dictionary of all words after removing common words can be constructed, which can facilitate subsequent processing. In addition, this embodiment also configures a mapping identifier (i.e., mapping id) for each word in the general dictionary to facilitate its identification.

[0063] In a specific embodiment, step 2 may specifically include steps 2.1 to 2.3, wherein:

[0064] Step 2.1: Perform Chinese word segmentation on all the first and second sentences to obtain multiple words.

[0065] Specifically, Chinese word segmentation is generally the first step in Chinese text processing, and the purpose is to quantify text through words, divide the boundaries of words, and let the computer understand Chinese text. There are many Chinese word segmentation tools, such as Jieba (stammering), HanLP, FoolNLTK and LTP, etc. Jiebade's word segmentation method is to form words by characters, and the implementation algorithm is the Viterbi algorithm based on the hidden Markov model. Preferably, the present embodiment uses the Jieba word segmentation tool. Such as "People would rather care about the eating, drinking, defecating and trivialities of a crappy movie actor, rather than knowing the turbulent inner world of an ordinary person." This sentence uses Jieba word segmentation with "," intervals, and the word segmentation result is ['people', 'prefer', 'to', 'care', 'a', 'crappy', 'movie actor', 'of', 'eating, drinking, defecating', 'and', 'trivia', ', ', 'but', 'unwilling', 'know', 'a', 'ordinary person', 'turbulent', 'of', 'inner world'].

[0066] Step 2.2: Remove the stop words from all the words obtained in step 2.1 to obtain the words after removing the stop words.

[0067] Specifically, read in stop word list (stop_words) file, common stop word list of Harbin Institute of Technology and Baidu stop word list, it is also possible to customize some meaningless words such as "of", "ground" and punctuation marks etc. to form a stop word list in the blank stop_words file, then judge each word in the sentence after word segmentation, whether it is in the stop word list, if so, remove the word. Preferably, the stop word list of Harbin Institute of Technology is used in the present embodiment. For the result after word segmentation result removal of stop words in step 2.1, ['care', 'one', 'poor', 'movie actor', 'eating, drinking, defecating and urinating', 'trivia', 'unwilling', 'understanding', 'one', 'ordinary person', 'rough waves', 'inner world'].

[0068] Step 2.3: construct a general dictionary based on all the words obtained in step 2.2 after removing the stop words, and configure a mapping identifier for each word in the general dictionary.

[0069] Specifically, a general dictionary is constructed for all the remaining words in the original document and the comparison document after processing in step 2.2. A mapping id (i.e., mapping identifier) ​​is assigned to each word in the general dictionary, which is used to encode the sentence into an id sequence corresponding to the word sequence. For example, a general dictionary is generated for all the words obtained from the sentence in step 2.2, and the mapping relationship between the words and the numbers is: {'one': 0, 'unwilling': 1, 'understand': 2, 'care': 3, 'inner world': 4, 'eating, drinking, defecating and urinating': 5, 'ordinary people': 6, 'rough waves': 7, 'movie actor': 8, 'crappy': 9, 'trivial': 10}.

[0070] Step 3: According to the word frequencies and parts of speech of the remaining words after removing the stop words in the first sentence and the second sentence, obtain the common marking of the word frequencies and parts of speech of the words.

[0071] Specifically, in order to count not only the frequency of each word (the frequency is the number of times the word appears) but also the part of speech of each word, this embodiment marks each word with the frequency and part of speech.

[0072] In a specific embodiment, step 3 may specifically include steps 3.1 to 3.3, wherein:

[0073] Step 3.1, count the number of occurrences of each word remaining after removing stop words in the first sentence and the second sentence, record the number as the word frequency, and convert the first sentence and the second sentence into a bag-of-words format to obtain a first label according to the word frequency of each word.

[0074] Specifically, count the number of occurrences of each word in a sentence, record it as the word frequency, convert the sentence into a bag-of-words format, and the first annotation is to annotate the words using the mapping identifier and the word frequency. For example, for the sentence in step 2.1, the output is [(0, 2), (1, 1), (2, 1), (3, 1), (4, 1), (5, 1), (6, 1), (7, 1), (8, 1), (9, 1), (10, 1)]. Here, (0, 2) means that the word '一个' with id 0 appears 2 times in the sentence, and (0, 2) is the first annotation of the word '一个'.

[0075] Step 3.2: Annotate the part-of-speech for each remaining word after removing stop words in the first sentence and the second sentence, and assign the corresponding second annotation to each word according to the priority order of the part-of-speech.

[0076] Specifically, annotate each word in the word segmentation result with a correct part-of-speech. There are some existing part-of-speech annotation tools such as Jieba, SnowNLP, and THULAC, etc. In this embodiment, the Jieba part-of-speech annotation tool is used. There are a total of 24 part-of-speech tags, as shown in Table 1 below. For the sentence in step 2.1, the part-of-speech annotation result is: [关心:n 一个:m 蹩脚:a 电影演员:n 吃喝拉撒:n 鸡毛蒜皮:i 不愿:v 了解:v 一个:m 普通人:n 波涛汹涌:i 内心世界:n].[[]END]]

[0077] In Chinese, the priority order of part-of-speech can be noun, verb, adjective, adverb, others. To reflect the priority of different parts of speech, in this embodiment, the noun part-of-speech is annotated as 5, the verb part-of-speech is annotated as 4, the adjective part-of-speech is annotated as 3, the adverb part-of-speech is annotated as 2, and other words are annotated as 1. For example, the part-of-speech annotation result of the example sentence after being processed by step 2 in step 3.1 is: {'关心': 5, '一个': 1, '蹩脚': 3, '电影演员': 5, '吃喝拉撒': 5, '鸡毛蒜皮': 1, '不愿': 4, '了解': 4, '普通人': 5, '波涛汹涌': 1, '内心世界': 5}.

[0078] Table 1: Part-of-speech annotation correspondence table

[0079] Label meaning Label meaning Label meaning Label meaning n Common Noun f Direction Noun s Place Noun t time nr Name ns Place name nt Organization Name nw Title nz Other proper names v Common Verbs vd Adverb v Noun Verb a adjective ad Adverb an adjective d adverb m Quantifiers q quantifier r pronoun p preposition c conjunction u particle xc Other function words w Punctuation PER Name LOC Place name ORG Organization Name TIME time

[0080] Step 3.3: Obtain the combined annotation of the word according to the first annotation and the second annotation of the word.

[0081] Specifically, the sum of the first and second annotations obtained for each word is used as a common annotation, so that the common annotation is finally used to annotate the word. The present invention not only takes into account the word frequency, but also the part of speech, and gives different weights to words of different parts of speech. Therefore, this embodiment adds the word frequency and part of speech annotations as the common annotation of a word. For example, the word bag format finally input in step 3.1 is: [(0,3), (1,5), (2,5), (3,6), (4,6), (5,6), (6,6), (7,2), (8,6), (9,4), (10,2)], where (0,3) represents the word frequency of the word 'one' with id 0, the first annotation of which is 2, and the second annotation of which is 1, so the final common annotation is 3.

[0082] Step 4: Based on the common annotations of the words, the word frequency and inverse document frequency values ​​of the words are obtained, and the word frequency and inverse document frequency vectors of the first sentence and the second sentence are obtained based on the word frequency and inverse document frequency values ​​of the words.

[0083] Specifically, the word frequency and inverse document frequency value (i.e., tfidf value) of each word are calculated using the calculation formula of word frequency and inverse document frequency value. The calculation formula of word frequency and inverse document frequency value is:

[0084]

[0085] Among them, tfidf(w) is the word frequency and inverse document frequency of word w, tf(d,w) is the common annotation of word w in the sentence, is the total number of all sentences in the document where word w is located, and N(w) is the total number of sentences in which word w appears in the document.

[0086] In addition, the term frequency and inverse document frequency vectors (TF-IDF vectors) of all first sentences of the original document and all second sentences of the comparison document are composed of the mapping id of the term and the tfidf value of the term [(id, tfidf)].

[0087] Step 5: Based on the word frequency and inverse document frequency vectors, find the second sentence that is most similar to each first sentence of the original document from the comparison document to obtain a first similarity score between the original document and the comparison document.

[0088] Specifically, this embodiment finds the second sentence most similar to each first sentence through the word frequency and inverse document frequency vector of the first sentence and the second sentence, thereby determining the similarity score between the original document and the comparison document through the most similar second sentence found.

[0089] In a specific embodiment, step 5 may specifically include steps 5.1 to 5.2, wherein:

[0090] Step 5.1, calculate the cosine similarity between the word frequency and inverse document frequency vector of each first sentence in the original document and the word frequency and inverse document frequency vector of all second sentences in the comparison document, and take the second sentence corresponding to the largest cosine similarity as the sentence most similar to the first sentence.

[0091] Specifically, the word frequency and inverse document frequency vectors of all sentences in the original document and the comparison document are matched pairwise to find the second sentence in the comparison document that is most similar to each first sentence in the original document. The cosine similarity between the word frequency and inverse document frequency vectors of the two sentences is the similarity score. The larger the cosine similarity, the more similar they are.

[0092] Step 5.2: Count the cosine similarities greater than a preset score among the maximum cosine similarities corresponding to the first sentence, and take the ratio of the number of sentences corresponding to the cosine similarities greater than the preset score to the total number of sentences in the original document as the first similarity score.

[0093] Specifically, sentences with similarity scores greater than a preset score are counted, and the number of similar sentences is marked, and the ratio of the number of similar sentences to the total number of sentences in the original document is calculated. This ratio is the TF-IDF similarity score (i.e., the first similarity score) of the original document and the comparison document.

[0094] Preferably, the preset score is 0.5. It should be noted that the preset score can be selected according to actual needs, and this embodiment does not make any specific limitation on this.

[0095] Step 6: Input the original document and the comparison document after Chinese word segmentation and stop word removal into Doc2Vec respectively, and iteratively update Doc2Vec to obtain the final document vector output by Doc2Vec.

[0096] Specifically, the original document and the comparison document processed in step 2 are respectively input into Doc2Vec, so that Doc2Vec will continuously update its parameters until the loss function finally stabilizes, and the document vector output by Doc2Vec is the final document vector.

[0097] In a specific embodiment, step 6 may specifically include steps 6.1 to 6.6, wherein:

[0098] Step 6.1: Input the original document and the comparison document after Chinese word segmentation and stop word removal into Doc2Vec respectively.

[0099] Specifically, the original document after Chinese word segmentation and stop word removal is input into a Doc2Vec, and the comparison document after Chinese word segmentation and stop word removal is input into another Doc2Vec.

[0100] Step 6.2: Doc2Vec samples a number of words from the first sentence of the original document and the second sentence of the comparison document according to the preset number.

[0101] Specifically, each time a fixed number (i.e., a preset number) of words are sampled from a sentence processed by step 2, that is, after the current sentence is processed by subsequent steps, the sliding window will continue to sample another fixed number of words in sequence until all sentences are sampled.

[0102] Step 6.3: Select a word from the sampled words as the prediction word, and use the remaining words as input words.

[0103] Specifically, a sampled word may be randomly selected as a prediction word, and the remaining sampled words may be used as input words.

[0104] Step 6.4: Calculate the loss value of the loss function based on the first word vector and the second word vector of the predicted word, where the second word vector is obtained based on the current document vector and the word vector corresponding to the input word.

[0105] Specifically, Doc2Vec will generate the word vector (word vector) of the currently sampled word and the current document vector (Paragraph vector). The predicted word corresponds to the first word vector, and the word vector corresponding to the input word and the corresponding document vector are used as the input of the input layer. Thus, Doc2Vec will generate the second word vector, which can be obtained by adding the document vector and the word vector of the input word sampled this time and averaging them, or by adding the document vector and the word vector of the input word sampled this time in sequence. Thus, the loss value of the loss function can be calculated based on the difference between the first word vector and the second word vector of the predicted word, and the objective function (i.e., the loss function) is to maximize the average probability.

[0106] Step 6.5: Optimize the loss function using stochastic gradient descent.

[0107] Specifically, stochastic gradient descent is used to optimize the parameters, and the loss value of the loss function is minimized through multiple iterations.

[0108] Step 6.6: Slide the window by a preset number to sample new words from the first sentence of the original document and the second sentence of the comparison document, and repeat steps 6.3 to 6.5 until the final document vector output by Doc2Vec is obtained.

[0109] Specifically, through the above continuous iterative cycle process, the loss value of the loss function is minimized, thereby obtaining the parameter-optimized Doc2Vec, and the document vector outputted by it is the final document vector. Therefore, according to the above steps, the final document vector of the original document and the final document vector of the comparison document can be obtained respectively.

[0110] In addition, when predicting a new document, the paragraph vector is randomly initialized, put into the model, and then iterated again according to the stochastic gradient descent to obtain the final stable sentence vector.

[0111] Step 7: Obtain a second similarity score according to the final document vector of the original document and the final document vector of the comparison document.

[0112] Specifically, the cosine similarity between the final document vector of the original document and the final document vector of the comparison document is calculated, and the obtained cosine similarity is used as the second similarity score.

[0113] Step 8: Determine a similarity result according to a relationship between an average value of the first similarity score and the second similarity score and a score threshold.

[0114] In a specific embodiment, step 8 may specifically include steps 8.1 to 8.2, wherein:

[0115] Step 8.1: Obtain a final similarity score between the original document and the comparison document according to the average of the first similarity score and the second similarity score.

[0116] Specifically, an average of the first similarity score and the second similarity score is calculated, and the average is used as the final similarity score.

[0117] Step 8.2: Determine the relationship between the final similarity score and the score threshold. If the final similarity score is greater than or equal to the score threshold, the similarity result is similar, otherwise it is not similar.

[0118] Preferably, the score threshold is 0.7. It should be noted that the score threshold can be selected according to actual needs, and this embodiment does not make any specific limitation on this.

[0119] The present invention proposes a method for calculating document similarity by combining sentence-by-sentence comparison based on TF-IDF and part-of-speech tagging to generate sentence vectors and direct comparison based on Doc2vec to generate document vectors. While considering word frequency, the present invention also introduces part-of-speech tagging, so that when calculating word weights, part-of-speech and word frequency are considered at the same time. The sentence-by-sentence matching method of the present invention is suitable for texts of different lengths, and does not require compression of the original text or deletion of part of the text. In addition, in order to consider semantic information, the Doc2vec method is combined on this basis to directly extract document vectors for similarity calculation. At the same time, the sentence-by-sentence matching method also improves the problem of weak generalization ability of the Doc2vec method model to a certain extent, that is, if the new text has a large difference in distribution with the text in the training corpus, it is difficult for the model to learn a high-quality semantic vector. Therefore, the present invention comprehensively considers word frequency and part-of-speech, improves the ability to distinguish, and improves the generalization of the model, and can effectively process long texts at the chapter level. In addition, the methods adopted by the present invention are all unsupervised methods, and a general model can be trained with a very small amount of data that does not require annotation.

[0120] Embodiment 2

[0121] The present invention also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned embodiment 1 are implemented.

[0122] Generally, the computer readable storage medium can be placed in a computer device, see Figure 3 The computer device may include units or modules such as a processor, a communication interface, a computer-readable storage medium, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus.

[0123] A computer-readable storage medium for storing a computer program;

[0124] The processor is used to implement the following steps when executing the program stored on the computer-readable storage medium:

[0125] Step 1, obtaining m first sentences of the original document and n second sentences of the comparison document;

[0126] Step 2: performing Chinese word segmentation and stop word removal processing on all the first sentences and the second sentences to obtain a general dictionary consisting of all words after the stop words are removed, wherein each word in the general dictionary is configured with a mapping identifier;

[0127] Step 3: obtaining a common labeling of the word frequency and part of speech of the word according to the word frequency and part of speech of the remaining words after removing the stop words in the first sentence and the second sentence;

[0128] Step 4: obtaining the word frequency and inverse document frequency value of the word based on the common annotation of the word, and obtaining the word frequency and inverse document frequency vectors of the first sentence and the second sentence based on the word frequency and inverse document frequency value of the word;

[0129] Step 5: based on the word frequency and the inverse document frequency vector, find the second sentence most similar to each first sentence of the original document from the comparison document to obtain a first similarity score between the original document and the comparison document;

[0130] Step 6: Input the original document and the comparison document after Chinese word segmentation and stop word removal into Doc2Vec respectively, and iteratively update the Doc2Vec to obtain the final document vector output by the final Doc2Vec;

[0131] Step 7: Obtain a second similarity score according to the final document vector of the original document and the final document vector of the compared document;

[0132] Step 8: Determine a similarity result according to a relationship between an average value of the first similarity score and the second similarity score and a score threshold.

[0133] The communication bus mentioned in the above computer device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0134] The communication interface is used for communication between the above electronic device and other devices.

[0135] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0136] The computer device may be: a desktop computer, a portable computer, an intelligent mobile terminal, a server, etc. This is not limited here, and any electronic device that can implement the present invention belongs to the protection scope of the present invention.

[0137] As for the computer device / storage medium embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment 1.

[0138] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0139] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristic data points described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristic data points described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification.

[0140] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, equipment, or computer program products. Therefore, the present application may adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in conjunction with software and hardware aspects, which are collectively referred to as "modules" or "systems" herein. Moreover, the present application may adopt the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program codes. Computer program storage / distribution is in suitable media, provided together with other hardware or as a part of hardware, and other distribution forms may also be adopted, such as by Internet or other wired or wireless telecommunication systems.

[0141] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices, and computer program products of the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0142] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A method for calculating chapter-level document similarity, characterized in that: The method for calculating chapter-level document similarity includes: Step 1, obtaining m first sentences of the original document and n second sentences of the comparison document; Step 2: performing Chinese word segmentation and stop word removal processing on all the first sentences and the second sentences to obtain a general dictionary consisting of all words after the stop words are removed, wherein each word in the general dictionary is configured with a mapping identifier; Step 3: obtaining a common labeling of the word frequency and part of speech of the word according to the word frequency and part of speech of the remaining words after removing the stop words in the first sentence and the second sentence; Step 4: obtaining the word frequency and inverse document frequency value of the word based on the common annotation of the word, and obtaining the word frequency and inverse document frequency vectors of the first sentence and the second sentence based on the word frequency and inverse document frequency value of the word; Step 5: based on the word frequency and the inverse document frequency vector, find the second sentence most similar to each first sentence of the original document from the comparison document to obtain a first similarity score between the original document and the comparison document; Step 6: Input the original document and the comparison document after Chinese word segmentation and stop word removal into Doc2Vec respectively, and iteratively update the Doc2Vec to obtain the final document vector output by the final Doc2Vec; Step 7: Obtain a second similarity score according to the final document vector of the original document and the final document vector of the compared document; Step 8: Determine a similarity result according to a relationship between an average value of the first similarity score and the second similarity score and a score threshold.

2. The method for calculating chapter-level document similarity according to claim 1, characterized in that: The step 1 comprises: Step 1.1, obtaining the original document and the comparison document; Step 1.2: performing regularized sentence segmentation on the original document and the comparison document to obtain m first sentences of the original document and n second sentences of the comparison document.

3. The method for calculating chapter-level document similarity according to claim 1, characterized in that: The step 2 comprises: Step 2.1, performing Chinese word segmentation on all the first sentences and the second sentences to obtain a plurality of words; Step 2.2, removing stop words from all the words obtained in step 2.1, to obtain the words after removing the stop words; Step 2.3: construct the general dictionary based on all the words obtained in step 2.2 after removing the stop words, and configure a mapping identifier for each word in the general dictionary.

4. The method for calculating chapter-level document similarity according to claim 1, characterized in that: The step 3 comprises: Step 3.1, counting the number of occurrences of each word remaining after removing stop words in the first sentence and the second sentence, recording the number as the word frequency, and converting the first sentence and the second sentence into a bag-of-words format to obtain a first label according to the word frequency of each word; Step 3.2, marking the part of speech for each word remaining after removing the stop words in the first sentence and the second sentence, and assigning a corresponding second tag to each word according to the priority order of the part of speech; Step 3.3: Obtain a common annotation of the word according to the first annotation and the second annotation of the word.

5. The method for calculating chapter-level document similarity according to claim 1, characterized in that: The calculation formula of the word frequency and inverse document frequency value is: Among them, tfidf(w) is the word frequency and inverse document frequency of word w, tf(d,w) is the common annotation of word w in the sentence, N is the total number of all sentences in the document where word w is located, and N(w) is the total number of sentences where word w appears in the document.

6. The method for calculating chapter-level document similarity according to claim 1, characterized in that: The step 5 comprises: Step 5.1, calculating the cosine similarity between the word frequency and inverse document frequency vector of each first sentence in the original document and the word frequency and inverse document frequency vector of all second sentences in the comparison document, and taking the second sentence corresponding to the largest cosine similarity as the sentence most similar to the first sentence; Step 5.2: Count the cosine similarities greater than a preset score among the maximum cosine similarities corresponding to the first sentence, and take the ratio of the number of sentences corresponding to the cosine similarities greater than the preset score to the total number of sentences in the original document as the first similarity score.

7. The method for calculating chapter-level document similarity according to claim 1, characterized in that: The step 6 comprises: Step 6.1, inputting the original document and the comparison document after Chinese word segmentation and stop word removal into Doc2Vec respectively; Step 6.2, the Doc2Vec samples a number of words from the first sentence of the original document and the second sentence of the comparison document respectively according to a preset number; Step 6.3, select a word from the sampled words as the prediction word, and use the remaining words as input words; Step 6.4, calculating the loss value of the loss function according to the first word vector and the second word vector of the predicted word, wherein the second word vector is obtained according to the current document vector and the word vector corresponding to the input word; Step 6.5: Optimize the loss function using stochastic gradient descent. Step 6.6, slide the window according to a preset number to sample new words from the first sentence of the original document and the second sentence of the comparison document respectively, and loop through steps 6.3 to 6.5 until the final document vector output by Doc2Vec is obtained.

8. The method for calculating chapter-level document similarity according to claim 1, characterized in that: The step 7 comprises: The cosine similarity between the final document vector of the original document and the final document vector of the comparison document is calculated, and the obtained cosine similarity is used as the second similarity score.

9. The method for calculating chapter-level document similarity according to claim 1, characterized in that: The step 8 comprises: Step 8.1, obtaining a final similarity score between the original document and the comparison document according to an average value of the first similarity score and the second similarity score; Step 8.2: Determine the relationship between the final similarity score and the score threshold. If the final similarity score is greater than or equal to the score threshold, the similarity result is similar; otherwise, it is dissimilar.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any one of the methods of claims 1-9 are implemented.

Images