Method and device for determining document similarity based on semantic analysis

Through the semantic analysis method, dividing the document parts, conducting semantic analysis and calculating the weights, the problem of inaccurate manually setting weights in the prior art is solved, and the accuracy of the similarity of the document is improved.

CN114580557BActive Publication Date: 2024-12-03BEIJING ZHONGZHI SMART TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210240186.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-10
Publication Date
2024-12-03
Estimated Expiration
2042-03-10

AI Technical Summary

Technical Problem

When determining the similarity of literature, the prior art relies on manual experience to set fixed weights, resulting in inaccurate weight setting, which in turn affects the accuracy of the similarity of literature.

Method used

The semantic analysis method is used to divide the literature into multiple parts, conduct semantic analysis, determine the weight value of each part based on the semantic analysis results, and calculate the similarity between the literature by weighted average.

Benefits of technology

It realizes accurate determination of the weights of different parts of the literature, improves the accuracy of the similarity of the literature, and solves the inaccuracy problem caused by manual experience setting of weights.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114580557B_ABST
    Figure CN114580557B_ABST
Patent Text Reader

Abstract

The present invention discloses a method and device for determining document similarity based on semantic analysis, wherein the method comprises: dividing each document to be compared into multiple parts; performing semantic analysis on each part to obtain the semantic analysis result of each part; determining the weight value of each part of each document to be compared according to the semantic analysis result of each part; obtaining the weighted average result of each document to be compared according to the weight value of each part of each document to be compared; and determining the similarity between the documents to be compared according to the weighted average result of each document to be compared. The present invention can accurately determine the weights of different parts of a document based on semantic analysis, and further accurately determine the document similarity.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and device for determining document similarity based on semantic analysis. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the invention recited in the claims. No admission is made that the description herein is prior art by inclusion in this section.

[0003] At present, when determining the similarity of documents, the existing technology pre-sets different weights for different parts of the document based on manual experience, and finally determines the document similarity by weighted summation of the similarities of the contents of each part based on the fixed weight set manually. The existing method for determining the similarity of documents sets weights based on experience, which has the problem of inaccurate weight setting, and thus leads to inaccurate determination of document similarity. Summary of the invention

[0004] The embodiment of the present invention provides a method for determining document similarity based on semantic analysis, which is used to accurately determine the weights of different parts of a document based on semantic analysis, and further accurately determine the document similarity. The method includes:

[0005] Divide each document to be compared into multiple parts;

[0006] Performing semantic analysis on each part to obtain the semantic analysis result of each part;

[0007] According to the semantic analysis results of each part, determine the weight value of each part of each document to be compared;

[0008] According to the weight value of each part of each document to be compared, the weighted average result of each document to be compared is obtained;

[0009] The similarity between the documents to be compared is determined based on the weighted average result of each document to be compared.

[0010] The embodiment of the present invention further provides a document similarity determination device based on semantic analysis, which is used to accurately determine the weights of different parts of a document based on semantic analysis, and further accurately determine the document similarity. The device includes:

[0011] A division unit is used to divide each document to be compared into multiple parts;

[0012] A semantic analysis unit, used for performing semantic analysis on each part to obtain a semantic analysis result of each part;

[0013] A weight value determination unit, used to determine the weight value of each part of each document to be compared according to the semantic analysis result of each part;

[0014] A processing unit, used for obtaining a weighted average result of each document to be compared according to the weight value of each part of each document to be compared;

[0015] The similarity determination unit is used to determine the similarity between the documents to be compared according to the weighted average result of each document to be compared.

[0016] An embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method for determining document similarity based on semantic analysis when executing the computer program.

[0017] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for determining document similarity based on semantic analysis is implemented.

[0018] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the method for determining document similarity based on semantic analysis is implemented.

[0019] In the embodiment of the present invention, the document similarity determination scheme based on semantic analysis is compared with the technical scheme in the prior art that pre-sets different fixed weights for different parts of the document based on experience to determine the document similarity, which has the problem of inaccurate weight setting and thus leads to inaccurate document similarity determination. By: dividing each document to be compared into multiple parts; performing semantic analysis on each part to obtain a semantic analysis result of each part; determining a weight value of each part of each document to be compared based on the semantic analysis result of each part; obtaining a weighted average result of each document to be compared based on the weight value of each part of each document to be compared; and determining the similarity between the documents to be compared based on the weighted average result of each document to be compared, it is possible to accurately determine the weights of different parts of the document based on semantic analysis, and thus accurately determine the document similarity. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0021] Figure 1 Schematic diagram of a flow chart of a method for determining document similarity based on semantic analysis in an embodiment of the present invention;

[0022] Figure 2 A schematic diagram of a flow chart of performing semantic analysis on each part to obtain a semantic analysis result of each part in an embodiment of the present invention;

[0023] Figure 3 It is a flowchart of performing semantic analysis on each part to obtain the semantic analysis result of each part in another embodiment of the present invention;

[0024] Figure 4 Schematic diagram of the process of document preprocessing in an embodiment of the present invention;

[0025] Figure 5 Schematic diagram of the structure of a device for determining document similarity based on semantic analysis in an embodiment of the present invention;

[0026] Figure 6 This is a schematic diagram of the structure of a semantic analysis unit in an embodiment of the present invention;

[0027] Figure 7 A schematic diagram of the structure of a semantic analysis unit in another embodiment of the present invention;

[0028] Figure 8 Schematic diagram of the Chinese word segmentation training process in a professional field in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0030] Figure 1 FIG. 4 is a flow chart of a method for determining document similarity based on semantic analysis in an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0031] Step 101: Divide each document to be compared into multiple parts;

[0032] Step 102: Perform semantic analysis on each part to obtain a semantic analysis result of each part;

[0033] Step 103: Determine the weight value of each part of each document to be compared according to the semantic analysis result of each part;

[0034] Step 104: Obtaining a weighted average result of each document to be compared according to the weight value of each part of each document to be compared;

[0035] Step 105: Determine the similarity between the documents to be compared based on the weighted average result of each document to be compared.

[0036] The method for determining document similarity based on semantic analysis provided by an embodiment of the present invention, when working: each document to be compared is divided into multiple parts (which can be called sub-documents); semantic analysis is performed on each part to obtain the semantic analysis result of each part; based on the semantic analysis result of each part, the weight value of each part of each document to be compared is determined; based on the weight value of each part of each document to be compared, the weighted average result of each document to be compared is obtained; based on the weighted average result of each document to be compared, the similarity between the documents to be compared is determined.

[0037] Compared with the prior art that pre-sets different fixed weights for different parts of the document based on experience to determine the document similarity, which has the problem of inaccurate weight setting and thus inaccurate document similarity determination, the document similarity determination method based on semantic analysis provided in the embodiment of the present invention can accurately determine the weights of different parts of the document based on semantic analysis, and thus accurately determine the document similarity. The document similarity determination method based on semantic analysis is described in detail below.

[0038] 1. First, the above step 101 is introduced.

[0039] In specific implementation, the documents in the embodiments of the present invention may be patent documents, trademark documents, non-patent documents, etc. Taking patent documents as an example, each document to be compared may be divided into multiple parts, such as the invention title, abstract, claims, and description.

[0040] 2. Next, the above step 102 is introduced.

[0041] In one embodiment, Figure 2 As shown, performing semantic analysis on each part to obtain the semantic analysis result of each part may include the following steps:

[0042] Step 1021: perform word segmentation processing on each part to obtain multiple keywords corresponding to each part;

[0043] Step 1022: extract multiple types of key features from each part according to multiple keywords corresponding to each part and a preset document feature extraction strategy to form a feature set corresponding to each part;

[0044] Step 1024: According to the feature set corresponding to each part, semantic analysis is performed on each part at the word level, syntax level and paragraph level to obtain the semantic analysis result of each part.

[0045] In the specific implementation, in the above step 1021, the document is segmented: according to the subject word library, the document is segmented, and function words that lack practical meaning, rarely appearing low-frequency words and frequently used high-frequency words are deleted, and finally multiple keywords corresponding to each part are obtained.

[0046] In specific implementation, a pre-trained word segmentation model can be used for word segmentation processing. In order to better understand how the word segmentation processing of the present invention is implemented, the word segmentation model is introduced below.

[0047] 1. Evaluation indicators.

[0048] Chinese word segmentation uses the precision and recall evaluation indicators, where:

[0049] Precision = number of correctly segmented words / total number of segmented words;

[0050] Recall = the number of correctly segmented words / the total number of words that should be segmented;

[0051] Comprehensive performance indicator F-measure;

[0052] Fβ=(β2+1)×Precision×Recall / (β2×Precision+Recall);

[0053] β is a weight factor. If we treat precision and recall equally, we take β = 1 and we get the most commonly used F1-measure.

[0054] F1=2×Precision×Recall / (Precision+Recall).

[0055] The following is an introduction using the Chinese word segmentation model as an example.

[0056] 2. Chinese word segmentation model training method.

[0057] Commonly used word segmentation methods can be divided into two categories: dictionary-based word segmentation and statistical-based word segmentation. Statistical-based word segmentation methods have become the mainstream word segmentation method in recent years because they have greatly improved in ambiguous segmentation and unregistered word recognition compared to dictionary-based word segmentation. Commonly used statistical models for word segmentation include hidden Markov models, conditional random field models, maximum entropy models, neural network models, etc. However, when the fields of the test corpus and the training corpus are inconsistent, the accuracy of word segmentation and the performance of unregistered word recognition will drop significantly. Therefore, when using statistical-based word segmentation methods to segment professional field texts, it is necessary to produce annotated training corpora for the corresponding fields. However, annotating training corpora in professional fields will consume a lot of manpower and material resources, and the number of professional fields that have completed annotation work is scarce at this stage.

[0058] Domain terms concentrate on reflecting and carrying the core knowledge of a subject area, while professional domain dictionaries are a collection of professional domain terms. The methods for constructing professional domain dictionaries can be divided into: constructing domain dictionaries using dictionary resources, constructing domain dictionaries using corpora and statistical methods, and constructing domain dictionaries using encyclopedia mining. The inventor proposes a method for constructing professional domain dictionaries that combines dictionary resources with encyclopedias, using encyclopedias to obtain new terms and using existing dictionary resources to improve completeness.

[0059] The Chinese word segmentation training process in professional fields can be as follows Figure 8 As shown:

[0060] (1) Construct a domain dictionary by mining encyclopedia entries and combining them with existing dictionary resources.

[0061] (2) The word segmentation result in the basic natural language processing model is used as the first word segmentation result. The domain dictionary is used to perform reverse maximum matching segmentation on the result, and the second word segmentation result is obtained by combining the ambiguity resolution rules.

[0062] After obtaining the word segmentation results using the basic natural language processing model word segmenter, the domain dictionary is used to perform reverse maximum matching on the word segmentation results to improve the domain adaptability of the word segmentation. The reverse maximum matching algorithm extracts the string of the word segmentation results from right to left, matches it in the domain dictionary, and divides the string into words if the match is successful. For example: "Rock wool composite board is replaced with benzene board composite board." The word segmentation using the basic natural language processing model word segmenter results in: "Rock / Wool / Composite / Board / Replaced / To / Benzene / Board / Composite / Board / ." The domain dictionary contains "rock wool", "benzene board", and "composite board". Perform reverse maximum matching, and the adjusted word segmentation result is: "Rock wool / Composite board / Replaced / To / Benzene / Composite board / ."

[0063] Directly using the domain dictionary to perform reverse maximum matching adjustment on the first word segmentation result often leads to ambiguity problems. For example, the first word segmentation result of "after the project is over" is: "at / work / period is over / after". Assuming that "project" and "period are over" are both in the domain dictionary, there are two possible results: "at / work / period is over / after" and "at / project / period is over / after". Directly using the reverse maximum matching method, the former result is obtained, however, the correct segmentation result should be the latter. Therefore, it is necessary to design a disambiguation algorithm to improve the accuracy of word segmentation.

[0064] Design principle: When the maximum inverse matching algorithm adjusts the initial word segmentation result, the following situations may occur: the initial word segmentation result remains unchanged; the words in the initial word segmentation result are merged without affecting other words; the words in the initial word segmentation result are re-segmented to form new words. The disambiguation algorithm we proposed is mainly aimed at situation three, and uses the disambiguation algorithm to decide whether to merge the character string currently matched to the domain dictionary into a new word. The disambiguation algorithm we proposed mainly considers the number of words before and after the adjustment, the number of single-character words, and the position characteristics of the characters in the words. If the number of words increases after the adjustment, no adjustment will be made. If the number of single-character words increases after the adjustment, no adjustment will be made. The positional features of characters refer to six character tables for the first character, middle character and last character, which are the first character table with a high frequency of being segmented into single characters (L1), the first character table with a low frequency of being segmented into single characters (L2), the middle character table with a high frequency of being combined with the first character into words (L3), the middle character table with a high frequency of being combined with the last character into words (L4), the last character table with a high frequency of being segmented into single characters (L5) and the last character table with a low frequency of being segmented into single characters (L6).

[0065] 3. Chinese word segmentation model training process.

[0066] The result of the first word segmentation is S: a1a2a3 / a4 / … / aM-1aM, W0 represents the word currently matched in the domain dictionary, HD(W0) represents the first word of W0, TL(W0) represents the last word of W0, LEN(W0) represents the length of W0, minSub(S,W0) represents the smallest subsequence of S containing W0, and the subsequence starts and ends with the segmentation interval. combine(S,W0) represents the sequence after merging W0 in S, LMatch(ai) represents the word obtained by reverse maximum matching in the domain dictionary with ai as the starting point, and the left segmentation point of the word does not fall in the existing word. new(S,W0) represents the set of sequences with a number of words greater than or equal to 2 generated after merging W0 in S. Assume that the result of a word segmentation is a1a2a3 / a4 a5 / a6a7a8, and W0 is a5a6, then HD(W0) is a5, TL(W0) is a6, minSub(S,W0) is a4a5 / a6a7a8, combine(minSub(S,W0),W0) is a4 / a5a6 / a7a8, and the leftmost end of combine(minSub(S,W0),W0) is the single word a4. If a2a3a4 is in the domain dictionary, LMatch(a4) is empty because the left segmentation point falls in the existing word a1a2a3. new(S,W0)={a7a8}.

[0067] When LEN(W0)=3, when the matching word length is 3 characters, priority is given to whether single characters and character position features are generated on the left and right sides, and then the situation where minSub(S,W0) is composed of single characters and 2-character words is considered. Finally, the changes in the number of single-character words and the number of words are considered. The specific rules are as follows.

[0068] Rule 1: If the leftmost end of combine(minSub(S,W0),W0) is a single word, denoted by aL, and LMatch(aL) is empty, and the rightmost end of combine(minSub(S,W0),W0) is not a single word, then if aL ∈ L1, and the number of words in combine(minSub(S,W0),W0) is less than or equal to minSub(S,W0), merge W0.

[0069] Rule 2: If the rightmost end of combine(minSub(S,W0),W0) is a single word (aR) and the leftmost end is not a single word, or the leftmost end of combine(minSub(S,W0),W0) is a single word (aL), but LMatch(aL) is not empty, then if aR is in the L5 word table and the number of words in combine(minSub(S,W0),W0) is less than or equal to minSub(S,W0), merge W0.

[0070] Rule 3: minSub(S,W0) is A / BC, LMatch(A) is not empty, W0 is not merged, and the next match starts from A.

[0071] Rule 4: If the number of single words in combine(minSub(S,W0),W0) is 0, and the number of words is less than or equal to minSub(S,W0), and the sequence in new(S,W0) is in the NLPIR dictionary or domain dictionary, then merge W0. If the leftmost end of combine(minSub(S,W0),W0) is a single word aL, and LMatch(aL) is not empty, aL is not counted as a single word.

[0072] Rule 5: If W0 is AB, minSub(S, W0) is A / B, * indicates a string of length greater than 2. If A is in L1, do not merge W0, otherwise, merge W0.

[0073] Rule 6: If W0 is AB, minSub(S,W0) is *A / B, * represents a string of length greater than 2, then if B is in L5, do not merge W0, otherwise, merge W0.

[0074] Rule 7: Except for rules 2 to 6, W0 is not merged.

[0075] The above rules are executed in sequence. In actual effect verification, compared with directly using the word segmenter in the basic natural language processing model, the domain Chinese word segmentation model we trained has greatly improved in accuracy, recall rate, and F value.

[0076] In specific implementation, in the above step 1022, feature extraction:

[0077] In one embodiment, the preset document feature extraction strategy may include: extracting document features based on the frequency of occurrence of keywords in the document, the inverse document frequency of keywords, the part of speech of keywords, whether the keywords are professional words, the position of keywords in the document, the text-rank value of keywords, the information entropy value of keywords, the word vector and overall deviation value of keywords, the length of keywords, keywords as components of sentences, whether keywords are further divided into sub-keywords, the length of the first and last occurrence positions of keywords in the document, and one or any combination of keyword distribution deviations.

[0078] In specific implementation, the above-mentioned document feature extraction strategy can improve the accuracy of feature extraction, thereby improving the accuracy of semantic analysis, and further improving the accuracy of determining the weight value of each part of each document to be compared, and ultimately improving the accuracy of determining document similarity.

[0079] In one embodiment, when keywords can be further divided into sub-keywords, the preset document feature extraction strategy can also include: extracting document features based on the word frequency-inverse document frequency of the sub-keyword, the part of speech of the sub-keyword, whether the sub-keyword is one of the professional words or any combination thereof.

[0080] In specific implementation, the above-mentioned document feature extraction strategy that takes sub-keywords into consideration can further improve the accuracy of feature extraction, thereby further improving the accuracy of semantic analysis, and further improving the accuracy of determining the weight value of each part of each document to be compared, and ultimately further improving the accuracy of determining document similarity.

[0081] In summary, in order to facilitate understanding of how the present invention is implemented, the document feature extraction strategy is further described in the form of the following Table 1.

[0082] In the specific implementation, after analyzing the data characteristics and business scenarios of patent documents, we proposed a specific strategy for extracting features from patent documents using natural language processing technology, as shown in Table 1 below:

[0083]

[0084]

[0085] Table 1

[0086] As shown in Table 1 above, compared with the prior art solution of setting fixed weights for different parts of a document based on experience to determine document similarity, the embodiment of the present invention is equivalent to adjusting the weights based on semantic analysis, thereby obtaining a more accurate weight value, thereby improving the accuracy of determining document similarity.

[0087] In one embodiment, the multiple types of key features may include: static features of the document, features associated with the document and the query, and features of the query.

[0088] In specific implementation, features can be the source of nutrients for algorithms and models. The quality of feature selection is directly related to the effect of the model learned by algorithm training. Unlike traditional text classification, MLR in the patent field outputs the ranking of document sets for a given query (query or retrieval feature), which not only considers the characteristics of the document itself, but also the characteristics of the relationship between the query and the document. In general, MLR in the patent field needs to consider three aspects of features (static features of documents, features of the relationship between documents and queries, and features of queries):

[0089] The feature selection of patent information refers to the purpose of grasping the development status of a certain technology by summarizing, deducing, analyzing, synthesizing, abstracting and generalizing the inherent characteristics of patent documents, that is, the patent technology content. Specifically, according to the technical content such as the technical subject, patent country, patent inventor, patent assignee, patent classification number, patent application date, patent authorization date and patent citation documents provided by the patent documents, extensive information collection is carried out, and the collected content is read and excerpted. On this basis, further research activities such as classification, comparison and analysis of this information are carried out to form an organic information collection. Then, those representative, key and typical patent documents are studied in a focused manner, and the inherent and even potential relationships between patent information are finally found, so as to form a relatively complete understanding.

[0090] 1. The static features of the document itself (static features of the document), including the text features of the document, such as weighted word vectors, TF, IDF, BM25 and other language model scores of different domains of the document (title, bibliographic abstract, claims, patent specification, patent text, etc.), as well as the quality score of the document, the importance score of the patent claims, etc. Regarding the quality score of the document, the search has different calculation indicators according to different patent classifications. For example, the quality score calculation of patent documents in the field of machinery not only considers the text richness of the patent itself, but also considers various related contents of the patent, such as changes in part of speech, word position, syntactic structure in the field of machinery, etc.

[0091] 2. Features associated with documents and queries (features associated with documents and queries), such as the TD-IDF score and BM25 score of the document corresponding to the query.

[0092] 3. Features of the query itself (query features), such as policy text features, weighted patent field professional word vectors, query length, categories described in the query, query's BM25 sum / avg / min / max / median scores, query popularity last month, etc.

[0093] In the feature engineering of query and document, in addition to lexical analysis, it is also necessary to analyze and extract the semantics of the "explained" lexical "really want to express", that is, the concept. For example, polysemy, synonyms and near-synonyms, the same word expresses different meanings in different scenarios, and different words in different scenarios may also express the same meaning. LSA (latent semantic analysis) is a well-known technology for dealing with such problems. Its main idea is to map the high-dimensional vector space to the low-dimensional latent semantic space or concept space, that is, to reduce the dimension. The specific method is to perform singular value decomposition (SVD) on the term-document matrix. In the singular value decomposition, C is a matrix with documents as rows and terms as columns (assuming M x N), and the elements are the term frequency-inverse document frequency value of the term. C is decomposed into 3 small matrices and multiplied; each column of U represents a topic, and each non-zero element in it represents the relevance of a topic to an article. The larger the value, the more relevant it is; V represents the relevance of the keyword to all terms; ∑ represents the relevance between the article topic and the keyword.

[0094] In one embodiment, Figure 3 As shown, the above-mentioned method for determining document similarity based on semantic analysis may further include step 1023: using principal component analysis, linear discriminant analysis and mutual information method, screening and combining features of the feature set corresponding to each part to obtain the feature set corresponding to each part after feature dimension reduction processing;

[0095] According to the feature set corresponding to each part, semantic analysis is performed on each part at the word level, syntactic level and paragraph level to obtain the semantic analysis result of each part, which may include: according to the feature set corresponding to each part after feature dimensionality reduction processing, semantic analysis is performed on each part at the word level, syntactic level and paragraph level to obtain the semantic analysis result of each part.

[0096] In specific implementation, the number of words that make up the text is generally very large, so the dimension of the vector space representing the text is also quite large, which can reach thousands of dimensions, so dimensionality reduction processing must be performed. Generally, dimensionality reduction is performed through feature extraction methods, and the characteristic indicators representing the vocabulary can include: document frequency, information acquisition, mutual information, square root fit test, term strength, etc. By calculating any of the above indicators of the vocabulary, and then sorting from large to small, a specified number of words or words with indicator values ​​greater than a specified threshold are selected to form a feature set. The following is a further introduction to dimensionality reduction processing.

[0097] In specific implementation, after converting the patent document into a feature set, that is, in step 1023 after step 1022, many algorithms can directly enter the retrieval and sorting stage. However, since the data of the patent document is too large and each patent document has many features, the calculation speed of the algorithm is crucial, otherwise the algorithm may look beautiful but have no practical effect. In order to further speed up the calculation speed, based on the feature set, principal component analysis, linear discriminant analysis, mutual information and other methods are used to screen and combine features to obtain new features and achieve the purpose of feature dimensionality reduction.

[0098] In specific implementation, dimensionality reduction is a method of preprocessing high-dimensional feature data. Dimensionality reduction is to retain the most important features of high-dimensional data and remove noise and unimportant features, so as to achieve the purpose of improving data processing speed. In actual production and application, dimensionality reduction can save us a lot of time and cost within a certain range of information loss. Dimensionality reduction has also become a very widely used data preprocessing method.

[0099] In practice, dimensionality reduction has the following advantages:

[0100] 1. Make the dataset easier to use.

[0101] 2. Reduce the computational overhead of the algorithm.

[0102] 3. Remove noise.

[0103] 4. Make the results easy to understand.

[0104] In the above step 1024, a natural language processing engine is used to perform semantic analysis of the patent document at the word level, syntax level, and chapter level. The word level analysis includes Chinese word segmentation, named entity recognition, part-of-speech tagging, synonym analysis, word vector analysis, n-gram analysis, word granularity analysis, stop word analysis, etc.; the syntactic level analysis of the patent document includes dependency grammar analysis, language model, short string analysis, etc.; the chapter level analysis of the patent document includes patent document tag extraction, topic model, text clustering, etc.

[0105] In the specific implementation, in the above step 1024, the feature evaluation weighting: the calculation method of the feature weight is mainly to use the word frequency-inverse document frequency formula. According to different mining purposes, there are currently a variety of word frequency-inverse document frequency formula construction methods. From the above content, it can be seen that the construction of the text vector space is completely carried out according to the law of probability statistics, and the relationship between words is not considered. Another key step in the text mining process is to calculate the similarity distance between text vectors. For any two vectors X = (x1, x2, ..., xn) and X' = (x'1, x'2, ..., x'n), there are mainly three most commonly used distance metrics: Euclidean distance, cosine distance and inner product. The above three distance metrics also do not involve the analysis of the relationship between words in the vector. Through the above analysis, it can be concluded that both the text vector construction process and the similarity distance measurement process are designed according to the principle of probability statistics, and the semantic relationship between features is not considered. In the use of natural language, in order to express the need, a large number of synonyms and related words (words that are often used together according to the context) often appear in the text, such as "computer" and "computer" are synonymous in IT technology; in the judicial field, the probability of "police" and "case" appearing at the same time is very high. These synonyms and related words will also appear in large numbers in the text feature vector, which increases the dimension of the text feature vector on the one hand, and reduces the accuracy of the text feature vector to express the document on the other hand. Although text feature extraction can reduce the dimension of the feature vector through a pre-set threshold, it is not based on the premise of ensuring semantic accuracy, and therefore often has the opposite effect. Although synonym dictionaries and implication word dictionaries can also be used to reduce synonyms and related words in the process of word segmentation, it also brings about the problem of dictionary maintenance and updating.

[0106] From the above analysis, we can see that associating synonyms and related words with a given word is very important for both user applications and text semantic mining. The function of the retrieval association model is mainly to solve this problem. The retrieval association model will be introduced below.

[0107] 3. Next, for ease of understanding, the above steps 103 to 105 are introduced together.

[0108] In specific implementation, the weight value of each part of each document to be compared is determined according to the semantic analysis result of each part; the weighted average result of each document to be compared is obtained according to the weight value of each part of each document to be compared; and the similarity between the documents to be compared is determined according to the weighted average result of each document to be compared.

[0109] 4. Next, further optimization steps are introduced.

[0110] The embodiment of the present invention can be used in semantic retrieval. Document preprocessing is performed on each document in the document database to be retrieved. Taking patent documents as an example, patent document preprocessing converts the documents into semi-structured data containing plain text by processing, extracting, encoding, normalizing, etc. These semi-structured data will be used as input for intelligent patent retrieval and searched in the pre-established semi-structured data. According to the document similarity determination method based on semantic analysis provided in the embodiment of the present invention, comparison and sorting are performed to obtain intelligent search results, which can also be called semantic search results, with high retrieval accuracy. Documents and their processing results are transmitted through the Kafka distributed message queue. The whole process is like Figure 4 As shown:

[0111] The main format of patent document data is XML. First, we need to extract and splice the required data from these two XML data tables.

[0112] XML files are mostly used to describe information, so after obtaining an XML document, extracting the corresponding information according to the elements in the XML is XML parsing. There are two ways to parse XML, one is DOM parsing and the other is SAX parsing. The embodiment of the present invention uses DOM parsing to process XML.

[0113] The XML parser based on DOM parsing converts it into a collection of object models and stores information in a tree data structure. Through the DOM interface, the application can access any part of the data in the XML document at any time, so this access method using the DOM interface is also called random access.

[0114] This method also has its flaws, because the DOM parser converts the entire XML file into a tree and stores it in memory. When the file structure is large or the data is complex, this method requires a lot of memory, and traversing a complex tree is also a very time-consuming operation. However, the tree structure used by DOM is consistent with the way XML stores information, and its random access can also be used, so the DOM interface still has a wide range of use value.

[0115] There are four core operation interfaces in DOM parsing:

[0116] 1. Document: This interface represents the entire XML document, which is the root of the entire DOM, that is, the entrance to the tree. Through this interface, you can access the contents of all elements in XML. Its common methods are as follows:

[0117] public NodeList getElementByTagName(String tagname);

[0118] Get the NodeList of the specified node name;

[0119] Public Element createElement(String tagName) throws DOMException;

[0120] Create a node with the specified name;

[0121] Public Text createTextNode(String data)throws DOMException;

[0122] Create a text content node;

[0123] Element createElement(String tagName) throws DOMException;

[0124] Create a node element;

[0125] Public Attr createAttribute(String name) throws DOMException;

[0126] Create a property:

[0127] 2.Node: This interface plays an important role in the entire DOM tree. The core interfaces of DOM operations are inherited from Node (Document, Element, Attr). In the DOM tree, each Node interface represents a DOM tree node.

[0128] Common methods of Node interface:

[0129] Node appendChilid(Node newChild)throws DOMException;

[0130] Add the next new node under the current node;

[0131] Public NodeList getChildNodes();

[0132] Get all child nodes under this node;

[0133] Public Node getFirstChild();

[0134] Get the first child node under this node;

[0135] Public Node getLastChild();

[0136] Get the last child node under this node;

[0137] Public boolean hasChildNodes();

[0138] Determine whether there are other nodes;

[0139] String getNodeValue() throws DOMException;

[0140] Get the node content:

[0141] 3.NodeList: This interface represents a collection of points, generally used for a group of nodes in an ordered relationship. Common methods of NodeList:

[0142] public int getLength();

[0143] Get the number of nodes in NodeList;

[0144] Public Node item(int index);

[0145] Get the node object according to the index;

[0146] 4.NamedNodeMap: This interface represents a one-to-one relationship between a set of nodes and their unique names, and is mainly used to represent node attributes.

[0147] In addition to the above four core interfaces, if a program needs to perform DOM parsing operations, it needs to follow the following steps:

[0148] 1. Create DocumentBuilderFactor to obtain the DocumentBuilder object:

[0149] DocumentBuilderFactory factory=DocumentBuilderFactory.newInstance();

[0150] 2. Create DocumentBuidler:

[0151] DocumentBuilder builder=factory.newDocumentBuilder();

[0152] 3. Create a Document object and get the entry to the tree:

[0153] Document doc = builder.parse("Relative path or absolute path of XML file");

[0154] 4. Create NodeList:

[0155] NodeList n1 = doc.getElementByTagName("read node").

[0156] The following introduces the method for determining document similarity based on semantic analysis provided by an embodiment of the present invention for intelligent retrieval of patent documents.

[0157] 1. Retrieval recall algorithm.

[0158] The most important thing in the retrieval system is sorting, which is divided into primary sorting (i.e. retrieval recall) and learned sorting based on business understanding, NLP, and machine learning. Retrieval recall mainly uses algorithms such as word frequency-inverse document frequency and BM25.

[0159] The purpose of retrieval recall is to use a relatively lightweight algorithm to quickly obtain candidate items, so as to facilitate the subsequent use of complex sorting algorithms, optimize the search efficiency of the entire process, and avoid unnecessary operations. The candidate items should be larger than the target items we are ultimately looking for. For example, if our goal is to find the most likely 50 patent documents, then the candidate items can be set to 500.

[0160] (1) Retrieval and recall based on word frequency-inverse document frequency

[0161] Term frequency-inverse document frequency is a common weighting technique used in information retrieval and text mining. Term frequency-inverse document frequency is a statistical method used to assess the importance of a word to a document set or a document in a corpus. In a given document, the term frequency (tf) refers to the frequency of a given word in the document. This number is normalized to prevent it from being biased towards long documents. For a term ti in a particular document, its importance can be expressed as:

[0162]

[0163] In the above formula, ni,j is the number of times the word appears in file dj, and the denominator is the sum of the number of times all words appear in file dj.

[0164] Inverse document frequency (idf) is a measure of the general importance of a word. The idf of a particular word can be calculated by dividing the total number of documents by the number of documents containing the word and taking the base 10 logarithm of the quotient:

[0165]

[0166] in:

[0167] |D|: total number of documents in the corpus;

[0168] |{j:t i ∈d j}|: The number of documents containing the word ti (i.e., the number of documents where ni, j!=0). If the word is not in the data, the denominator will be zero, so 1+|{j:t i ∈d j}|;

[0169] Then tfidfi,j=tfi,j×idfi;

[0170] A high term frequency within a particular document, combined with a low document frequency of that term in the entire document collection, can produce a highly weighted term frequency-inverse document frequency. Therefore, term frequency-inverse document frequency tends to filter out common terms and retain important terms.

[0171] (2) Retrieval and recall based on BM25

[0172] BM25 is a common formula for relevance scoring, which is equivalent to an expansion of tfidf. It mainly calculates the relevance of all words and documents in a query, and then accumulates the scores. The relevance score of each word is mainly affected by tf / idf. The original formula of BM25 is as follows:

[0173]

[0174] Where ri is the number of relevant documents containing term i, ni is the number of documents containing term i, N is the number of all documents in the entire document dataset, and R is the number of documents related to this query. In typical cases, there is no relevant information, that is, ri and R are both 0, and in normal queries, no term will appear more than 1 times, so the scoring formula score becomes:

[0175]

[0176] The main operation of the BM25 algorithm when used for search relevance scoring is: perform morpheme analysis on the query to generate the morpheme qi; then, for each search result D, calculate the relevance score of each morpheme qi with D; finally, perform a weighted summation of the relevance scores of qi relative to D to obtain the relevance score of the query with D. The above formula can be translated into the general formula of BM25 as follows:

[0177]

[0178] Among them, Q represents Query, qi represents a morpheme after Q is parsed (for Chinese, we can regard the word segmentation of Query as morpheme analysis and each word as morpheme qi); d represents a search result document; Wi represents the weight of the morpheme; R(qi, d) represents the relevance score between morpheme qi and document d.

[0179] The following defines Wi, which represents the weight of the relevance between a word and a document. There are many ways to do this. If it is used as a general formula for BM25, it can be described in a form similar to IDF. The formula is as follows:

[0180]

[0181] Where N is the total number of documents in the index, and n(qi) is the number of documents containing qi. According to the definition of IDF, for a given document set, the more documents contain qi, the lower the weight of qi. In other words, when many documents contain qi, the discrimination of qi is not high, so the importance of using qi to judge relevance is lower. Let's look at the relevance score R(qi, d) between morpheme qi and document d. First, let's look at the general form of the relevance score in BM25:

[0182]

[0183]

[0184] Among them, k1.k2.b are adjustment factors, which are usually set according to experience, generally k1 = 2, b = 0.75; fi is the frequency of qi in d, and qfi is the frequency of qi in Query. dl is the length of document d, and avgdl is the average length of all documents. Since in most cases, qi will only appear once in Query, that is, qfi = 1, the formula can be simplified to:

[0185]

[0186] From the definition of K, we can see that the role of parameter b is to adjust the impact of document length on relevance. The larger b is, the greater the impact of document length on relevance score, and vice versa; the larger the relative length of the document, the larger the K value will be, and the smaller the relevance score will be. This can be understood as, when the document is longer, the greater the chance of containing qi, so under the same fi, the correlation between long documents and qi should be weaker than the correlation between short documents and qi.

[0187] In summary, the relevance score formula of the BM25 algorithm can be summarized as:

[0188]

[0189] From the BM25 formula, we can see that by using different morpheme analysis methods, morpheme weight determination methods, and morpheme-document relevance determination methods, we can derive different search relevance score calculation methods, which provides us with greater flexibility in designing algorithms.

[0190] (3) Function-based sorting

[0191] Many ranking models are based on functions. For example, the decay function can reduce the weight by fractional decay based on dimensions such as time and number. Common decay functions include Gauss decay and linear decay.

[0192] Gauss attenuation: The attenuation function derived from the Gaussian function is defined as follows:

[0193]

[0194] The score of this formula will be attenuated after exceeding origin + / - offset; when the scale (attenuation limit) and decay (attenuation degree) are given, the shape of the attenuation function is also given, so the calculation method of σ is as follows:

[0195] σ 2 =-scale 2 / (2×ln(decay))

[0196] Exponential decay:

[0197] S(doc)=exp(λ×max(0,|fieldvalue doc -origin|-offset))

[0198] Similarly, the score of this formula will be attenuated after exceeding origin + / - offset; when the scale (attenuation limit) and decay (attenuation degree) are given, the shape of the attenuation function is also given, so the calculation method of λ is as follows:

[0199] λ=ln(decay) / scale

[0200] Linear Falloff:

[0201]

[0202] The calculation method of s is as follows:

[0203] s = scale / (1.0 - decay));

[0204] Compared with the previous two formulas, the score of this formula will become 0 when the fieldvalue exceeds twice the scale.

[0205] 2. Retrieve sorting algorithm.

[0206] The essence of the retrieval system is to achieve semantic matching based on the understanding of the query and patents, and then sort the matching candidate results. The sorting is divided into two stages: recall and refinement. The recall is completed and the TopN candidate results are returned. In the refinement stage, the TopN results are accurately sorted based on the machine learning ranking model to select the TopM (N>M) results, referred to as LTR (learning to rank).

[0207] The following is a detailed introduction to constructing the retrieval association model mentioned above using latent semantic analysis and association rule mining.

[0208] 1. Latent semantic analysis.

[0209] Latent semantic analysis (LSA) reduces synonym noise by introducing a concept space. LSA uses the contextual relevance of words, that is, words that appear in similar contexts are considered to be similar in usage and meaning. To implement the LSA idea, first construct a word-document matrix:

[0210] A=|a ij |m×n;

[0211] m represents the total vocabulary, n represents the number of documents, where a ijis a non - negative value, representing the weight of the i - th word in the j - th document. Different words correspond to different rows of matrix A, and each document corresponds to a column of matrix A. Usually, aij should consider contributions from two aspects, namely the local weight L(i, j) and the global weight C(i, j). In the VSM model, there are different weight calculation methods for the local weight L(i, j) and the global weight C(i, j), such as IDF, TFIDF, etc. Since each word only appears in a small number of documents, A is usually a high - order sparse matrix. Let the i - th and j - th words correspond to the i - th row and j - th row of the word - document matrix respectively, denoted as ti=(ai1, ai2,…, ain) and tj=(aj1, aj2,…, ajn), and their similarity is defined as

[0212]

[0213] Before calculating the similarity, usually aij is first transformed into log(aij + 1), and then divided by its entropy. Such pre - processing can take into account the context of the word and highlight the usage environment of the word in the article. After the information entropy transformation, an ordered word - document matrix A′ = a′ij m×n is obtained, where:

[0214]

[0215] The theoretical basis of latent semantic analysis is the singular value decomposition (SVD) of the matrix. Singular value decomposition is a commonly used method in mathematical statistics. After the word - document matrix A′ is established, the k - rank approximation matrix A′k (k << min(m, n)) of A′ is calculated using singular value decomposition. After singular value decomposition, matrix A′ can be expressed as the product of three matrices:

[0216] A' = U∑V T ;

[0217] In the formula, U and V are the left and right singular vector matrices corresponding to the singular values of A′ respectively; Σ is the standard form; VT is the transpose of V; the singular values of A′ are arranged in descending order to form the diagonal matrix Σk. Take the first k columns of U and V to construct the k - rank approximation matrix of A′:

[0218]

[0219] In the formula, the column vectors of Uk and Vk are all orthogonal vectors. Assuming the rank of A′ is r, then there are:

[0220] U T U = V T V = I r (I r is an r×r order identity matrix);

[0221] Use A' k Approximate representation of the original word-document matrix A′, U k and V k The row vectors in are used as word vectors and document vectors respectively, and text classification and other document processing are performed on this basis. This is the latent semantic indexing technology. Although LSI also uses the words contained in the document to represent the semantics of the document, the LSI model does not regard all the words in the document as reliable representations of the document concept. Since the diversity of words in a document largely obscures the semantic structure of the document, LSI uses singular value decomposition and k-rank approximation matrices to, on the one hand, eliminate the "noise" factor contained in the original word-document matrix, thereby highlighting the semantic relationship between words and documents; on the other hand, it greatly reduces the word and document vector space, which can improve the efficiency of text mining.

[0222] 2. Association rule mining.

[0223] Association rules are a major mining technique in data mining. They can discover potentially useful associations or correlations from massive amounts of data. Let I = {i1, i2, …, im} be a set of items, and let D be a set of transactions T, where transaction T is a set of items and T∈I. Each transaction has a unique identifier, denoted as TID. Let X be a set of items in I. If X∈T, then transaction T is said to contain X. An association rule is an implication of the form X→Y, where And the support of the rule X→Y in the transaction set D is the ratio of the number of transactions containing X and Y to the number of all transactions in the transaction set, denoted as support(X→Y), that is:

[0224]

[0225] The confidence of rule X→Y in the transaction set refers to the ratio of the number of transactions containing X and Y to the number of transactions containing X, denoted as confidence(X→Y), that is:

[0226]

[0227] Given a transaction set D, the association rule mining problem is to generate association rules whose support and credibility are respectively greater than the minimum support and minimum credibility given by the user. Commonly used algorithms include the Apriori algorithm and the FP-Growth algorithm.

[0228] For the mining of associated words, suppose the mined association rules are in the form of {ti→tj,s,c}, which means that if word ti appears in a document, then the support of word tj in the same document is s (0≤s≤1) and the confidence is c (0≤c≤1). If the support and confidence are greater than the specified threshold, it can be considered that their association is very large, so large that even if the word is ignored, there will be no information loss, which ensures the independence of the word.

[0229] 3. Algorithm process.

[0230] According to the above principle analysis, the construction algorithm of synonyms and associated word sets is described as follows:

[0231] Algorithm 1: Synonym set construction algorithm.

[0232] enter:

[0233] 1) Training document set;

[0234] 2) The dimension k of the concept space;

[0235] 3) Synonymous merging retains the number N of feature words.

[0236] Output:

[0237] 1) A feature word set with a total of N elements;

[0238] 2) Merging scheme set (determines which retained feature word the merged synonyms should be merged into).

[0239] step:

[0240] 1) Construct a word-document matrix based on the words contained in the training documents;

[0241] 2) Order the word-document matrix;

[0242] 3) Use SVD to perform singular value decomposition on the word-document matrix to obtain the left and right singular vectors and singular value standard form of the word-document matrix;

[0243] 4) Keep the first k columns of data of the left singular vector and clear all other columns of data;

[0244] 5) Keep the first k data on the diagonal of the singular value standard form, and clear all other diagonal data;

[0245] 6) Keep the first k columns of data of the right singular vector, and clear all other columns of data;

[0246] 7) Multiply the cleared left and right singular vectors and the standard form of the singular values ​​to obtain a new word-document matrix;

[0247] 8) Order the new word-document matrix;

[0248] 9) When the number of words exceeds N, perform steps 10 to 14 to perform synonym merging;

[0249] 10) Find the two feature words with the greatest similarity in the feature word set;

[0250] 11) Delete any one of the feature word pairs with the greatest similarity in the feature word set;

[0251] 12) Searching for a merging scheme containing any feature word in the feature word pair with the greatest similarity in the merging scheme set;

[0252] 13) If found, add another feature word to the merged feature word set of the merge scheme;

[0253] 14) If no merge solution is matched, a merge solution is constructed with the two feature words and put into the merge solution set.

[0254] Algorithm 2: Algorithm for constructing associated word sets.

[0255] enter:

[0256] 1) Training document set;

[0257] 2) Association merging threshold: support s, confidence c.

[0258] Output:

[0259] 1) Feature word set after association and merging;

[0260] 2) The set of merged solutions after associated merge.

[0261] step:

[0262] 1) Use the Aprior algorithm to obtain all single-dimensional associations whose support and confidence are greater than s and c respectively;

[0263] 2) Perform steps 3 to 6 for each single-dimensional association rule whose support and confidence are greater than s and c respectively, and perform association merging;

[0264] 3) Delete the feature words on the right side of the association rule in the feature word set;

[0265] 4) Search for a merging solution containing the feature words on any side of the association rule in the merging solution set;

[0266] 5) If found, add the other feature word to the merged feature word set of the merge scheme;

[0267] 6) If no merging scheme is matched, a merging scheme is constructed with the two feature words on the left and right sides of the association rule and put into the merging scheme set.

[0268] Practice has proved that the construction of retrieval association models can be effectively realized through latent semantic analysis and association rule mining.

[0269] In summary, the method for determining document similarity based on semantic analysis provided by the embodiment of the present invention can accurately determine the weights of different parts of a document based on semantic analysis, and further accurately determine the document similarity.

[0270] The present invention also provides a document similarity determination device based on semantic analysis, as described in the following embodiments. Since the principle of the device to solve the problem is similar to the document similarity determination method based on semantic analysis, the implementation of the device can refer to the implementation of the document similarity determination method based on semantic analysis, and the repeated parts will not be repeated.

[0271] Figure 5 FIG. 1 is a schematic diagram of the structure of a device for determining document similarity based on semantic analysis in an embodiment of the present invention. Figure 5 As shown, the device comprises:

[0272] A division unit 01 is used to divide each document to be compared into multiple parts;

[0273] Semantic analysis unit 02, used for performing semantic analysis on each part to obtain a semantic analysis result of each part;

[0274] The weight value determination unit 03 is used to determine the weight value of each part of each document to be compared according to the semantic analysis result of each part;

[0275] Processing unit 04, used for obtaining a weighted average result of each document to be compared according to the weight value of each part of each document to be compared;

[0276] The similarity determination unit 05 is used to determine the similarity between the documents to be compared according to the weighted average result of each document to be compared.

[0277] In one embodiment, Figure 6 As shown, the semantic analysis unit 02 may include:

[0278] The word segmentation processing module 021 is used to perform word segmentation processing on each part to obtain multiple keywords corresponding to each part;

[0279] The feature extraction module 022 is used to extract multiple types of key features from each part according to multiple keywords corresponding to each part and a preset document feature extraction strategy to form a feature set corresponding to each part;

[0280] The feature evaluation module 024 is used to perform semantic analysis on each part at the word level, syntactic level and paragraph level according to the feature set corresponding to each part, so as to obtain the semantic analysis result of each part.

[0281] In one embodiment, Figure 7 As shown, the semantic analysis unit 02 may further include a feature dimension reduction module 023, which is used to screen and combine the features of the feature set corresponding to each part by using the principal component analysis method, the linear discriminant analysis method and the mutual information method, so as to obtain the feature set corresponding to each part after the feature dimension reduction processing;

[0282] The feature evaluation module 024 is specifically used for: performing semantic analysis on each part at the word level, syntactic level and paragraph level according to the feature set corresponding to each part after the feature dimensionality reduction processing, to obtain the semantic analysis result of each part.

[0283] In one embodiment, the multiple types of key features may include: static features of the document, features associated with the document and the query, and features of the query.

[0284] In one embodiment, the preset document feature extraction strategy may include: extracting document features based on the frequency of occurrence of keywords in the document, the inverse document frequency of keywords, the part of speech of keywords, whether the keywords are professional words, the position of keywords in the document, the text-rank value of keywords, the information entropy value of keywords, the word vector and overall deviation value of keywords, the length of keywords, keywords as components of sentences, whether keywords are further divided into sub-keywords, the length of the first and last occurrence positions of keywords in the document, and one or any combination of keyword distribution deviations.

[0285] In one embodiment, when keywords can be further divided into sub-keywords, the preset document feature extraction strategy can also include: extracting document features based on the word frequency-inverse document frequency of the sub-keyword, the part of speech of the sub-keyword, whether the sub-keyword is one of the professional words or any combination thereof.

[0286] The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0287] An embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method for determining document similarity based on semantic analysis when executing the computer program.

[0288] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for determining document similarity based on semantic analysis is implemented.

[0289] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the method for determining document similarity based on semantic analysis is implemented.

[0290] In the embodiment of the present invention, the document similarity determination scheme based on semantic analysis is compared with the technical scheme in the prior art that pre-sets different fixed weights for different parts of the document based on experience to determine the document similarity, which has the problem of inaccurate weight setting and thus leads to inaccurate document similarity determination. By: dividing each document to be compared into multiple parts; performing semantic analysis on each part to obtain a semantic analysis result of each part; determining a weight value of each part of each document to be compared based on the semantic analysis result of each part; obtaining a weighted average result of each document to be compared based on the weight value of each part of each document to be compared; and determining the similarity between the documents to be compared based on the weighted average result of each document to be compared, it is possible to accurately determine the weights of different parts of the document based on semantic analysis, and thus accurately determine the document similarity.

[0291] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0292] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0293] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0294] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0295] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for determining document similarity based on semantic analysis, characterized in that: include: Divide each document to be compared into multiple parts; Performing semantic analysis on each part to obtain the semantic analysis result of each part, which includes: using a pre-trained word segmentation model to perform word segmentation processing on each part to obtain multiple keywords corresponding to each part, and using a disambiguation algorithm to adjust the number of previous and next words, the number of single words and the position characteristics of the word in the word when training the word segmentation model; according to the multiple keywords corresponding to each part and the preset document feature extraction strategy, extracting multiple types of key features from each part to form a feature set corresponding to each part; according to the feature set corresponding to each part, performing word-level, syntactic-level and paragraph-level semantic analysis on each part to obtain the semantic analysis result of each part; According to the semantic analysis results of each part, determine the weight value of each part of each document to be compared; According to the weight value of each part of each document to be compared, the weighted average result of each document to be compared is obtained; The similarity between the documents to be compared is determined based on the weighted average result of each document to be compared.

2. The method for determining document similarity based on semantic analysis according to claim 1, characterized in that: Also includes: Using principal component analysis, linear discriminant analysis and mutual information method, the feature set corresponding to each part is screened and combined to obtain the feature set corresponding to each part after feature dimension reduction processing; According to the feature set corresponding to each part, semantic analysis is performed on each part at the word level, syntactic level and paragraph level to obtain the semantic analysis result of each part, including: according to the feature set corresponding to each part after feature dimensionality reduction processing, semantic analysis is performed on each part at the word level, syntactic level and paragraph level to obtain the semantic analysis result of each part.

3. The method for determining document similarity based on semantic analysis according to claim 1, characterized in that: The multiple types of key features include: static features of documents, features associated with documents and queries, and features of queries.

4. The method for determining document similarity based on semantic analysis according to claim 1, characterized in that: The preset document feature extraction strategy includes: extracting document features based on the frequency of keyword appearance in the document, the inverse document frequency of the keyword, the part of speech of the keyword, whether the keyword is a professional word, the position of the keyword in the document, the text-rank value of the keyword, the information entropy value of the keyword, the word vector and the overall deviation value of the keyword, the length of the keyword as a sentence component, whether the keyword is further divided into sub-keywords, the length of the first and last appearance positions of the keyword in the document, and the keyword distribution deviation, or any combination thereof.

5. The method for determining document similarity based on semantic analysis according to claim 4, characterized in that: When the keywords can be further divided into sub-keywords, the preset document feature extraction strategy also includes: extracting document features based on the word frequency-inverse document frequency of the sub-keywords, the part of speech of the sub-keywords, whether the sub-keywords are professional words or any combination thereof.

6. A document similarity determination device based on semantic analysis, characterized in that: include: A division unit is used to divide each document to be compared into multiple parts; The semantic analysis unit is used to perform semantic analysis on each part to obtain the semantic analysis result of each part, which includes: using a pre-trained word segmentation model to perform word segmentation processing on each part to obtain multiple keywords corresponding to each part, and using a disambiguation algorithm to adjust the number of previous and next words, the number of single words and the position characteristics of the word in the word when training the word segmentation model; according to the multiple keywords corresponding to each part and the preset document feature extraction strategy, multiple types of key features are extracted from each part to form a feature set corresponding to each part; according to the feature set corresponding to each part, semantic analysis is performed on each part at the word level, the syntax level and the paragraph level to obtain the semantic analysis result of each part; A weight value determination unit, used to determine the weight value of each part of each document to be compared according to the semantic analysis result of each part; A processing unit, used for obtaining a weighted average result of each document to be compared according to the weight value of each part of each document to be compared; The similarity determination unit is used to determine the similarity between the documents to be compared according to the weighted average result of each document to be compared.

7. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

9. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.