Document similarity detecting method based on kernel function

Abstract

The invention discloses a document similarity detecting method based on a kernel function, belongs to the technical field of information retrieval and mainly solves the problem that the comprehensive evaluation effect of text retrieval is not good. The method comprises the following steps of: 1, creating a document set, wherein a set consisting of lexical items of the document set forms a dictionary, and the storage capacity is N; 2, indicating a document dX and a document dZ which are to be compared into document vector sum; 3, describing the similarity of the two documents through the product of the two document vectors and Euclidean distance between the two document vectors, wherein a new S_Wang kernel function suitable for document similarity detection is formed, sigma (sigma) 0) in a formula is a width parameter, and the width parameter is used for controlling the radial action range of the function, and adjusting the influence degree of the distance between the two documents on the similarity because words are different; and 4, finishing a document similarity detecting task through the formed kernel function. The method has the advantages of high detection precision, high recall rate and good comprehensive behavior, and can be applied to document similarity calculation, document classification, document information filtration, mode identification and artificial intelligence.

Description

technical field [0001] The invention relates to the field of information retrieval, specifically a method for using the S_Wang kernel function constructed by the invention for document similarity detection. Background technique [0002] The idea of ​​the kernel method is to convert a non-linearly separable problem in a low-dimensional space to a high-dimensional space, that is, to map it to a high-dimensional space so that it becomes linearly separable in a high-dimensional space, and then in the feature space Using a linear learning machine to establish an optimized hyperplane, using the inner product in the high-dimensional feature space to classify the problem in the low-dimensional space, so as to solve the problem. The most critical part of the transformation is to find the mapping method from x in the input space to φ(x) in the high-dimensional space. There is no systematic way to find this mapping φ. In fact, the mapping function is often not easy to find, and may no...

AI Technical Summary

Problems solved by technology

CLA compound kernel [Wang Xiuhong, Ju Guangguang. Fusion of distributed information retrieval results based on hybrid kernel function [J]. Journal of Communications, 2011, 32(4): 112-118, 125.] Although it is similar to latent semantic kernel and ANOVA kernel Compared with the improvement in the accuracy rate and recall rate of similar detection, the accuracy rate, recall rate and comprehensive performance of similar detection still need to be improved

Images