Large-scale document similarity detection method

A detection method and similarity technology, applied in the field of computer algorithms, can solve problems such as insufficient accuracy and low efficiency, and achieve the effects of high accuracy, high execution efficiency, and improved execution efficiency.

Active Publication Date: 2018-09-28
NANJING UNIV OF POSTS & TELECOMM
View PDF3 Cites 15 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

In essence, the Simhash algorithm is an improved hash algorithm designed to solve the deduplication of similar data, but after it is actually...

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Large-scale document similarity detection method
  • Large-scale document similarity detection method
  • Large-scale document similarity detection method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0039] In order to describe in more detail the load balancing method executed between the service nodes of the server cluster proposed by the present invention. to combine figure 1 , as detailed below.

[0040] A large-scale document similarity detection method, comprising the following steps:

[0041] S1. Input a document set, and calculate the similarity of other information of the documents in the document set.

[0042] S2. The content of each document in the document set corresponds to a signature S initialized to 0 and a length of f, and an f-dimensional vector V initialized to 0.

[0043] S3. Use the NLPIR word segmentation system to perform word segmentation processing on the document content, filter out modal particles and auxiliary words, and remove interference symbols to convert the document content into a set of feature words.

[0044] S4. The weight of the feature word x uses TF-IDF technology and the topic correlation calculation of words comprehensively, and ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a large-scale document similarity detection method. The method comprises the steps of S1, calculating the similarity of other information of documents in a document set; S2, enabling each document content to correspond to a signature S and a f-dimensional vector V; S3, performing word segmentation processing on the document content; S4, comprehensively calculating a weight of a feature word x; S5, mapping the feature word x into a signature h by using a hash function, traversing all bits of the h, and adjusting the V; S6, traversing the V, adjusting the signature S, andfinally generating a signature value, corresponding to the document content, of the signature S; S7, dividing the signature value corresponding to the document content into n blocks, mapping the blocks to a bucket by using the hash function, and judging whether double hash is performed or not; S8, taking the documents of the same bucket as a candidate pair, and calculating the similarity; and S9,judging whether the documents are similar documents or not. The method is high in detection accuracy and high in executive efficiency, and can be widely used in the internet large-scale data mining.

Description

technical field [0001] The invention relates to a detection method, in particular to a large-scale document similarity detection method, which belongs to the field of computer algorithms. Background technique [0002] With the advent of the era of big data, data-based information is growing rapidly, and data takes up more and more space. Such a large amount of data has brought about huge storage problems. The study found that the proportion of redundant data in the stored data is greater than 60%, and the proportion of redundancy will continue to increase in the future. Redundant data reduces the efficiency of user retrieval and query data, and a large amount of storage resources are wasted in storing redundant data, and users do not want to see a bunch of retrieval results with the same or similar content. On the other hand, data mining developers also face the problem of data duplication and redundancy when crawling data through the web. Therefore, document similarity de...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
Inventor 王诚王宇成
Owner NANJING UNIV OF POSTS & TELECOMM
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap