Measure of similarity of documentation based on document structure

A technology of similarity measurement and document structure, applied in the fields of instrumentation, calculation, electrical and digital data processing, etc., can solve the problem of low accuracy of measurement methods, and achieve the effect of improving accuracy

Active Publication Date: 2007-05-09
PEKING UNIV
View PDF0 Cites 17 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Therefore, the accuracy of the existing measurement methods is not high

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Measure of similarity of documentation based on document structure
  • Measure of similarity of documentation based on document structure
  • Measure of similarity of documentation based on document structure

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0034] In preferred embodiment one of the present invention, a text block segmentation method (TextTiling) is adopted to analyze the document structure, and the flow process includes the following steps as shown in Figure 1:

[0035] 1. Read in the two documents X and Y that need to be compared. For the two documents X and Y that need to be compared, use the text block segmentation method (TextTiling) to obtain the subtopic sequence X={x 1 , x 2 ,...,x n} and Y={y 1 ,y 2 ,...y m}, the specific steps are:

[0036] ① Segment the read document X, divide every 20 words into a word string, and the size of the word string can be selected according to needs.

[0037] 2. Calculate a similarity value for the position between each two word strings by the following method: for the position between word string i and word string i+1, calculate the text block composed of word string i-k to word string i The cosine similarity value between the text blocks composed of word string i+1 to...

Embodiment 2

[0058] The second preferred embodiment of the present invention uses clustering technology to analyze the document structure, including the following steps:

[0059] 1. Read in the two documents X and Y that need to be compared, and use the clustering method to obtain the document subtopic sequence for the two documents X and Y respectively. The specific algorithm steps are:

[0060] ① Segment the read document and divide the document into n sentences;

[0061] ② Calculate the cosine similarity value between any two sentences;

[0062] ③Using the data clustering method to cluster the sentences, the text composed of all the sentences in each category

[0063] This block is a subtopic. In this embodiment, the aggregated clustering method is used to cluster sentences, and the steps are:

[0064] a. Initially, each sentence is classified into one category, and there are k clusters in total;

[0065] b. The two clusters with the largest similarity value among the existing k clu...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

A method for measuring file similarity based on file structure includes finding out subsubject sequence of each file from X and Y files to be compared by utilizing file structure analysis means, utilizing similarity measurement means to calculate similarity value between two subsubjects of different files, setting up weighted bipartite graph G={ X,Y, E } on obtained subsubject sequence and similarity value, solving out optimum match for weighted bipartite graph and carrying out normalization treatment on total weight value of optimum match so as to obtain similarity value of X and Y files.

Description

technical field [0001] The invention relates to a data processing method for document structure similarity measurement. Background technique [0002] Document similarity measurement is a core issue in the field of text information processing. Many text applications, including document clustering, document retrieval, and document filtering, all rely on accurate measurement of document similarity. At present, many document similarity measurement methods have been proposed and applied, such as cosine measure, Jaccard measure, and Dice measure (references: W.B. Frakes and R. Baeza-Yates: Information Retrieval, Data Structure and Algorithms, 1992), methods based on information theory (references: J.A.Aslam and M.Frost: AnInformation-theoretic Measure for Document Similarity.In Proceedings of SIGIR 2003), etc., among which the cosine metric method is the most widely used. [0003] Document similarity measures are documented in: [0004] Dice Metric: Author: W.B.Frakes and R.Baez...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30G06F17/27
Inventor 万小军彭宇新杨建武吴於茜陈晓鸥
Owner PEKING UNIV
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap