Data similarity detection method based on sliding window

A data similarity and sliding window technology, applied in the field of data cleaning, can solve the problems of no uniform standard for window size setting and no consideration of impact differences, so as to save detection time and improve detection efficiency

Inactive Publication Date: 2013-10-02
JIANGSU UNIV
View PDF3 Cites 12 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

But there are also problems: ① The impact of different recorded attributes on the detection effect is not co...

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data similarity detection method based on sliding window
  • Data similarity detection method based on sliding window
  • Data similarity detection method based on sliding window

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0037] The present invention will be described in detail below in conjunction with various embodiments shown in the drawings. However, these embodiments do not limit the present invention, and any structural, method, or functional changes made by those skilled in the art according to these embodiments are included in the protection scope of the present invention.

[0038] ginseng figure 1 Shown, a kind of data similarity detection method based on sliding window of the present invention, this method comprises:

[0039] S1. Using the hierarchical method to calculate the experience vector G of the attribute;

[0040] S2. Calculate the statistical vector C of the attribute by mathematical statistics;

[0041] S3. Computing the final weight vector W by integrating the experience vector G and the statistical vector C;

[0042] S4. Calculate the window upper bound of the queue with variable window size.

[0043] S5. Create multiple threads according to the number of attributes; ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a data similarity detection method based on a sliding window, which comprises the steps that S1, an empirical vector G of an attribute is computed by a ranking method; S2,a statistic vector C of the attribute is computed by a mathematical statistic method; S3, the empirical vector G and the statistic vector C are integrated, and a final weight vector W is computed; S4, a window upper bound of a queue of sizes of variable windows is computed; S5, a plurality of threads are created according to the number of the attributes; S6, a record set is scanned sequentially in each thread, and a similarity degree of a current record and a record in the variable queue is computed; and S7, duplicated record sets detected from the threads are merged. Multi-round detection is replaced by the detection algorithm based on multithread concurrence, so that the detection efficiency is improved, and the detection time is saved.

Description

technical field [0001] The invention relates to the technical field of data cleaning, in particular to a data similarity detection method based on a sliding window under massive data. Background technique [0002] Data similarity detection is to detect similar duplicate records in the database to eliminate redundant data. Similar duplicate records are different manifestations of the same real entity in the data set. Due to their differences in format and spelling, the database management system cannot correctly identify them, which in turn affects the correct processing of the data. The measurement indicators for similar duplicate record detection include recall rate, precision rate and time efficiency, etc., and the three are often mutually restricted. Data detection under massive data is particularly prominent in terms of recall rate and time efficiency. It is necessary to optimize the detection algorithm from many aspects to improve the detection effect and detection ef...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
Inventor 周莲英周典瑞
Owner JIANGSU UNIV
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap