Spark-based parallel association mining optimization method

An optimization method, k-1 technology, applied in special data processing applications, instruments, electrical digital data processing, etc., can solve problems such as low efficiency, achieve high return on investment, reduce system I/O, and improve marketing decisions Effect

Inactive Publication Date: 2017-10-24
NANJING UNIV OF POSTS & TELECOMM
View PDF5 Cites 5 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

At the same time, under the current big data background, in the face of massive data, the efficiency of this traditional serial method is extremely low, and the parallel transformation of the algorithm has become a research hotspot.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Spark-based parallel association mining optimization method
  • Spark-based parallel association mining optimization method
  • Spark-based parallel association mining optimization method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0016] In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

[0017] The steps of the Spark-based parallel association mining optimization method are as follows:

[0018] (1) Scan the transaction database D, clean the source data, simplify the data records, extract valid information, replace all data items with their corresponding numbers to generate a new transaction database D, and store it in HDFS. The new coded data are shown in Table 1.

[0019] Table 1

[0020]

[0021] (2) Read the data set to be processed in HDFS and store it in the memory of each node of the cluster in the form of RDD, and realize the data structure conversion at the same time. ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a Spark-based parallel association mining optimization method. According to the method, first, a transaction databases is preprocessed, data cleaning is performed according to a business demand, brief effective information is extracted, all transaction item data is read into a memory after being encoded, and the encoded transaction item data is converted into an RDD model; in the process of generating a frequent one-item set, a new data structure is constructed to store transaction serial numbers of the one-item set; in the process of connecting and pruning frequent item sets to generate candidate sets, a generation process of candidate item sets is abandoned, and the item sets with the transaction serial numbers meeting minimum support after connection are screened out; and the process is repeated till a larger item set meeting the requirement is not generated. Through the method, the defect of an Apriori algorithm is overcome, and mining efficiency is improved.

Description

technical field [0001] The invention relates to the field of big data association mining algorithms, in particular to a spark-based parallel association mining optimization method. Background technique [0002] Big data is usually used to describe a large amount of semi-structured and unstructured data, which has obvious characteristics: large volume, various types, fast generation speed, high real-time requirements, and low value density, which means that traditional data association mining Algorithms can no longer meet the processing needs of big data, and multi-machine, parallel, and distributed big data processing methods are becoming more and more important. Therefore, it is very urgent to study and propose new association mining algorithms that can adapt to the big data environment. important. [0003] The Apriori algorithm is the most classic association rule mining algorithm. The core of the algorithm is to generate the largest item set, and search for frequent item...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/215G06F16/2465
Inventor 肖甫许平沙乐天王少辉韩崇王汝传
Owner NANJING UNIV OF POSTS & TELECOMM
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap