Multi-objective parallel attribute reduction method based on Spark and ant colony optimization

Abstract

The invention discloses a multi-objective parallel attribute reduction method based on Spark and ant colony optimization. A thought of combining a cloud computing Spark parallel technology and an intelligent ant colony algorithm is introduced into rough set theoretical attribute reduction; on the basis, the information gain rate is used as heuristic information, and an innovative strategy of redundancy detection is carried out on the selected attribute and each generation of optimal solution, so that the algorithm can be quickly converged to the global optimal solution, the possibility that the redundancy attribute is added to a reduction set can be effectively avoided, and redundancy caused by random selection of the initial attribute is eliminated. Besides, a multi-target parallel solving strategy is adopted when heuristic information is calculated, the heuristic information of multiple attributes relative to the current attribute can be solved at the same time, and the time complexity is reduced from O (| n2 |) to O (| n |).

Description

Technical field [0001] The invention relates to the technical field of big data cloud, in particular to a multi-objective parallel attribute reduction method based on Spark and ant colony optimization. Background technique [0002] Attribute reduction is one of the important research contents of rough set theory and a key step in knowledge acquisition. The so-called attribute reduction refers to the deletion of unnecessary knowledge under the condition that the classification ability of the information system remains unchanged. By deleting redundant attribute information, the potential clarity of the information system can be improved, high-quality cleaning data can be obtained, and effective information with theoretical analysis and application value can be mined. [0003] The development of information technology and the continuous increase of data scale make traditional data mining methods, including attribute reduction algorithm, face the challenge of data scale and computing ...

AI Technical Summary

Problems solved by technology

[0005] Aiming at the problem that traditional algorithms cannot effectively deal with big data attribute reduction and the combined optimization problem faced when solving attribute reduction, the present invention provides a multi-objective parallel attribute reduction method based on Spark and ant colony optimization for solving big data The attribute reduction problem can effectively obtain the minimum attribute reduction while processing large data, and at the same time reduce the time complexity of calculating the attribute importance from O(n 2 ) down to O(|n|)

Images