Spark-based data mining method for oil and gas exploitation big data

Abstract

The invention provides a Spark-based data mining method for oil and gas exploitation big data, which comprises a data preprocessing module, a model training module and a model application module, andcomprises the following steps: data preprocessing: preprocessing the data through the data preprocessing module; Performing model training, and performing training of a corresponding algorithm to obtain a final model; Model application: analyzing and predicting by using the trained model; And result display: displaying the obtained result to the user. And establishing a big data mining analysis and knowledge discovery model framework, and establishing a general big data analysis mode to support knowledge mining of oil and gas exploitation big data. According to the Spark-based data mining method for the oil and gas exploitation big data, the data mining method and the oil and gas exploitation big data are combined, algorithm parallelization is carried out through Spark, and CPU-is utilized; And through the collaborative computing capability of the GPU, high-efficiency data mining is carried out, and the algorithm speed is increased.

Description

technical field [0001] The invention relates to Spark, data mining and oil and gas exploitation big data, in particular to a Spark-based data mining method for oil and gas exploitation big data. Background technique [0002] Knowledge Discovery in Database (KDD) is a broader term of the so-called "data mining", which is to obtain knowledge according to different needs from the information represented by various media. The purpose of knowledge discovery is to shield the user from the tedious details of the original data, extract meaningful and concise knowledge from the original data, and report it directly to the user. A general process should accept raw data input, select significant data items, reduce, preprocess, and enrich data sets, transform data into appropriate formats, find patterns in data, and evaluate and interpret findings. The main tasks of knowledge discovery include classification, clustering, prediction, association analysis, etc., and the core technologies...

AI Technical Summary

Problems solved by technology

Of course, because MapReduce is calculated in the form of offline batch processing, it cannot meet the requirements for real-time occasions. That is to say, the traditional method based on MapReduce cannot meet Velocity, a characteristic of big data, that is, the real-time performance of data mining.

Images