A significant mining method for characteristic variables of nonlinear systems based on data stream processing

Abstract

The invention discloses a nonlinear system characteristic variable significance mining method based on data stream processing, wherein the method comprises the following steps: collecting the historical data and preprocessing the historical data; DBSCAN parameters are optimized by closed-loop structure, and the data set is divided by DBSCAN clustering. A nonlinear model based on LGBMRegressor is established for each type of data, and the number of times each feature variable is used in the iterative training process of the model is recorded to characterize its significance, and the sequence ofsignificant feature variables is established. Determine the main distribution intervals of each variable value, form the eigenvalue distribution interval model of a single class, integrate the eigenvalue distribution interval models of all classes to establish the grid model, and map the salient eigenvariable series to the corresponding class in the grid model to form the composite grid model; Build the real-time computing framework based on Storm, design the Topology (topology) based on stream processing, and load the composite mesh model at the same time.

Description

technical field [0001] The invention relates to the fields of data stream processing and data mining, in particular to a method for mining the significance of nonlinear system characteristic variables based on data stream processing. Background technique [0002] "Big data" became popular all the way in 2011, and it shined even more in 2012, becoming a well-deserved focus of the industry. With the rapid development of the Internet and Web technology, the wide use of technologies such as web logs, Internet search indexes, e-commerce, and social networking sites has brought about a sharp increase in the amount of data. The widespread use of computer technology in all walks of life has also led to the generation of a large amount of data. The data is growing at an alarming rate, which indicates that we have entered the era of big data. According to the monitoring of the International Data Information (IDC) company, the global data volume doubles approximately every two years. ...

AI Technical Summary

Problems solved by technology

At the same time, various instantiated systems in the above fields are mostly nonlinear systems composed of high-dimensional features. Therefore, this also brings challenges to the saliency mining of feature variables in nonlinear systems: (1) It is impossible to use simple correlation (2) PCA and other means cannot be used to achieve dimensionality reduction to simplify the nonlinear system under study; (3) The current application scenarios of significance testing are not the ones in the above-mentioned nonlinear systems. Significance mining of characteristic variables; (4) Although various current machine learning algorithms for nonlinear problems can fit any complex nonlinear model with high precision, the data structure describing the weights of the input variables of the nonlinear system is too cumbersome , such as neural networks; (5) The structure of most nonlinear systems is not designed by humans, but is determined by their business purpose and existing data structures, such as classification, clustering problems, and even natural language processing, machine vision and other fields The problems are all nonlinear, and the original data model corresponding to the specific problem is complex and unknown. Therefore, it is unrealistic to mine the influence of characteristic variables on the system output by fitting the nonlinear relationship.

Images