Algorithm flow based complicated multi-variable data processing method

Abstract

The invention discloses a complex data processing method based on algorithm flow, which is suitable for "three high" data (high-dimensional, high-throughput and high-complexity) analysis processing and information extraction and mining, and belongs to the field of analytical chemometrics. The present invention realizes intelligent data analysis and information mining through the integration and optimization of the data processing flow, that is, by constructing a flow optimization combination including different data processing methods, including data batch loading, preprocessing, feature selection, model building and Unknown sample prediction, etc., set method parameters, and then "inject" the data to be analyzed into the algorithm flow (training set, calibration set, verification set, prediction set, etc.), to achieve fast, convenient, accurate and intelligent analysis of "big data". In particular, changes in the structure of the algorithm flow can realize one-key processing and multi-model processing of complex data, the impact of data processing methods and parameters on the analysis results, and the impact of the same data processing method (algorithm flow) on the processing of different types of data sets etc., to truly achieve the intelligent optimization combination of personalized data and data processing methods.

Description

technical field [0001] The invention relates to a complex multivariate data processing method based on algorithm flow, and belongs to the field of chemometrics in analytical chemistry. Specifically, it integrates and optimizes the complex multivariate data that needs to be processed, from data loading to data preprocessing, from key feature selection to model construction and generalization, and creates an algorithm flow for complex multivariate data processing. , to achieve fast and intelligent data processing. For the analysis of actual complex high-throughput data, it is only necessary to add target data to the algorithm flow to realize intelligent data processing and information mining such as one-click processing and multi-model analysis. Background technique [0002] Complex multivariate data processing and information extraction and mining strongly rely on the application and development of mathematics, statistics, artificial intelligence, chemistry and bioinformatic...

AI Technical Summary

Problems solved by technology

The traditional method of gradually selecting data processing algorithms and actual data leads to a long and complicated data analysis process, and it is difficult to achieve intelligent and fast

Taking the construction of y=f(X) model as an example, the diversity of f and X makes traditional methods unable to truly realize the needs of "big data" intelligent processing and information extraction

For example, data processing software such as spectroscopy, chromatography, and mass spectrometry are designed, organized, and implemented according to the above-mentioned traditional methods, including the current international mainstream chemometric analysis software, such as The Unscrambler and SIMCA, and the operation is extremely time-consuming and complicated. , each data processing method requires repeated operations and manual search for the optimal combination of methods

Images