Flowchart-based data structure prediction transfer and automatic data processing method

Abstract

The present invention proposes a method for predicting transmission of data structure and automatic data processing based on flow chart, including: acquiring a modeling experience data table of a target business scenario; after setting a data source node, selecting and connecting an operator node, and setting the operator node The parameters of the operator node are generated to generate a flowchart; according to the parameters of the operator node, the data table structure of the operation output of the current operator node is predicted, and passed to all downstream operator nodes of the current operator node. , the downstream operator node configures parameters according to the transmitted predicted data table structure; the operator calculation is performed, and the downstream operator node obtains the data flowing from the upstream operator node. After each operator node runs, the output data is generated and transmitted to the downstream operator nodes. The invention conveniently and flexibly realizes data transmission and data structure prediction transmission, and provides a simple and fast construction method for data processing and model design for business personnel and technical personnel.

Description

technical field [0001] The invention relates to the technical field of big data processing, in particular to a flow chart-based data structure prediction transfer and automatic data processing method. Background technique [0002] With the advent of the era of big data, artificial intelligence technology is deeply practiced and applied to financial business scenarios, and machine learning is an inevitable product of the development of artificial intelligence research to a certain stage. It is committed to improving system applications by means of computing and using experience in corresponding performance of the scene. "Experience" usually exists in the form of "data". Through machine learning algorithms, "models" can be generated from the data, and the data needs to be converted into machine learning samples including various feature variables. However, the characteristic variables in the empirical data often need to be processed with statistical algorithms before the mach...

AI Technical Summary

Problems solved by technology

[0003] By using machine learning technology to mine value from data, traditional machine learning or data processing requires technicians to write codes to complete the overall process before building a model and generate valid data and variables for the model. This process requires human input. High cost, high technical threshold and low production efficiency

For business personnel, they have a clearer understanding of real business scenario requirements, while technical personnel often have a biased understanding of business scenario requirements, and the designed system often does not fully meet the needs of business scenarios

[0004] For example, traditional data processing methods need to write program codes for data processing of characteristic variables in data tables. There are a lot of dirty, messy, missing data in real data or data with many dimensions and large data volume. In related data processing software , cannot automatically filter and output field variables, lacks effective variable transmission and processing methods, and cannot effectively predict preprocessed field variables and pass them downstream

[0005] Existing related system platforms do not have table structure prediction and transmission. When calculations are performed to a certain operator node, new data structures and data are generated. At this time, downstream data needs to obtain other structures from rows and configure parameters. In complex data Very poor experience during processing and waste of time

Images