A method of deriving machine learning features from structured data in real time

Abstract

The invention discloses a method for deriving machine learning features from structured data in real time. The method comprises the following steps: defining a computer language comprising a pluralityof computing functions for feature processing in machine learning; developing feature calculation logic using the computer language; generating an executable program code according to the feature calculation logic; executing the program code to apply the feature calculation logic to the corresponding raw data to derive machine learning features. The invention has the beneficial effects that the complexity and development period of feature development are greatly reduced, and modeling personnel of a data analysis background can flexibly and conveniently generate required machine learning features.

Description

technical field [0001] The invention belongs to the field of machine learning, and in particular relates to a method for deriving features for machine learning from structured data in real time. Background technique [0002] Machine learning is an implementation method of artificial intelligence. It is a method that uses probability and statistics to obtain laws (generally called models) from data, and uses this law to reason about unknown data. The application fields of machine learning are very extensive, and have applications in finance, medicine, social public affairs, etc. [0003] When using machine learning to solve practical problems, feature index processing or feature index calculation is a very important step. Machine learning methods such as commonly used xgboost, gbdt, lightgbm, etc. generally do not directly use the original data (generally called training data), but process the data into the form of feature indicators. The characteristic index is generally a...

AI Technical Summary

Problems solved by technology

First, the development of feature derivation is a lot of work, and the developers of machine learning models are mostly statistical and mathematical machines, which require the assistance of additional development engineers, and the development cycle is long; second, after the development of machine learning models by developers, they need to be produced Go online and enter the online decision-making system. At this time, the feature processing work done during the model development process cannot be reused here, because there is no unified method for feature processing during the model development process, and the code written focuses more on batch features. Computing, for the online real-time system, the due performance index cannot be achieved, and it needs to be redeveloped, which increases the online cycle

In the current environment of rapid development of various businesses in social industries, the requirements for model tuning and update frequency are getting higher and higher, and the original method is becoming more and more difficult to meet business needs.

Images