Machine learning automatic process management and optimization system and method based on micro-service

Abstract

The invention provides a machine learning automatic process management and optimization system and method based on micro-service. The system comprises a data set, a data processing flow constructor, adata processing flow service unit, an algorithm constructor, an algorithm service unit, a model constructor, a model service unit, a dirty data model constructor, a service unit orchestration optimizer and a container operation environment. Based on SQL-like data structure conversion, data type conversion, data mapping, data aggregation, data screening and data splitting processes, an algorithm input-oriented data conversion processing flow service unit is automatically constructed, the readability of the data processing process is improved, and the later maintenance cost is reduced.

Description

technical field [0001] The invention relates to the technical field of data analysis in the automobile industry, in particular to a microservice-based machine learning automation process management and optimization system and method. In particular, it relates to a microservice-based machine learning automation process management and optimization system and method for the automotive industry. Background technique [0002] A large number of related companies in the automotive industry lack tools for service-based packaging of algorithm models and digital management and monitoring. [0003] In the past 10 years, due to the development of big data distributed storage computing technology and parallel technology. Enterprises can perceive and accumulate data to grow exponentially. More and more enterprises have begun to carry out data mining, and even began to use complex algorithm models for data analysis, hoping to realize the value of these data. [0004] However, the proces...

AI Technical Summary

Problems solved by technology

[0005] (1) Algorithm development environment hardware and software environment interference problem: Multiple algorithm developers often share a development environment. If debugging and training are required at the same time, mutual software dependencies will interfere with each other. Manually limit task resource usage, or activate virtual machines separately

[0006] (2) Algorithm development has a low degree of automation and low versatility: Algorithm development requires professionals to spend a lot of energy and time on data cleaning and parameter tuning, and the algorithm is limited to specific scenarios, and needs to be redeveloped when the scenario changes The data cleaning process and algorithm program are time-consuming and labor-intensive

[0007] (3) The algorithm release process is complicated: professional microservice program developers are required to manually write codes for service-oriented packaging of algorithm modules, and to test and release

[0008] (4) The iterative comparison process of the algorithm model is long and complicated: the algorithm model needs to manually record the model results, compare the results manually, and then iterate continuously

[0009] (5) The online algorithm and offline algorithm service management process is cumbersome, and the service upgrade experience is poor: manual management algorithm service upgrade, online, algorithm service rollback and other processes, the difference between algorithm service versions is large, and the service management process is cumbersome. Difficult to manage

Images