Data preprocessing method for machine learning algorithm and related equipment

Abstract

The invention provides a data preprocessing method for a machine learning algorithm and related equipment. The method comprises the following steps: acquiring to-be-processed original data; performing screening based on the missing value of each feature of the original data to obtain first screening data; performing screening based on the same value of each feature in the first screening data to determine second screening data; filling the missing value of each feature in the second screening data to obtain complete data; and performing standardization processing on the complete data according to a preset standardization processing rule to obtain the complete data after standardization processing. According to the embodiment of the invention, data preprocessing can be carried out on the structured data, the data availability and the data quality are improved by processing the abnormal value of the data, and a large amount of manpower and material resources during development of a machine learning algorithm are saved.

Description

technical field [0001] The present disclosure relates to the technical field of data processing, in particular to a data preprocessing method and related equipment for machine learning algorithms. Background technique [0002] With the development of machine learning technology, the demand for machine learning technology in more and more industries is becoming more and more urgent. However, in the development of machine learning algorithms based on structured data such as data mining and user profiling, there will be a certain degree of data loss during data collection. If the machine learning algorithm is directly developed without screening, it will cause due to Invalid data leads to increased training time, or decreased performance, or even training cannot be performed when there are missing values. For filling missing values, brainstorming or consulting with relevant experts is often used to fill in, resulting in a lot of waste of manpower and material resources. In the...

AI Technical Summary

Problems solved by technology

However, in the development of machine learning algorithms based on structured data such as data mining and user profiling, there will be a certain degree of data loss during data collection. If the machine learning algorithm is directly developed without screening, it will cause due to Invalid data leads to increased training time, or performance degradation, and even training cannot be performed when there are missing values. For filling missing values, brainstorming is often used at present, or relevant experts are often used to fill in, resulting in a lot of waste of manpower and material resources.

In the existing technology, principal component analysis (PCA) dimensionality reduction method and convolutional neural network (CNN) are used to reduce the influence of invalid features on the results. The new features generated by the former are difficult to establish with the actual application scenarios, and manual settings are required. , Adjust the threshold of cumulative explainable variance; the latter has insufficient biological support, no memory function, and the CNN full connection mode is too redundant and inefficient; in addition, both of the above two methods exist: when there are missing values ​​in the data set, dimensionality reduction The problem that the method will fail

Images